Restore improvements #3986
Draft
Restore improvements #3986
Commits on Sep 2, 2024
-
refactor(restore): wrap smaller stages into separate functions
This decreases main function complexity. Noticed by: cognitive complexity 66 of func `(*tablesWorker).restore` is high (> 50) (gocognit).
-
feat(restore_test): validate that compaction and tombstone_gc are dis…
…abled and enabled As a preparation for restoring data, SM should disable tombstone_gc and compaction. They should be re-enabled after the restore finishes.
-
feat(restore): new indexing and batching approach
The idea is to first index all files to be restored, so that we can create better batches. Restore workload is aggregated first by table, then by remote sstable dir. From batching we expect: - batch contains X*shard_cnt sstables - batch contains similarly sized sstables - batch is created from any manifest/table - no waiting for manifest/table restore to finish - workload across different nodes is evenly distributed - --batch-size is ignored, batches are aiming to have size equal to 5% of expected node workload New flag --table-parallel - it allows for running multiple download and l&s jobs from the same node (no documentation as it might not be exposed later on).
-
-
-
-
feat(restoretest): enable --unpin-agent-cpu, --stream-to-all-replicas…
… and --table-parallel flags
-
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.