Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add re-tries to restore #4065

Closed
Michal-Leszczynski opened this issue Oct 9, 2024 · 0 comments · Fixed by #4071
Closed

Add re-tries to restore #4065

Michal-Leszczynski opened this issue Oct 9, 2024 · 0 comments · Fixed by #4071
Assignees
@Michal-Leszczynski
Labels
restore
Milestone
3.4

Comments

@Michal-Leszczynski
Copy link
Collaborator

Right now, when a given node fails to restore a batch, the restore ends with an error.
We should make it more robust so that:

  • failed batch can be re-tried by other nodes in the context of a single restore task execution
  • failed node can still work on other batches, but originating from different backup datacenters

The first one is self-explanatory.
The second one comes from the fact that the failure might be dc-related. An example of such failure could be seen in #3871. This approach also reduces the amount of failed batches when the source of the problem is node related.
@Michal-Leszczynski Michal-Leszczynski self-assigned this Oct 9, 2024
@Michal-Leszczynski Michal-Leszczynski added this to the 3.4 milestone Oct 9, 2024
Michal-Leszczynski added a commit that referenced this issue Oct 16, 2024
@Michal-Leszczynski
feat(restore): make batches retryable
0feb1bc 
Now, if batch restoration failed on one node,
it can still be retried by other nodes.
Failed node is no longer used for the restore.

Fixes #4065
Michal-Leszczynski added a commit that referenced this issue Oct 16, 2024
@Michal-Leszczynski
feat(restore): make batches retryable
53a05e9 
Now, if batch restoration failed on one node,
it can still be retried by other nodes.
Failed node is no longer used for the restore.

Fixes #4065
@Michal-Leszczynski Michal-Leszczynski mentioned this issue Oct 16, 2024
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@Michal-Leszczynski Michal-Leszczynski

Labels
restore
Projects
None yet
Milestone
3.4
Development

Successfully merging a pull request may close this issue.

Restore improvement: batch retry scylladb/scylla-manager
1 participant
@Michal-Leszczynski