Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Migrate to rclone and add helm-docs #262

Merged
merged 11 commits into from
Jul 3, 2023
Merged

Migrate to rclone and add helm-docs #262

merged 11 commits into from
Jul 3, 2023

Conversation

bakhtin
Copy link
Contributor

@bakhtin bakhtin commented Jun 29, 2023

Migrate node snapshot restoration scripts to rclone. rclone is compatible with numerous cloud storage systems and allows to work with them using a unified syntax.

rclone replaces multiple different utilities that we have been using to download chain snapshots: wget for HTTP, gcs for Google Cloud Storage (GCS).

Add support for other providers -- for now just s3-compatible storages but can be easily expanded to support more.

Add support for downloading snapshots over HTTP using a file with a list of files. It should significantly reduce cost of restoring a snapshot from S3/GCS as no expensive API requests will be consumed.

Bump major version of the chart introducing breaking changes (see README)

Bonus: Add helm-docs to auto-generate values section of README (see README.md and README.md.gotmpl)

TODO:

  • Update Deploying a node with data synced from a snapshot archive section of README
  • Test different scenarios for backup restoration

Closes: #237

Copy link
Contributor

@dblane-digicatapult dblane-digicatapult left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Excellent set of changes.

charts/node/values.yaml Outdated Show resolved Hide resolved
@bakhtin bakhtin marked this pull request as ready for review July 3, 2023 09:31
Signed-off-by: bakhtin <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Implement S3 support for backups
5 participants