Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Keep previous tileset as backup #11

Open
1ec5 opened this issue Jun 17, 2024 · 3 comments
Open

Keep previous tileset as backup #11

1ec5 opened this issue Jun 17, 2024 · 3 comments
Labels
enhancement New feature or request

Comments

@1ec5
Copy link
Member

1ec5 commented Jun 17, 2024

Would it be feasible to keep the previous tranche of tiles around after an update, to temporarily swap in if necessary using s3 lifecycle configuration rules? If we find out about particularly bad vandalism or, say, someone draining one of the Great Lakes, someone could trigger an action to swap in the previous tiles while the next update is churning away. Related to this idea would be some sort of trigger to halt the production of a known bad dataset.

@1ec5 1ec5 added the enhancement New feature or request label Jun 17, 2024
@ZeLonewolf
Copy link
Collaborator

ZeLonewolf commented Jun 17, 2024

From a technical standpoint, this should be easy to implement. We can just have the upload script first make a copy of the planet within the bucket to e.g. planet.1.pmtiles. If you want multiple backups, you could copy e.g. 2 to 3, then 1 to 2, then planet to 1.

Then, we could add a GitHub action to restore an older planet, so you wouldn't even have to do arcane AWS stuff - anyone with GH access could run the action and restore the old planet. I'm not sure if it's possible to manually pass a parameter, so we'd likely need an action for each Nth planet we want to restore.

The "restore" process would be to copy an older planet and overwrite the current one and await the next planet upload by the render server.

File storage on s3 is 2.3 cents per GB, so the only real question is how many planet archives at $1.72 per month we want to keep on hand in the bucket.

@quincylvania
Copy link
Contributor

One backup sounds prudent, multiple seems like overkill at this point. However, we might not necessarily want the last planet to immediately become the backup ever time; say, if two subsequent builds have the same vandalism. We could replace the backup every 24 or 48 hours or so to reduce the chances of this issue.

@ZeLonewolf
Copy link
Collaborator

In that case, the script can just check the last-modified date of the current planet file in the s3 bucket, and if it's older than $X, it can overwrite it with the current planet before pushing a new one up. That still leaves a little wiggle room for the planet/vandalism timeline happening at JUSSSST the wrong timeframe but a longer gap is probably a better hedge than a shorter one.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

3 participants