Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update dump-to-s3 step function to "unparallelize" entire thing #1027

Open
shawncrawley opened this issue Jan 7, 2025 · 1 comment
Open

Comments

@shawncrawley
Copy link
Collaborator

All of the dump to s3 logic (i.e. the pg_dump command) occurs on the same exact machine, which will quickly get strapped for resources when dealing with the size of some of the dumps. Thus, we need to update the step function to synchronize the processing of all of the dumps - namely, ensuring that the "egis" dumps and "viz" dumps are not parallel processed, and same with processing schemas versus tables. Right now the step function branches out in to essentially four tracks, but it should really just be a single track.

@RobHanna-NOAA
Copy link
Contributor

Note: If we are able to get larger EC2's for processing, which is expected as the current "bastion" server is far, far too small. TBD

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants