Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Stages can open multiple connections if not used in parallel mode #318

Open
odscjames opened this issue Feb 29, 2024 · 0 comments
Open

Stages can open multiple connections if not used in parallel mode #318

odscjames opened this issue Feb 29, 2024 · 0 comments

Comments

@odscjames
Copy link
Contributor

Notes

See validate() func in validate stage.
It opens a new connection and only closes it at the end of it's work.

But in single process mode, it calls process_hash_list() and process_hash_list() opens it's own connection!

It needs to do this in multiple process mode, so each process has it's own psql connection.

But in single mode, we have an unused connection that is idle and may timeout.

BUT in validate() function, the connection is not used after getUnvalidatedDatasets is called. Can we close it straight away, before any actual work is done?

This may apply to other stages

Severity
Low - it results in one extra connection per stage so not a big deal, but the unused connections may hang due to idleness (see #317 )

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant