[WIP] Added a MultiStore to DataServer #502
Draft
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
When running large numbers of calculations simultaneously,
rlaunch multi
takes care to only create one MongoClient instance and take advantage of connection pooling to prevent large numbers of concurrent connections to the mongodb.However, when a user would like data to be saved to another database or collection, it is typically achieved by creating maggma Stores within a
Firetask
and inserting the document (for example jobflow's JobFiretask). This results in poor control over the number of connections made to a database and the potential for an unlimited number of connections.To solve this, I have implemented a class which keeps track of the Stores in use and allows individual fireworks to share these Stores. This extends the
DataServer
used to share aLaunchPad
, by registering an additional callableMultiStore
. No other fireworks code utilizes this functionality, but it is required that the main process keeps track of it and therefore it cannot be put in another code.Because I have put this class in maggma, this does require fireworks to have maggma as a dependency. I could also put this class in jobflow, but jobflow (and implicitly maggma) would need to be dependencies.
These are the pull requests for maggma and jobflow