Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add ref store size threshold property to initiate purge of old streams #4610

Open
at055612 opened this issue Nov 21, 2024 · 2 comments
Open
Labels
enhancement A new feature or enhancement to an existing feature f:ref data Issues relating to reference data loads or lookups
Milestone

Comments

@at055612
Copy link
Member

When a ref store reaches stroom.pipeline.referenceData.lmdb.maxStoreSize loads will fail. It would be good if there was an additional threshold % property (e.g. 90%) such that prior to a load, if the store size (as a % of the maxStoreSize) is greater than this threshold, then it will purge old streams until below the threshold.

@at055612 at055612 added enhancement A new feature or enhancement to an existing feature f:ref data Issues relating to reference data loads or lookups labels Nov 21, 2024
@at055612 at055612 added this to the v7.7 milestone Nov 21, 2024
@at055612
Copy link
Member Author

Raised for @p-kimberley

@p-kimberley
Copy link
Contributor

For context, there are two issues here:

  1. Disk pressure caused by multiple ref DBs (one per feed), each growing at their own rate up to maxStoreSize. Once a volume fills up, ref data writes will fail. Therefore it would be useful to be able to cap ref usage on a per-node basis as well as per-feed/DB.
  2. A single DB reaching maxStoreSize and subsequently failing to load until entries expire and are purged.

Firstly, I suggest maxStoreSize be renamed to maxDbSize. When ref stores were combined (not split into feeds), this property made sense as a combined limit. Now there can be multiple DBs, there should be a separate property governing the size of individual DBs.

I propose two additional settings be created, both of which will cause streams to be purged until the DB(s) are within limits:

  1. stroom.pipeline.referenceData.lmdb.maxStoreSize. Maximum size of the ref store, encompassing all ref DBs. This will enable a global cap to prevent disk pressure from affecting node health if ref data grows in an uncontrolled manner. If this limit is reached, Stroom should start purging from the largest ref DBs until the aggregate DB size falls below. Maybe purge iteratively in batches, each time from the largest DB.
  2. stroom.pipeline.referenceData.lmdb.dbHighWaterMarkPercent. Once an individual ref DB reaches this percentage of dbMaxSize, purge streams until its size falls below.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement A new feature or enhancement to an existing feature f:ref data Issues relating to reference data loads or lookups
Projects
None yet
Development

No branches or pull requests

2 participants