store-gateway: wait for query gate after loading blocks #5507
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
A surge in lazy loading index headers can saturate the query concurrency limit and effectively block any other queries from even starting. This was evident in a recent similar incident where the request rate on the full query path was impacted by store-gateways lazy-loading blocks. Below are screenshots of request rate, latency, lazy loading latency and query gate latency reported by store-gateways. We already have a concurrency gate for lazy loading index headers added by Heather, which prevents from overloading the disk. If we decouple the query gate and the header loading gate, queries that require loading many index headers will be waiting on a separate gate and not impact queries which are querying already loaded blocks. #### Note to reviewers This is a draft PR because I suspect it will conflict with the changes in stream-store, so I will rebase it after merging that PR first. Signed-off-by: Dimitar Dimitrov <[email protected]>
Signed-off-by: Dimitar Dimitrov <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
A surge in lazy loading index headers can saturate the query concurrency limit and effectively block any other queries from even starting. This was evident in a recent similar incident where the request rate on the full query path was impacted by store-gateways lazy-loading blocks. During the time between 00:00 and 00:30 99.2% of queries were issues for the last 12 hours only.
Below are screenshots of request rate, latency, lazy loading latency and query gate latency reported by store-gateways.
We already have a concurrency gate for lazy loading index headers added by @heatheryuan , which prevents from overloading the disk. If we decouple the query gate and the header loading gate, queries that require loading many index headers will be waiting on a separate gate and not impact queries which are querying already loaded blocks.
Notes to reviewers
ifbecause the query gate is either a noop gate or a real gate, but never nil.