quantile sharding; fix bug related to # of steps (instant query mem usage) and use CollapsingLowestDense store for ddsketch #11905
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
jeschkies
approved these changes
Feb 9, 2024
Signed-off-by: Callum Styan <[email protected]>
evaluate more steps than we needed to, especially in the case of instant queries! Signed-off-by: Callum Styan <[email protected]>
cstyan
force-pushed
the
callum-quantile-collapsing-lowest
branch
from
dd682ab to
fbdb52a
Compare
cstyan
changed the title
jeschkies
requested changes
Feb 17, 2024
pkg/logql/downstream_test.go
Outdated
| @@ -181,6 +181,70 @@ func TestMappingEquivalenceSketches(t *testing.T) { | |||
| } | |||
| } | |||
|
|
|||
| func TestMappingEquivalenceSketches_Instant(t *testing.T) { | |||
There was a problem hiding this comment.
What makes this an instant query? What do you think about song another loop to the original equivalence tests?
There was a problem hiding this comment.
The query params right after this line, in the case of an instant query we set start and end to the same value.
| @@ -283,6 +283,11 @@ func JoinQuantileSketchVector(next bool, r StepResult, stepEvaluator StepEvaluat | |||
| if stepEvaluator.Error() != nil { | |||
| return nil, stepEvaluator.Error() | |||
| } | |||
| // Don't continue to evaluate more steps if we already filled the | |||
| // # of steps we should have based on start/end and step params. | |||
| if len(result) == stepCount { | |||
There was a problem hiding this comment.
How's that done in the original join?
There was a problem hiding this comment.
Found it
Line 388 in 918e736
Signed-off-by: Callum Styan <[email protected]>
Signed-off-by: Callum Styan <[email protected]>
Signed-off-by: Callum Styan <[email protected]>
cstyan
added a commit
that referenced
this pull request
Feb 20, 2024
…sage) and use CollapsingLowestDense store for ddsketch (#11905) Signed-off-by: Callum Styan <[email protected]>
cstyan
added a commit
that referenced
this pull request
Feb 20, 2024
…uery mem usage) and use CollapsingLowestDense store for ddsketch (#11905) Signed-off-by: Callum Styan <[email protected]>
cstyan
added a commit
that referenced
this pull request
Feb 20, 2024
rhnasc
pushed a commit
to inloco/loki
that referenced
this pull request
Apr 12, 2024
…sage) and use CollapsingLowestDense store for ddsketch (grafana#11905) Signed-off-by: Callum Styan <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
EDIT: I also found the bug that was causing us to OOM when executing instant queries, see the 2nd commit
This should reduce the memory allocated by the vector iteration by at least 25% but as much as ~80%. The quantile vector iteration accounts for at least 15% of all in use space last time I tested (and during that time queriers we're OOM'ing as a result of quantile sharding).
However, with the usage of this alternate store type we give up some amount of relative accuracy guarantees for lower quantiles. From their readme:
Benchmark:
And profiling
