CNDB-11613 SAI compressed indexes #1474
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
pkolaczk
force-pushed
the
c11613-sai-compression
branch
from
b6c7de2 to
cc3605e
Compare
What was ported: - current compaction throughput measurement by CompactionManager - exposing current compaction throughput in StorageService and CompactionMetrics - nodetool getcompactionthroughput, including tests Not ported: - changes to `nodetool compactionstats`, because that would require porting also the tests which are currently missing in CC and porting those tests turned out to be a complex task without porting the other changes in the CompactionManager API - Code for getting / setting compaction throughput as double
This commit introduces a new AdaptiveCompressor class.
AdaptiveCompressor uses ZStandard compression with a dynamic
compression level based on the current write load. AdaptiveCompressor's
goal is to provide similar write performance as LZ4Compressor
for write heavy workloads, but a significantly better compression ratio
for databases with a moderate amount of writes or on systems
with a lot of spare CPU power.
If the memtable flush queue builds up, and it turns out the compression
is a significant bottleneck, then the compression level used for
flushing is decreased to gain speed. Similarly, when pending
compaction tasks build up, then the compression level used
for compaction is decreased.
In order to enable adaptive compression:
- set `-Dcassandra.default_sstable_compression=adaptive` JVM option
to automatically select `AdaptiveCompressor` as the main compressor
for flushes and new tables, if not overriden by specific options in
cassandra.yaml or table schema
- set `flush_compression: adaptive` in cassandra.yaml to enable it
for flushing
- set `AdaptiveCompressor` in Table options to enable it
for compaction
Caution: this feature is not turned on by default because it
may impact read speed negatively in some rare cases.
Fixes riptano/cndb#11532
Reduces some overhead of setting up / tearing down those contexts that happened inside the calls to Zstd.compress / Zstd.decompress. Makes a difference with very small chunks. Additionally, added some compression/decompression rate metrics.
pkolaczk
force-pushed
the
c11613-sai-compression
branch
from
cc3605e to
93bbdb4
Compare
|
❌ Build ds-cassandra-pr-gate/PR-1474 rejected by Butler2 new test failure(s) in 3 builds Found 2 new test failures
Found 228 known test failures |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What is the issue
SAI Indexes consume too much storage space, sometimes.
What does this PR fix and why was it fixed
This PR allows to compress both the per-sstable and per-index components of SAI.
Use
index_compressiontable param to control the per-sstable components compression.Use
compressionproperty on the index to control the per-index components compression.Checklist before you submit for review
NoSpamLoggerfor log lines that may appear frequently in the logs