Skip to content

Commit

Permalink
Merge branch '8.10.tikv' into bot/update-owners-1729750177503
Browse files Browse the repository at this point in the history
  • Loading branch information
ti-chi-bot authored Nov 5, 2024
2 parents 8af0e95 + d425a86 commit 8a7262f
Show file tree
Hide file tree
Showing 6 changed files with 19 additions and 47 deletions.
3 changes: 1 addition & 2 deletions HISTORY.md
Original file line number Diff line number Diff line change
Expand Up @@ -1030,8 +1030,6 @@ Note: The next release will be major release 7.0. See https://github.com/faceboo
* BackupEngineOptions::sync (default true) now applies to restoring backups in addition to creating backups. This could slow down restores, but ensures they are fully persisted before returning OK. (Consider increasing max_background_operations to improve performance.)

## 6.23.0 (2021-07-16)
### Behavior Changes
* Obsolete keys in the bottommost level that were preserved for a snapshot will now be cleaned upon snapshot release in all cases. This form of compaction (snapshot release triggered compaction) previously had an artificial limitation that multiple tombstones needed to be present.
### Bug Fixes
* Blob file checksums are now printed in hexadecimal format when using the `manifest_dump` `ldb` command.
* `GetLiveFilesMetaData()` now populates the `temperature`, `oldest_ancester_time`, and `file_creation_time` fields of its `LiveFileMetaData` results when the information is available. Previously these fields always contained zero indicating unknown.
Expand All @@ -1056,6 +1054,7 @@ Note: The next release will be major release 7.0. See https://github.com/faceboo
### Behavior Changes
* Added two additional tickers, MEMTABLE_PAYLOAD_BYTES_AT_FLUSH and MEMTABLE_GARBAGE_BYTES_AT_FLUSH. These stats can be used to estimate the ratio of "garbage" (outdated) bytes in the memtable that are discarded at flush time.
* Added API comments clarifying safe usage of Disable/EnableManualCompaction and EventListener callbacks for compaction.

### Bug Fixes
* fs_posix.cc GetFreeSpace() always report disk space available to root even when running as non-root. Linux defaults often have disk mounts with 5 to 10 percent of total space reserved only for root. Out of space could result for non-root users.
* Subcompactions are now disabled when user-defined timestamps are used, since the subcompaction boundary picking logic is currently not timestamp-aware, which could lead to incorrect results when different subcompactions process keys that only differ by timestamp.
Expand Down
19 changes: 8 additions & 11 deletions db/external_sst_file_ingestion_job.cc
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,6 @@
// COPYING file in the root directory) and Apache 2.0 License
// (found in the LICENSE.Apache file in the root directory).


#include "db/external_sst_file_ingestion_job.h"

#include <algorithm>
Expand Down Expand Up @@ -912,16 +911,7 @@ Status ExternalSstFileIngestionJob::AssignLevelAndSeqnoForIngestedFile(
if (lvl > 0 && lvl < vstorage->base_level()) {
continue;
}
if (cfd_->RangeOverlapWithCompaction(
file_to_ingest->smallest_internal_key.user_key(),
file_to_ingest->largest_internal_key.user_key(), lvl)) {
// We must use L0 or any level higher than `lvl` to be able to overwrite
// the compaction output keys that we overlap with in this level, We also
// need to assign this file a seqno to overwrite the compaction output
// keys in level `lvl`
overlap_with_db = true;
break;
} else if (vstorage->NumLevelFiles(lvl) > 0) {
if (vstorage->NumLevelFiles(lvl) > 0) {
bool overlap_with_level = false;
status = sv->current->OverlapWithLevelIterator(
ro, env_options_, file_to_ingest->smallest_internal_key.user_key(),
Expand Down Expand Up @@ -1123,6 +1113,13 @@ bool ExternalSstFileIngestionJob::IngestedFileFitInLevel(
return false;
}

if (cfd_->RangeOverlapWithCompaction(file_smallest_user_key,
file_largest_user_key, level)) {
// File overlap with a running compaction output that will be stored
// in this level, we cannot add this file to this level
return false;
}

// File did not overlap with level files, nor compaction output
return true;
}
Expand Down
3 changes: 2 additions & 1 deletion db/version_set.cc
Original file line number Diff line number Diff line change
Expand Up @@ -4216,7 +4216,8 @@ void VersionStorageInfo::ComputeBottommostFilesMarkedForCompaction(

for (auto& level_and_file : bottommost_files_) {
if (!level_and_file.second->being_compacted &&
level_and_file.second->fd.largest_seqno != 0) {
level_and_file.second->fd.largest_seqno != 0 &&
level_and_file.second->num_deletions > 1) {
// largest_seqno might be nonzero due to containing the final key in an
// earlier compaction, whose seqnum we didn't zero out.
if (level_and_file.second->fd.largest_seqno < oldest_snapshot_seqnum_) {
Expand Down
11 changes: 6 additions & 5 deletions options/db_options.cc
Original file line number Diff line number Diff line change
Expand Up @@ -327,6 +327,9 @@ static std::unordered_map<std::string, OptionTypeInfo>
{offsetof(struct ImmutableDBOptions, enable_pipelined_write),
OptionType::kBoolean, OptionVerificationType::kNormal,
OptionTypeFlags::kNone}},
{"enable_pipelined_commit",
{0, OptionType::kBoolean, OptionVerificationType::kDeprecated,
OptionTypeFlags::kNone}},
{"enable_multi_batch_write",
{offsetof(struct ImmutableDBOptions, enable_multi_batch_write),
OptionType::kBoolean, OptionVerificationType::kNormal,
Expand Down Expand Up @@ -922,8 +925,7 @@ void ImmutableDBOptions::Dump(Logger* log) const {
ROCKS_LOG_HEADER(log, " Options.wal_compression: %d",
wal_compression);
ROCKS_LOG_HEADER(log, " Options.atomic_flush: %d", atomic_flush);
ROCKS_LOG_HEADER(log,
" Options.avoid_unnecessary_blocking_io: %d",
ROCKS_LOG_HEADER(log, " Options.avoid_unnecessary_blocking_io: %d",
avoid_unnecessary_blocking_io);
ROCKS_LOG_HEADER(log, " Options.persist_stats_to_disk: %u",
persist_stats_to_disk);
Expand Down Expand Up @@ -1060,14 +1062,13 @@ void MutableDBOptions::Dump(Logger* log) const {
ROCKS_LOG_HEADER(log,
" Options.wal_bytes_per_sync: %" PRIu64,
wal_bytes_per_sync);
ROCKS_LOG_HEADER(log,
" Options.strict_bytes_per_sync: %d",
ROCKS_LOG_HEADER(log, " Options.strict_bytes_per_sync: %d",
strict_bytes_per_sync);
ROCKS_LOG_HEADER(log,
" Options.compaction_readahead_size: %" ROCKSDB_PRIszt,
compaction_readahead_size);
ROCKS_LOG_HEADER(log, " Options.max_background_flushes: %d",
max_background_flushes);
max_background_flushes);
ROCKS_LOG_HEADER(log, "Options.daily_offpeak_time_utc: %s",
daily_offpeak_time_utc.c_str());
}
Expand Down
12 changes: 2 additions & 10 deletions utilities/blob_db/blob_db_test.cc
Original file line number Diff line number Diff line change
Expand Up @@ -570,6 +570,7 @@ TEST_F(BlobDBTest, EnableDisableCompressionGC) {
Random rnd(301);
BlobDBOptions bdb_options;
bdb_options.min_blob_size = 0;
bdb_options.enable_garbage_collection = true;
bdb_options.garbage_collection_cutoff = 1.0;
bdb_options.disable_background_tasks = true;
bdb_options.compression = kSnappyCompression;
Expand Down Expand Up @@ -598,11 +599,6 @@ TEST_F(BlobDBTest, EnableDisableCompressionGC) {
ASSERT_EQ(2, blob_files.size());
ASSERT_EQ(kNoCompression, blob_files[1]->GetCompressionType());

// Enable GC. If we do it earlier the snapshot release triggered compaction
// may compact files and trigger GC before we can verify there are two files.
bdb_options.enable_garbage_collection = true;
Reopen(bdb_options);

// Trigger compaction
ASSERT_OK(blob_db_->CompactRange(CompactRangeOptions(), nullptr, nullptr));
blob_db_impl()->TEST_DeleteObsoleteFiles();
Expand Down Expand Up @@ -641,6 +637,7 @@ TEST_F(BlobDBTest, ChangeCompressionGC) {
Random rnd(301);
BlobDBOptions bdb_options;
bdb_options.min_blob_size = 0;
bdb_options.enable_garbage_collection = true;
bdb_options.garbage_collection_cutoff = 1.0;
bdb_options.disable_background_tasks = true;
bdb_options.compression = kLZ4Compression;
Expand Down Expand Up @@ -670,11 +667,6 @@ TEST_F(BlobDBTest, ChangeCompressionGC) {
ASSERT_EQ(2, blob_files.size());
ASSERT_EQ(kSnappyCompression, blob_files[1]->GetCompressionType());

// Enable GC. If we do it earlier the snapshot release triggered compaction
// may compact files and trigger GC before we can verify there are two files.
bdb_options.enable_garbage_collection = true;
Reopen(bdb_options);

ASSERT_OK(blob_db_->CompactRange(CompactRangeOptions(), nullptr, nullptr));
VerifyDB(data);

Expand Down
18 changes: 0 additions & 18 deletions utilities/transactions/write_prepared_transaction_test.cc
Original file line number Diff line number Diff line change
Expand Up @@ -2582,7 +2582,6 @@ TEST_P(WritePreparedTransactionTest, ReleaseSnapshotDuringCompaction) {
const size_t snapshot_cache_bits = 7; // same as default
const size_t commit_cache_bits = 0; // minimum commit cache
UpdateTransactionDBOptions(snapshot_cache_bits, commit_cache_bits);
options.disable_auto_compactions = true;
ASSERT_OK(ReOpen());

ASSERT_OK(db->Put(WriteOptions(), "key1", "value1_1"));
Expand All @@ -2602,13 +2601,7 @@ TEST_P(WritePreparedTransactionTest, ReleaseSnapshotDuringCompaction) {
VerifyKeys({{"key1", "value1_1"}}, snapshot2);
// Add a flush to avoid compaction to fallback to trivial move.

// The callback might be called twice, record the calling state to
// prevent double calling.
bool callback_finished = false;
auto callback = [&](void*) {
if (callback_finished) {
return;
}
// Release snapshot1 after CompactionIterator init.
// CompactionIterator need to figure out the earliest snapshot
// that can see key1:value1_2 is kMaxSequenceNumber, not
Expand All @@ -2617,7 +2610,6 @@ TEST_P(WritePreparedTransactionTest, ReleaseSnapshotDuringCompaction) {
// Add some keys to advance max_evicted_seq.
ASSERT_OK(db->Put(WriteOptions(), "key3", "value3"));
ASSERT_OK(db->Put(WriteOptions(), "key4", "value4"));
callback_finished = true;
};
SyncPoint::GetInstance()->SetCallBack("CompactionIterator:AfterInit",
callback);
Expand All @@ -2639,7 +2631,6 @@ TEST_P(WritePreparedTransactionTest, ReleaseSnapshotDuringCompaction2) {
const size_t snapshot_cache_bits = 7; // same as default
const size_t commit_cache_bits = 0; // minimum commit cache
UpdateTransactionDBOptions(snapshot_cache_bits, commit_cache_bits);
options.disable_auto_compactions = true;
ASSERT_OK(ReOpen());

ASSERT_OK(db->Put(WriteOptions(), "key1", "value1"));
Expand Down Expand Up @@ -2690,7 +2681,6 @@ TEST_P(WritePreparedTransactionTest, ReleaseSnapshotDuringCompaction3) {
const size_t snapshot_cache_bits = 7; // same as default
const size_t commit_cache_bits = 1; // commit cache size = 2
UpdateTransactionDBOptions(snapshot_cache_bits, commit_cache_bits);
options.disable_auto_compactions = true;
ASSERT_OK(ReOpen());

// Add a dummy key to evict v2 commit cache, but keep v1 commit cache.
Expand Down Expand Up @@ -2720,18 +2710,11 @@ TEST_P(WritePreparedTransactionTest, ReleaseSnapshotDuringCompaction3) {
add_dummy();
auto* s2 = db->GetSnapshot();

// The callback might be called twice, record the calling state to
// prevent double calling.
bool callback_finished = false;
auto callback = [&](void*) {
if (callback_finished) {
return;
}
db->ReleaseSnapshot(s1);
// Add some dummy entries to trigger s1 being cleanup from old_commit_map.
add_dummy();
add_dummy();
callback_finished = true;
};
SyncPoint::GetInstance()->SetCallBack("CompactionIterator:AfterInit",
callback);
Expand All @@ -2749,7 +2732,6 @@ TEST_P(WritePreparedTransactionTest, ReleaseEarliestSnapshotDuringCompaction) {
const size_t snapshot_cache_bits = 7; // same as default
const size_t commit_cache_bits = 0; // minimum commit cache
UpdateTransactionDBOptions(snapshot_cache_bits, commit_cache_bits);
options.disable_auto_compactions = true;
ASSERT_OK(ReOpen());

ASSERT_OK(db->Put(WriteOptions(), "key1", "value1"));
Expand Down

0 comments on commit 8a7262f

Please sign in to comment.