From 7f78d3cc09d6e33bbebd81f1576ce1a7a11065b6 Mon Sep 17 00:00:00 2001 From: Grace Cai Date: Wed, 30 Apr 2025 16:37:38 +0800 Subject: [PATCH 1/2] This is an automated cherry-pick of #20871 Signed-off-by: ti-chi-bot --- releases/release-7.6.0.md | 3 ++ releases/release-8.0.0.md | 3 +- releases/release-8.1.0.md | 1 + releases/release-8.1.2.md | 9 +++- tikv-configuration-file.md | 85 ++++++++++++++++++++++++++++++++++---- 5 files changed, 91 insertions(+), 10 deletions(-) diff --git a/releases/release-7.6.0.md b/releases/release-7.6.0.md index b53d9f10c58ae..fa7d0f8018ce7 100644 --- a/releases/release-7.6.0.md +++ b/releases/release-7.6.0.md @@ -281,12 +281,15 @@ Quick access: [Quick start](https://docs.pingcap.com/tidb/v7.6/quick-start-with- | Configuration file | Configuration parameter | Change type | Description | | -------- | -------- | -------- | -------- | | TiDB | [`tls-version`](/tidb-configuration-file.md#tls-version) | Modified | The default value is "". The default supported TLS versions of TiDB are changed from `TLS1.1` or higher to `TLS1.2` or higher. | +| TiKV | [`raftstore.report-min-resolved-ts-interval`](https://docs.pingcap.com/tidb/v7.5/tikv-configuration-file/#report-min-resolved-ts-interval-new-in-v600) | Renamed | To make the name more accurate, this configuration item is renamed to [`raftstore.pd-report-min-resolved-ts-interval`](/tikv-configuration-file.md#pd-report-min-resolved-ts-interval-new-in-v760). `raftstore.report-min-resolved-ts-interval` is no longer effective. | | TiKV | [`blob-file-compression`](/tikv-configuration-file.md#blob-file-compression) | Modified | The algorithm used for compressing values in Titan, which takes value as the unit. Starting from TiDB v7.6.0, the default compression algorithm is `zstd`. | | TiKV | [`rocksdb.defaultcf.titan.min-blob-size`](/tikv-configuration-file.md#min-blob-size) | Modified | Starting from TiDB v7.6.0, the default value for new clusters is `32KB`. For existing clusters upgrading to v7.6.0, the default value `1KB` remains unchanged. | | TiKV | [`rocksdb.titan.enabled`](/tikv-configuration-file.md#enabled) | Modified | Enables or disables Titan. For v7.5.0 and earlier versions, the default value is `false`. Starting from v7.6.0, the default value is `true` for only new clusters. Existing clusters upgraded to v7.6.0 or later versions will retain the original configuration. | +| TiKV | [`cdc.incremental-scan-concurrency-limit`](/tikv-configuration-file.md#incremental-scan-concurrency-limit-new-in-v760) | Newly added | Sets the maximum queue length for the tasks of incrementally scanning historical data waiting to be executed. The default value is `10000`, which means that at most 10000 tasks can be queued for execution. | | TiKV | [`gc.num-threads`](/tikv-configuration-file.md#num-threads-new-in-v658-v714-v751-and-v760) | Newly added | When `enable-compaction-filter` is set to `false`, this parameter controls the number of GC threads. The default value is `1`. | | TiKV | [`raftstore.periodic-full-compact-start-times`](/tikv-configuration-file.md#periodic-full-compact-start-times-new-in-v760) | Newly added | Sets the specific times that TiKV initiates periodic full compaction. The default value `[]` means periodic full compaction is disabled. | | TiKV | [`raftstore.periodic-full-compact-start-max-cpu`](/tikv-configuration-file.md#periodic-full-compact-start-max-cpu-new-in-v760) | Newly added | Limits the maximum CPU usage rate for TiKV periodic full compaction. The default value is `0.1`. | +| TiKV | [`raftstore.pd-report-min-resolved-ts-interval`](/tikv-configuration-file.md#pd-report-min-resolved-ts-interval-new-in-v760) | Newly added | Renamed from [`raftstore.report-min-resolved-ts-interval`](https://docs.pingcap.com/tidb/v7.5/tikv-configuration-file#report-min-resolved-ts-interval-new-in-v600). It specifies the minimum interval for TiKV to report Resolved TS to the PD leader. The default value is `"1s"`. | | TiKV | [`zstd-dict-size`](/tikv-configuration-file.md#zstd-dict-size) | Newly added | Specifies the `zstd` dictionary compression size. The default value is `"0KB"`, which means to disable the `zstd` dictionary compression. | | TiFlash | [`logger.level`](/tiflash/tiflash-configuration.md#configure-the-tiflashtoml-file) | Modified | Changes the default value from `"debug"` to `"INFO"` to reduce the cost of logging. | | TiDB Lightning| [`tidb.pd-addr`](/tidb-lightning/tidb-lightning-configuration.md#tidb-lightning-task) | Modified | Configures the addresses of the PD Servers. Starting from v7.6.0, TiDB supports setting multiple PD addresses. | diff --git a/releases/release-8.0.0.md b/releases/release-8.0.0.md index 477b09d4f1d4b..c82bd5d0bef41 100644 --- a/releases/release-8.0.0.md +++ b/releases/release-8.0.0.md @@ -336,7 +336,9 @@ Quick access: [Quick start](https://docs.pingcap.com/tidb/v8.0/quick-start-with- | TiKV | [`log-backup.initial-scan-rate-limit`](/tikv-configuration-file.md#initial-scan-rate-limit-new-in-v620) | Modified | Adds a limit of `1MiB` as the minimum value. | | TiKV | [`raftstore.store-io-pool-size`](/tikv-configuration-file.md#store-io-pool-size-new-in-v530) | Modified | Changes the default value from `0` to `1` to improve TiKV performance, meaning that the size of the StoreWriter thread pool now defaults to `1`. | | TiKV | [`rocksdb.defaultcf.titan.blob-cache-size`](/tikv-configuration-file.md#blob-cache-size) | Modified | Starting from v8.0.0, TiKV introduces the `shared-blob-cache` configuration item and enables it by default, so there is no need to set `blob-cache-size` separately. The configuration of `blob-cache-size` only takes effect when `shared-blob-cache` is set to `false`. | +| TiKV | [`rocksdb.titan.max-background-gc`](/tikv-configuration-file.md#max-background-gc) | Modified | Changes the default value from `4` to `1` to reduce the occupation of thread resources by the Titan GC process. | | TiKV | [`security.encryption.master-key.vendor`](/encryption-at-rest.md#specify-a-master-key-via-kms) | Modified | Adds `gcp` as an available type for the service provider. | +| TiKV | [`storage.block-cache.low-pri-pool-ratio`](/tikv-configuration-file.md#low-pri-pool-ratio-new-in-v800) | Newly added | Specifies the proportion of the entire block cache that the Titan component can use. The default value is `0.2`. | | TiKV | [`rocksdb.defaultcf.titan.shared-blob-cache`](/tikv-configuration-file.md#shared-blob-cache-new-in-v800) | Newly added | Controls whether to enable the shared cache for Titan blob files and RocksDB block files. The default value is `true`. | | TiKV | [`security.encryption.master-key.gcp.credential-file-path`](/encryption-at-rest.md#specify-a-master-key-via-kms) | Newly added | Specifies the path to the Google Cloud authentication credentials file when `security.encryption.master-key.vendor` is `gcp`. | | PD | [`schedule.enable-heartbeat-breakdown-metrics`](/pd-configuration-file.md#enable-heartbeat-breakdown-metrics-new-in-v800) | Newly added | Controls whether to enable breakdown metrics for Region heartbeats. These metrics measure the time consumed in each stage of Region heartbeat processing, facilitating analysis through monitoring. The default value is `true`. | @@ -347,7 +349,6 @@ Quick access: [Quick start](https://docs.pingcap.com/tidb/v8.0/quick-start-with- | TiDB Lightning | [`logical-import-batch-size`](/tidb-lightning/tidb-lightning-configuration.md#tidb-lightning-task) | Newly added | Controls the maximum size of each SQL query executed on the downstream TiDB server in the logical import mode. The default value is `"96KiB"`. The unit can be KB, KiB, MB, or MiB. | | Data Migration | [`secret-key-path`](/dm/dm-master-configuration-file.md) | Newly added | Specifies the file path of the secret key, which is used to encrypt and decrypt upstream and downstream passwords. The file must contain a 64-character hexadecimal AES-256 secret key. | | TiCDC | [`debezium-disable-schema`](/ticdc/ticdc-changefeed-config.md) | Newly added | Controls whether to disable the output of schema information. This parameter only takes effect when the sink type is MQ and the output protocol is Debezium. | - | TiCDC | [`tls-certificate-file`](/ticdc/ticdc-sink-to-pulsar.md) | Newly added | Specifies the path to the encrypted certificate file on the client, which is required when Pulsar enables TLS encrypted transmission. | | TiCDC | [`tls-key-file-path`](/ticdc/ticdc-sink-to-pulsar.md) | Newly added | Specifies the path to the encrypted private key on the client, which is required when Pulsar enables TLS encrypted transmission. | diff --git a/releases/release-8.1.0.md b/releases/release-8.1.0.md index 148b07bdab94b..a3e55fea1f227 100644 --- a/releases/release-8.1.0.md +++ b/releases/release-8.1.0.md @@ -187,6 +187,7 @@ Compared with the previous LTS 7.5.0, 8.1.0 includes new features, improvements, | PD | [`enable-telemetry`](/pd-configuration-file.md#enable-telemetry) | Deprecated | Starting from v8.1.0, the telemetry feature in TiDB Dashboard is removed, and this configuration item is no longer functional. It is retained solely for compatibility with earlier versions. | | TiDB Lightning | [`conflict.max-record-rows`](/tidb-lightning/tidb-lightning-configuration.md#tidb-lightning-configuration) | Modified | Starting from v8.1.0, there is no need to configure `conflict.max-record-rows` manually, because TiDB Lightning automatically assigns the value of `conflict.max-record-rows` with the value of `conflict.threshold`, regardless of the user input. `conflict.max-record-rows` will be deprecated in a future release. | | TiDB Lightning | [`conflict.threshold`](/tidb-lightning/tidb-lightning-configuration.md#tidb-lightning-task) | Modified | Changes the default value from `9223372036854775807` to `10000` to quickly interrupt abnormal tasks so that you can make corresponding adjustments as soon as possible. This saves time and computational resources by avoiding the scenario where a large amount of conflicting data is discovered after the import, caused by abnormal data sources or incorrect table schema definitions. | +| TiKV | [`memory.enable-thread-exclusive-arena`](/tikv-configuration-file.md#enable-thread-exclusive-arena-new-in-v810) | Newly added | Controls whether to display the memory allocation status at the TiKV thread level to track the memory usage of each TiKV thread. The default value is `true`. | | TiCDC | [`security.client-allowed-user`](/ticdc/ticdc-server-config.md#cdc-server-configuration-file-parameters) | Newly added | Lists the usernames that are allowed for client authentication. Authentication requests with usernames not in this list will be rejected. The default value is null. | | TiCDC | [`security.client-user-required`](/ticdc/ticdc-server-config.md#cdc-server-configuration-file-parameters) | Newly added | Controls whether to use username and password for client authentication. The default value is `false`. | | TiCDC | [`security.mtls`](/ticdc/ticdc-server-config.md#cdc-server-configuration-file-parameters) | Newly added | Controls whether to enable the TLS client authentication. The default value is `false`. | diff --git a/releases/release-8.1.2.md b/releases/release-8.1.2.md index db5c947269cdc..e7f993e297844 100644 --- a/releases/release-8.1.2.md +++ b/releases/release-8.1.2.md @@ -1,6 +1,6 @@ --- title: TiDB 8.1.2 Release Notes -summary: Learn about the improvements and bug fixes in TiDB 8.1.2. +summary: Learn about the compatibility changes, improvements, and bug fixes in TiDB 8.1.2. --- # TiDB 8.1.2 Release Notes @@ -11,6 +11,13 @@ TiDB version: 8.1.2 Quick access: [Quick start](https://docs.pingcap.com/tidb/v8.1/quick-start-with-tidb) | [Production deployment](https://docs.pingcap.com/tidb/v8.1/production-deployment-using-tiup) +## Compatibility changes + +- Add the TiKV configuration item [`server.snap-min-ingest-size`](/tikv-configuration-file.md#snap-min-ingest-size-new-in-v812), which specifies the minimum threshold for whether TiKV adopts the ingest method when processing snapshots. The default value is `2MiB`. + + - When the snapshot size exceeds this threshold, TiKV adopts the ingest method, which imports SST files from the snapshot into RocksDB. This method is faster for large files. + - When the snapshot size does not exceed this threshold, TiKV adopts the direct write method, which writes each piece of data into RocksDB individually. This method is more efficient for small files. + ## Improvements + TiDB diff --git a/tikv-configuration-file.md b/tikv-configuration-file.md index 05a2b3e7036da..8dfcb78d84fd6 100644 --- a/tikv-configuration-file.md +++ b/tikv-configuration-file.md @@ -223,6 +223,17 @@ This document only describes the parameters that are not included in command-lin + Unit: KiB|MiB|GiB + Minimum value: `"1KiB"` +### `snap-min-ingest-size` New in v8.1.2 + ++ Specifies the minimum threshold for whether TiKV adopts the ingest method when processing snapshots. + + + When the snapshot size exceeds this threshold, TiKV adopts the ingest method, which imports SST files from the snapshot into RocksDB. This method is faster for large files. + + When the snapshot size does not exceed this threshold, TiKV adopts the direct write method, which writes each piece of data into RocksDB individually. This method is more efficient for small files. + ++ Default value: `"2MiB"` ++ Unit: KiB|MiB|GiB ++ Minimum value: `0` + ### `enable-request-batch` + Determines whether to process requests in batches @@ -515,6 +526,11 @@ Configuration items related to the sharing of block cache among multiple RocksDB + Unit: KiB|MiB|GiB +### `low-pri-pool-ratio` New in v8.0.0 + ++ Controls the proportion of the entire block cache that the Titan component can use. ++ Default value: `0.2` + ## storage.flow-control Configuration items related to the flow control mechanism in TiKV. This mechanism replaces the write stall mechanism in RocksDB and controls flow at the scheduler layer, which avoids secondary disasters caused by the stuck Raftstore or Apply threads. @@ -818,6 +834,17 @@ Configuration items related to Raftstore. + Default value: `"10s"` + Minimum value: `0` +### `pd-report-min-resolved-ts-interval` New in v7.6.0 + +> **Note:** +> +> This configuration item is renamed from [`report-min-resolved-ts-interval`](https://docs.pingcap.com/tidb/v7.5/tikv-configuration-file/#report-min-resolved-ts-interval-new-in-v600). Starting from v7.6.0, `report-min-resolved-ts-interval` is no longer effective. + ++ Specifies the minimum interval for TiKV to report Resolved TS to the PD leader. Setting it to `0` disables the reporting. ++ Default value: `"1s"`, which is the minimum positive value. Before v6.3.0, the default value is `"0s"`. ++ Minimum value: `0` ++ Unit: second + ### `snap-mgr-gc-tick-interval` + The time interval at which the recycle of expired snapshot files is triggered. `0` means that this feature is disabled. @@ -1018,13 +1045,6 @@ Configuration items related to Raftstore. + Default value: `1MiB` + Minimum value: `0` -### `report-min-resolved-ts-interval` New in v6.0.0 - -+ Determines the interval at which the minimum resolved timestamp is reported to the PD leader. If this value is set to `0`, it means that the reporting is disabled. -+ Default value: Before v6.3.0, the default value is `"0s"`. Starting from v6.3.0, the default value is `"1s"`, which is the smallest positive value. -+ Minimum value: `0` -+ Unit: second - ### `evict-cache-on-memory-ratio` New in v7.5.0 + When the memory usage of TiKV exceeds 90% of the system available memory, and the memory occupied by Raft entry cache exceeds the used memory * `evict-cache-on-memory-ratio`, TiKV evicts the Raft entry cache. @@ -1379,7 +1399,7 @@ Configuration items related to Titan. ### `max-background-gc` + The maximum number of GC threads in Titan. From the **TiKV Details** > **Thread CPU** > **RocksDB CPU** panel, if you observe that the Titan GC threads are at full capacity for a long time, consider increasing the size of the Titan GC thread pool. -+ Default value: `4` ++ Default value: `1`. Before v8.0.0, the default value is `4`. + Minimum value: `1` ## rocksdb.defaultcf | rocksdb.writecf | rocksdb.lockcf @@ -2257,6 +2277,12 @@ Configuration items related to TiCDC. + Default value: `6`, which means 6 tasks can be concurrent executed at most. + Note: The value of `incremental-scan-concurrency` must be greater than or equal to that of `incremental-scan-threads`; otherwise, TiKV will report an error at startup. +### `incremental-scan-concurrency-limit` New in v7.6.0 + ++ The maximum queue length for the tasks of incrementally scanning historical data waiting to be executed. When the number of tasks waiting to be executed exceeds this limit, new tasks will be rejected. ++ Default value: `10000`, which means that at most 10000 tasks can be queued for execution. ++ Note: `incremental-scan-concurrency-limit` must be greater than or equal to [`incremental-scan-concurrency`](#incremental-scan-concurrency); otherwise, TiKV uses `incremental-scan-concurrency` to override this configuration. + ## resolved-ts Configuration items related to maintaining the Resolved TS to serve Stale Read requests. @@ -2464,3 +2490,46 @@ Configuration items related to [Load Base Split](/configure-load-base-split.md). + Specifies the amount of data sampled by Heap Profiling each time, rounding up to the nearest power of 2. + Default value: `512KiB` +<<<<<<< HEAD +======= + +### `enable-thread-exclusive-arena` New in v8.1.0 + ++ Controls whether to display the memory allocation status at the TiKV thread level to track the memory usage of each TiKV thread. ++ Default value: `true` + +## in-memory-engine New in v8.5.0 + +TiKV MVCC in-memory engine (IME) configuration items related to the storage layer. + +### `enable` New in v8.5.0 + +> **Note:** +> +> You can configure this configuration item in the configuration file, but cannot query it via SQL statements. + ++ Whether to enable the in-memory engine to accelerate multi-version queries. For more information about the in-memory engine, see [TiKV MVCC In-Memory Engine](/tikv-in-memory-engine.md) ++ Default value: `false` (in-memory engine is disabled) ++ It is recommended to configure at least 8 GiB of memory for the TiKV node, with 32 GiB or more for optimal performance. ++ If the available memory for the TiKV node is insufficient, the in-memory engine will not be enabled even if this configuration item is set to `true`. In such cases, check the TiKV log file for messages containing `"in-memory engine is disabled because"` to learn why the in-memory engine is not enabled. + +### `capacity` New in v8.5.0 + +> **Note:** +> +> + After the in-memory engine is enabled, `block-cache.capacity` automatically decreases by 10%. +> + If you manually configure `capacity`, `block-cache.capacity` does not automatically decrease. In this case, you need to manually adjust its value to avoid OOM. + ++ Controls the maximum memory size that the in-memory engine can use. The maximum value is 5 GiB. You can manually configure it to use more memory. ++ Default value: 10% of the system memory. + +### `gc-run-interval` New in v8.5.0 + ++ Controls the time interval that the in-memory engine GC caches MVCC versions. Reducing this parameter can increase the GC frequency, and decrease the number of MVCC versions, but will increase CPU consumption for GC and increase the probability of in-memory engine cache miss. ++ Default value: `"3m"` + +### `mvcc-amplification-threshold` New in v8.5.0 + ++ Controls the threshold for MVCC read amplification when the in-memory engine selects and loads Regions. The default value is `10`, indicating that if reading a single row in a Region requires processing more than 10 MVCC versions, this Region might be loaded into the in-memory engine. ++ Default value: `10` +>>>>>>> a5b3f19d33 (tikv: recorrect the settings of some configs and supplement missing annotations for several configs. (#20871)) From d00264e453e3b47adcc73600f03f219202515f5c Mon Sep 17 00:00:00 2001 From: Grace Cai Date: Wed, 30 Apr 2025 17:51:28 +0800 Subject: [PATCH 2/2] Update tikv-configuration-file.md --- tikv-configuration-file.md | 38 -------------------------------------- 1 file changed, 38 deletions(-) diff --git a/tikv-configuration-file.md b/tikv-configuration-file.md index 8dfcb78d84fd6..977667ffa3d58 100644 --- a/tikv-configuration-file.md +++ b/tikv-configuration-file.md @@ -2490,46 +2490,8 @@ Configuration items related to [Load Base Split](/configure-load-base-split.md). + Specifies the amount of data sampled by Heap Profiling each time, rounding up to the nearest power of 2. + Default value: `512KiB` -<<<<<<< HEAD -======= ### `enable-thread-exclusive-arena` New in v8.1.0 + Controls whether to display the memory allocation status at the TiKV thread level to track the memory usage of each TiKV thread. + Default value: `true` - -## in-memory-engine New in v8.5.0 - -TiKV MVCC in-memory engine (IME) configuration items related to the storage layer. - -### `enable` New in v8.5.0 - -> **Note:** -> -> You can configure this configuration item in the configuration file, but cannot query it via SQL statements. - -+ Whether to enable the in-memory engine to accelerate multi-version queries. For more information about the in-memory engine, see [TiKV MVCC In-Memory Engine](/tikv-in-memory-engine.md) -+ Default value: `false` (in-memory engine is disabled) -+ It is recommended to configure at least 8 GiB of memory for the TiKV node, with 32 GiB or more for optimal performance. -+ If the available memory for the TiKV node is insufficient, the in-memory engine will not be enabled even if this configuration item is set to `true`. In such cases, check the TiKV log file for messages containing `"in-memory engine is disabled because"` to learn why the in-memory engine is not enabled. - -### `capacity` New in v8.5.0 - -> **Note:** -> -> + After the in-memory engine is enabled, `block-cache.capacity` automatically decreases by 10%. -> + If you manually configure `capacity`, `block-cache.capacity` does not automatically decrease. In this case, you need to manually adjust its value to avoid OOM. - -+ Controls the maximum memory size that the in-memory engine can use. The maximum value is 5 GiB. You can manually configure it to use more memory. -+ Default value: 10% of the system memory. - -### `gc-run-interval` New in v8.5.0 - -+ Controls the time interval that the in-memory engine GC caches MVCC versions. Reducing this parameter can increase the GC frequency, and decrease the number of MVCC versions, but will increase CPU consumption for GC and increase the probability of in-memory engine cache miss. -+ Default value: `"3m"` - -### `mvcc-amplification-threshold` New in v8.5.0 - -+ Controls the threshold for MVCC read amplification when the in-memory engine selects and loads Regions. The default value is `10`, indicating that if reading a single row in a Region requires processing more than 10 MVCC versions, this Region might be loaded into the in-memory engine. -+ Default value: `10` ->>>>>>> a5b3f19d33 (tikv: recorrect the settings of some configs and supplement missing annotations for several configs. (#20871))