From 0a2debe06301e1ec47acf52995a044b697d2bc4e Mon Sep 17 00:00:00 2001 From: houfaxin Date: Sat, 12 Oct 2024 14:55:20 +0800 Subject: [PATCH 1/8] fix txn limit size --- basic-features.md | 2 +- develop/dev-guide-transaction-restraints.md | 11 +++++++++-- tidb-configuration-file.md | 1 + 3 files changed, 11 insertions(+), 3 deletions(-) diff --git a/basic-features.md b/basic-features.md index 9f21339c7193c..e7c827d0d2915 100644 --- a/basic-features.md +++ b/basic-features.md @@ -142,7 +142,7 @@ You can try out TiDB features on [TiDB Playground](https://play.tidbcloud.com/?u |---|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:| | [Async commit](/system-variables.md#tidb_enable_async_commit-new-in-v50) | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | | [1PC](/system-variables.md#tidb_enable_1pc-new-in-v50) | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | -| [Large transactions (10GB)](/transaction-overview.md#transaction-size-limit) | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | +| [Large transactions (1 TiB)](/transaction-overview.md#transaction-size-limit) | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | | [Pessimistic transactions](/pessimistic-transaction.md) | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | | [Optimistic transactions](/optimistic-transaction.md) | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | | [Repeatable-read isolation (snapshot isolation)](/transaction-isolation-levels.md) | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | diff --git a/develop/dev-guide-transaction-restraints.md b/develop/dev-guide-transaction-restraints.md index 673628921bcc3..0fd3630ee1c34 100644 --- a/develop/dev-guide-transaction-restraints.md +++ b/develop/dev-guide-transaction-restraints.md @@ -709,8 +709,15 @@ mysql> SELECT * FROM T2; The basic principle is to limit the size of the transaction. At the KV level, TiDB has a restriction on the size of a single transaction. At the SQL level, one row of data is mapped to one KV entry, and each additional index will add one KV entry. The restriction is as follows at the SQL level: -- The maximum single row record size is `120 MB`. You can adjust it by using the `performance.txn-entry-size-limit` configuration parameter of tidb-server for TiDB v4.0.10 and later v4.0.x versions, TiDB v5.0.0 and later versions. The value is `6 MB` for versions earlier than v4.0.10. -- The maximum single transaction size supported is `10 GB`. You can configure it by `performance.txn-total-size-limit` for TiDB v4.0 and later versions. The value is `100 MB` for earlier versions. +- The maximum single row record size is 120 MiB. + + - You can adjust it by using the [`performance.txn-entry-size-limit`](/tidb-configuration-file.md#txn-entry-size-limit-new-in-v4010-and-v500) configuration parameter of tidb-server for TiDB v4.0.10 and later v4.0.x versions, TiDB v5.0.0 and later versions. The value is `6 MB` for versions earlier than v4.0.10. + - Starting from v7.6.0, you can use the [`tidb_txn_entry_size_limit`](/system-variables.md#tidb_txn_entry_size_limit-introduced-from-v760-version) system variable to dynamically modify the value of this configuration item. + +- The maximum single transaction size supported is 1 TiB. + + - You can configure it by [`performance.txn-total-size-limit`](/tidb-configuration-file.md#txn-total-size-limit) for TiDB v4.0 and later versions. The value is `100 MB` for earlier versions. + - In TiDB v6.5.0 and later versions, this configuration is no longer recommended. For more information, see [`performance.txn-total-size-limit`](/tidb-configuration-file.md#txn-total-size-limit). Note that for both the size restrictions and row restrictions, you should also consider the overhead of encoding and additional keys for the transaction during the transaction execution. To achieve optimal performance, it is recommended to write one transaction every 100 ~ 500 rows. diff --git a/tidb-configuration-file.md b/tidb-configuration-file.md index fb16412774738..1d9d45fca8be8 100644 --- a/tidb-configuration-file.md +++ b/tidb-configuration-file.md @@ -522,6 +522,7 @@ Configuration items related to performance. - In TiDB v6.5.0 and later versions, this configuration is no longer recommended. The memory size of a transaction will be accumulated into the memory usage of the session, and the [`tidb_mem_quota_query`](/system-variables.md#tidb_mem_quota_query) variable will take effect when the session memory threshold is exceeded. To be compatible with previous versions, this configuration works as follows when you upgrade from an earlier version to TiDB v6.5.0 or later: - If this configuration is not set or is set to the default value (`104857600`), after an upgrade, the memory size of a transaction will be accumulated into the memory usage of the session, and the `tidb_mem_quota_query` variable will take effect. - If this configuration is not defaulted (`104857600`), it still takes effect and its behavior on controlling the size of a single transaction remains unchanged before and after the upgrade. This means that the memory size of the transaction is not controlled by the `tidb_mem_quota_query` variable. +- When TiDB executes transactions in the [`tidb_dml_type`](/system-variables.md#tidb_dml_type-new-in-v800) `"bulk"` mode, transaction size is not limited by the TiDB configuration item [`txn-total-size-limit`](https://docs.pingcap.com/tidb/stable/tidb-configuration-file#txn-total-size-limit). ### `tcp-keep-alive` From 41fb6c471c5db29c446bea88b9489e9de9a00888 Mon Sep 17 00:00:00 2001 From: xixirangrang Date: Sat, 12 Oct 2024 15:17:44 +0800 Subject: [PATCH 2/8] Apply suggestions from code review Co-authored-by: crazycs --- develop/dev-guide-transaction-restraints.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/develop/dev-guide-transaction-restraints.md b/develop/dev-guide-transaction-restraints.md index 0fd3630ee1c34..e37eafb364b91 100644 --- a/develop/dev-guide-transaction-restraints.md +++ b/develop/dev-guide-transaction-restraints.md @@ -716,8 +716,8 @@ The basic principle is to limit the size of the transaction. At the KV level, Ti - The maximum single transaction size supported is 1 TiB. - - You can configure it by [`performance.txn-total-size-limit`](/tidb-configuration-file.md#txn-total-size-limit) for TiDB v4.0 and later versions. The value is `100 MB` for earlier versions. - - In TiDB v6.5.0 and later versions, this configuration is no longer recommended. For more information, see [`performance.txn-total-size-limit`](/tidb-configuration-file.md#txn-total-size-limit). + - For TiDB v4.0 and later versions, You can configure it by [`performance.txn-total-size-limit`](/tidb-configuration-file.md#txn-total-size-limit). The value is `100 MB` for earlier versions. + - For TiDB v6.5.0 and later versions, this configuration is no longer recommended. For more information, see [`performance.txn-total-size-limit`](/tidb-configuration-file.md#txn-total-size-limit). Note that for both the size restrictions and row restrictions, you should also consider the overhead of encoding and additional keys for the transaction during the transaction execution. To achieve optimal performance, it is recommended to write one transaction every 100 ~ 500 rows. From 22293f995c51a34d71f8ad16e2fc1a313cf67fdb Mon Sep 17 00:00:00 2001 From: houfaxin Date: Sat, 12 Oct 2024 15:37:46 +0800 Subject: [PATCH 3/8] Update basic-features.md --- basic-features.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/basic-features.md b/basic-features.md index e7c827d0d2915..9584d81f62e41 100644 --- a/basic-features.md +++ b/basic-features.md @@ -89,6 +89,8 @@ You can try out TiDB features on [TiDB Playground](https://play.tidbcloud.com/?u | [`ALTER TABLE ... COMPACT`](/sql-statements/sql-statement-alter-table-compact.md) | Y | Y | Y | Y | Y | Y | E | N | N | N | N | | [Table Lock](/sql-statements/sql-statement-lock-tables-and-unlock-tables.md) | E | E | E | E | E | E | E | E | E | E | E | | [TiFlash Query Result Materialization](/tiflash/tiflash-results-materialization.md) | Y | Y | Y | Y | Y | E | N | N | N | N | N | +| [Standard DML execution mode (`tidb_dml_type = "standard"`)](/system-variables.md#tidb_dml_type-new-in-v800) | Y | Y | Y | N | N | N | N | N | N | N | N | +| [Bulk DML execution mode (`tidb_dml_type = "bulk"`)](/system-variables.md#tidb_dml_type-new-in-v800) | E | E | E | N | N | N | N | N | N | N | N | ## Advanced SQL features From 9ba2a394b2bcb1003aed708219aa4dc1f319613a Mon Sep 17 00:00:00 2001 From: xixirangrang Date: Sat, 12 Oct 2024 15:42:07 +0800 Subject: [PATCH 4/8] Update basic-features.md --- basic-features.md | 1 - 1 file changed, 1 deletion(-) diff --git a/basic-features.md b/basic-features.md index 9584d81f62e41..1782d4930a670 100644 --- a/basic-features.md +++ b/basic-features.md @@ -89,7 +89,6 @@ You can try out TiDB features on [TiDB Playground](https://play.tidbcloud.com/?u | [`ALTER TABLE ... COMPACT`](/sql-statements/sql-statement-alter-table-compact.md) | Y | Y | Y | Y | Y | Y | E | N | N | N | N | | [Table Lock](/sql-statements/sql-statement-lock-tables-and-unlock-tables.md) | E | E | E | E | E | E | E | E | E | E | E | | [TiFlash Query Result Materialization](/tiflash/tiflash-results-materialization.md) | Y | Y | Y | Y | Y | E | N | N | N | N | N | -| [Standard DML execution mode (`tidb_dml_type = "standard"`)](/system-variables.md#tidb_dml_type-new-in-v800) | Y | Y | Y | N | N | N | N | N | N | N | N | | [Bulk DML execution mode (`tidb_dml_type = "bulk"`)](/system-variables.md#tidb_dml_type-new-in-v800) | E | E | E | N | N | N | N | N | N | N | N | ## Advanced SQL features From f6cff4f6675ac7eb084e64347a487b485c881a54 Mon Sep 17 00:00:00 2001 From: houfaxin Date: Sat, 12 Oct 2024 15:54:19 +0800 Subject: [PATCH 5/8] Update basic-features.md --- basic-features.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/basic-features.md b/basic-features.md index 1782d4930a670..8086516acbef8 100644 --- a/basic-features.md +++ b/basic-features.md @@ -89,7 +89,6 @@ You can try out TiDB features on [TiDB Playground](https://play.tidbcloud.com/?u | [`ALTER TABLE ... COMPACT`](/sql-statements/sql-statement-alter-table-compact.md) | Y | Y | Y | Y | Y | Y | E | N | N | N | N | | [Table Lock](/sql-statements/sql-statement-lock-tables-and-unlock-tables.md) | E | E | E | E | E | E | E | E | E | E | E | | [TiFlash Query Result Materialization](/tiflash/tiflash-results-materialization.md) | Y | Y | Y | Y | Y | E | N | N | N | N | N | -| [Bulk DML execution mode (`tidb_dml_type = "bulk"`)](/system-variables.md#tidb_dml_type-new-in-v800) | E | E | E | N | N | N | N | N | N | N | N | ## Advanced SQL features @@ -149,6 +148,7 @@ You can try out TiDB features on [TiDB Playground](https://play.tidbcloud.com/?u | [Repeatable-read isolation (snapshot isolation)](/transaction-isolation-levels.md) | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | | [Read-committed isolation](/transaction-isolation-levels.md) | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | | [Automatically terminating long-running idle transactions](/system-variables.md#tidb_idle_transaction_timeout-new-in-v760) | Y | Y | Y | N | N | N | N | N | N | N | N | +| [Bulk DML execution mode (`tidb_dml_type = "bulk"`)](/system-variables.md#tidb_dml_type-new-in-v800) | E | E | E | N | N | N | N | N | N | N | N | ## Partitioning From d38e74d334b6ce3cbf1d0bb66cd6d0434804b147 Mon Sep 17 00:00:00 2001 From: houfaxin Date: Sat, 12 Oct 2024 17:06:28 +0800 Subject: [PATCH 6/8] Update dev-guide-transaction-restraints.md --- develop/dev-guide-transaction-restraints.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/develop/dev-guide-transaction-restraints.md b/develop/dev-guide-transaction-restraints.md index e37eafb364b91..fdc27a4e80e6e 100644 --- a/develop/dev-guide-transaction-restraints.md +++ b/develop/dev-guide-transaction-restraints.md @@ -712,7 +712,7 @@ The basic principle is to limit the size of the transaction. At the KV level, Ti - The maximum single row record size is 120 MiB. - You can adjust it by using the [`performance.txn-entry-size-limit`](/tidb-configuration-file.md#txn-entry-size-limit-new-in-v4010-and-v500) configuration parameter of tidb-server for TiDB v4.0.10 and later v4.0.x versions, TiDB v5.0.0 and later versions. The value is `6 MB` for versions earlier than v4.0.10. - - Starting from v7.6.0, you can use the [`tidb_txn_entry_size_limit`](/system-variables.md#tidb_txn_entry_size_limit-introduced-from-v760-version) system variable to dynamically modify the value of this configuration item. + - Starting from v7.6.0, you can use the [`tidb_txn_entry_size_limit`](/system-variables.md#tidb_txn_entry_size_limit-new-in-v760) system variable to dynamically modify the value of this configuration item. - The maximum single transaction size supported is 1 TiB. From 6fb9d7e2dfcb87bb047fb5e2d5a8b96fb9a65bfc Mon Sep 17 00:00:00 2001 From: xixirangrang Date: Sat, 12 Oct 2024 21:13:14 +0800 Subject: [PATCH 7/8] Update develop/dev-guide-transaction-restraints.md --- develop/dev-guide-transaction-restraints.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/develop/dev-guide-transaction-restraints.md b/develop/dev-guide-transaction-restraints.md index fdc27a4e80e6e..e496fc1260291 100644 --- a/develop/dev-guide-transaction-restraints.md +++ b/develop/dev-guide-transaction-restraints.md @@ -716,8 +716,8 @@ The basic principle is to limit the size of the transaction. At the KV level, Ti - The maximum single transaction size supported is 1 TiB. - - For TiDB v4.0 and later versions, You can configure it by [`performance.txn-total-size-limit`](/tidb-configuration-file.md#txn-total-size-limit). The value is `100 MB` for earlier versions. - - For TiDB v6.5.0 and later versions, this configuration is no longer recommended. For more information, see [`performance.txn-total-size-limit`](/tidb-configuration-file.md#txn-total-size-limit). + - For TiDB v4.0 and later versions, You can configure it by [`performance.txn-total-size-limit`](https://docs.pingcap.com/tidb/stable/tidb-configuration-file#txn-total-size-limit)). The value is `100 MB` for earlier versions. + - For TiDB v6.5.0 and later versions, this configuration is no longer recommended. For more information, see [`performance.txn-total-size-limit`](https://docs.pingcap.com/tidb/stable/tidb-configuration-file#txn-total-size-limit)). Note that for both the size restrictions and row restrictions, you should also consider the overhead of encoding and additional keys for the transaction during the transaction execution. To achieve optimal performance, it is recommended to write one transaction every 100 ~ 500 rows. From ed195dbab926dd03fc9841164303ca356b308731 Mon Sep 17 00:00:00 2001 From: houfaxin Date: Sat, 12 Oct 2024 21:51:36 +0800 Subject: [PATCH 8/8] Update dev-guide-transaction-restraints.md --- develop/dev-guide-transaction-restraints.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/develop/dev-guide-transaction-restraints.md b/develop/dev-guide-transaction-restraints.md index e496fc1260291..e11359a8ca065 100644 --- a/develop/dev-guide-transaction-restraints.md +++ b/develop/dev-guide-transaction-restraints.md @@ -711,7 +711,7 @@ The basic principle is to limit the size of the transaction. At the KV level, Ti - The maximum single row record size is 120 MiB. - - You can adjust it by using the [`performance.txn-entry-size-limit`](/tidb-configuration-file.md#txn-entry-size-limit-new-in-v4010-and-v500) configuration parameter of tidb-server for TiDB v4.0.10 and later v4.0.x versions, TiDB v5.0.0 and later versions. The value is `6 MB` for versions earlier than v4.0.10. + - You can adjust it by using the [`performance.txn-entry-size-limit`](https://docs.pingcap.com/tidb/stable/tidb-configuration-file#txn-entry-size-limit-new-in-v4010-and-v500) configuration parameter of tidb-server for TiDB v4.0.10 and later v4.0.x versions, TiDB v5.0.0 and later versions. The value is `6 MB` for versions earlier than v4.0.10. - Starting from v7.6.0, you can use the [`tidb_txn_entry_size_limit`](/system-variables.md#tidb_txn_entry_size_limit-new-in-v760) system variable to dynamically modify the value of this configuration item. - The maximum single transaction size supported is 1 TiB.