From 2a8c757d82f13c98187bf6b50b7225368c1f19ac Mon Sep 17 00:00:00 2001 From: houfaxin Date: Sat, 12 Oct 2024 14:55:20 +0800 Subject: [PATCH 1/8] fix txn limit size --- basic-features.md | 2 +- develop/dev-guide-transaction-restraints.md | 11 +++++++++-- tidb-configuration-file.md | 1 + 3 files changed, 11 insertions(+), 3 deletions(-) diff --git a/basic-features.md b/basic-features.md index 714fff63d6197..f1bc37b19dbe8 100644 --- a/basic-features.md +++ b/basic-features.md @@ -143,7 +143,7 @@ You can try out TiDB features on [TiDB Playground](https://play.tidbcloud.com/?u |---|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:| | [Async commit](/system-variables.md#tidb_enable_async_commit-new-in-v50) | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | | [1PC](/system-variables.md#tidb_enable_1pc-new-in-v50) | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | -| [Large transactions (10GB)](/transaction-overview.md#transaction-size-limit) | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | +| [Large transactions (1 TiB)](/transaction-overview.md#transaction-size-limit) | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | | [Pessimistic transactions](/pessimistic-transaction.md) | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | | [Optimistic transactions](/optimistic-transaction.md) | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | | [Repeatable-read isolation (snapshot isolation)](/transaction-isolation-levels.md) | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | diff --git a/develop/dev-guide-transaction-restraints.md b/develop/dev-guide-transaction-restraints.md index 673628921bcc3..0fd3630ee1c34 100644 --- a/develop/dev-guide-transaction-restraints.md +++ b/develop/dev-guide-transaction-restraints.md @@ -709,8 +709,15 @@ mysql> SELECT * FROM T2; The basic principle is to limit the size of the transaction. At the KV level, TiDB has a restriction on the size of a single transaction. At the SQL level, one row of data is mapped to one KV entry, and each additional index will add one KV entry. The restriction is as follows at the SQL level: -- The maximum single row record size is `120 MB`. You can adjust it by using the `performance.txn-entry-size-limit` configuration parameter of tidb-server for TiDB v4.0.10 and later v4.0.x versions, TiDB v5.0.0 and later versions. The value is `6 MB` for versions earlier than v4.0.10. -- The maximum single transaction size supported is `10 GB`. You can configure it by `performance.txn-total-size-limit` for TiDB v4.0 and later versions. The value is `100 MB` for earlier versions. +- The maximum single row record size is 120 MiB. + + - You can adjust it by using the [`performance.txn-entry-size-limit`](/tidb-configuration-file.md#txn-entry-size-limit-new-in-v4010-and-v500) configuration parameter of tidb-server for TiDB v4.0.10 and later v4.0.x versions, TiDB v5.0.0 and later versions. The value is `6 MB` for versions earlier than v4.0.10. + - Starting from v7.6.0, you can use the [`tidb_txn_entry_size_limit`](/system-variables.md#tidb_txn_entry_size_limit-introduced-from-v760-version) system variable to dynamically modify the value of this configuration item. + +- The maximum single transaction size supported is 1 TiB. + + - You can configure it by [`performance.txn-total-size-limit`](/tidb-configuration-file.md#txn-total-size-limit) for TiDB v4.0 and later versions. The value is `100 MB` for earlier versions. + - In TiDB v6.5.0 and later versions, this configuration is no longer recommended. For more information, see [`performance.txn-total-size-limit`](/tidb-configuration-file.md#txn-total-size-limit). Note that for both the size restrictions and row restrictions, you should also consider the overhead of encoding and additional keys for the transaction during the transaction execution. To achieve optimal performance, it is recommended to write one transaction every 100 ~ 500 rows. diff --git a/tidb-configuration-file.md b/tidb-configuration-file.md index a8d1f99caa43d..0f6c8a10efdfa 100644 --- a/tidb-configuration-file.md +++ b/tidb-configuration-file.md @@ -523,6 +523,7 @@ Configuration items related to performance. - In TiDB v6.5.0 and later versions, this configuration is no longer recommended. The memory size of a transaction will be accumulated into the memory usage of the session, and the [`tidb_mem_quota_query`](/system-variables.md#tidb_mem_quota_query) variable will take effect when the session memory threshold is exceeded. To be compatible with previous versions, this configuration works as follows when you upgrade from an earlier version to TiDB v6.5.0 or later: - If this configuration is not set or is set to the default value (`104857600`), after an upgrade, the memory size of a transaction will be accumulated into the memory usage of the session, and the `tidb_mem_quota_query` variable will take effect. - If this configuration is not defaulted (`104857600`), it still takes effect and its behavior on controlling the size of a single transaction remains unchanged before and after the upgrade. This means that the memory size of the transaction is not controlled by the `tidb_mem_quota_query` variable. +- When TiDB executes transactions in the [`tidb_dml_type`](/system-variables.md#tidb_dml_type-new-in-v800) `"bulk"` mode, transaction size is not limited by the TiDB configuration item [`txn-total-size-limit`](https://docs.pingcap.com/tidb/stable/tidb-configuration-file#txn-total-size-limit). ### `tcp-keep-alive` From 073afdecd1854fabde555b92135e0e99a18bf482 Mon Sep 17 00:00:00 2001 From: xixirangrang Date: Sat, 12 Oct 2024 15:17:44 +0800 Subject: [PATCH 2/8] Apply suggestions from code review Co-authored-by: crazycs --- develop/dev-guide-transaction-restraints.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/develop/dev-guide-transaction-restraints.md b/develop/dev-guide-transaction-restraints.md index 0fd3630ee1c34..e37eafb364b91 100644 --- a/develop/dev-guide-transaction-restraints.md +++ b/develop/dev-guide-transaction-restraints.md @@ -716,8 +716,8 @@ The basic principle is to limit the size of the transaction. At the KV level, Ti - The maximum single transaction size supported is 1 TiB. - - You can configure it by [`performance.txn-total-size-limit`](/tidb-configuration-file.md#txn-total-size-limit) for TiDB v4.0 and later versions. The value is `100 MB` for earlier versions. - - In TiDB v6.5.0 and later versions, this configuration is no longer recommended. For more information, see [`performance.txn-total-size-limit`](/tidb-configuration-file.md#txn-total-size-limit). + - For TiDB v4.0 and later versions, You can configure it by [`performance.txn-total-size-limit`](/tidb-configuration-file.md#txn-total-size-limit). The value is `100 MB` for earlier versions. + - For TiDB v6.5.0 and later versions, this configuration is no longer recommended. For more information, see [`performance.txn-total-size-limit`](/tidb-configuration-file.md#txn-total-size-limit). Note that for both the size restrictions and row restrictions, you should also consider the overhead of encoding and additional keys for the transaction during the transaction execution. To achieve optimal performance, it is recommended to write one transaction every 100 ~ 500 rows. From decaf9df5d03e3453c2aab529e88c27e8a7b620e Mon Sep 17 00:00:00 2001 From: houfaxin Date: Sat, 12 Oct 2024 15:37:46 +0800 Subject: [PATCH 3/8] Update basic-features.md --- basic-features.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/basic-features.md b/basic-features.md index f1bc37b19dbe8..73d0f88b0afe0 100644 --- a/basic-features.md +++ b/basic-features.md @@ -90,6 +90,8 @@ You can try out TiDB features on [TiDB Playground](https://play.tidbcloud.com/?u | [`ALTER TABLE ... COMPACT`](/sql-statements/sql-statement-alter-table-compact.md) | Y | Y | Y | Y | Y | Y | E | N | N | N | N | | [Table Lock](/sql-statements/sql-statement-lock-tables-and-unlock-tables.md) | E | E | E | E | E | E | E | E | E | E | E | | [TiFlash Query Result Materialization](/tiflash/tiflash-results-materialization.md) | Y | Y | Y | Y | Y | E | N | N | N | N | N | +| [Standard DML execution mode (`tidb_dml_type = "standard"`)](/system-variables.md#tidb_dml_type-new-in-v800) | Y | Y | Y | N | N | N | N | N | N | N | N | +| [Bulk DML execution mode (`tidb_dml_type = "bulk"`)](/system-variables.md#tidb_dml_type-new-in-v800) | E | E | E | N | N | N | N | N | N | N | N | ## Advanced SQL features From eb5ce15908e63b408e7d5cc764b9493d205e783b Mon Sep 17 00:00:00 2001 From: xixirangrang Date: Sat, 12 Oct 2024 15:42:07 +0800 Subject: [PATCH 4/8] Update basic-features.md --- basic-features.md | 1 - 1 file changed, 1 deletion(-) diff --git a/basic-features.md b/basic-features.md index 73d0f88b0afe0..0b3c3005cab93 100644 --- a/basic-features.md +++ b/basic-features.md @@ -90,7 +90,6 @@ You can try out TiDB features on [TiDB Playground](https://play.tidbcloud.com/?u | [`ALTER TABLE ... COMPACT`](/sql-statements/sql-statement-alter-table-compact.md) | Y | Y | Y | Y | Y | Y | E | N | N | N | N | | [Table Lock](/sql-statements/sql-statement-lock-tables-and-unlock-tables.md) | E | E | E | E | E | E | E | E | E | E | E | | [TiFlash Query Result Materialization](/tiflash/tiflash-results-materialization.md) | Y | Y | Y | Y | Y | E | N | N | N | N | N | -| [Standard DML execution mode (`tidb_dml_type = "standard"`)](/system-variables.md#tidb_dml_type-new-in-v800) | Y | Y | Y | N | N | N | N | N | N | N | N | | [Bulk DML execution mode (`tidb_dml_type = "bulk"`)](/system-variables.md#tidb_dml_type-new-in-v800) | E | E | E | N | N | N | N | N | N | N | N | ## Advanced SQL features From 4cc2c30acfd9511344761cff163a97ab317da4d7 Mon Sep 17 00:00:00 2001 From: houfaxin Date: Sat, 12 Oct 2024 15:54:19 +0800 Subject: [PATCH 5/8] Update basic-features.md --- basic-features.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/basic-features.md b/basic-features.md index 0b3c3005cab93..5ee86d3527651 100644 --- a/basic-features.md +++ b/basic-features.md @@ -90,7 +90,6 @@ You can try out TiDB features on [TiDB Playground](https://play.tidbcloud.com/?u | [`ALTER TABLE ... COMPACT`](/sql-statements/sql-statement-alter-table-compact.md) | Y | Y | Y | Y | Y | Y | E | N | N | N | N | | [Table Lock](/sql-statements/sql-statement-lock-tables-and-unlock-tables.md) | E | E | E | E | E | E | E | E | E | E | E | | [TiFlash Query Result Materialization](/tiflash/tiflash-results-materialization.md) | Y | Y | Y | Y | Y | E | N | N | N | N | N | -| [Bulk DML execution mode (`tidb_dml_type = "bulk"`)](/system-variables.md#tidb_dml_type-new-in-v800) | E | E | E | N | N | N | N | N | N | N | N | ## Advanced SQL features @@ -150,6 +149,7 @@ You can try out TiDB features on [TiDB Playground](https://play.tidbcloud.com/?u | [Repeatable-read isolation (snapshot isolation)](/transaction-isolation-levels.md) | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | | [Read-committed isolation](/transaction-isolation-levels.md) | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | | [Automatically terminating long-running idle transactions](/system-variables.md#tidb_idle_transaction_timeout-new-in-v760) | Y | Y | Y | N | N | N | N | N | N | N | N | +| [Bulk DML execution mode (`tidb_dml_type = "bulk"`)](/system-variables.md#tidb_dml_type-new-in-v800) | E | E | E | N | N | N | N | N | N | N | N | ## Partitioning From 957846e6eaf9bc2bc582acd8b1ae6f9eab71a22a Mon Sep 17 00:00:00 2001 From: houfaxin Date: Sat, 12 Oct 2024 17:06:28 +0800 Subject: [PATCH 6/8] Update dev-guide-transaction-restraints.md --- develop/dev-guide-transaction-restraints.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/develop/dev-guide-transaction-restraints.md b/develop/dev-guide-transaction-restraints.md index e37eafb364b91..fdc27a4e80e6e 100644 --- a/develop/dev-guide-transaction-restraints.md +++ b/develop/dev-guide-transaction-restraints.md @@ -712,7 +712,7 @@ The basic principle is to limit the size of the transaction. At the KV level, Ti - The maximum single row record size is 120 MiB. - You can adjust it by using the [`performance.txn-entry-size-limit`](/tidb-configuration-file.md#txn-entry-size-limit-new-in-v4010-and-v500) configuration parameter of tidb-server for TiDB v4.0.10 and later v4.0.x versions, TiDB v5.0.0 and later versions. The value is `6 MB` for versions earlier than v4.0.10. - - Starting from v7.6.0, you can use the [`tidb_txn_entry_size_limit`](/system-variables.md#tidb_txn_entry_size_limit-introduced-from-v760-version) system variable to dynamically modify the value of this configuration item. + - Starting from v7.6.0, you can use the [`tidb_txn_entry_size_limit`](/system-variables.md#tidb_txn_entry_size_limit-new-in-v760) system variable to dynamically modify the value of this configuration item. - The maximum single transaction size supported is 1 TiB. From 5888cecd8815de022d28c0860d46492fe68318b3 Mon Sep 17 00:00:00 2001 From: xixirangrang Date: Sat, 12 Oct 2024 21:13:14 +0800 Subject: [PATCH 7/8] Update develop/dev-guide-transaction-restraints.md --- develop/dev-guide-transaction-restraints.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/develop/dev-guide-transaction-restraints.md b/develop/dev-guide-transaction-restraints.md index fdc27a4e80e6e..e496fc1260291 100644 --- a/develop/dev-guide-transaction-restraints.md +++ b/develop/dev-guide-transaction-restraints.md @@ -716,8 +716,8 @@ The basic principle is to limit the size of the transaction. At the KV level, Ti - The maximum single transaction size supported is 1 TiB. - - For TiDB v4.0 and later versions, You can configure it by [`performance.txn-total-size-limit`](/tidb-configuration-file.md#txn-total-size-limit). The value is `100 MB` for earlier versions. - - For TiDB v6.5.0 and later versions, this configuration is no longer recommended. For more information, see [`performance.txn-total-size-limit`](/tidb-configuration-file.md#txn-total-size-limit). + - For TiDB v4.0 and later versions, You can configure it by [`performance.txn-total-size-limit`](https://docs.pingcap.com/tidb/stable/tidb-configuration-file#txn-total-size-limit)). The value is `100 MB` for earlier versions. + - For TiDB v6.5.0 and later versions, this configuration is no longer recommended. For more information, see [`performance.txn-total-size-limit`](https://docs.pingcap.com/tidb/stable/tidb-configuration-file#txn-total-size-limit)). Note that for both the size restrictions and row restrictions, you should also consider the overhead of encoding and additional keys for the transaction during the transaction execution. To achieve optimal performance, it is recommended to write one transaction every 100 ~ 500 rows. From bc3c15849d67747b91b24fd6c871a0c7df9e9dc3 Mon Sep 17 00:00:00 2001 From: houfaxin Date: Sat, 12 Oct 2024 21:51:36 +0800 Subject: [PATCH 8/8] Update dev-guide-transaction-restraints.md --- develop/dev-guide-transaction-restraints.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/develop/dev-guide-transaction-restraints.md b/develop/dev-guide-transaction-restraints.md index e496fc1260291..e11359a8ca065 100644 --- a/develop/dev-guide-transaction-restraints.md +++ b/develop/dev-guide-transaction-restraints.md @@ -711,7 +711,7 @@ The basic principle is to limit the size of the transaction. At the KV level, Ti - The maximum single row record size is 120 MiB. - - You can adjust it by using the [`performance.txn-entry-size-limit`](/tidb-configuration-file.md#txn-entry-size-limit-new-in-v4010-and-v500) configuration parameter of tidb-server for TiDB v4.0.10 and later v4.0.x versions, TiDB v5.0.0 and later versions. The value is `6 MB` for versions earlier than v4.0.10. + - You can adjust it by using the [`performance.txn-entry-size-limit`](https://docs.pingcap.com/tidb/stable/tidb-configuration-file#txn-entry-size-limit-new-in-v4010-and-v500) configuration parameter of tidb-server for TiDB v4.0.10 and later v4.0.x versions, TiDB v5.0.0 and later versions. The value is `6 MB` for versions earlier than v4.0.10. - Starting from v7.6.0, you can use the [`tidb_txn_entry_size_limit`](/system-variables.md#tidb_txn_entry_size_limit-new-in-v760) system variable to dynamically modify the value of this configuration item. - The maximum single transaction size supported is 1 TiB.