From f4972cfca863fdac9eb88479970cf6672c776fd8 Mon Sep 17 00:00:00 2001 From: Peter Oyebanji Date: Tue, 9 Jan 2024 22:44:11 +0100 Subject: [PATCH 01/41] functions-and-operators: enrich instructions for string functions CHAR_LENGTH() and CHARACTER_LENGTH() improve content on string functions --- functions-and-operators/string-functions.md | 28 +++++++++++++++++++-- 1 file changed, 26 insertions(+), 2 deletions(-) diff --git a/functions-and-operators/string-functions.md b/functions-and-operators/string-functions.md index 78d027cd6e9c0..fb06647f58a76 100644 --- a/functions-and-operators/string-functions.md +++ b/functions-and-operators/string-functions.md @@ -115,11 +115,35 @@ Return the character for each integer passed. ### [`CHAR_LENGTH()`](https://dev.mysql.com/doc/refman/8.0/en/string-functions.html#function_char-length) -Return number of characters in argument. +The `CHAR_LENGTH()` function is used to get the number of characters in a specified expression. It returns the total number of strings in a specified expression as an integer value. + +Examples: + +```sql +SELECT CHAR_LENGTH("TiDB") AS LengthOfString; + ++----------+ +| LengthOfString | ++----------+ +| 4 | ++----------+ +``` + +```sql +SELECT CustomerName, CHAR_LENGTH(CustomerName) AS LenghtOfName FROM Customers; + +| CustomerName|LengthOfName | +|-------------|--------------| +| Albert Einstein | 15 +| Robert Oppenheimer | 18 +``` +> **Note:** +> +> The second example operates under the assumption that there is a MySQL database with a record titled `Customers` and a field inside titled `CustomerName` ### [`CHARACTER_LENGTH()`](https://dev.mysql.com/doc/refman/8.0/en/string-functions.html#function_character-length) -Synonym for `CHAR_LENGTH()`. +The `CHARACTER_LENGTH()` function is the same as the `CHAR_LENGTH` function. Both functions can be used synonymously because they give the same outcome. ### [`CONCAT()`](https://dev.mysql.com/doc/refman/8.0/en/string-functions.html#function_concat) From ac47fca9c79b61c0490574960304b207d58d0e17 Mon Sep 17 00:00:00 2001 From: Peter Oyebanji Date: Tue, 9 Jan 2024 23:22:33 +0100 Subject: [PATCH 02/41] functions-and-operators: fix table rendering error --- functions-and-operators/string-functions.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/functions-and-operators/string-functions.md b/functions-and-operators/string-functions.md index fb06647f58a76..a0c7ad3a4daff 100644 --- a/functions-and-operators/string-functions.md +++ b/functions-and-operators/string-functions.md @@ -134,8 +134,8 @@ SELECT CustomerName, CHAR_LENGTH(CustomerName) AS LenghtOfName FROM Customers; | CustomerName|LengthOfName | |-------------|--------------| -| Albert Einstein | 15 -| Robert Oppenheimer | 18 +| Albert Einstein | 15 | +| Robert Oppenheimer | 18 | ``` > **Note:** > From 372b54db5dcc41f29e44840dae8b5c698a69fa80 Mon Sep 17 00:00:00 2001 From: Peter Oyebanji <127598375+PitifulPete@users.noreply.github.com> Date: Wed, 10 Jan 2024 09:57:14 +0100 Subject: [PATCH 03/41] Apply suggestions from code review MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Implemented corrections suggested by @dveeden and responded to one comment Co-authored-by: Daniël van Eeden --- functions-and-operators/string-functions.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/functions-and-operators/string-functions.md b/functions-and-operators/string-functions.md index a0c7ad3a4daff..c9484a5b6334d 100644 --- a/functions-and-operators/string-functions.md +++ b/functions-and-operators/string-functions.md @@ -115,7 +115,7 @@ Return the character for each integer passed. ### [`CHAR_LENGTH()`](https://dev.mysql.com/doc/refman/8.0/en/string-functions.html#function_char-length) -The `CHAR_LENGTH()` function is used to get the number of characters in a specified expression. It returns the total number of strings in a specified expression as an integer value. +The `CHAR_LENGTH()` function is used to get the number of characters in a specified expression. It returns the total number of characters in a specified expression as an integer value. Examples: @@ -137,9 +137,10 @@ SELECT CustomerName, CHAR_LENGTH(CustomerName) AS LenghtOfName FROM Customers; | Albert Einstein | 15 | | Robert Oppenheimer | 18 | ``` + > **Note:** > -> The second example operates under the assumption that there is a MySQL database with a record titled `Customers` and a field inside titled `CustomerName` +> The second example operates under the assumption that there is a database with a record titled `Customers` and a field inside titled `CustomerName` ### [`CHARACTER_LENGTH()`](https://dev.mysql.com/doc/refman/8.0/en/string-functions.html#function_character-length) From 031d59d0e2f935295315a1ae56cd58fd9c843a37 Mon Sep 17 00:00:00 2001 From: Peter Oyebanji Date: Wed, 10 Jan 2024 14:46:00 +0100 Subject: [PATCH 04/41] functions-and-operators: enrich instructions for BIT_LENGTH and CHAR() improve string functions documentation --- functions-and-operators/string-functions.md | 47 ++++++++++++++++++++- 1 file changed, 45 insertions(+), 2 deletions(-) diff --git a/functions-and-operators/string-functions.md b/functions-and-operators/string-functions.md index c9484a5b6334d..5c5457dd5ae44 100644 --- a/functions-and-operators/string-functions.md +++ b/functions-and-operators/string-functions.md @@ -107,11 +107,54 @@ Return a string containing binary representation of a number. ### [`BIT_LENGTH()`](https://dev.mysql.com/doc/refman/8.0/en/string-functions.html#function_bit-length) -Return length of argument in bits. +The `BIT_LENGTH()` function is used to return the lenght of a given string in bits. It counts the total number of characters in a given string and then provides the result in bits. + +Examples: + +```sql +SELECT BIT_LENGTH("TiDB"); ++----------+ +| BIT_LENGTH("TiDB") | ++----------+ +| 8 (T) + 8 (i) + 8 (D) + 8 (B) = 40 bits | ++----------+ +``` +```sql +SELECT CustomerName, BIT_LENGTH(CustomerName) AS BitLengthOfName FROM Customers; +| CustomerName|BitLengthOfName | +|-------------|----------------| +| Albert Einstein | 120 bits | +| Robert Oppenheimer | 144 bits | +``` + +> **Note:** +> +> The second example operates under the assumption that there is a database with a record titled `Customers` and a field inside titled `CustomerName` ### [`CHAR()`](https://dev.mysql.com/doc/refman/8.0/en/string-functions.html#function_char) -Return the character for each integer passed. +The `CHAR()` function is used to get the character value of a specific number. It performs the opposite function of `ASCII()`. Unlike `ASCII()`, which can get the ASCII value of a specific character, `CHAR()` can get the character of a specific ASCII value. + +Examples: + +```sql +SELECT CHAR(65); + ++------------+ +| CHAR(65) | ++------------+ +| A | ++------------+ +``` +```sql +SELECT CHAR(84); + ++------------+ +| CHAR(84) | ++------------+ +| T | ++------------+ +``` ### [`CHAR_LENGTH()`](https://dev.mysql.com/doc/refman/8.0/en/string-functions.html#function_char-length) From a9cb5d17531d4a7fbff5a5a32ee936a1473b8e57 Mon Sep 17 00:00:00 2001 From: Peter Oyebanji Date: Tue, 9 Jan 2024 22:44:11 +0100 Subject: [PATCH 05/41] functions-and-operators: enrich instructions for string functions CHAR_LENGTH() and CHARACTER_LENGTH() improve content on string functions --- functions-and-operators/string-functions.md | 28 +++++++++++++++++++-- 1 file changed, 26 insertions(+), 2 deletions(-) diff --git a/functions-and-operators/string-functions.md b/functions-and-operators/string-functions.md index 78d027cd6e9c0..fb06647f58a76 100644 --- a/functions-and-operators/string-functions.md +++ b/functions-and-operators/string-functions.md @@ -115,11 +115,35 @@ Return the character for each integer passed. ### [`CHAR_LENGTH()`](https://dev.mysql.com/doc/refman/8.0/en/string-functions.html#function_char-length) -Return number of characters in argument. +The `CHAR_LENGTH()` function is used to get the number of characters in a specified expression. It returns the total number of strings in a specified expression as an integer value. + +Examples: + +```sql +SELECT CHAR_LENGTH("TiDB") AS LengthOfString; + ++----------+ +| LengthOfString | ++----------+ +| 4 | ++----------+ +``` + +```sql +SELECT CustomerName, CHAR_LENGTH(CustomerName) AS LenghtOfName FROM Customers; + +| CustomerName|LengthOfName | +|-------------|--------------| +| Albert Einstein | 15 +| Robert Oppenheimer | 18 +``` +> **Note:** +> +> The second example operates under the assumption that there is a MySQL database with a record titled `Customers` and a field inside titled `CustomerName` ### [`CHARACTER_LENGTH()`](https://dev.mysql.com/doc/refman/8.0/en/string-functions.html#function_character-length) -Synonym for `CHAR_LENGTH()`. +The `CHARACTER_LENGTH()` function is the same as the `CHAR_LENGTH` function. Both functions can be used synonymously because they give the same outcome. ### [`CONCAT()`](https://dev.mysql.com/doc/refman/8.0/en/string-functions.html#function_concat) From 83b3f6b2326ad4df79cedcae1153bb2ae71b5c90 Mon Sep 17 00:00:00 2001 From: Peter Oyebanji Date: Tue, 9 Jan 2024 23:22:33 +0100 Subject: [PATCH 06/41] functions-and-operators: fix table rendering error --- functions-and-operators/string-functions.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/functions-and-operators/string-functions.md b/functions-and-operators/string-functions.md index fb06647f58a76..a0c7ad3a4daff 100644 --- a/functions-and-operators/string-functions.md +++ b/functions-and-operators/string-functions.md @@ -134,8 +134,8 @@ SELECT CustomerName, CHAR_LENGTH(CustomerName) AS LenghtOfName FROM Customers; | CustomerName|LengthOfName | |-------------|--------------| -| Albert Einstein | 15 -| Robert Oppenheimer | 18 +| Albert Einstein | 15 | +| Robert Oppenheimer | 18 | ``` > **Note:** > From f2ad58b6dbfd3b795916580089ee7150ef2cca74 Mon Sep 17 00:00:00 2001 From: Peter Oyebanji <127598375+PitifulPete@users.noreply.github.com> Date: Wed, 10 Jan 2024 09:57:14 +0100 Subject: [PATCH 07/41] Apply suggestions from code review MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Implemented corrections suggested by @dveeden and responded to one comment Co-authored-by: Daniël van Eeden --- functions-and-operators/string-functions.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/functions-and-operators/string-functions.md b/functions-and-operators/string-functions.md index a0c7ad3a4daff..c9484a5b6334d 100644 --- a/functions-and-operators/string-functions.md +++ b/functions-and-operators/string-functions.md @@ -115,7 +115,7 @@ Return the character for each integer passed. ### [`CHAR_LENGTH()`](https://dev.mysql.com/doc/refman/8.0/en/string-functions.html#function_char-length) -The `CHAR_LENGTH()` function is used to get the number of characters in a specified expression. It returns the total number of strings in a specified expression as an integer value. +The `CHAR_LENGTH()` function is used to get the number of characters in a specified expression. It returns the total number of characters in a specified expression as an integer value. Examples: @@ -137,9 +137,10 @@ SELECT CustomerName, CHAR_LENGTH(CustomerName) AS LenghtOfName FROM Customers; | Albert Einstein | 15 | | Robert Oppenheimer | 18 | ``` + > **Note:** > -> The second example operates under the assumption that there is a MySQL database with a record titled `Customers` and a field inside titled `CustomerName` +> The second example operates under the assumption that there is a database with a record titled `Customers` and a field inside titled `CustomerName` ### [`CHARACTER_LENGTH()`](https://dev.mysql.com/doc/refman/8.0/en/string-functions.html#function_character-length) From 0c232f2c81237de0257c4569013c96f6b95923db Mon Sep 17 00:00:00 2001 From: Peter Oyebanji Date: Wed, 10 Jan 2024 14:46:00 +0100 Subject: [PATCH 08/41] functions-and-operators: enrich instructions for BIT_LENGTH and CHAR() improve string functions documentation --- functions-and-operators/string-functions.md | 47 ++++++++++++++++++++- 1 file changed, 45 insertions(+), 2 deletions(-) diff --git a/functions-and-operators/string-functions.md b/functions-and-operators/string-functions.md index c9484a5b6334d..5c5457dd5ae44 100644 --- a/functions-and-operators/string-functions.md +++ b/functions-and-operators/string-functions.md @@ -107,11 +107,54 @@ Return a string containing binary representation of a number. ### [`BIT_LENGTH()`](https://dev.mysql.com/doc/refman/8.0/en/string-functions.html#function_bit-length) -Return length of argument in bits. +The `BIT_LENGTH()` function is used to return the lenght of a given string in bits. It counts the total number of characters in a given string and then provides the result in bits. + +Examples: + +```sql +SELECT BIT_LENGTH("TiDB"); ++----------+ +| BIT_LENGTH("TiDB") | ++----------+ +| 8 (T) + 8 (i) + 8 (D) + 8 (B) = 40 bits | ++----------+ +``` +```sql +SELECT CustomerName, BIT_LENGTH(CustomerName) AS BitLengthOfName FROM Customers; +| CustomerName|BitLengthOfName | +|-------------|----------------| +| Albert Einstein | 120 bits | +| Robert Oppenheimer | 144 bits | +``` + +> **Note:** +> +> The second example operates under the assumption that there is a database with a record titled `Customers` and a field inside titled `CustomerName` ### [`CHAR()`](https://dev.mysql.com/doc/refman/8.0/en/string-functions.html#function_char) -Return the character for each integer passed. +The `CHAR()` function is used to get the character value of a specific number. It performs the opposite function of `ASCII()`. Unlike `ASCII()`, which can get the ASCII value of a specific character, `CHAR()` can get the character of a specific ASCII value. + +Examples: + +```sql +SELECT CHAR(65); + ++------------+ +| CHAR(65) | ++------------+ +| A | ++------------+ +``` +```sql +SELECT CHAR(84); + ++------------+ +| CHAR(84) | ++------------+ +| T | ++------------+ +``` ### [`CHAR_LENGTH()`](https://dev.mysql.com/doc/refman/8.0/en/string-functions.html#function_char-length) From 4500edc141e286ca779e810f3f844245452bf41e Mon Sep 17 00:00:00 2001 From: Peter Oyebanji Date: Thu, 11 Jan 2024 14:51:15 +0100 Subject: [PATCH 09/41] functions-and-operators: correct errors in descriptions of both functions and add new details add details and examples that modify previous descriptions --- functions-and-operators/string-functions.md | 44 ++++++++++++++++++++- 1 file changed, 42 insertions(+), 2 deletions(-) diff --git a/functions-and-operators/string-functions.md b/functions-and-operators/string-functions.md index 5c5457dd5ae44..834f7da554306 100644 --- a/functions-and-operators/string-functions.md +++ b/functions-and-operators/string-functions.md @@ -107,7 +107,7 @@ Return a string containing binary representation of a number. ### [`BIT_LENGTH()`](https://dev.mysql.com/doc/refman/8.0/en/string-functions.html#function_bit-length) -The `BIT_LENGTH()` function is used to return the lenght of a given string in bits. It counts the total number of characters in a given string and then provides the result in bits. +The `BIT_LENGTH()` function is used to return the lenght of a given argument in bits. It calculates the length of the argument and then returns the result in bits. Examples: @@ -119,6 +119,7 @@ SELECT BIT_LENGTH("TiDB"); | 8 (T) + 8 (i) + 8 (D) + 8 (B) = 40 bits | +----------+ ``` + ```sql SELECT CustomerName, BIT_LENGTH(CustomerName) AS BitLengthOfName FROM Customers; | CustomerName|BitLengthOfName | @@ -127,13 +128,22 @@ SELECT CustomerName, BIT_LENGTH(CustomerName) AS BitLengthOfName FROM Customers; | Robert Oppenheimer | 144 bits | ``` +```sql +SELECT BIT_LENGTH("PingCap 123") ++----------+ +| BIT_LENGTH("PingCap 123") | ++----------+ +| 8 (P) + 8 (i) + 8 (n) + 8 (g) + 8 (C) + 8 (a) + 8 (p) + 8 () + 8 (1) + 8 (2) + 8 (3) = 88 bits | ++----------+ +``` + > **Note:** > > The second example operates under the assumption that there is a database with a record titled `Customers` and a field inside titled `CustomerName` ### [`CHAR()`](https://dev.mysql.com/doc/refman/8.0/en/string-functions.html#function_char) -The `CHAR()` function is used to get the character value of a specific number. It performs the opposite function of `ASCII()`. Unlike `ASCII()`, which can get the ASCII value of a specific character, `CHAR()` can get the character of a specific ASCII value. +The `CHAR()` function is used to get the corresponding character value of a specific ASCII value. It performs the opposite function of `ASCII()`. Unlike `ASCII()`, which can get the ASCII value of a specific character, `CHAR()` can get the character of a specific ASCII value. Examples: @@ -146,6 +156,7 @@ SELECT CHAR(65); | A | +------------+ ``` + ```sql SELECT CHAR(84); @@ -155,6 +166,35 @@ SELECT CHAR(84); | T | +------------+ ``` +> **Note:** +> +> The `CHAR()` function can also be used to get the corresponding character values of ASCII values that extend beyond the standard ASCII range (0 - 127). It can also get the corresponding character value of a unicode value. + +Further Examples: + +```sql +/*For extended ASCII: */ + +SELECT CHAR(128); + ++------------+ +| CHAR(128) | ++------------+ +| Ç | ++------------+ +``` + +```sql +/* For Unicode: */ + +SELECT CHAR (233); + ++------------+ +| CHAR(233) | ++------------+ +| é | ++------------+ +``` ### [`CHAR_LENGTH()`](https://dev.mysql.com/doc/refman/8.0/en/string-functions.html#function_char-length) From 600dccf1fad2bbdce33ddb24c82e60274cfba22f Mon Sep 17 00:00:00 2001 From: Peter Oyebanji <127598375+PitifulPete@users.noreply.github.com> Date: Thu, 11 Jan 2024 17:57:49 +0100 Subject: [PATCH 10/41] Apply suggestions from code review MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-authored-by: Daniël van Eeden --- functions-and-operators/string-functions.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/functions-and-operators/string-functions.md b/functions-and-operators/string-functions.md index f6ddb31daa042..390ab3e823444 100644 --- a/functions-and-operators/string-functions.md +++ b/functions-and-operators/string-functions.md @@ -107,7 +107,7 @@ Return a string containing binary representation of a number. ### [`BIT_LENGTH()`](https://dev.mysql.com/doc/refman/8.0/en/string-functions.html#function_bit-length) -The `BIT_LENGTH()` function is used to return the lenght of a given argument in bits. It calculates the length of the argument and then returns the result in bits. +The `BIT_LENGTH()` function is used to return the length of a given argument in bits. It calculates the length of the argument and then returns the result in bits. Examples: @@ -188,7 +188,7 @@ SELECT CHAR(128); ```sql /* For Unicode: */ -SELECT CHAR (233); +SELECT CHAR(233); +------------+ | CHAR(233) | From 4b015b07506d0fb71f61b91263b7ebedfc489363 Mon Sep 17 00:00:00 2001 From: Peter Oyebanji Date: Thu, 11 Jan 2024 18:52:32 +0100 Subject: [PATCH 11/41] functions-and-operators: correct one example in BIT_LENGTH and two examples in CHAR --- functions-and-operators/string-functions.md | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-) diff --git a/functions-and-operators/string-functions.md b/functions-and-operators/string-functions.md index 390ab3e823444..19d6d3e28857b 100644 --- a/functions-and-operators/string-functions.md +++ b/functions-and-operators/string-functions.md @@ -116,7 +116,7 @@ SELECT BIT_LENGTH("TiDB"); +----------+ | BIT_LENGTH("TiDB") | +----------+ -| 8 (T) + 8 (i) + 8 (D) + 8 (B) = 40 bits | +| 8 (T) + 8 (i) + 8 (D) + 8 (B) = 32 bits | +----------+ ``` @@ -129,7 +129,7 @@ SELECT CustomerName, BIT_LENGTH(CustomerName) AS BitLengthOfName FROM Customers; ``` ```sql -SELECT BIT_LENGTH("PingCap 123") +SELECT BIT_LENGTH("PingCap 123"); +----------+ | BIT_LENGTH("PingCap 123") | +----------+ @@ -181,17 +181,19 @@ SELECT CHAR(128); +------------+ | CHAR(128) | +------------+ -| Ç | +| 0x80 | +------------+ ``` ```sql /* For Unicode: */ -SELECT CHAR(233); +--skip-binary-as-hex + +SELECT CHAR(50089); +------------+ -| CHAR(233) | +| CHAR(50089) | +------------+ | é | +------------+ From 8280c1d56eab22540c008d1a79a6fe335f455753 Mon Sep 17 00:00:00 2001 From: Peter Oyebanji Date: Fri, 12 Jan 2024 08:57:31 +0100 Subject: [PATCH 12/41] functions-and-operators: fix errors in BIT_LENGTH() examples --- functions-and-operators/string-functions.md | 12 ++++++++---- 1 file changed, 8 insertions(+), 4 deletions(-) diff --git a/functions-and-operators/string-functions.md b/functions-and-operators/string-functions.md index 19d6d3e28857b..3f44eee72d1d4 100644 --- a/functions-and-operators/string-functions.md +++ b/functions-and-operators/string-functions.md @@ -116,16 +116,18 @@ SELECT BIT_LENGTH("TiDB"); +----------+ | BIT_LENGTH("TiDB") | +----------+ -| 8 (T) + 8 (i) + 8 (D) + 8 (B) = 32 bits | +| 32 | +----------+ ``` +8 bits per character x 4 characters = 32 bits + ```sql SELECT CustomerName, BIT_LENGTH(CustomerName) AS BitLengthOfName FROM Customers; | CustomerName|BitLengthOfName | |-------------|----------------| -| Albert Einstein | 120 bits | -| Robert Oppenheimer | 144 bits | +| Albert Einstein | 120 | +| Robert Oppenheimer | 144 | ``` ```sql @@ -133,10 +135,12 @@ SELECT BIT_LENGTH("PingCap 123"); +----------+ | BIT_LENGTH("PingCap 123") | +----------+ -| 8 (P) + 8 (i) + 8 (n) + 8 (g) + 8 (C) + 8 (a) + 8 (p) + 8 () + 8 (1) + 8 (2) + 8 (3) = 88 bits | +| 88 | +----------+ ``` +8 bits per character (space is counted as it is non-alphanumeric character) x 11 characters = 88 bits + > **Note:** > > The second example operates under the assumption that there is a database with a record titled `Customers` and a field inside titled `CustomerName` From 86b275acb5620858609763351f8073947aeca63c Mon Sep 17 00:00:00 2001 From: Peter Oyebanji Date: Fri, 12 Jan 2024 09:57:29 +0100 Subject: [PATCH 13/41] functions-and-operators: fix wordings in BIT_LENGTH() --- functions-and-operators/string-functions.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/functions-and-operators/string-functions.md b/functions-and-operators/string-functions.md index 3f44eee72d1d4..32c189e4a530a 100644 --- a/functions-and-operators/string-functions.md +++ b/functions-and-operators/string-functions.md @@ -139,11 +139,11 @@ SELECT BIT_LENGTH("PingCap 123"); +----------+ ``` -8 bits per character (space is counted as it is non-alphanumeric character) x 11 characters = 88 bits +8 bits per character (space is counted as it is a non-alphanumeric character) x 11 characters = 88 bits > **Note:** > -> The second example operates under the assumption that there is a database with a record titled `Customers` and a field inside titled `CustomerName` +> The second example operates under the assumption that there is a database with a row titled `Customers` and a column inside titled `CustomerName` ### [`CHAR()`](https://dev.mysql.com/doc/refman/8.0/en/string-functions.html#function_char) From e197638574e0e9c9ab9998f3cdb56c35a1902640 Mon Sep 17 00:00:00 2001 From: Peter Oyebanji Date: Thu, 11 Jan 2024 14:51:15 +0100 Subject: [PATCH 14/41] functions-and-operators: enrich instructions for BIT_LENGTH() and CHAR() rewrite instructions for BIT_LENGTH and CHAR() by editing the descriptions, adding examples, and linking the database that one of my examples was based on. also fix all issues raised after submitting pull request, these issues can be seen in the comment section of the pull request. finally squashed my commits into one single commit, which is this, and then rewrite the commit message, which is what you have just read --- functions-and-operators/string-functions.md | 126 +++++++++++++++++++- 1 file changed, 122 insertions(+), 4 deletions(-) diff --git a/functions-and-operators/string-functions.md b/functions-and-operators/string-functions.md index 78d027cd6e9c0..61702150faad5 100644 --- a/functions-and-operators/string-functions.md +++ b/functions-and-operators/string-functions.md @@ -107,19 +107,137 @@ Return a string containing binary representation of a number. ### [`BIT_LENGTH()`](https://dev.mysql.com/doc/refman/8.0/en/string-functions.html#function_bit-length) -Return length of argument in bits. +The `BIT_LENGTH()` function is used to return the length of a given argument in bits. It calculates the length of the argument and then returns the result in bits. + +Examples: + +```sql +SELECT BIT_LENGTH("TiDB"); + ++----------+ +| BIT_LENGTH("TiDB") | ++----------+ +| 32 | ++----------+ +``` + +8 bits per character x 4 characters = 32 bits + +```sql +SELECT CustomerName, BIT_LENGTH(CustomerName) AS BitLengthOfName FROM Customers; + +| CustomerName|BitLengthOfName | +|-------------|----------------| +| Albert Einstein | 120 | +| Robert Oppenheimer | 144 | +``` + +```sql +SELECT BIT_LENGTH("PingCap 123"); + ++----------+ +| BIT_LENGTH("PingCap 123") | ++----------+ +| 88 | ++----------+ +``` + +8 bits per character (space is counted as it is a non-alphanumeric character) x 11 characters = 88 bits + +> **Note:** +> +> The second example operates under the assumption that there is a [database](https://www.w3schools.com/sql/trymysql.asp?filename=trysql_func_mysql_char_length2) with a table titled `Customers` and a column inside the table titled `CustomerName` ### [`CHAR()`](https://dev.mysql.com/doc/refman/8.0/en/string-functions.html#function_char) -Return the character for each integer passed. +The `CHAR()` function is used to get the corresponding character value of a specific ASCII value. It performs the opposite function of `ASCII()`. Unlike `ASCII()`, which can get the ASCII value of a specific character, `CHAR()` can get the character of a specific ASCII value. + +Examples: + +```sql +SELECT CHAR(65); + ++------------+ +| CHAR(65) | ++------------+ +| A | ++------------+ +``` + +```sql +SELECT CHAR(84); + ++------------+ +| CHAR(84) | ++------------+ +| T | ++------------+ +``` + +> **Note:** +> +> The `CHAR()` function can also be used to get the corresponding character values of ASCII values that extend beyond the standard ASCII range (0 - 127). It can also get the corresponding character value of a unicode value. + +Further Examples: + +```sql +/*For extended ASCII: */ + +SELECT CHAR(128); + ++------------+ +| CHAR(128) | ++------------+ +| 0x80 | ++------------+ +``` + +```sql +/* For Unicode: */ + +--skip-binary-as-hex + +SELECT CHAR(50089); + ++------------+ +| CHAR(50089) | ++------------+ +| é | ++------------+ +``` ### [`CHAR_LENGTH()`](https://dev.mysql.com/doc/refman/8.0/en/string-functions.html#function_char-length) -Return number of characters in argument. +The `CHAR_LENGTH()` function is used to get the number of characters in a specified expression. It returns the total number of characters in a specified expression as an integer value. + +Examples: + +```sql +SELECT CHAR_LENGTH("TiDB") AS LengthOfString; + ++----------+ +| LengthOfString | ++----------+ +| 4 | ++----------+ +``` + +```sql +SELECT CustomerName, CHAR_LENGTH(CustomerName) AS LenghtOfName FROM Customers; + +| CustomerName|LengthOfName | +|-------------|--------------| +| Albert Einstein | 15 | +| Robert Oppenheimer | 18 | +``` + +> **Note:** +> +> The second example operates under the assumption that there is a [database](https://www.w3schools.com/sql/trymysql.asp?filename=trysql_func_mysql_char_length2) with a table titled `Customers` and a column inside the table titled `CustomerName` ### [`CHARACTER_LENGTH()`](https://dev.mysql.com/doc/refman/8.0/en/string-functions.html#function_character-length) -Synonym for `CHAR_LENGTH()`. +The `CHARACTER_LENGTH()` function is the same as the `CHAR_LENGTH()` function. Both functions can be used synonymously because they provide the same outputs. ### [`CONCAT()`](https://dev.mysql.com/doc/refman/8.0/en/string-functions.html#function_concat) From 37bdafe8a34ecb7befe7f84e3bc3848f85a4b0a0 Mon Sep 17 00:00:00 2001 From: Lilian Lee Date: Tue, 16 Jan 2024 10:57:15 +0800 Subject: [PATCH 15/41] develop: make format and wording consistent (#16154) --- develop/dev-guide-choose-driver-or-orm.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/develop/dev-guide-choose-driver-or-orm.md b/develop/dev-guide-choose-driver-or-orm.md index b8b4d585a4cea..f2a06af728a23 100644 --- a/develop/dev-guide-choose-driver-or-orm.md +++ b/develop/dev-guide-choose-driver-or-orm.md @@ -44,7 +44,7 @@ Support level: **Full** The authentication based on SM3 is only supported in TiDB's TiDB-JDBC. -If you use Maven, add the following content to the `` section in the `pom.xml` file: +If you are using Maven, add the following content to the `` section in the `pom.xml` file: ```xml @@ -100,7 +100,7 @@ Support level: **Full** To avoid manually managing complex relationships between different dependencies of an application, you can use [Gradle](https://gradle.org/install) or [Maven](https://maven.apache.org/install.html) to get all dependencies of your application, including those indirect ones. Note that only Hibernate `6.0.0.Beta2` or above supports the TiDB dialect. -If you are using **Maven**, add the following to your ``: +If you are using Maven, add the following to your ``: ```xml @@ -116,7 +116,7 @@ If you are using **Maven**, add the following to your ` ``` -If you are using **Gradle**, add the following to your `dependencies`: +If you are using Gradle, add the following to your `dependencies`: ```gradle implementation 'org.hibernate:hibernate-core:6.0.0.CR2' @@ -183,7 +183,7 @@ Currently, tidb-loadbalance supports the following policies: roundrobin, random, > > tidb-loadbalance must be used with [mysql-connector-j](https://github.com/pingcap/mysql-connector-j). -If you use Maven, add the following content to the element body of `` in the `pom.xml` file: +If you are using Maven, add the following content to the element body of `` in the `pom.xml` file: ```xml @@ -198,7 +198,7 @@ If you use Maven, add the following content to the element body of ` ``` -If you use Gradle, add the following content to `dependencies`: +If you are using Gradle, add the following content to `dependencies`: ```gradle implementation group: 'io.github.lastincisor', name: 'mysql-connector-java', version: '8.0.29-tidb-1.0.0' From 674fe77b0e2e27012b752dcd4b0afed7b8a58c7f Mon Sep 17 00:00:00 2001 From: Liu Linkang <135819329+llkdd1@users.noreply.github.com> Date: Tue, 16 Jan 2024 11:18:15 +0800 Subject: [PATCH 16/41] add ticdc aws glue schema registry config docs (#16088) --- ticdc/ticdc-changefeed-config.md | 8 ++++++++ ticdc/ticdc-sink-to-kafka.md | 22 ++++++++++++++++++++++ 2 files changed, 30 insertions(+) diff --git a/ticdc/ticdc-changefeed-config.md b/ticdc/ticdc-changefeed-config.md index 9ab2b979463ee..068247055be35 100644 --- a/ticdc/ticdc-changefeed-config.md +++ b/ticdc/ticdc-changefeed-config.md @@ -246,6 +246,14 @@ sasl-oauth-scopes = ["producer.kafka", "consumer.kafka"] sasl-oauth-grant-type = "client_credentials" # The audience in the Kafka SASL OAUTHBEARER authentication. The default value is empty. This parameter is optional when the OAUTHBEARER authentication is used. sasl-oauth-audience = "kafka" +# The following configuration is only required when using Avro as the protocol and AWS Glue Schema Registry: +# Please refer to the section "Integrate TiCDC with AWS Glue Schema Registry" in the document "Sync Data to Kafka": https://docs.pingcap.com/tidb/dev/ticdc-sink-to-kafka#integrate-ticdc-with-aws-glue-schema-registry +# [sink.kafka-config.glue-schema-registry-config] +# region="us-west-1" +# registry-name="ticdc-test" +# access-key="xxxx" +# secret-access-key="xxxx" +# token="xxxx" # The following parameters take effect only when the downstream is Pulsar. [sink.pulsar-config] diff --git a/ticdc/ticdc-sink-to-kafka.md b/ticdc/ticdc-sink-to-kafka.md index 110b446a3d8c5..0f6af2f2464d9 100644 --- a/ticdc/ticdc-sink-to-kafka.md +++ b/ticdc/ticdc-sink-to-kafka.md @@ -157,6 +157,28 @@ dispatchers = [ For detailed integration guide, see [Quick Start Guide on Integrating TiDB with Confluent Platform](/ticdc/integrate-confluent-using-ticdc.md). +### Integrate TiCDC with AWS Glue Schema Registry + +Starting from v7.4.0, TiCDC supports using the [AWS Glue Schema Registry](https://docs.aws.amazon.com/glue/latest/dg/schema-registry.html) as the Schema Registry when users choose the Avro protocol for data replication. The configuration example is as follows: + +```shell +./cdc cli changefeed create --server=127.0.0.1:8300 --changefeed-id="kafka-glue-test" --sink-uri="kafka://127.0.0.1:9092/topic-name?&protocol=avro&replication-factor=3" --config changefeed_glue.toml +``` + +```toml +[sink] +[sink.kafka-config.glue-schema-registry-config] +region="us-west-1" +registry-name="ticdc-test" +access-key="xxxx" +secret-access-key="xxxx" +token="xxxx" +``` + +In the above configuration, `region` and `registry-name` are required fields, while `access-key`, `secret-access-key`, and `token` are optional fields. The best practice is to set the AWS credentials as environment variables or store them in the `~/.aws/credentials` file instead of setting them in the changefeed configuration file. + +For more information, refer to the [official AWS SDK for Go V2 documentation](https://aws.github.io/aws-sdk-go-v2/docs/configuring-sdk/#specifying-credentials). + ## Customize the rules for Topic and Partition dispatchers of Kafka Sink ### Matcher rules From 7a01833158539bf4b424306c706bf73db3e53f9a Mon Sep 17 00:00:00 2001 From: Cheng Rui <44940678+RobertCheng-956@users.noreply.github.com> Date: Tue, 16 Jan 2024 11:45:15 +0800 Subject: [PATCH 17/41] restore: add faq for rule not found (#16091) --- faq/backup-and-restore-faq.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/faq/backup-and-restore-faq.md b/faq/backup-and-restore-faq.md index 34f1ffde41e7c..13e57039f5dec 100644 --- a/faq/backup-and-restore-faq.md +++ b/faq/backup-and-restore-faq.md @@ -296,6 +296,10 @@ Note that even if you configures [table filter](/table-filter.md#syntax), **BR d - System variable tables (`mysql.tidb`, `mysql.global_variables`) - [Other system tables](https://github.com/pingcap/tidb/blob/master/br/pkg/restore/systable_restore.go#L31) +### How to deal with the error of `cannot find rewrite rule` during restoration? + +Examine whether there are tables in the restoration cluster sharing the same name as other tables in the backup data but having inconsistent structures. In most cases, this issue is caused by missing indexes in the tables of the restoration cluster. A recommended approach is to delete such tables in the restoration cluster first and then retry restoration. + ## Other things you may want to know about backup and restore ### What is the size of the backup data? Are there replicas of the backup? From 4c2fa0c7135fc8731be4ef771a1996f9ccffd582 Mon Sep 17 00:00:00 2001 From: cyberslack_lee Date: Tue, 16 Jan 2024 17:33:16 +0800 Subject: [PATCH 18/41] tiup: add instructions for the --local option (#16097) --- tiup/tiup-component-cluster-template.md | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/tiup/tiup-component-cluster-template.md b/tiup/tiup-component-cluster-template.md index 628002ccaee56..3fe0c7a1dc332 100644 --- a/tiup/tiup-component-cluster-template.md +++ b/tiup/tiup-component-cluster-template.md @@ -28,6 +28,11 @@ If this option is not specified, the output default template contains the follow - Outputs a detailed topology template that is commented with configurable parameters. To enable this option, add it to the command. - If this option is not specified, the simple topology template is output by default. +### --local + +- Outputs a simple topology template for the local cluster, which can be used directly, and the `global` parameter can be adjusted as needed. +- This template creates a PD service, a TiDB service, a TiKV service, a monitoring service, and a Grafana service. + ### --multi-dc - Outputs the topology template of multiple data centers. To enable this option, add it to the command. From b7caa9da61069f74fedf1d9a60742aaa1da2521e Mon Sep 17 00:00:00 2001 From: xhe Date: Tue, 16 Jan 2024 22:22:45 +0800 Subject: [PATCH 19/41] tidb: fix config value for tiproxy (#15635) --- tidb-configuration-file.md | 12 ++++-------- 1 file changed, 4 insertions(+), 8 deletions(-) diff --git a/tidb-configuration-file.md b/tidb-configuration-file.md index 13904103ce49c..01387c713a375 100644 --- a/tidb-configuration-file.md +++ b/tidb-configuration-file.md @@ -444,19 +444,15 @@ Configuration items related to security. ### `session-token-signing-cert` New in v6.4.0 -> **Warning:** -> -> The feature controlled by this parameter is under development. **Do not modify the default value**. - ++ The certificate file path, which is used by [TiProxy](/tiproxy/tiproxy-overview.md) for session migration. + Default value: "" ++ Empty value will cause TiProxy session migration to fail. To enable session migration, all TiDB nodes must set this to the same certificate and key. This means that you should store the same certificate and key on every TiDB node. ### `session-token-signing-key` New in v6.4.0 -> **Warning:** -> -> The feature controlled by this parameter is under development. **Do not modify the default value**. - ++ The key file path used by [TiProxy](/tiproxy/tiproxy-overview.md) for session migration. + Default value: "" ++ Refer to the descriptions of [`session-token-signing-cert`](#session-token-signing-cert-new-in-v640). ## Performance From 3e8dea2539c4c261d5f4c260b3da362c816ffa36 Mon Sep 17 00:00:00 2001 From: Ray Paik Date: Wed, 17 Jan 2024 18:27:16 -0800 Subject: [PATCH 20/41] Update config.yaml for Docs Dash (#16181) --- .vaunt/config.yaml | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/.vaunt/config.yaml b/.vaunt/config.yaml index 460f1ce7f75a4..b9c7741a8a86c 100644 --- a/.vaunt/config.yaml +++ b/.vaunt/config.yaml @@ -20,7 +20,7 @@ actions: action: pull_request condition: "created_at >= 01-09-2024 08:00:00 & created_at <= 01-12-2024 07:59:00 - & merged = true & merged_at <= 2024-01-26 20:00:00 + & merged = true & merged_at <= 2024-02-01 20:00:00 & labels in {'2024-tidb-docs-dash','contribution'}" value: 10 - action: @@ -70,7 +70,7 @@ actions: action: pull_request condition: "created_at >= 01-09-2024 08:00:00 & created_at <= 01-12-2024 07:59:00 - & merged = true & merged_at <= 2024-01-26 20:00:00 + & merged = true & merged_at <= 2024-02-01 20:00:00 & labels in !['2024-tidb-docs-dash'] & labels in ['contribution']" value: 5 @@ -109,8 +109,8 @@ actions: action: pull_request condition: "created_at >= 01-09-2024 08:00:00 & created_at <= 01-12-2024 07:59:00 - & merged = true & merged_at <= 2024-01-26 20:00:00 - & labels in ['contribution','tidb-docs-dash-bonus']" + & merged = true & merged_at <= 2024-02-01 20:00:00 + & labels in {'contribution','tidb-docs-dash-bonus'}" value: 20 achievements: - achievement: From b73e77a01c58f2ba9ecdc5f96e88c3777ef63dc4 Mon Sep 17 00:00:00 2001 From: xixirangrang Date: Thu, 18 Jan 2024 18:18:18 +0800 Subject: [PATCH 21/41] tidb-lightning: tidb.pd-addr supports multiple addresses (#16145) --- get-started-with-tidb-lightning.md | 4 ++-- tidb-lightning/tidb-lightning-command-line-full.md | 2 +- tidb-lightning/tidb-lightning-configuration.md | 4 ++-- tidb-lightning/tidb-lightning-physical-import-mode-usage.md | 4 ++-- 4 files changed, 7 insertions(+), 7 deletions(-) diff --git a/get-started-with-tidb-lightning.md b/get-started-with-tidb-lightning.md index d2b406961a544..14e9ed639c541 100644 --- a/get-started-with-tidb-lightning.md +++ b/get-started-with-tidb-lightning.md @@ -79,8 +79,8 @@ The TiDB Lightning installation package is included in the TiDB Toolkit. To down password = "rootroot" # Table schema information is fetched from TiDB via this status-port. status-port = 10080 - # The PD address of the cluster - pd-addr = "172.16.31.3:2379" + # The PD address of the cluster. Starting from v7.6.0, TiDB supports setting multiple PD addresses. + pd-addr = "172.16.31.3:2379,56.78.90.12:3456" ``` 4. After configuring the parameters properly, use a `nohup` command to start the `tidb-lightning` process. If you directly run the command in the command-line, the process might exit because of the SIGHUP signal received. Instead, it's preferable to run a bash script that contains the `nohup` command: diff --git a/tidb-lightning/tidb-lightning-command-line-full.md b/tidb-lightning/tidb-lightning-command-line-full.md index b3895427e3511..4f4b3778a4998 100644 --- a/tidb-lightning/tidb-lightning-command-line-full.md +++ b/tidb-lightning/tidb-lightning-command-line-full.md @@ -23,7 +23,7 @@ You can configure the following parameters using `tidb-lightning`: | `--backend ` | Select an import mode. `local` refers to [physical import mode](/tidb-lightning/tidb-lightning-physical-import-mode.md); `tidb` refers to [logical import mode](/tidb-lightning/tidb-lightning-logical-import-mode.md). | `tikv-importer.backend` | | `--log-file ` | Log file path. By default, it is `/tmp/lightning.log.{timestamp}`. If set to '-', it means that the log files will be output to stdout. | `lightning.log-file` | | `--status-addr ` | Listening address of the TiDB Lightning server | `lightning.status-port` | -| `--pd-urls ` | PD endpoint address | `tidb.pd-addr` | +| `--pd-urls ` | PD endpoint address. Starting from v7.6.0, TiDB supports setting multiple PD addresses. | `tidb.pd-addr` | | `--tidb-host ` | TiDB server host | `tidb.host` | | `--tidb-port ` | TiDB server port (default = 4000) | `tidb.port` | | `--tidb-status ` | TiDB status port (default = 10080) | `tidb.status-port` | diff --git a/tidb-lightning/tidb-lightning-configuration.md b/tidb-lightning/tidb-lightning-configuration.md index be3fa7d35a9b4..3855f9479be0a 100644 --- a/tidb-lightning/tidb-lightning-configuration.md +++ b/tidb-lightning/tidb-lightning-configuration.md @@ -359,8 +359,8 @@ user = "root" password = "" # Table schema information is fetched from TiDB via this status-port. status-port = 10080 -# Address of any PD server from the cluster. -pd-addr = "172.16.31.4:2379" +# Address of any PD server from the cluster. Starting from v7.6.0, TiDB supports setting multiple PD addresses. +pd-addr = "172.16.31.4:2379,56.78.90.12:3456" # tidb-lightning imports TiDB as a library and generates some logs itself. # This setting controls the log level of the TiDB library. log-level = "error" diff --git a/tidb-lightning/tidb-lightning-physical-import-mode-usage.md b/tidb-lightning/tidb-lightning-physical-import-mode-usage.md index 5918ebf39f9cf..2b4fef7fb6b59 100644 --- a/tidb-lightning/tidb-lightning-physical-import-mode-usage.md +++ b/tidb-lightning/tidb-lightning-physical-import-mode-usage.md @@ -67,8 +67,8 @@ user = "root" password = "" # Required. Table schema information is fetched from TiDB via this status-port. status-port = 10080 -# Required. The address of any pd-server from the cluster. -pd-addr = "172.16.31.4:2379" +# Required. The address of any pd-server from the cluster. Starting from v7.6.0, TiDB supports setting multiple PD addresses. +pd-addr = "172.16.31.4:2379,56.78.90.12:3456" # tidb-lightning imports the TiDB library, and generates some logs. # Set the log level of the TiDB library. log-level = "error" From c420b5ac491a869733cf9d1d3c8bfb5576b102cd Mon Sep 17 00:00:00 2001 From: xixirangrang Date: Thu, 18 Jan 2024 21:22:47 +0800 Subject: [PATCH 22/41] Update bdr mode && add description related bdr sql (#15925) --- TOC.md | 1 + error-codes.md | 4 + keywords.md | 2 +- .../sql-statement-admin-bdr-role.md | 88 +++++++++++ ticdc/ticdc-bidirectional-replication.md | 140 +++++++++++++----- 5 files changed, 200 insertions(+), 35 deletions(-) create mode 100644 sql-statements/sql-statement-admin-bdr-role.md diff --git a/TOC.md b/TOC.md index 56f00a87e34bb..fde63327c6cd8 100644 --- a/TOC.md +++ b/TOC.md @@ -718,6 +718,7 @@ - [`ADMIN PAUSE DDL`](/sql-statements/sql-statement-admin-pause-ddl.md) - [`ADMIN RECOVER INDEX`](/sql-statements/sql-statement-admin-recover.md) - [`ADMIN RESUME DDL`](/sql-statements/sql-statement-admin-resume-ddl.md) + - [`ADMIN [SET|SHOW|UNSET] BDR ROLE`](/sql-statements/sql-statement-admin-bdr-role.md) - [`ADMIN SHOW DDL [JOBS|JOB QUERIES]`](/sql-statements/sql-statement-admin-show-ddl.md) - [`ADMIN SHOW TELEMETRY`](/sql-statements/sql-statement-admin-show-telemetry.md) - [`ALTER DATABASE`](/sql-statements/sql-statement-alter-database.md) diff --git a/error-codes.md b/error-codes.md index 735156891f207..1069950f6a552 100644 --- a/error-codes.md +++ b/error-codes.md @@ -526,6 +526,10 @@ TiDB is compatible with the error codes in MySQL, and in most cases returns the DDL is paused by `ADMIN PAUSE` and cannot be paused again. +* Error Number: 8263 + + This DDL cannot be executed under a specific BDR role. Make sure that the cluster is in [bidirectional replication](/ticdc/ticdc-bidirectional-replication.md). If the cluster is not in bidirectional replication, you can use `ADMIN UNSET BDR ROLE;` to make DDL normal. + * Error Number: 9001 The complete error message: `ERROR 9001 (HY000): PD Server Timeout` diff --git a/keywords.md b/keywords.md index 29bf539dbb565..b468aa0e972c4 100644 --- a/keywords.md +++ b/keywords.md @@ -395,7 +395,6 @@ The following list shows the keywords in TiDB. Reserved keywords are marked with - LIST - LOAD (R) - LOCAL -- LOCAL_ONLY - LOCALTIME (R) - LOCALTIMESTAMP (R) - LOCATION @@ -734,6 +733,7 @@ The following list shows the keywords in TiDB. Reserved keywords are marked with - UNIQUE (R) - UNKNOWN - UNLOCK (R) +- UNSET - UNSIGNED (R) - UNTIL (R) - UPDATE (R) diff --git a/sql-statements/sql-statement-admin-bdr-role.md b/sql-statements/sql-statement-admin-bdr-role.md new file mode 100644 index 0000000000000..4781794542dfa --- /dev/null +++ b/sql-statements/sql-statement-admin-bdr-role.md @@ -0,0 +1,88 @@ +--- +title: ADMIN [SET|SHOW|UNSET] BDR ROLE +summary: An overview of the usage of ADMIN [SET|SHOW|UNSET] BDR ROLE for the TiDB database. +--- + +# ADMIN [SET|SHOW|UNSET] BDR ROLE + +- Use `ADMIN SET BDR ROLE` to set the BDR role of the cluster. Currently, you can set the following BDR roles for a TiDB cluster: `PRIMARY` and `SECONDARY`. For more information about BDR roles, see [DDL Synchronization in TiCDC Bidirectional Replication](/ticdc/ticdc-bidirectional-replication.md#ddl-replication). +- Use `ADMIN SHOW BDR ROLE` to show the BDR role of the cluster. +- Use `ADMIN UNSET BDR ROLE` to unset the BDR role of the cluster. + +> **Warning:** +> +> This feature is experimental. It is not recommended that you use it in the production environment. This feature might be changed or removed without prior notice. If you find a bug, you can report an [issue](https://github.com/pingcap/tidb/issues) on GitHub. + +## Synopsis + +```ebnf+diagram +AdminShowBDRRoleStmt ::= + 'ADMIN' 'SHOW' 'BDR' 'ROLE' + +AdminSetBDRRoleStmt ::= + 'ADMIN' 'SET' 'BDR' 'ROLE' ('PRIMARY' | 'SECONDARY') + +AdminUnsetBDRRoleStmt ::= + 'ADMIN' 'UNSET' 'BDR' 'ROLE' +``` + +## Examples + +By default, a TiDB cluster has no BDR role. Run the following command to show the BDR role of the cluster. + +```sql +ADMIN SHOW BDR ROLE; +``` + +```sql ++------------+ +| BDR_ROLE | ++------------+ +| | ++------------+ +1 row in set (0.01 sec) +``` + +Run the following command to set the BDR role to `PRIMARY`. + +```sql +ADMIN SET BDR ROLE PRIMARY; +``` + +```sql +Query OK, 0 rows affected (0.01 sec) +``` + +```sql +ADMIN SHOW BDR ROLE; ++----------+ +| BDR_ROLE | ++----------+ +| primary | ++----------+ +1 row in set (0.00 sec) +``` + +Run the following command to unset the BDR role of the cluster. + +```sql +ADMIN UNSET BDR ROLE; +``` + +```sql +Query OK, 0 rows affected (0.01 sec) +``` + +```sql +ADMIN SHOW BDR ROLE; ++----------+ +| BDR_ROLE | ++----------+ +| | ++----------+ +1 row in set (0.01 sec) +``` + +## MySQL compatibility + +This statement is a TiDB extension to MySQL syntax. diff --git a/ticdc/ticdc-bidirectional-replication.md b/ticdc/ticdc-bidirectional-replication.md index 508fc45511968..607f23dda2aae 100644 --- a/ticdc/ticdc-bidirectional-replication.md +++ b/ticdc/ticdc-bidirectional-replication.md @@ -5,7 +5,7 @@ summary: Learn how to use bidirectional replication of TiCDC. # Bidirectional Replication -Starting from v6.5.0, TiCDC supports bi-directional replication among two TiDB clusters. Based on this feature, you can create a multi-active TiDB solution using TiCDC. +TiCDC supports bi-directional replication (BDR) among two TiDB clusters. Based on this feature, you can create a multi-active TiDB solution using TiCDC. This section describes how to use bi-directional replication taking two TiDB clusters as an example. @@ -34,47 +34,119 @@ TiCDC only replicates incremental data changes that occur after a specified time After the configuration takes effect, the clusters can perform bi-directional replication. -## Execute DDL - -After the bidirectional replication is enabled, TiCDC does not replicate any DDL statements. You need to execute DDL statements in the upstream and downstream clusters respectively. - -Note that some DDL statements might cause table structure changes or data change time sequence problems, which might lead to data inconsistency after the replication. Therefore, after enabling bidirectional replication, only the DDL statements in the following table can be executed without stopping the write operations of the application. - -| Event | Does it cause changefeed errors | Note | -|---|---|---| -| create database | Yes | After you manually execute the DDL statements in the upstream and downstream clusters, the errors can be automatically recovered. | -| drop database | Yes | You need to manually restart the changefeed and specify `--overwrite-checkpoint-ts` as the `commitTs` of the DDL statement to recover the errors. | -| create table | Yes | After you manually execute the DDL statements in the upstream and downstream clusters, the errors can be automatically recovered. | -| drop table | Yes | You need to manually restart the changefeed and specify `--overwrite-checkpoint-ts` as the `commitTs` of the DDL statement to recover the errors. | -| alter table comment | No | | -| rename index | No | | -| alter table index visibility | No | | -| add partition | Yes | After you manually execute the DDL statements in the upstream and downstream clusters, the errors can be automatically recovered. | -| drop partition | No | | -| create view | No | | -| drop view | No | | -| alter column default value | No | | -| reorganize partition | Yes | After you manually execute the DDL statements in the upstream and downstream clusters, the errors can be automatically recovered. | -| alter table ttl | No | | -| alter table remove ttl | No | | -| add **not unique** index | No | | -| drop **not unique** index | No | | - -If you need to execute DDL statements that are not in the preceding table, take the following steps: - -1. Pause the write operations in the tables that need to execute DDL in all clusters. -2. After the write operations of the corresponding tables in all clusters have been replicated to other clusters, manually execute all DDL statements in each TiDB cluster. -3. After the DDL statements are executed, resume the write operations. +## DDL types + +Starting from v7.6.0, to support DDL replication as much as possible in bi-directional replication, TiDB divides the [DDLs that TiCDC originally supports](/ticdc/ticdc-ddl.md) into two types: replicable DDLs and non-replicable DDLs, according to the impact of DDLs on the business. + +### Replicable DDLs + +Replicable DDLs are the DDLs that can be directly executed and replicated to other TiDB clusters in bi-directional replication. + +Replicable DDLs include: + +- `CREATE DATABASE` +- `CREATE TABLE` +- `ADD COLUMN`: the column can be `null`, or has `not null` and `default value` at the same time +- `ADD NON-UNIQUE INDEX` +- `DROP INDEX` +- `MODIFY COLUMN`: you can only modify the `default value` and `comment` of the column +- `ALTER COLUMN DEFAULT VALUE` +- `MODIFY TABLE COMMENT` +- `RENAME INDEX` +- `ADD TABLE PARTITION` +- `DROP PRIMARY KEY` +- `ALTER TABLE INDEX VISIBILITY` +- `ALTER TABLE TTL` +- `ALTER TABLE REMOVE TTL` +- `CREATE VIEW` +- `DROP VIEW` + +### Non-replicable DDLs + +Non-replicable DDLs are the DDLs that have a great impact on the business, and might cause data inconsistency between clusters. Non-replicable DDLs cannot be directly replicated to other TiDB clusters in bi-directional replication through TiCDC. Non-replicable DDLs must be executed through specific operations. + +Non-replicable DDLs include: + +- `DROP DATABASE` +- `DROP TABLE` +- `ADD COLUMN`: the column is `not null` and does not have a `default value` +- `DROP COLUMN` +- `ADD UNIQUE INDEX` +- `TRUNCATE TABLE` +- `MODIFY COLUMN`: you can modify the attributes of the column except `default value` and `comment` +- `RENAME TABLE` +- `DROP PARTITION` +- `TRUNCATE PARTITION` +- `ALTER TABLE CHARACTER SET` +- `ALTER DATABASE CHARACTER SET` +- `RECOVER TABLE` +- `ADD PRIMARY KEY` +- `REBASE AUTO ID` +- `EXCHANGE PARTITION` +- `REORGANIZE PARTITION` + +## DDL replication + +To solve the problem of replicable DDLs and non-replicable DDLs, TiDB introduces the following BDR roles: + +- `PRIMARY`: you can execute replicable DDLs, but cannot execute non-replicable DDLs. Replicable DDLs will be replicated to the downstream by TiCDC. +- `SECONDARY`: you cannot execute replicable DDLs or non-replicable DDLs, but can execute the DDLs replicated by TiCDC. + +When no BDR role is set, you can execute any DDL. But after you set `bdr_mode=true` on TiCDC, the executed DDL will not be replicated by TiCDC. + +> **Warning:** +> +> This feature is experimental. It is not recommended that you use it in the production environment. This feature might be changed or removed without prior notice. If you find a bug, you can report an [issue](https://github.com/pingcap/tidb/issues) on GitHub. + +### Replication scenarios of replicable DDLs + +1. Choose a TiDB cluster and execute `ADMIN SET BDR ROLE PRIMARY` to set it as the primary cluster. +2. On other TiDB clusters, execute `ADMIN SET BDR ROLE SECONDARY` to set them as the secondary clusters. +3. Execute **replicable DDLs** on the primary cluster. The successfully executed DDLs will be replicated to the secondary clusters by TiCDC. + +> **Note:** +> +> To prevent misuse: +> +> - If you try to execute **non-replicable DDLs** on the primary cluster, you will get the [Error 8263](/error-codes.md). +> - If you try to execute **replicable DDLs** or **non-replicable DDLs** on the secondary clusters, you will get the [Error 8263](/error-codes.md). + +### Replication scenarios of non-replicable DDLs + +1. Execute `ADMIN UNSET BDR ROLE` on all TiDB clusters to unset the BDR role. +2. Stop writing data to the tables that need to execute DDLs in all clusters. +3. Wait until all writes to the corresponding tables in all clusters are replicated to other clusters, and then manually execute all DDLs on each TiDB cluster. +4. Wait until the DDLs are completed, and then resume writing data. +5. Follow the steps in [Replication scenarios of replicable DDLs](#replication-scenarios-of-replicable-ddls) to switch back to the replication scenario of replicable DDLs. + +> **Warning:** +> +> After you execute `ADMIN UNSET BDR ROLE` on all TiDB clusters, none of the DDLs are replicated by TiCDC. You need to manually execute the DDLs on each cluster separately. ## Stop bi-directional replication After the application has stopped writing data, you can insert a special record into each cluster. By checking the two special records, you can make sure that data in two clusters are consistent. -After the check is completed, you can stop the changefeed to stop bi-directional replication. +After the check is completed, you can stop the changefeed to stop bi-directional replication, and execute `ADMIN UNSET BDR ROLE` on all TiDB clusters. ## Limitations -- For the limitations of DDL, see [Execute DDL](#execute-ddl). +- Use BDR role only in the following scenarios: + + - 1 `PRIMARY` cluster and n `SECONDARY` clusters (replication scenarios of replicable DDLs) + - n clusters that have no BDR roles (replication scenarios in which you can manually execute non-replicable DDLs on each cluster) + + > **Note:** + > + > Do not set the BDR role in other scenarios, for example, setting `PRIMARY`, `SECONDARY`, and no BDR roles at the same time. If you set the BDR role incorrectly, TiDB cannot guarantee data correctness and consistency during data replication. + +- Usually do not use `AUTO_INCREMENT` or `AUTO_RANDOM` to avoid data conflicts in the replicated tables. If you need to use `AUTO_INCREMENT` or `AUTO_RANDOM`, you can set different `auto_increment_increment` and `auto_increment_offset` for different clusters to ensure that different clusters can be assigned different primary keys. For example, if there are three TiDB clusters (A, B, and C) in bi-directional replication, you can set them as follows: + + - In Cluster A, set `auto_increment_increment=3` and `auto_increment_offset=2000` + - In Cluster B, set `auto_increment_increment=3` and `auto_increment_offset=2001` + - In Cluster C, set `auto_increment_increment=3` and `auto_increment_offset=2002` + + This way, A, B, and C will not conflict with each other in the implicitly assigned `AUTO_INCREMENT` ID and `AUTO_RANDOM` ID. If you need to add a cluster in BDR mode, you need to temporarily stop writing data of the related application, set the appropriate values for `auto_increment_increment` and `auto_increment_offset` on all clusters, and then resume writing data of the related application. - Bi-directional replication clusters cannot detect write conflicts, which might cause undefined behaviors. Therefore, you must ensure that there are no write conflicts from the application side. From 528e4287804a31eae400046db34587fbb5343563 Mon Sep 17 00:00:00 2001 From: Weizhen Wang Date: Thu, 18 Jan 2024 21:36:17 +0800 Subject: [PATCH 23/41] system-variable: add `tidb_analyze_distsql_scan_concurrency` (#15883) --- system-variables.md | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-) diff --git a/system-variables.md b/system-variables.md index c6f1ecfc735a4..e7682826ac85d 100644 --- a/system-variables.md +++ b/system-variables.md @@ -1085,6 +1085,16 @@ MPP is a distributed computing framework provided by the TiFlash engine, which a - Default value: `OFF` - This variable is used to set whether the `AUTO_INCREMENT` property of a column is allowed to be removed by executing `ALTER TABLE MODIFY` or `ALTER TABLE CHANGE` statements. It is not allowed by default. +### tidb_analyze_distsql_scan_concurrency New in v7.6.0 + +- Scope: GLOBAL +- Persists to cluster: Yes +- Applies to hint [SET_VAR](/optimizer-hints.md#set_varvar_namevar_value): No +- Type: Integer +- Default value: `4` +- Range: `[1, 4294967295]` +- This variable is used to set the concurrency of the `scan` operation when executing the `ANALYZE` operation. + ### tidb_analyze_partition_concurrency > **Warning:** @@ -5350,7 +5360,7 @@ For details, see [Identify Slow Queries](/identify-slow-queries.md). - Applies to hint [SET_VAR](/optimizer-hints.md#set_varvar_namevar_value): No - Type: Integer - Default value: `1` -- Range: `[1, 256]` +- Range: `[1, 4294967295]`. The maximum value for v7.5.0 and earlier versions is `256`. - This variable is used to set the concurrency of scan operations performed when TiDB executes internal SQL statements (such as an automatic update of statistics). ### tidb_table_cache_lease New in v6.0.0 From 80da287123a9b7dcc3225642c12383d23384e09f Mon Sep 17 00:00:00 2001 From: cyberslack_lee Date: Fri, 19 Jan 2024 00:04:47 +0800 Subject: [PATCH 24/41] translate changes in #15883 from docs-cn (#16047) --- sql-mode.md | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/sql-mode.md b/sql-mode.md index 5b6c4a3fb2989..acc529b2d66d9 100644 --- a/sql-mode.md +++ b/sql-mode.md @@ -8,11 +8,13 @@ aliases: ['/docs/dev/sql-mode/','/docs/dev/reference/sql/sql-mode/'] TiDB servers operate in different SQL modes and apply these modes differently for different clients. SQL mode defines the SQL syntaxes that TiDB supports and the type of data validation check to perform, as described below: -After TiDB is started, modify `SET [ SESSION | GLOBAL ] sql_mode='modes'` to set SQL mode. +After TiDB is started, you can use the `SET [ SESSION | GLOBAL ] sql_mode='modes'` statement to set SQL mode. -Ensure that you have `SUPER` privilege when setting SQL mode at `GLOBAL` level, and your setting at this level only affects the connections established afterwards. Changes to SQL mode at `SESSION` level only affect the current client. +- Ensure that you have `SUPER` privilege when setting SQL mode at `GLOBAL` level, and your setting at this level only affects the connections established afterwards. -`Modes` are a series of different modes separated by commas (','). You can use the `SELECT @@sql_mode` statement to check the current SQL mode. The default value of SQL mode: `ONLY_FULL_GROUP_BY, STRICT_TRANS_TABLES, NO_ZERO_IN_DATE, NO_ZERO_DATE, ERROR_FOR_DIVISION_BY_ZERO, NO_AUTO_CREATE_USER, NO_ENGINE_SUBSTITUTION`. +- Changes to SQL mode at `SESSION` level only affect the current client. + +In this statement, `modes` are a series of different modes separated by commas (','). You can use the `SELECT @@sql_mode` statement to check the current SQL mode. The default value of SQL mode: `ONLY_FULL_GROUP_BY, STRICT_TRANS_TABLES, NO_ZERO_IN_DATE, NO_ZERO_DATE, ERROR_FOR_DIVISION_BY_ZERO, NO_AUTO_CREATE_USER, NO_ENGINE_SUBSTITUTION`. ## Important `sql_mode` values From 07c6b302fee3e84c0db4fb9460baa2146267273b Mon Sep 17 00:00:00 2001 From: lidezhu <47731263+lidezhu@users.noreply.github.com> Date: Fri, 19 Jan 2024 08:25:47 +0800 Subject: [PATCH 25/41] ticdc: add explanation for csv delimiter config (#15877) --- ticdc/ticdc-csv.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/ticdc/ticdc-csv.md b/ticdc/ticdc-csv.md index 83b093343940d..a55c78b89e693 100644 --- a/ticdc/ticdc-csv.md +++ b/ticdc/ticdc-csv.md @@ -23,7 +23,7 @@ protocol = "csv" terminator = "\n" [sink.csv] -delimiter = ',' +delimiter = ',' # Before v7.6.0, you can only set the delimiter to a single character. Starting from v7.6.0, you can set it to 1-3 characters. For example, `$^` or `|@|`. quote = '"' null = '\N' include-commit-ts = true From 2706a586a65e4e141a2ba51e0364f70d07f9a527 Mon Sep 17 00:00:00 2001 From: xixirangrang Date: Fri, 19 Jan 2024 09:43:47 +0800 Subject: [PATCH 26/41] *: add a new system variable `tidb_enable_global_index` (#15675) --- system-variables.md | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/system-variables.md b/system-variables.md index e7682826ac85d..ac5c5b737a399 100644 --- a/system-variables.md +++ b/system-variables.md @@ -2055,6 +2055,16 @@ mysql> SELECT job_info FROM mysql.analyze_jobs ORDER BY end_time DESC LIMIT 1; - Default value: `OFF` - This variable controls whether to enable GC-Aware memory track. +### tidb_enable_global_index New in v7.6.0 + +- Scope: SESSION | GLOBAL +- Persists to cluster: Yes +- Applies to hint [SET_VAR](/optimizer-hints.md#set_varvar_namevar_value): No +- Type: Boolean +- Default value: `OFF` +- Possible values: `OFF`, `ON` +- This variable controls whether to support creating `Global indexes` for partitioned tables. `Global index` is currently in the development stage. **It is not recommended to modify the value of this system variable**. + ### tidb_enable_non_prepared_plan_cache - Scope: SESSION | GLOBAL From 14b6d03402d3aade319271184458fcfe82c44df3 Mon Sep 17 00:00:00 2001 From: xixirangrang Date: Fri, 19 Jan 2024 11:21:17 +0800 Subject: [PATCH 27/41] resource_control: enhance observability (#15934) --- identify-slow-queries.md | 7 +++++++ .../information-schema-slow-query.md | 14 ++++++++++++-- mysql-schema.md | 4 ++++ statement-summary-tables.md | 15 +++++++++++++-- system-variables.md | 1 + 5 files changed, 37 insertions(+), 4 deletions(-) diff --git a/identify-slow-queries.md b/identify-slow-queries.md index aaab48e497eeb..62923d1158b53 100644 --- a/identify-slow-queries.md +++ b/identify-slow-queries.md @@ -161,6 +161,13 @@ TiKV Coprocessor Task fields: * `tikvDiskFull`: The backoff caused by that the TiKV disk is full. * `txnLockFast`: The backoff caused by that locks are encountered during data reads. +Fields related to Resource Control: + +* `Resource_group`: the resource group that the statement is bound to. +* `Request_unit_read`: the total read RUs consumed by the statement. +* `Request_unit_write`: the total write RUs consumed by the statement. +* `Time_queued_by_rc`: the total time that the statement waits for available resources. + ## Related system variables * [`tidb_slow_log_threshold`](/system-variables.md#tidb_slow_log_threshold): Sets the threshold for the slow log. The SQL statement whose execution time exceeds this threshold is recorded in the slow log. The default value is 300 (ms). diff --git a/information-schema/information-schema-slow-query.md b/information-schema/information-schema-slow-query.md index 9678b20f6a25c..0dc2c9dc6f3b8 100644 --- a/information-schema/information-schema-slow-query.md +++ b/information-schema/information-schema-slow-query.md @@ -33,6 +33,7 @@ The output is as follows: | User | varchar(64) | YES | | NULL | | | Host | varchar(64) | YES | | NULL | | | Conn_ID | bigint(20) unsigned | YES | | NULL | | +| Session_alias | varchar(64) | YES | | NULL | | | Exec_retry_count | bigint(20) unsigned | YES | | NULL | | | Exec_retry_time | double | YES | | NULL | | | Query_time | double | YES | | NULL | | @@ -97,13 +98,17 @@ The output is as follows: | Plan_from_cache | tinyint(1) | YES | | NULL | | | Plan_from_binding | tinyint(1) | YES | | NULL | | | Has_more_results | tinyint(1) | YES | | NULL | | +| Resource_group | varchar(64) | YES | | NULL | | +| Request_unit_read | double | YES | | NULL | | +| Request_unit_write | double | YES | | NULL | | +| Time_queued_by_rc | double | YES | | NULL | | | Plan | longtext | YES | | NULL | | | Plan_digest | varchar(128) | YES | | NULL | | | Binary_plan | longtext | YES | | NULL | | | Prev_stmt | longtext | YES | | NULL | | | Query | longtext | YES | | NULL | | +-------------------------------+---------------------+------+------+---------+-------+ -74 rows in set (0.001 sec) +79 rows in set (0.00 sec) ``` The maximum statement length of the `Query` column is limited by the [`tidb_stmt_summary_max_sql_length`](/system-variables.md#tidb_stmt_summary_max_sql_length-new-in-v40) system variable. @@ -138,6 +143,7 @@ The output is as follows: | User | varchar(64) | YES | | NULL | | | Host | varchar(64) | YES | | NULL | | | Conn_ID | bigint(20) unsigned | YES | | NULL | | +| Session_alias | varchar(64) | YES | | NULL | | | Exec_retry_count | bigint(20) unsigned | YES | | NULL | | | Exec_retry_time | double | YES | | NULL | | | Query_time | double | YES | | NULL | | @@ -202,13 +208,17 @@ The output is as follows: | Plan_from_cache | tinyint(1) | YES | | NULL | | | Plan_from_binding | tinyint(1) | YES | | NULL | | | Has_more_results | tinyint(1) | YES | | NULL | | +| Resource_group | varchar(64) | YES | | NULL | | +| Request_unit_read | double | YES | | NULL | | +| Request_unit_write | double | YES | | NULL | | +| Time_queued_by_rc | double | YES | | NULL | | | Plan | longtext | YES | | NULL | | | Plan_digest | varchar(128) | YES | | NULL | | | Binary_plan | longtext | YES | | NULL | | | Prev_stmt | longtext | YES | | NULL | | | Query | longtext | YES | | NULL | | +-------------------------------+---------------------+------+------+---------+-------+ -75 rows in set (0.001 sec) +80 rows in set (0.00 sec) ``` When the cluster system table is queried, TiDB does not obtain data from all nodes, but pushes down the related calculation to other nodes. The execution plan is as follows: diff --git a/mysql-schema.md b/mysql-schema.md index 588e102e31b8c..02d95816d31e5 100644 --- a/mysql-schema.md +++ b/mysql-schema.md @@ -102,6 +102,10 @@ Currently, the `help_topic` is NULL. * `tidb_background_subtask`: the metadata of the current DXF subtask * `tidb_background_subtask_history`: the metadata of the historical DXF subtasks +## System tables related to Resource Control + +* `request_unit_by_group`: the history records of consumed resource units (RUs) of all resource groups + ## Miscellaneous system tables diff --git a/statement-summary-tables.md b/statement-summary-tables.md index 8b28afdd2a57b..0a4f814738941 100644 --- a/statement-summary-tables.md +++ b/statement-summary-tables.md @@ -24,7 +24,7 @@ This document details these tables and introduces how to use them to troubleshoo ## `statements_summary` -`statements_summary` is a system table in `information_schema`. `statements_summary` groups the SQL statements by the SQL digest and the plan digest, and provides statistics for each SQL category. +`statements_summary` is a system table in `information_schema`. `statements_summary` groups the SQL statements by the resource group, the SQL digest and the plan digest, and provides statistics for each SQL category. The "SQL digest" here means the same as used in slow logs, which is a unique identifier calculated through normalized SQL statements. The normalization process ignores constant, blank characters, and is case insensitive. Therefore, statements with consistent syntaxes have the same digest. For example: @@ -86,7 +86,8 @@ The following is a sample output of querying `statements_summary`: > **Note:** > -> In TiDB, the time unit of fields in statement summary tables is nanosecond (ns), whereas in MySQL the time unit is picosecond (ps). +> - In TiDB, the time unit of fields in statement summary tables is nanosecond (ns), whereas in MySQL the time unit is picosecond (ps). +> - Starting from v7.6.0, for clusters with [resource control](/tidb-resource-control.md) enabled, `statements_summary` will be aggregated by resource group, for example, the same statements executed in different resource groups will be collected as different records. ## `statements_summary_history` @@ -402,6 +403,16 @@ Transaction-related fields: - `AVG_AFFECTED_ROWS`: The average number of rows affected. - `PREV_SAMPLE_TEXT`: When the current SQL statement is `COMMIT`, `PREV_SAMPLE_TEXT` is the previous statement to `COMMIT`. In this case, SQL statements are grouped by the digest and `prev_sample_text`. This means that `COMMIT` statements with different `prev_sample_text` are grouped to different rows. When the current SQL statement is not `COMMIT`, the `PREV_SAMPLE_TEXT` field is an empty string. +Fields related to Resource Control: + +- `AVG_REQUEST_UNIT_WRITE`: the average number of write RUs consumed by SQL statements. +- `MAX_REQUEST_UNIT_WRITE`: the maximum number of write RUs consumed by SQL statements. +- `AVG_REQUEST_UNIT_READ`: the average number of read RUs consumed by SQL statements. +- `MAX_REQUEST_UNIT_READ`: the maximum number of read RUs consumed by SQL statements. +- `AVG_QUEUED_RC_TIME`: the average waiting time for available RU when executing SQL statements. +- `MAX_QUEUED_RC_TIME`: the maximum waiting time for available RU when executing SQL statements. +- `RESOURCE_GROUP`: the resource group bound to SQL statements. + ### `statements_summary_evicted` fields description - `BEGIN_TIME`: Records the starting time. diff --git a/system-variables.md b/system-variables.md index ac5c5b737a399..772eba4b7e9ce 100644 --- a/system-variables.md +++ b/system-variables.md @@ -3277,6 +3277,7 @@ For a system upgraded to v5.0 from an earlier version, if you have not modified - `start_ts`: The start timestamp of the transaction. - `for_update_ts`: The `for_update_ts` of the previously executed DML statement. This is an internal term of TiDB used for tests. Usually, you can ignore this information. - `error`: The error message, if any. + - `ru_consumption`: Consumed [RU](/tidb-resource-control.md#what-is-request-unit-ru) for executing the statement. ### tidb_last_txn_info New in v4.0.9 From 5664815efd26e20c761a76adf56a2d6d6021a5a8 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Dani=C3=ABl=20van=20Eeden?= Date: Fri, 19 Jan 2024 05:19:47 +0100 Subject: [PATCH 28/41] Typo fix for show status (#16180) --- sql-statements/sql-statement-show-status.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sql-statements/sql-statement-show-status.md b/sql-statements/sql-statement-show-status.md index 269befdc45b59..b478ec5da1598 100644 --- a/sql-statements/sql-statement-show-status.md +++ b/sql-statements/sql-statement-show-status.md @@ -47,7 +47,7 @@ mysql> SHOW SESSION STATUS; +-------------------------------+--------------------------------------+ 13 rows in set (0.00 sec) -sql> SHOW GLOBAL STATUS; +mysql> SHOW GLOBAL STATUS; +-----------------------+--------------------------------------+ | Variable_name | Value | +-----------------------+--------------------------------------+ From 65a4e32387252ab51d1baaad4e9b8c2252902678 Mon Sep 17 00:00:00 2001 From: xixirangrang Date: Fri, 19 Jan 2024 12:22:47 +0800 Subject: [PATCH 29/41] resource_control: add more info about sql ru consumption (#16039) --- tidb-resource-control.md | 100 +++++++++++++++++++++++++++++++++++++-- 1 file changed, 96 insertions(+), 4 deletions(-) diff --git a/tidb-resource-control.md b/tidb-resource-control.md index 36ce8312ae0d3..18396aea2a034 100644 --- a/tidb-resource-control.md +++ b/tidb-resource-control.md @@ -99,10 +99,6 @@ Request Unit (RU) is a unified abstraction unit in TiDB for system resources, wh > - The preceding table lists only the resources involved in RU calculation for TiDB Self-Hosted clusters, excluding the network and storage consumption. For TiDB Serverless RUs, see [TiDB Serverless Pricing Details](https://www.pingcap.com/tidb-cloud-serverless-pricing-details/). > - Currently, TiFlash resource control only considers SQL CPU, which is the CPU time consumed by the execution of pipeline tasks for queries, and read request payload. -## Estimate RU consumption of SQL statements - -You can use the [`EXPLAIN ANALYZE`](/sql-statements/sql-statement-explain-analyze.md#ru-request-unit-consumption) statement to get the amount of RUs consumed during SQL execution. Note that the amount of RUs is affected by the cache (for example, [coprocessor cache](/coprocessor-cache.md)). When the same SQL is executed multiple times, the amount of RUs consumed by each execution might be different. The RU value does not represent the exact value for each execution, but can be used as a reference for estimation. - ## Parameters for resource control The resource control feature introduces the following system variables or parameters: @@ -522,6 +518,102 @@ By default, the task types that are marked as background tasks are `""`, and the +## View RU consumption + +You can view information about RU consumption. + +### View the RU consumption by SQL + +You can view the RU consumption of SQL statements in the following ways: + +- The system variable `tidb_last_query_info` +- `EXPLAIN ANALYZE` +- Slow queries and corresponding system table +- `statements_summary` + +#### View the RUs consumed by the last SQL execution by querying the system variable `tidb_last_query_info` + +TiDB provides the system variable [`tidb_last_query_info`](/system-variables.md#tidb_last_query_info-new-in-v4014). This system variable records the information of the last DML statement executed, including the RUs consumed by the SQL execution. + +Example: + +1. Run the `UPDATE` statement: + + ```sql + UPDATE sbtest.sbtest1 SET k = k + 1 WHERE id = 1; + ``` + + ``` + Query OK, 1 row affected (0.01 sec) + Rows matched: 1 Changed: 1 Warnings: 0 + ``` + +2. Query the system variable `tidb_last_query_info` to view the information of the last executed statement: + + ```sql + SELECT @@tidb_last_query_info; + ``` + + ``` + +------------------------------------------------------------------------------------------------------------------------+ + | @@tidb_last_query_info | + +------------------------------------------------------------------------------------------------------------------------+ + | {"txn_scope":"global","start_ts":446809472210829315,"for_update_ts":446809472210829315,"ru_consumption":4.34885578125} | + +------------------------------------------------------------------------------------------------------------------------+ + 1 row in set (0.01 sec) + ``` + + In the result, `ru_consumption` is the RUs consumed by the execution of this SQL statement. + +#### View RUs consumed during SQL execution by `EXPLAIN ANALYZE` + +You can use the [`EXPLAIN ANALYZE`](/sql-statements/sql-statement-explain-analyze.md#ru-request-unit-consumption) statement to get the amount of RUs consumed during SQL execution. Note that the amount of RUs is affected by the cache (for example, [coprocessor cache](/coprocessor-cache.md)). When the same SQL is executed multiple times, the amount of RUs consumed by each execution might be different. The RU value does not represent the exact value for each execution, but can be used as a reference for estimation. + +#### Slow queries and the corresponding system table + + + +When you enable resource control, the [slow query log](/identify-slow-queries.md) of TiDB and the corresponding system table [`INFORMATION_SCHEMA.SLOW_QUERY`](/information-schema/information-schema-slow-query.md) contain the resource group, RU consumption of the corresponding SQL, and the time spent waiting for available RUs. + + + + + +When you enable resource control, the system table [`INFORMATION_SCHEMA.SLOW_QUERY`](/information-schema/information-schema-slow-query.md) contains the resource group, RU consumption of the corresponding SQL, and the time spent waiting for available RUs. + + + +#### View RU statistics by `statements_summary` + +The system table [`INFORMATION_SCHEMA.statements_summary`](/statement-summary-tables.md#statements_summary) in TiDB stores the normalized and aggregated statistics of SQL statements. You can use the system table to view and analyze the execution performance of SQL statements. It also contains statistics about resource control, including the resource group name, RU consumption, and the time spent waiting for available RUs. For more details, see [`statements_summary` fields description](/statement-summary-tables.md#statements_summary-fields-description). + +### View the RU consumption of resource groups + +Starting from v7.6.0, TiDB provides the system table [`mysql.request_unit_by_group`](/mysql-schema.md#system-tables-related-to-resource-control) to store the historical records of the RU consumption of each resource group. + +Example: + +```sql +SELECT * FROM request_unit_by_group LIMIT 5; +``` + +``` ++----------------------------+----------------------------+----------------+----------+ +| start_time | end_time | resource_group | total_ru | ++----------------------------+----------------------------+----------------+----------+ +| 2024-01-01 00:00:00.000000 | 2024-01-02 00:00:00.000000 | default | 334147 | +| 2024-01-01 00:00:00.000000 | 2024-01-02 00:00:00.000000 | rg1 | 4172 | +| 2024-01-01 00:00:00.000000 | 2024-01-02 00:00:00.000000 | rg2 | 34028 | +| 2024-01-02 00:00:00.000000 | 2024-01-03 00:00:00.000000 | default | 334088 | +| 2024-01-02 00:00:00.000000 | 2024-01-03 00:00:00.000000 | rg1 | 3850 | ++----------------------------+----------------------------+----------------+----------+ +5 rows in set (0.01 sec) +``` + +> **Note:** +> +> The data of `mysql.request_unit_by_group` is automatically imported by a TiDB scheduled task at the end of each day. If the RU consumption of a resource group is 0 on a certain day, no record is generated. By default, this table stores data for the last three months (up to 92 days). Data that exceeds this period is automatically cleared. + ## Monitoring metrics and charts From c53a938e6c362c253ebd5256b746a46248cb17b0 Mon Sep 17 00:00:00 2001 From: xixirangrang Date: Fri, 19 Jan 2024 12:36:18 +0800 Subject: [PATCH 30/41] DM: support for MySQL 8.0 GA (#15959) --- dm/dm-compatibility-catalog.md | 2 +- dm/dm-overview.md | 7 +++++-- 2 files changed, 6 insertions(+), 3 deletions(-) diff --git a/dm/dm-compatibility-catalog.md b/dm/dm-compatibility-catalog.md index a4a83a6ba2e68..c3fbe60afc57f 100644 --- a/dm/dm-compatibility-catalog.md +++ b/dm/dm-compatibility-catalog.md @@ -19,7 +19,7 @@ DM supports migrating data from different sources to TiDB clusters. Based on the |MySQL ≤ 5.5|Not tested|| |MySQL 5.6|GA|| |MySQL 5.7|GA|| -|MySQL 8.0|Experimental|| +|MySQL 8.0|GA|Does not support binlog transaction compression [Transaction_payload_event](https://dev.mysql.com/doc/refman/8.0/en/binary-log-transaction-compression.html)| |MariaDB < 10.1.2|Incompatible|Incompatible with binlog of the time type| |MariaDB 10.1.2 ~ 10.5.10|Experimental|| |MariaDB > 10.5.10|Incompatible|Permission errors reported in the check procedure| diff --git a/dm/dm-overview.md b/dm/dm-overview.md index 62726e1d3f1e4..b1e9ba04f930d 100644 --- a/dm/dm-overview.md +++ b/dm/dm-overview.md @@ -41,8 +41,7 @@ Before using the DM tool, note the following restrictions: + Database version requirements - - MySQL version 5.5 ~ 5.7 - - MySQL version 8.0 (experimental features) + - MySQL version 5.6 ~ 8.0 - MariaDB version >= 10.1.2 (experimental features) > **Note:** @@ -64,6 +63,10 @@ Before using the DM tool, note the following restrictions: - DM does not support migrating `charset=GBK` tables to TiDB clusters earlier than v5.4.0. ++ Binlog compatibility + + - DM does not support the MySQL 8.0 new feature binlog [Transaction_payload_event](https://dev.mysql.com/doc/refman/8.0/en/binary-log-transaction-compression.html). Using binlog Transaction_payload_event might result in data inconsistency between upstream and downstream. + ## Contributing You are welcome to participate in the DM open sourcing project. Your contribution would be highly appreciated. For more details, see [CONTRIBUTING.md](https://github.com/pingcap/tiflow/blob/master/dm/CONTRIBUTING.md). From 54092815a7e8eb16db4249c8bfb3813d252399ea Mon Sep 17 00:00:00 2001 From: GMHDBJD <35025882+GMHDBJD@users.noreply.github.com> Date: Fri, 19 Jan 2024 12:37:47 +0800 Subject: [PATCH 31/41] ddl: add description of ddl v2 (#16073) --- TOC-tidb-cloud.md | 7 +++++- TOC.md | 1 + ddl-v2.md | 60 +++++++++++++++++++++++++++++++++++++++++++++ system-variables.md | 14 +++++++++++ 4 files changed, 81 insertions(+), 1 deletion(-) create mode 100644 ddl-v2.md diff --git a/TOC-tidb-cloud.md b/TOC-tidb-cloud.md index 2c43f6bc9ae39..1dbb4039cdf8f 100644 --- a/TOC-tidb-cloud.md +++ b/TOC-tidb-cloud.md @@ -283,10 +283,14 @@ - [Billing from AWS or GCP Marketplace](/tidb-cloud/tidb-cloud-billing.md#billing-from-aws-marketplace-or-google-cloud-marketplace) - [Billing for Changefeed](/tidb-cloud/tidb-cloud-billing-ticdc-rcu.md) - [Billing for Data Migration](/tidb-cloud/tidb-cloud-billing-dm.md) +- API - API - [API Overview](/tidb-cloud/api-overview.md) - API Reference - - [v1beta1](https://docs.pingcap.com/tidbcloud/api/v1beta1) + - v1beta1 + - [Billing](https://docs.pingcap.com/tidbcloud/api/v1beta1/billing) + - [IAM](https://docs.pingcap.com/tidbcloud/api/v1beta1/apikey) + - [MSP](https://docs.pingcap.com/tidbcloud/api/msp/v1beta1) - [v1beta](https://docs.pingcap.com/tidbcloud/api/v1beta) - Integrations - [Airbyte](/tidb-cloud/integrate-tidbcloud-with-airbyte.md) @@ -598,6 +602,7 @@ - [`SESSION_CONNECT_ATTRS`](/performance-schema/performance-schema-session-connect-attrs.md) - [Metadata Lock](/metadata-lock.md) - [Use UUIDs](/best-practices/uuid.md) + - [TiDB DDL V2](/ddl-v2.md) - [System Variables](/system-variables.md) - [Server Status Variables](/status-variables.md) - Storage Engines diff --git a/TOC.md b/TOC.md index fde63327c6cd8..e613aaf4cb0b4 100644 --- a/TOC.md +++ b/TOC.md @@ -976,6 +976,7 @@ - [Overview](/performance-schema/performance-schema.md) - [`SESSION_CONNECT_ATTRS`](/performance-schema/performance-schema-session-connect-attrs.md) - [Metadata Lock](/metadata-lock.md) + - [TiDB DDL V2](/ddl-v2.md) - UI - TiDB Dashboard - [Overview](/dashboard/dashboard-intro.md) diff --git a/ddl-v2.md b/ddl-v2.md new file mode 100644 index 0000000000000..961d7f776b722 --- /dev/null +++ b/ddl-v2.md @@ -0,0 +1,60 @@ +--- +title: Use TiDB DDL V2 to Accelerate Table Creation +summary: Learn the concept, principles, and implementation details of TiDB DDL V2 for acceleration table creation. +--- + +# Use TiDB DDL V2 to Accelerate Table Creation + +Starting from v7.6.0, the new version V2 of TiDB DDL supports creating tables quickly, which improves the efficiency of bulk table creation. + +TiDB uses the online asynchronous schema change algorithm to change the metadata. All DDL jobs are submitted to the `mysql.tidb_ddl_job` table, and the owner node pulls the DDL job to execute. After executing each phase of the online DDL algorithm, the DDL job is marked as completed and moved to the `mysql.tidb_ddl_history` table. Therefore, DDL statements can only be executed on the owner node and cannot be linearly extended. + +However, for some DDL statements, it is not necessary to strictly follow the online DDL algorithm. For example, the `CREATE TABLE` statement only has two states for the job: `none` and `public`. Therefore, TiDB can simplify the execution process of DDL, and executes the `CREATE TABLE` statement on a non-owner node to accelerate table creation. + +> **Warning:** +> +> This feature is currently an experimental feature and it is not recommended to use in a production environment. This feature might change or be removed without prior notice. If you find a bug, please give feedback by raising an [issue](https://github.com/pingcap/tidb/issues) on GitHub. + + + +## Compatibility with TiDB tools + +- [TiCDC](/ticdc/ticdc-overview.md) does not support replicating the tables that are created by TiDB DDL V2. + + + +## Limitation + +You can now use TiDB DDL V2 only in the [`CREATE TABLE`](/sql-statements/sql-statement-create-table.md) statement, and this statement must not include any foreign key constraints. + +## Use TiDB DDL V2 + +You can enable or disable TiDB DDL V2 by specifying the value of the system variable [`tidb_ddl_version`](/system-variables.md#tidb_ddl_version-new-in-v760) . + +To enable TiDB DDL V2, set the value of this variable to `2`: + +```sql +SET GLOBAL tidb_ddl_version = 2; +``` + +To disable TiDB DDL V2, set the value of this variable to `1`: + +```sql +SET GLOBAL tidb_ddl_version = 1; +``` + +## Implementation principle + +The detailed implementation principle of TiDB DDL V2 for accelerating table creation is as follows: + +1. Create a `CREATE TABLE` Job. + + This step is the same as that of the V1 implementation. The corresponding DDL Job is generated by parsing the `CREATE TABLE` statement. + +2. Execute the `CREATE TABLE` job. + + Different from V1, in TiDB DDL V2, the TiDB node that receives the `CREATE TABLE` statement executes it directly, and then persists the table structure to TiKV. At the same time, the `CREATE TABLE` job is marked as completed and inserted into the `mysql.tidb_ddl_history` table. + +3. Synchronize the table information. + + TiDB notifies other nodes to synchronize the newly created table structure. diff --git a/system-variables.md b/system-variables.md index 772eba4b7e9ce..59c0d0a9bf6eb 100644 --- a/system-variables.md +++ b/system-variables.md @@ -1692,6 +1692,20 @@ mysql> SELECT job_info FROM mysql.analyze_jobs ORDER BY end_time DESC LIMIT 1; - Unit: Threads - This variable is used to set the concurrency of the DDL operation in the `re-organize` phase. +### `tidb_ddl_version` New in v7.6.0 + +> **Warning:** +> +> This variable is currently an experimental feature and it is not recommended to use in a production environment. This feature might change or be removed without prior notice. If you find a bug, please give feedback by raising an [issue](https://github.com/pingcap/tidb/issues) on GitHub. + +- Scope: GLOBAL +- Persists to cluster: Yes +- Applies to hint [SET_VAR](/optimizer-hints.md#set_varvar_namevar_value): No +- Default value: `1` +- Value range: `1` or `2` +- This variable is used to control whether to enable [TiDB DDL V2](/ddl-v2.md). Setting it to `2` enables this feature, and setting it to `1` disables it. The default value is `1`. After you enable it, TiDB uses TiDB DDL V2 to execute DDL statements. +- Starting from v7.6.0, TiDB only supports accelerating table creation by the [`CREATE TABLE`](/sql-statements/sql-statement-create-table.md) statement. + ### tidb_default_string_match_selectivity New in v6.2.0 - Scope: SESSION | GLOBAL From 9874fde3275518f85f71e798e44fe3df59db5ed7 Mon Sep 17 00:00:00 2001 From: xixirangrang Date: Fri, 19 Jan 2024 12:39:17 +0800 Subject: [PATCH 32/41] fix 2 issues (#16183) --- br/backup-and-restore-overview.md | 2 +- br/br-pitr-guide.md | 8 +++----- dr-solution-introduction.md | 2 +- ticdc/ticdc-open-protocol.md | 14 +++++++------- 4 files changed, 12 insertions(+), 14 deletions(-) diff --git a/br/backup-and-restore-overview.md b/br/backup-and-restore-overview.md index 0f4753bbf6d86..41e51f8e935ee 100644 --- a/br/backup-and-restore-overview.md +++ b/br/backup-and-restore-overview.md @@ -94,7 +94,7 @@ Corresponding to the backup features, you can perform two types of restore: full #### Restore performance and impact on TiDB clusters - Data restore is performed at a scalable speed. Generally, the speed is 100 MiB/s per TiKV node. `br` only supports restoring data to a new cluster and uses the resources of the target cluster as much as possible. For more details, see [Restore performance and impact](/br/br-snapshot-guide.md#performance-and-impact-of-snapshot-restore). -- On each TiKV node, PITR can restore log data at 30 GiB/h. For more details, see [PITR performance and impact](/br/br-pitr-guide.md#performance-and-impact-of-pitr). +- On each TiKV node, PITR can restore log data at 30 GiB/h. For more details, see [PITR performance and impact](/br/br-pitr-guide.md#performance-capabilities-of-pitr). ## Backup storage diff --git a/br/br-pitr-guide.md b/br/br-pitr-guide.md index 56426e8733cf6..76f2f4b704702 100644 --- a/br/br-pitr-guide.md +++ b/br/br-pitr-guide.md @@ -103,9 +103,7 @@ The following steps describe how to clean up backup data that exceeds the backup rm -rf s3://backup-101/snapshot-${date} ``` -## Performance and impact of PITR - -### Capabilities +## Performance capabilities of PITR - On each TiKV node, PITR can restore snapshot data at a speed of 280 GB/h and log data 30 GB/h. - BR deletes outdated log backup data at a speed of 600 GB/h. @@ -121,7 +119,7 @@ The following steps describe how to clean up backup data that exceeds the backup > The default replica number for all clusters in the test is 3. > To improve the overall restore performance, you can modify the [`import.num-threads`](/tikv-configuration-file.md#import) item in the TiKV configuration file and the [`concurrency`](/br/use-br-command-line-tool.md#common-options) option in the BR command. -Testing scenario 1 (on [TiDB Cloud](https://tidbcloud.com)): +Testing scenario 1 (on [TiDB Cloud](https://tidbcloud.com)) is as follows: - The number of TiKV nodes (8 core, 16 GB memory): 21 - TiKV configuration item `import.num-threads`: 8 @@ -130,7 +128,7 @@ Testing scenario 1 (on [TiDB Cloud](https://tidbcloud.com)): - New log data created in the cluster: 10 GB/h - Write (INSERT/UPDATE/DELETE) QPS: 10,000 -Testing scenario 2 (on TiDB Self-Hosted): +Testing scenario 2 (on TiDB Self-Hosted) is as follows: - The number of TiKV nodes (8 core, 64 GB memory): 6 - TiKV configuration item `import.num-threads`: 8 diff --git a/dr-solution-introduction.md b/dr-solution-introduction.md index ca3a80cc91e0c..dd70a150d0c78 100644 --- a/dr-solution-introduction.md +++ b/dr-solution-introduction.md @@ -93,7 +93,7 @@ Of course, if the error tolerance objective is multiple regions and RPO must be In this architecture, TiDB cluster 1 is deployed in region 1. BR regularly backs up the data of cluster 1 to region 2, and continuously backs up the data change logs of this cluster to region 2 as well. When region 1 encounters a disaster and cluster 1 cannot be recovered, you can use the backup data and data change logs to restore a new cluster (cluster 2) in region 2 to provide services. -The DR solution based on BR provides an RPO lower than 5 minutes and an RTO that varies with the size of the data to be restored. For BR v6.5.0, you can refer to [Performance and impact of snapshot restore](/br/br-snapshot-guide.md#performance-and-impact-of-snapshot-restore) and [Performance and impact of PITR](/br/br-pitr-guide.md#performance-and-impact-of-pitr) to learn about the restore speed. Usually, the feature of backup across regions is considered the last resort of data security and also a must-have solution for most systems. For more information about this solution, see [DR solution based on BR](/dr-backup-restore.md). +The DR solution based on BR provides an RPO lower than 5 minutes and an RTO that varies with the size of the data to be restored. For BR v6.5.0, you can refer to [Performance and impact of snapshot restore](/br/br-snapshot-guide.md#performance-and-impact-of-snapshot-restore) and [Performance and impact of PITR](/br/br-pitr-guide.md#performance-capabilities-of-pitr) to learn about the restore speed. Usually, the feature of backup across regions is considered the last resort of data security and also a must-have solution for most systems. For more information about this solution, see [DR solution based on BR](/dr-backup-restore.md). Meanwhile, starting from v6.5.0, BR supports [restoring a TiDB cluster from EBS volume snapshots](https://docs.pingcap.com/tidb-in-kubernetes/stable/restore-from-aws-s3-by-snapshot). If your cluster is running on Kubernetes and you want to restore the cluster as fast as possible without affecting the cluster, you can use this feature to reduce the RTO of your system. diff --git a/ticdc/ticdc-open-protocol.md b/ticdc/ticdc-open-protocol.md index 54ad9facf27bf..7e345531e09b6 100644 --- a/ticdc/ticdc-open-protocol.md +++ b/ticdc/ticdc-open-protocol.md @@ -50,7 +50,7 @@ This section introduces the formats of Row Changed Event, DDL Event, and Resolve + **Key:** - ``` + ```json { "ts":, "scm":, @@ -69,7 +69,7 @@ This section introduces the formats of Row Changed Event, DDL Event, and Resolve `Insert` event. The newly added row data is output. - ``` + ```json { "u":{ :{ @@ -90,7 +90,7 @@ This section introduces the formats of Row Changed Event, DDL Event, and Resolve `Update` event. The newly added row data ("u") and the row data before the update ("p") are output. - ``` + ```json { "u":{ :{ @@ -125,7 +125,7 @@ This section introduces the formats of Row Changed Event, DDL Event, and Resolve `Delete` event. The deleted row data is output. - ``` + ```json { "d":{ :{ @@ -156,7 +156,7 @@ This section introduces the formats of Row Changed Event, DDL Event, and Resolve + **Key:** - ``` + ```json { "ts":, "scm":, @@ -173,7 +173,7 @@ This section introduces the formats of Row Changed Event, DDL Event, and Resolve + **Value:** - ``` + ```json { "q":, "t": @@ -189,7 +189,7 @@ This section introduces the formats of Row Changed Event, DDL Event, and Resolve + **Key:** - ``` + ```json { "ts":, "t":3 From c5aeead22d09cdb7ad393fc99ed9bba5dfe97ded Mon Sep 17 00:00:00 2001 From: Weizhen Wang Date: Fri, 19 Jan 2024 13:43:48 +0800 Subject: [PATCH 33/41] system-variables: set default tidb_auto_analyze_partition_batch_size as 128 (#15865) --- system-variables.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/system-variables.md b/system-variables.md index 59c0d0a9bf6eb..f642a72d80802 100644 --- a/system-variables.md +++ b/system-variables.md @@ -1190,7 +1190,7 @@ mysql> SELECT job_info FROM mysql.analyze_jobs ORDER BY end_time DESC LIMIT 1; - Scope: GLOBAL - Persists to cluster: Yes - Applies to hint [SET_VAR](/optimizer-hints.md#set_varvar_namevar_value): No -- Default value: `1` +- Default value: `128`. Before v7.6.0, the default value is `1`. - Range: `[1, 1024]` - This variable specifies the number of partitions that TiDB [automatically analyzes](/statistics.md#automatic-update) when analyzing a partitioned table (which means automatically collecting statistics on a partitioned table). - If the value of this variable is smaller than the number of partitions, TiDB automatically analyzes all partitions of the partitioned table in multiple batches. If the value of this variable is greater than or equal to the number of partitions, TiDB analyzes all partitions of the partitioned table at the same time. From ffcb52d73acc77d36d7c7dc703e93f3fdbfd5486 Mon Sep 17 00:00:00 2001 From: Grace Cai Date: Fri, 19 Jan 2024 13:45:17 +0800 Subject: [PATCH 34/41] =?UTF-8?q?txn:=20add=20deprecation=20description=20?= =?UTF-8?q?of=20system=20variable=20tidb=5Fdisable=5Ftxn=5F=E2=80=A6=20(#1?= =?UTF-8?q?6130)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- system-variables.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/system-variables.md b/system-variables.md index f642a72d80802..1e0512bea6d30 100644 --- a/system-variables.md +++ b/system-variables.md @@ -1722,6 +1722,10 @@ mysql> SELECT job_info FROM mysql.analyze_jobs ORDER BY end_time DESC LIMIT 1; ### tidb_disable_txn_auto_retry +> **Warning:** +> +> Starting from TiDB v8.0.0, this variable will be deprecated, and TiDB will no longer support automatic retries of optimistic transactions. As an alternative, when encountering transaction conflicts, you can capture the error and retry transactions in your application, or use the [Pessimistic transaction mode](/pessimistic-transaction.md) instead. + - Scope: SESSION | GLOBAL - Persists to cluster: Yes - Applies to hint [SET_VAR](/optimizer-hints.md#set_varvar_namevar_value): No From 2669a4d791d36c21286fa620bce6b3aefc2e0575 Mon Sep 17 00:00:00 2001 From: Grace Cai Date: Fri, 19 Jan 2024 14:16:47 +0800 Subject: [PATCH 35/41] Add two json function which supports push down to tiflash now (#15904) --- tiflash/tiflash-supported-pushdown-calculations.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/tiflash/tiflash-supported-pushdown-calculations.md b/tiflash/tiflash-supported-pushdown-calculations.md index 6d61bfda294e4..e24b6967e2b2b 100644 --- a/tiflash/tiflash-supported-pushdown-calculations.md +++ b/tiflash/tiflash-supported-pushdown-calculations.md @@ -40,9 +40,9 @@ TiFlash supports the following push-down expressions: | [Bitwise operations](/functions-and-operators/bit-functions-and-operators.md) | `&` (bitand), \| (bitor), `~` (bitneg), `^` (bitxor) | | [String functions](/functions-and-operators/string-functions.md) | `SUBSTR()`, `CHAR_LENGTH()`, `REPLACE()`, `CONCAT()`, `CONCAT_WS()`, `LEFT()`, `RIGHT()`, `ASCII()`, `LENGTH()`, `TRIM()`, `LTRIM()`, `RTRIM()`, `POSITION()`, `FORMAT()`, `LOWER()`, `UCASE()`, `UPPER()`, `SUBSTRING_INDEX()`, `LPAD()`, `RPAD()`, `STRCMP()` | | [Regular expression functions and operators](/functions-and-operators/string-functions.md) | `REGEXP`, `REGEXP_LIKE()`, `REGEXP_INSTR()`, `REGEXP_SUBSTR()`, `REGEXP_REPLACE()` | -| [Date functions](/functions-and-operators/date-and-time-functions.md) | `DATE_FORMAT()`, `TIMESTAMPDIFF()`, `FROM_UNIXTIME()`, `UNIX_TIMESTAMP(int)`, `UNIX_TIMESTAMP(decimal)`, `STR_TO_DATE(date)`, `STR_TO_DATE(datetime)`, `DATEDIFF()`, `YEAR()`, `MONTH()`, `DAY()`, `EXTRACT(datetime)`, `DATE()`, `HOUR()`, `MICROSECOND()`, `MINUTE()`, `SECOND()`, `SYSDATE()`, `DATE_ADD/ADDDATE(datetime, int)`, `DATE_ADD/ADDDATE(string, int/real)`, `DATE_SUB/SUBDATE(datetime, int)`, `DATE_SUB/SUBDATE(string, int/real)`, `QUARTER()`, `DAYNAME()`, `DAYOFMONTH()`, `DAYOFWEEK()`, `DAYOFYEAR()`, `LAST_DAY()`, `MONTHNAME()`, `TO_SECONDS()`, `TO_DAYS()`, `FROM_DAYS()`, `WEEKOFYEAR()` -| [JSON function](/functions-and-operators/json-functions.md) | `JSON_LENGTH()`, `->`, `->>`, `JSON_EXTRACT()`, `JSON_ARRAY()`, `JSON_DEPTH()`, `JSON_VALID()`, `JSON_KEYS()`, `JSON_CONTAINS_PATH()` | -| [Conversion functions](/functions-and-operators/cast-functions-and-operators.md) | `CAST(int AS DOUBLE), CAST(int AS DECIMAL)`, `CAST(int AS STRING)`, `CAST(int AS TIME)`, `CAST(double AS INT)`, `CAST(double AS DECIMAL)`, `CAST(double AS STRING)`, `CAST(double AS TIME)`, `CAST(string AS INT)`, `CAST(string AS DOUBLE), CAST(string AS DECIMAL)`, `CAST(string AS TIME)`, `CAST(decimal AS INT)`, `CAST(decimal AS STRING)`, `CAST(decimal AS TIME)`, `CAST(time AS INT)`, `CAST(time AS DECIMAL)`, `CAST(time AS STRING)`, `CAST(time AS REAL)`, `CAST(json AS JSON)`, `CAST(int AS JSON)`, `CAST(real AS JSON)`, `CAST(decimal AS JSON)`, `CAST(string AS JSON)`, `CAST(time AS JSON)`, `CAST(duration AS JSON)` | +| [Date functions](/functions-and-operators/date-and-time-functions.md) | `DATE_FORMAT()`, `TIMESTAMPDIFF()`, `FROM_UNIXTIME()`, `UNIX_TIMESTAMP(int)`, `UNIX_TIMESTAMP(decimal)`, `STR_TO_DATE(date)`, `STR_TO_DATE(datetime)`, `DATEDIFF()`, `YEAR()`, `MONTH()`, `DAY()`, `EXTRACT(datetime)`, `DATE()`, `HOUR()`, `MICROSECOND()`, `MINUTE()`, `SECOND()`, `SYSDATE()`, `DATE_ADD/ADDDATE(datetime, int)`, `DATE_ADD/ADDDATE(string, int/real)`, `DATE_SUB/SUBDATE(datetime, int)`, `DATE_SUB/SUBDATE(string, int/real)`, `QUARTER()`, `DAYNAME()`, `DAYOFMONTH()`, `DAYOFWEEK()`, `DAYOFYEAR()`, `LAST_DAY()`, `MONTHNAME()`, `TO_SECONDS()`, `TO_DAYS()`, `FROM_DAYS()`, `WEEKOFYEAR()` | +| [JSON function](/functions-and-operators/json-functions.md) | `JSON_LENGTH()`, `->`, `->>`, `JSON_EXTRACT()`, `JSON_ARRAY()`, `JSON_DEPTH()`, `JSON_VALID()`, `JSON_KEYS()`, `JSON_CONTAINS_PATH()`, `JSON_UNQUOTE()` | +| [Conversion functions](/functions-and-operators/cast-functions-and-operators.md) | `CAST(int AS DOUBLE), CAST(int AS DECIMAL)`, `CAST(int AS STRING)`, `CAST(int AS TIME)`, `CAST(double AS INT)`, `CAST(double AS DECIMAL)`, `CAST(double AS STRING)`, `CAST(double AS TIME)`, `CAST(string AS INT)`, `CAST(string AS DOUBLE), CAST(string AS DECIMAL)`, `CAST(string AS TIME)`, `CAST(decimal AS INT)`, `CAST(decimal AS STRING)`, `CAST(decimal AS TIME)`, `CAST(time AS INT)`, `CAST(time AS DECIMAL)`, `CAST(time AS STRING)`, `CAST(time AS REAL)`, `CAST(json AS JSON)`, `CAST(json AS STRING)`, `CAST(int AS JSON)`, `CAST(real AS JSON)`, `CAST(decimal AS JSON)`, `CAST(string AS JSON)`, `CAST(time AS JSON)`, `CAST(duration AS JSON)` | | [Aggregate functions](/functions-and-operators/aggregate-group-by-functions.md) | `MIN()`, `MAX()`, `SUM()`, `COUNT()`, `AVG()`, `APPROX_COUNT_DISTINCT()`, `GROUP_CONCAT()` | | [Miscellaneous functions](/functions-and-operators/miscellaneous-functions.md) | `INET_NTOA()`, `INET_ATON()`, `INET6_NTOA()`, `INET6_ATON()` | From 24c46aa156a4cbad3cc2c9c208ac9197c595474c Mon Sep 17 00:00:00 2001 From: xixirangrang Date: Fri, 19 Jan 2024 14:46:48 +0800 Subject: [PATCH 36/41] data migration: add an example Parquet file name (#16192) --- ddl-v2.md | 6 +----- migrate-from-parquet-files-to-tidb.md | 2 +- 2 files changed, 2 insertions(+), 6 deletions(-) diff --git a/ddl-v2.md b/ddl-v2.md index 961d7f776b722..59db024d00c94 100644 --- a/ddl-v2.md +++ b/ddl-v2.md @@ -15,13 +15,9 @@ However, for some DDL statements, it is not necessary to strictly follow the onl > > This feature is currently an experimental feature and it is not recommended to use in a production environment. This feature might change or be removed without prior notice. If you find a bug, please give feedback by raising an [issue](https://github.com/pingcap/tidb/issues) on GitHub. - - ## Compatibility with TiDB tools -- [TiCDC](/ticdc/ticdc-overview.md) does not support replicating the tables that are created by TiDB DDL V2. - - +- [TiCDC](https://docs.pingcap.com/tidb/stable/ticdc-overview) does not support replicating the tables that are created by TiDB DDL V2. ## Limitation diff --git a/migrate-from-parquet-files-to-tidb.md b/migrate-from-parquet-files-to-tidb.md index 0ca90cb019469..ef37bb9d10fd1 100644 --- a/migrate-from-parquet-files-to-tidb.md +++ b/migrate-from-parquet-files-to-tidb.md @@ -41,7 +41,7 @@ Each table in Hive can be exported to parquet files by annotating `STORED AS PAR DROP TABLE temp; ``` -3. The parquet files exported from Hive might not have the `.parquet` suffix and cannot be correctly identified by TiDB Lightning. Therefore, before importing the files, you need to rename the exported files and add the `.parquet` suffix. +3. The parquet files exported from Hive might not have the `.parquet` suffix and cannot be correctly identified by TiDB Lightning. Therefore, before importing the files, you need to rename the exported files and add the `.parquet` suffix to change the full filename to a format that TiDB Lightning recognizes, for example, `${db_name}. ${table_name}.parquet`. For more information about file types and patterns, see [TiDB Lightning Data Sources](/tidb-lightning/tidb-lightning-data-source.md). You can also match data files by setting correct [customized expressions](/tidb-lightning/tidb-lightning-data-source.md#match-customized-files). 4. Put all the parquet files in a unified directory, for example, `/data/my_datasource/` or `s3://my-bucket/sql-backup`. TiDB Lightning will recursively search for all `.parquet` files in this directory and its subdirectories. From 03b5d0da814738470e4bda0f3bccb6d27abb09e5 Mon Sep 17 00:00:00 2001 From: Peter Oyebanji Date: Fri, 19 Jan 2024 10:58:31 +0100 Subject: [PATCH 37/41] functions-and-operators: remove href link from BIT_LENGTH() --- functions-and-operators/string-functions.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/functions-and-operators/string-functions.md b/functions-and-operators/string-functions.md index 61702150faad5..1b655feb73952 100644 --- a/functions-and-operators/string-functions.md +++ b/functions-and-operators/string-functions.md @@ -146,7 +146,7 @@ SELECT BIT_LENGTH("PingCap 123"); > **Note:** > -> The second example operates under the assumption that there is a [database](https://www.w3schools.com/sql/trymysql.asp?filename=trysql_func_mysql_char_length2) with a table titled `Customers` and a column inside the table titled `CustomerName` +> The second example operates under the assumption that there is a database with a table titled `Customers` and a column inside the table titled `CustomerName` ### [`CHAR()`](https://dev.mysql.com/doc/refman/8.0/en/string-functions.html#function_char) From 2e13ed4ba7a0a78c4a3cec57fbb6d4e23e5f3e66 Mon Sep 17 00:00:00 2001 From: Peter Oyebanji <127598375+PitifulPete@users.noreply.github.com> Date: Wed, 24 Jan 2024 14:48:00 +0100 Subject: [PATCH 38/41] Update functions-and-operators/string-functions.md Co-authored-by: Grace Cai --- functions-and-operators/string-functions.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/functions-and-operators/string-functions.md b/functions-and-operators/string-functions.md index 1b655feb73952..ffb53964c79bf 100644 --- a/functions-and-operators/string-functions.md +++ b/functions-and-operators/string-functions.md @@ -133,10 +133,10 @@ SELECT CustomerName, BIT_LENGTH(CustomerName) AS BitLengthOfName FROM Customers; ``` ```sql -SELECT BIT_LENGTH("PingCap 123"); +SELECT BIT_LENGTH("PingCAP 123"); +----------+ -| BIT_LENGTH("PingCap 123") | +| BIT_LENGTH("PingCAP 123") | +----------+ | 88 | +----------+ From 9e9781079bb4fa3a58bb99a599584bcbcf560929 Mon Sep 17 00:00:00 2001 From: qiancai Date: Mon, 29 Jan 2024 15:05:23 +0800 Subject: [PATCH 39/41] refine the output formats --- functions-and-operators/string-functions.md | 67 +++++++++++---------- 1 file changed, 34 insertions(+), 33 deletions(-) diff --git a/functions-and-operators/string-functions.md b/functions-and-operators/string-functions.md index ffb53964c79bf..237446f54414d 100644 --- a/functions-and-operators/string-functions.md +++ b/functions-and-operators/string-functions.md @@ -107,50 +107,52 @@ Return a string containing binary representation of a number. ### [`BIT_LENGTH()`](https://dev.mysql.com/doc/refman/8.0/en/string-functions.html#function_bit-length) -The `BIT_LENGTH()` function is used to return the length of a given argument in bits. It calculates the length of the argument and then returns the result in bits. +The `BIT_LENGTH()` function is used to return the length of a given argument in bits. Examples: ```sql SELECT BIT_LENGTH("TiDB"); -+----------+ ++--------------------+ | BIT_LENGTH("TiDB") | -+----------+ -| 32 | -+----------+ ++--------------------+ +| 32 | ++--------------------+ ``` 8 bits per character x 4 characters = 32 bits ```sql -SELECT CustomerName, BIT_LENGTH(CustomerName) AS BitLengthOfName FROM Customers; +SELECT BIT_LENGTH("PingCAP 123"); -| CustomerName|BitLengthOfName | -|-------------|----------------| -| Albert Einstein | 120 | -| Robert Oppenheimer | 144 | ++---------------------------+ +| BIT_LENGTH("PingCAP 123") | ++---------------------------+ +| 88 | ++---------------------------+ ``` +8 bits per character (space is counted because it is a non-alphanumeric character) x 11 characters = 88 bits + ```sql -SELECT BIT_LENGTH("PingCAP 123"); +SELECT CustomerName, BIT_LENGTH(CustomerName) AS BitLengthOfName FROM Customers; -+----------+ -| BIT_LENGTH("PingCAP 123") | -+----------+ -| 88 | -+----------+ ++--------------------+-----------------+ +| CustomerName | BitLengthOfName | ++--------------------+-----------------+ +| Albert Einstein | 120 | +| Robert Oppenheimer | 144 | ++--------------------+-----------------+ ``` -8 bits per character (space is counted as it is a non-alphanumeric character) x 11 characters = 88 bits - > **Note:** > -> The second example operates under the assumption that there is a database with a table titled `Customers` and a column inside the table titled `CustomerName` +> The preceding example operates under the assumption that there is a database with a table named `Customers` and a column inside the table named `CustomerName`. ### [`CHAR()`](https://dev.mysql.com/doc/refman/8.0/en/string-functions.html#function_char) -The `CHAR()` function is used to get the corresponding character value of a specific ASCII value. It performs the opposite function of `ASCII()`. Unlike `ASCII()`, which can get the ASCII value of a specific character, `CHAR()` can get the character of a specific ASCII value. +The `CHAR()` function is used to get the corresponding character of a specific ASCII value. It performs the opposite operation of `ASCII()`, which returns the ASCII value of a specific character. Examples: @@ -158,9 +160,9 @@ Examples: SELECT CHAR(65); +------------+ -| CHAR(65) | +| CHAR(65) | +------------+ -| A | +| A | +------------+ ``` @@ -168,17 +170,13 @@ SELECT CHAR(65); SELECT CHAR(84); +------------+ -| CHAR(84) | +| CHAR(84) | +------------+ -| T | +| T | +------------+ ``` -> **Note:** -> -> The `CHAR()` function can also be used to get the corresponding character values of ASCII values that extend beyond the standard ASCII range (0 - 127). It can also get the corresponding character value of a unicode value. - -Further Examples: +The `CHAR()` function can also be used to get the corresponding character of ASCII values that extend beyond the standard ASCII range (`0` - `127`). ```sql /*For extended ASCII: */ @@ -192,6 +190,8 @@ SELECT CHAR(128); +------------+ ``` +The `CHAR()` function can also get the corresponding character value of a unicode value. + ```sql /* For Unicode: */ @@ -199,13 +199,14 @@ SELECT CHAR(128); SELECT CHAR(50089); -+------------+ ++--------------+ | CHAR(50089) | -+------------+ -| é | -+------------+ ++--------------+ +| é | ++--------------+ ``` + ### [`CHAR_LENGTH()`](https://dev.mysql.com/doc/refman/8.0/en/string-functions.html#function_char-length) The `CHAR_LENGTH()` function is used to get the number of characters in a specified expression. It returns the total number of characters in a specified expression as an integer value. From 32dc80e6e01b891f8bb7a9f2467edd916abc8e3f Mon Sep 17 00:00:00 2001 From: qiancai Date: Mon, 29 Jan 2024 15:08:12 +0800 Subject: [PATCH 40/41] revert changes of CHAR()` and `CHAR_LENGTH()` as they are covered in https://github.com/pingcap/docs/pull/16057 --- functions-and-operators/string-functions.md | 30 ++------------------- 1 file changed, 2 insertions(+), 28 deletions(-) diff --git a/functions-and-operators/string-functions.md b/functions-and-operators/string-functions.md index 237446f54414d..a4663d082e9c0 100644 --- a/functions-and-operators/string-functions.md +++ b/functions-and-operators/string-functions.md @@ -206,39 +206,13 @@ SELECT CHAR(50089); +--------------+ ``` - ### [`CHAR_LENGTH()`](https://dev.mysql.com/doc/refman/8.0/en/string-functions.html#function_char-length) -The `CHAR_LENGTH()` function is used to get the number of characters in a specified expression. It returns the total number of characters in a specified expression as an integer value. - -Examples: - -```sql -SELECT CHAR_LENGTH("TiDB") AS LengthOfString; - -+----------+ -| LengthOfString | -+----------+ -| 4 | -+----------+ -``` - -```sql -SELECT CustomerName, CHAR_LENGTH(CustomerName) AS LenghtOfName FROM Customers; - -| CustomerName|LengthOfName | -|-------------|--------------| -| Albert Einstein | 15 | -| Robert Oppenheimer | 18 | -``` - -> **Note:** -> -> The second example operates under the assumption that there is a [database](https://www.w3schools.com/sql/trymysql.asp?filename=trysql_func_mysql_char_length2) with a table titled `Customers` and a column inside the table titled `CustomerName` +Return the character for each integer passed. ### [`CHARACTER_LENGTH()`](https://dev.mysql.com/doc/refman/8.0/en/string-functions.html#function_character-length) -The `CHARACTER_LENGTH()` function is the same as the `CHAR_LENGTH()` function. Both functions can be used synonymously because they provide the same outputs. +Return number of characters in argument. ### [`CONCAT()`](https://dev.mysql.com/doc/refman/8.0/en/string-functions.html#function_concat) From d5e14f0976b1f86a6ec891c36f88b918f4e4d966 Mon Sep 17 00:00:00 2001 From: Grace Cai Date: Mon, 29 Jan 2024 15:09:29 +0800 Subject: [PATCH 41/41] revert changes --- functions-and-operators/string-functions.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/functions-and-operators/string-functions.md b/functions-and-operators/string-functions.md index a4663d082e9c0..616ef205dfc6b 100644 --- a/functions-and-operators/string-functions.md +++ b/functions-and-operators/string-functions.md @@ -208,11 +208,11 @@ SELECT CHAR(50089); ### [`CHAR_LENGTH()`](https://dev.mysql.com/doc/refman/8.0/en/string-functions.html#function_char-length) -Return the character for each integer passed. +Return number of characters in argument. ### [`CHARACTER_LENGTH()`](https://dev.mysql.com/doc/refman/8.0/en/string-functions.html#function_character-length) -Return number of characters in argument. +Synonym for `CHAR_LENGTH()`. ### [`CONCAT()`](https://dev.mysql.com/doc/refman/8.0/en/string-functions.html#function_concat)