From 34d89a3219ef8e6f0f6331267fe7bb7806dcb50c Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Dani=C3=ABl=20van=20Eeden?= Date: Wed, 22 Nov 2023 21:41:04 +0100 Subject: [PATCH 1/7] Clarify statement on backup-and-restore-using-dumpling page --- backup-and-restore-using-dumpling-lightning.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/backup-and-restore-using-dumpling-lightning.md b/backup-and-restore-using-dumpling-lightning.md index fa4942a0c17af..46b88869f9962 100644 --- a/backup-and-restore-using-dumpling-lightning.md +++ b/backup-and-restore-using-dumpling-lightning.md @@ -7,20 +7,20 @@ summary: Learn how to use Dumpling and TiDB Lightning to back up and restore ful This document introduces how to use Dumpling and TiDB Lightning to back up and restore full data of TiDB. -If you need to back up a small amount of data (for example, less than 50 GB) and do not require high backup speed, you can use [Dumpling](/dumpling-overview.md) to export data from the TiDB database and then use [TiDB Lightning](/tidb-lightning/tidb-lightning-overview.md) to import the data into another TiDB database. For more information about backup and restore, see [TiDB Backup & Restore Overview](/br/backup-and-restore-overview.md). +If you need to back up a small amount of data (for example, less than 50 GiB) and do not require high backup speed, you can use [Dumpling](/dumpling-overview.md) to export data from the TiDB database and then use [TiDB Lightning](/tidb-lightning/tidb-lightning-overview.md) to restore the data into another TiDB database. To backup larger databases the recommended method is to use BR, for more information about backup and restore with BR, see [TiDB Backup & Restore Overview](/br/backup-and-restore-overview.md). Note that Dumpling can be used to export large databases, but BR is better suited tool for that. ## Requirements -- Install and start Dumpling: +- Install Dumpling: ```shell - tiup install dumpling && tiup dumpling + tiup install dumpling ``` -- Install and start TiDB Lightning: +- Install TiDB Lightning: ```shell - tiup install tidb lightning && tiup tidb lightning + tiup install tidb tidb-lightning ``` - [Grant the source database privileges required for Dumpling](/dumpling-overview.md#export-data-from-tidb-or-mysql) From 2879fafb85e9a2e1dd6daea091f497a53bee64fa Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Dani=C3=ABl=20van=20Eeden?= Date: Wed, 22 Nov 2023 22:15:01 +0100 Subject: [PATCH 2/7] Format and improve SQL statements --- ...up-and-restore-using-dumpling-lightning.md | 31 ++++++++++++++++--- 1 file changed, 26 insertions(+), 5 deletions(-) diff --git a/backup-and-restore-using-dumpling-lightning.md b/backup-and-restore-using-dumpling-lightning.md index 46b88869f9962..824601d59122f 100644 --- a/backup-and-restore-using-dumpling-lightning.md +++ b/backup-and-restore-using-dumpling-lightning.md @@ -44,11 +44,32 @@ If you need to save data of one backup task to the local disk, note the followin **Note**: It is difficult to calculate the exact data volume exported by Dumpling from MySQL, but you can estimate the data volume by using the following SQL statement to summarize the `data-length` field in the `information_schema.tables` table: ```sql -/* Calculate the size of all schemas, in MiB. Replace ${schema_name} with your schema name. */ -SELECT table_schema,SUM(data_length)/1024/1024 AS data_length,SUM(index_length)/1024/1024 AS index_length,SUM(data_length+index_length)/1024/1024 AS SUM FROM information_schema.tables WHERE table_schema = "${schema_name}" GROUP BY table_schema; - -/* Calculate the size of the largest table, in MiB. Replace ${schema_name} with your schema name. */ -SELECT table_name,table_schema,SUM(data_length)/1024/1024 AS data_length,SUM(index_length)/1024/1024 AS index_length,SUM(data_length+index_length)/1024/1024 AS SUM from information_schema.tables WHERE table_schema = "${schema_name}" GROUP BY table_name,table_schema ORDER BY SUM DESC LIMIT 5; +-- Calculate the size of all schemas +SELECT + TABLE_SCHEMA, + FORMAT_BYTES(SUM(DATA_LENGTH)) AS 'Data Size', + FORMAT_BYTES(SUM(INDEX_LENGTH)) 'Index Size' +FROM + information_schema.tables +GROUP BY + TABLE_SCHEMA; + +-- Calculate the 5 largest tables +SELECT + TABLE_NAME, + TABLE_SCHEMA, + FORMAT_BYTES(SUM(data_length)) AS 'Data Size', + FORMAT_BYTES(SUM(index_length)) AS 'Index Size', + FORMAT_BYTES(SUM(data_length+index_length)) AS 'Total Size' +FROM + information_schema.tables +GROUP BY + table_name, + table_schema +ORDER BY + SUM(DATA_LENGTH+INDEX_LENGTH) DESC +LIMIT + 5; ``` ### Disk space for the target TiKV cluster From 7ff6e1b6748dab79e09137d31651b8b2bb9aadc4 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Dani=C3=ABl=20van=20Eeden?= Date: Wed, 22 Nov 2023 22:22:28 +0100 Subject: [PATCH 3/7] small fix --- backup-and-restore-using-dumpling-lightning.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/backup-and-restore-using-dumpling-lightning.md b/backup-and-restore-using-dumpling-lightning.md index 824601d59122f..bf8f9988407b5 100644 --- a/backup-and-restore-using-dumpling-lightning.md +++ b/backup-and-restore-using-dumpling-lightning.md @@ -64,8 +64,8 @@ SELECT FROM information_schema.tables GROUP BY - table_name, - table_schema + TABLE_NAME, + TABLE_SCHEMA ORDER BY SUM(DATA_LENGTH+INDEX_LENGTH) DESC LIMIT From 0e20b499b314adc4177937e07ad3aec2425d5918 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Dani=C3=ABl=20van=20Eeden?= Date: Wed, 22 Nov 2023 22:23:13 +0100 Subject: [PATCH 4/7] Another small fix --- backup-and-restore-using-dumpling-lightning.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/backup-and-restore-using-dumpling-lightning.md b/backup-and-restore-using-dumpling-lightning.md index bf8f9988407b5..026d5d598ebe5 100644 --- a/backup-and-restore-using-dumpling-lightning.md +++ b/backup-and-restore-using-dumpling-lightning.md @@ -41,7 +41,7 @@ If you need to save data of one backup task to the local disk, note the followin - Dumpling requires a disk space that can store the whole data source (or to store all upstream tables to be exported). To calculate the required space, see [Downstream storage space requirements](/tidb-lightning/tidb-lightning-requirements.md#storage-space-of-the-target-database). - During the import, TiDB Lightning needs temporary space to store the sorted key-value pairs. The disk space should be enough to hold the largest single table from the data source. -**Note**: It is difficult to calculate the exact data volume exported by Dumpling from MySQL, but you can estimate the data volume by using the following SQL statement to summarize the `data-length` field in the `information_schema.tables` table: +**Note**: It is difficult to calculate the exact data volume exported by Dumpling from MySQL, but you can estimate the data volume by using the following SQL statement to summarize the `DATA_LENGTH` field in the `information_schema.tables` table: ```sql -- Calculate the size of all schemas From d230c7796a5bf67f8a4536a6d86c431c8d7c13af Mon Sep 17 00:00:00 2001 From: xixirangrang Date: Thu, 23 Nov 2023 10:35:09 +0800 Subject: [PATCH 5/7] Update backup-and-restore-using-dumpling-lightning.md --- backup-and-restore-using-dumpling-lightning.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/backup-and-restore-using-dumpling-lightning.md b/backup-and-restore-using-dumpling-lightning.md index 026d5d598ebe5..03f3915a0a6ec 100644 --- a/backup-and-restore-using-dumpling-lightning.md +++ b/backup-and-restore-using-dumpling-lightning.md @@ -7,7 +7,9 @@ summary: Learn how to use Dumpling and TiDB Lightning to back up and restore ful This document introduces how to use Dumpling and TiDB Lightning to back up and restore full data of TiDB. -If you need to back up a small amount of data (for example, less than 50 GiB) and do not require high backup speed, you can use [Dumpling](/dumpling-overview.md) to export data from the TiDB database and then use [TiDB Lightning](/tidb-lightning/tidb-lightning-overview.md) to restore the data into another TiDB database. To backup larger databases the recommended method is to use BR, for more information about backup and restore with BR, see [TiDB Backup & Restore Overview](/br/backup-and-restore-overview.md). Note that Dumpling can be used to export large databases, but BR is better suited tool for that. +If you need to back up a small amount of data (for example, less than 50 GiB) and do not require high backup speed, you can use [Dumpling](/dumpling-overview.md) to export data from the TiDB database and then use [TiDB Lightning](/tidb-lightning/tidb-lightning-overview.md) to restore the data into another TiDB database. + +If you need to back up larger databases, the recommended method is to use [BR](/br/backup-and-restore-overview.md). Note that Dumpling can be used to export large databases, but BR is a better tool for that. ## Requirements From 5fb615414589af9ca46897717a2d438454c6751a Mon Sep 17 00:00:00 2001 From: xixirangrang <35301108+hfxsd@users.noreply.github.com> Date: Thu, 23 Nov 2023 11:21:28 +0800 Subject: [PATCH 6/7] update DATA_LENGTH --- dm/dm-hardware-and-software-requirements.md | 2 +- migrate-large-mysql-to-tidb.md | 2 +- tidb-lightning/tidb-lightning-requirements.md | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) diff --git a/dm/dm-hardware-and-software-requirements.md b/dm/dm-hardware-and-software-requirements.md index eb384aa2ee6e9..2a1113d923c29 100644 --- a/dm/dm-hardware-and-software-requirements.md +++ b/dm/dm-hardware-and-software-requirements.md @@ -55,7 +55,7 @@ The target TiKV cluster must have enough disk space to store the imported data. - Indexes might take extra space. - RocksDB has a space amplification effect. -You can estimate the data volume by using the following SQL statements to summarize the `data-length` field: +You can estimate the data volume by using the following SQL statements to summarize the `DATA_LENGTH` field: - Calculate the size of all schemas, in MiB. Replace `${schema_name}` with your schema name. diff --git a/migrate-large-mysql-to-tidb.md b/migrate-large-mysql-to-tidb.md index c7d86294e7723..5f687d214d00a 100644 --- a/migrate-large-mysql-to-tidb.md +++ b/migrate-large-mysql-to-tidb.md @@ -29,7 +29,7 @@ This document describes how to perform the full migration using Dumpling and TiD - During the import, TiDB Lightning needs temporary space to store the sorted key-value pairs. The disk space should be enough to hold the largest single table from the data source. - If the full data volume is large, you can increase the binlog storage time in the upstream. This is to ensure that the binlogs are not lost during the incremental replication. -**Note**: It is difficult to calculate the exact data volume exported by Dumpling from MySQL, but you can estimate the data volume by using the following SQL statement to summarize the `data-length` field in the `information_schema.tables` table: +**Note**: It is difficult to calculate the exact data volume exported by Dumpling from MySQL, but you can estimate the data volume by using the following SQL statement to summarize the `DATA_LENGTH` field in the `information_schema.tables` table: {{< copyable "" >}} diff --git a/tidb-lightning/tidb-lightning-requirements.md b/tidb-lightning/tidb-lightning-requirements.md index c7897a9794aba..86c9bb53af71f 100644 --- a/tidb-lightning/tidb-lightning-requirements.md +++ b/tidb-lightning/tidb-lightning-requirements.md @@ -84,7 +84,7 @@ The target TiKV cluster must have enough disk space to store the imported data. - Indexes might take extra space. - RocksDB has a space amplification effect. -It is difficult to calculate the exact data volume exported by Dumpling from MySQL. However, you can estimate the data volume by using the following SQL statement to summarize the data-length field in the information_schema.tables table: +It is difficult to calculate the exact data volume exported by Dumpling from MySQL. However, you can estimate the data volume by using the following SQL statement to summarize the `DATA_LENGTH` field in the information_schema.tables table: Calculate the size of all schemas, in MiB. Replace ${schema_name} with your schema name. From 061348f5bf6ee0d005410233f0289e2c2487c0f4 Mon Sep 17 00:00:00 2001 From: xixirangrang Date: Thu, 23 Nov 2023 11:22:14 +0800 Subject: [PATCH 7/7] Update backup-and-restore-using-dumpling-lightning.md Co-authored-by: Aolin --- backup-and-restore-using-dumpling-lightning.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/backup-and-restore-using-dumpling-lightning.md b/backup-and-restore-using-dumpling-lightning.md index 03f3915a0a6ec..640969e2a47c8 100644 --- a/backup-and-restore-using-dumpling-lightning.md +++ b/backup-and-restore-using-dumpling-lightning.md @@ -22,7 +22,7 @@ If you need to back up larger databases, the recommended method is to use [BR](/ - Install TiDB Lightning: ```shell - tiup install tidb tidb-lightning + tiup install tidb-lightning ``` - [Grant the source database privileges required for Dumpling](/dumpling-overview.md#export-data-from-tidb-or-mysql)