From f85af38c72cd83ee200afccab4c3cc65f493c61a Mon Sep 17 00:00:00 2001 From: xixirangrang <35301108+hfxsd@users.noreply.github.com> Date: Mon, 9 Oct 2023 22:05:03 +0800 Subject: [PATCH 01/11] draft --- ...migrate-from-mysql-using-data-migration.md | 35 +++++++++++++++---- tidb-cloud/tidb-cloud-billing-dm.md | 12 +++++++ 2 files changed, 41 insertions(+), 6 deletions(-) diff --git a/tidb-cloud/migrate-from-mysql-using-data-migration.md b/tidb-cloud/migrate-from-mysql-using-data-migration.md index 32305d0759709..ef533d0885376 100644 --- a/tidb-cloud/migrate-from-mysql-using-data-migration.md +++ b/tidb-cloud/migrate-from-mysql-using-data-migration.md @@ -14,26 +14,36 @@ If you want to migrate incremental data only, see [Migrate Incremental Data from ## Limitations +### Availability + - The Data Migration feature is available only for **TiDB Dedicated** clusters. - The Data Migration feature is only available to clusters that are created in [certain regions](https://www.pingcap.com/tidb-cloud-pricing-details/#dm-cost) after November 9, 2022. If your **project** was created before the date or if your cluster is in another region, this feature is not available to your cluster and the **Data Migration** tab will not be displayed on the cluster overview page in the TiDB Cloud console. -- Amazon Aurora MySQL writer instances support both existing data and incremental data migration. Amazon Aurora MySQL reader instances only support existing data migration and do not support incremental data migration. - - You can create up to 200 migration jobs for each organization. To create more migration jobs, you need to [file a support ticket](/tidb-cloud/tidb-cloud-support.md). +### Filtered out and deleted databases + - The system databases will be filtered out and not migrated to TiDB Cloud even if you select all of the databases to migrate. That is, `mysql`, `information_schema`, `information_schema`, and `sys` will not be migrated using this feature. +- When you delete a cluster in TiDB Cloud, all migration jobs in that cluster are automatically deleted and not recoverable. + +### Supported databases and size + +- Amazon Aurora MySQL writer instances support both existing data and incremental data migration. Amazon Aurora MySQL reader instances only support existing data migration and do not support incremental data migration. + +- When you use Data Migration, it is recommended to keep the size of your dataset smaller than 1 TiB. If the dataset size is larger than 1 TiB, the existing data migration will take a long time due to limited specifications. + +### Limitations of existing data migration + - During existing data migration, if the table to be migrated already exists in the target database with duplicated keys, the duplicate keys will be replaced. -- During incremental data migration, if the table to be migrated already exists in the target database with duplicated keys, an error is reported and the migration is interrupted. In this situation, you need to make sure whether the upstream data is accurate. If yes, click the "Restart" button of the migration job and the migration job will replace the downstream conflicting records with the upstream records. +### Limitations of incremental data migration -- When you delete a cluster in TiDB Cloud, all migration jobs in that cluster are automatically deleted and not recoverable. +- During incremental data migration, if the table to be migrated already exists in the target database with duplicated keys, an error is reported and the migration is interrupted. In this situation, you need to make sure whether the upstream data is accurate. If yes, click the "Restart" button of the migration job and the migration job will replace the downstream conflicting records with the upstream records. - During incremental replication (migrating ongoing changes to your cluster), if the migration job recovers from an abrupt error, it might open the safe mode for 60 seconds. During the safe mode, `INSERT` statements are migrated as `REPLACE`, `UPDATE` statements as `DELETE` and `REPLACE`, and then these transactions are migrated to the downstream cluster to make sure that all the data during the abrupt error has been migrated smoothly to the downstream cluster. In this scenario, for upstream tables without primary keys or not-null unique indexes, some data might be duplicated in the downstream cluster because the data might be inserted repeatedly to the downstream. -- When you use Data Migration, it is recommended to keep the size of your dataset smaller than 1 TiB. If the dataset size is larger than 1 TiB, the existing data migration will take a long time due to limited specifications. - - In the following scenarios, if the migration job takes longer than 24 hours, do not purge binary logs in the source database to ensure that Data Migration can get consecutive binary logs for incremental replication: - During existing data migration. @@ -195,10 +205,23 @@ In the **Choose the objects to be migrated** step, you can choose existing data To migrate data to TiDB Cloud once and for all, choose both **Existing data migration** and **Incremental data migration**, which ensures data consistency between the source and target databases. +You can use [physical mode](https://docs.pingcap.com/tidb/stable/tidb-lightning-physical-import-mode) or [logical mode](https://docs.pingcap.com/tidb/stable/tidb-lightning-logical-import-mode) to migrate existing data. + +- The default mode is **Logical mode**. This mode supports migrating data into a table with existing data. But the performance is slower than physical mode. + +- If you use **Physical mode**, the migration job will use [TiDB Lightning](https://docs.pingcap.com/tidb/stable/tidb-lightning-overview) to migrate the existing data. This mode is recommended for large datasets. When you use this mode, the target table must be empty. The performance is about 2.5 times faster than the logical mode. For more information about the performance, see [Performance of physical mode](/tidb-cloud/tidb-cloud-billing-dm.md#performance-of-physical-mode). + +There are limitations for using physical mode: + +- When you use physical mode, you cannot create a second migration job before the existing data migration is completed. +- Physical mode is available only for TiDB clusters in AWS regions. + ### Migrate only existing data To migrate only existing data of the source database to TiDB Cloud, choose **Existing data migration**. +You can choose to use [physical mode](https://docs.pingcap.com/tidb/stable/tidb-lightning-physical-import-mode) or [logical mode](https://docs.pingcap.com/tidb/stable/tidb-lightning-logical-import-mode) to migrate existing data. For more information, see [Migrate existing data and incremental data](#migrate-existing-data-and-incremental-data). + ### Migrate only incremental data To migrate only the incremental data of the source database to TiDB Cloud, choose **Incremental data migration**. In this case, the migration job does not migrate the existing data of the source database to TiDB Cloud, but only migrates the ongoing changes of the source database that are explicitly specified by the migration job. diff --git a/tidb-cloud/tidb-cloud-billing-dm.md b/tidb-cloud/tidb-cloud-billing-dm.md index f083eb212bc29..7b39567e02800 100644 --- a/tidb-cloud/tidb-cloud-billing-dm.md +++ b/tidb-cloud/tidb-cloud-billing-dm.md @@ -26,6 +26,18 @@ The Data Migration job measures full data migration performance in MiB/s. This u The Data Migration job measures incremental data migration performance in rows/s. This unit indicates the number of rows that are migrated to the target database per second. For example, if the upstream database executes `INSERT`, `UPDATE`, or `DELETE` statements of 10,000 rows in about 1 second, the Data Migration job of the corresponding specification can replicate the 10,000 rows to the downstream in about 1 second. +### Performance of physical mode + +[Physical mode](https://docs.pingcap.com/tidb/stable/tidb-lightning-physical-import-mode) exports the upstream data as fast as possible, so different specifications have different performance impacts on QPS and TPS of the upstream database. The following table shows the performance regression of each specification. + +| Migration specifications | Performance regression | Export speed (-t=2*rcu) | +|-------|-------|----------| +| Without DM | / | / | +| 2RCU | 15.6% | 80.84M/s | +| 4RCU | 20.0% | 214.2M/s | +| 8RCU | 28.9% | 365.5M/s | +| 16RCU | 46.7% | 424.6M/s | + ## Price To learn about the supported regions and the price of TiDB Cloud for each Data Migration RCU, see [Data Migration Cost](https://www.pingcap.com/tidb-cloud-pricing-details/#dm-cost). From 867620ed45cdfc7d641bbadcce5fcf3da757c81f Mon Sep 17 00:00:00 2001 From: xixirangrang <35301108+hfxsd@users.noreply.github.com> Date: Mon, 9 Oct 2023 22:14:56 +0800 Subject: [PATCH 02/11] Update migrate-from-mysql-using-data-migration.md --- .../migrate-from-mysql-using-data-migration.md | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-) diff --git a/tidb-cloud/migrate-from-mysql-using-data-migration.md b/tidb-cloud/migrate-from-mysql-using-data-migration.md index ef533d0885376..df47c5bd4c645 100644 --- a/tidb-cloud/migrate-from-mysql-using-data-migration.md +++ b/tidb-cloud/migrate-from-mysql-using-data-migration.md @@ -209,13 +209,23 @@ You can use [physical mode](https://docs.pingcap.com/tidb/stable/tidb-lightning- - The default mode is **Logical mode**. This mode supports migrating data into a table with existing data. But the performance is slower than physical mode. -- If you use **Physical mode**, the migration job will use [TiDB Lightning](https://docs.pingcap.com/tidb/stable/tidb-lightning-overview) to migrate the existing data. This mode is recommended for large datasets. When you use this mode, the target table must be empty. The performance is about 2.5 times faster than the logical mode. For more information about the performance, see [Performance of physical mode](/tidb-cloud/tidb-cloud-billing-dm.md#performance-of-physical-mode). +- If you use **Physical mode**, the migration job will use [TiDB Lightning](https://docs.pingcap.com/tidb/stable/tidb-lightning-overview) to migrate the existing data. This mode is recommended for large datasets. When you use this mode, the target table must be empty. The performance is about 2.5 times faster than the logical mode. There are limitations for using physical mode: - When you use physical mode, you cannot create a second migration job before the existing data migration is completed. - Physical mode is available only for TiDB clusters in AWS regions. +Physical mode exports the upstream data as fast as possible, so [different specifications](/tidb-cloud/tidb-cloud-billing-dm.md#specifications-for-data-migration) have different performance impacts on QPS and TPS of the upstream database during data export. The following table shows the performance regression of each specification. + +| Migration specifications | Performance regression | Export speed (-t=2*rcu) | +|-------|-------|----------| +| Without DM | / | / | +| 2RCU | 15.6% | 80.84M/s | +| 4RCU | 20.0% | 214.2M/s | +| 8RCU | 28.9% | 365.5M/s | +| 16RCU | 46.7% | 424.6M/s | + ### Migrate only existing data To migrate only existing data of the source database to TiDB Cloud, choose **Existing data migration**. From 8ee18f3eb2e8cd57252f656e7ca2d7b1741f6433 Mon Sep 17 00:00:00 2001 From: xixirangrang <35301108+hfxsd@users.noreply.github.com> Date: Mon, 9 Oct 2023 22:15:05 +0800 Subject: [PATCH 03/11] Update tidb-cloud-billing-dm.md --- tidb-cloud/tidb-cloud-billing-dm.md | 12 ------------ 1 file changed, 12 deletions(-) diff --git a/tidb-cloud/tidb-cloud-billing-dm.md b/tidb-cloud/tidb-cloud-billing-dm.md index 7b39567e02800..f083eb212bc29 100644 --- a/tidb-cloud/tidb-cloud-billing-dm.md +++ b/tidb-cloud/tidb-cloud-billing-dm.md @@ -26,18 +26,6 @@ The Data Migration job measures full data migration performance in MiB/s. This u The Data Migration job measures incremental data migration performance in rows/s. This unit indicates the number of rows that are migrated to the target database per second. For example, if the upstream database executes `INSERT`, `UPDATE`, or `DELETE` statements of 10,000 rows in about 1 second, the Data Migration job of the corresponding specification can replicate the 10,000 rows to the downstream in about 1 second. -### Performance of physical mode - -[Physical mode](https://docs.pingcap.com/tidb/stable/tidb-lightning-physical-import-mode) exports the upstream data as fast as possible, so different specifications have different performance impacts on QPS and TPS of the upstream database. The following table shows the performance regression of each specification. - -| Migration specifications | Performance regression | Export speed (-t=2*rcu) | -|-------|-------|----------| -| Without DM | / | / | -| 2RCU | 15.6% | 80.84M/s | -| 4RCU | 20.0% | 214.2M/s | -| 8RCU | 28.9% | 365.5M/s | -| 16RCU | 46.7% | 424.6M/s | - ## Price To learn about the supported regions and the price of TiDB Cloud for each Data Migration RCU, see [Data Migration Cost](https://www.pingcap.com/tidb-cloud-pricing-details/#dm-cost). From 8bc29378db422d06fcf92df69e35070dfdf3213c Mon Sep 17 00:00:00 2001 From: xixirangrang <35301108+hfxsd@users.noreply.github.com> Date: Mon, 9 Oct 2023 22:23:22 +0800 Subject: [PATCH 04/11] Update migrate-from-mysql-using-data-migration.md --- tidb-cloud/migrate-from-mysql-using-data-migration.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tidb-cloud/migrate-from-mysql-using-data-migration.md b/tidb-cloud/migrate-from-mysql-using-data-migration.md index df47c5bd4c645..7779667767fe9 100644 --- a/tidb-cloud/migrate-from-mysql-using-data-migration.md +++ b/tidb-cloud/migrate-from-mysql-using-data-migration.md @@ -209,7 +209,7 @@ You can use [physical mode](https://docs.pingcap.com/tidb/stable/tidb-lightning- - The default mode is **Logical mode**. This mode supports migrating data into a table with existing data. But the performance is slower than physical mode. -- If you use **Physical mode**, the migration job will use [TiDB Lightning](https://docs.pingcap.com/tidb/stable/tidb-lightning-overview) to migrate the existing data. This mode is recommended for large datasets. When you use this mode, the target table must be empty. The performance is about 2.5 times faster than the logical mode. +- If you use **Physical mode**, the migration job uses [TiDB Lightning](https://docs.pingcap.com/tidb/stable/tidb-lightning-overview) in background to migrate the existing data. This mode is recommended for large datasets. When you use this mode, the target table must be empty. The performance is about 2.5 times faster than logical mode. There are limitations for using physical mode: From a93adf6582dd9ce1c2e1d013e2ab4c127f593457 Mon Sep 17 00:00:00 2001 From: xixirangrang Date: Mon, 9 Oct 2023 22:36:11 +0800 Subject: [PATCH 05/11] Apply suggestions from code review Co-authored-by: Frank945946 <108602632+Frank945946@users.noreply.github.com> --- tidb-cloud/migrate-from-mysql-using-data-migration.md | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/tidb-cloud/migrate-from-mysql-using-data-migration.md b/tidb-cloud/migrate-from-mysql-using-data-migration.md index 7779667767fe9..633d2d70afb46 100644 --- a/tidb-cloud/migrate-from-mysql-using-data-migration.md +++ b/tidb-cloud/migrate-from-mysql-using-data-migration.md @@ -209,7 +209,7 @@ You can use [physical mode](https://docs.pingcap.com/tidb/stable/tidb-lightning- - The default mode is **Logical mode**. This mode supports migrating data into a table with existing data. But the performance is slower than physical mode. -- If you use **Physical mode**, the migration job uses [TiDB Lightning](https://docs.pingcap.com/tidb/stable/tidb-lightning-overview) in background to migrate the existing data. This mode is recommended for large datasets. When you use this mode, the target table must be empty. The performance is about 2.5 times faster than logical mode. +- It is recommended to use **Physical mode** for large datasets. When you use this mode, the target table must be empty. The performance is about 2.5 times faster than logical mode. There are limitations for using physical mode: @@ -218,9 +218,8 @@ There are limitations for using physical mode: Physical mode exports the upstream data as fast as possible, so [different specifications](/tidb-cloud/tidb-cloud-billing-dm.md#specifications-for-data-migration) have different performance impacts on QPS and TPS of the upstream database during data export. The following table shows the performance regression of each specification. -| Migration specifications | Performance regression | Export speed (-t=2*rcu) | +| Migration specifications | Performance regression of the upstream database | Maximum Export speed | |-------|-------|----------| -| Without DM | / | / | | 2RCU | 15.6% | 80.84M/s | | 4RCU | 20.0% | 214.2M/s | | 8RCU | 28.9% | 365.5M/s | From 9a6d318d2a35219a6512a164c556d6310c9ad273 Mon Sep 17 00:00:00 2001 From: xixirangrang <35301108+hfxsd@users.noreply.github.com> Date: Mon, 9 Oct 2023 22:39:38 +0800 Subject: [PATCH 06/11] Update migrate-from-mysql-using-data-migration.md --- .../migrate-from-mysql-using-data-migration.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/tidb-cloud/migrate-from-mysql-using-data-migration.md b/tidb-cloud/migrate-from-mysql-using-data-migration.md index 7779667767fe9..e039efd24b0f9 100644 --- a/tidb-cloud/migrate-from-mysql-using-data-migration.md +++ b/tidb-cloud/migrate-from-mysql-using-data-migration.md @@ -20,20 +20,20 @@ If you want to migrate incremental data only, see [Migrate Incremental Data from - The Data Migration feature is only available to clusters that are created in [certain regions](https://www.pingcap.com/tidb-cloud-pricing-details/#dm-cost) after November 9, 2022. If your **project** was created before the date or if your cluster is in another region, this feature is not available to your cluster and the **Data Migration** tab will not be displayed on the cluster overview page in the TiDB Cloud console. +- Amazon Aurora MySQL writer instances support both existing data and incremental data migration. Amazon Aurora MySQL reader instances only support existing data migration and do not support incremental data migration. + +### Supported sizes + - You can create up to 200 migration jobs for each organization. To create more migration jobs, you need to [file a support ticket](/tidb-cloud/tidb-cloud-support.md). +- When you use Data Migration, it is recommended to keep the size of your dataset smaller than 1 TiB. If the dataset size is larger than 1 TiB, the existing data migration will take a long time due to limited specifications. + ### Filtered out and deleted databases - The system databases will be filtered out and not migrated to TiDB Cloud even if you select all of the databases to migrate. That is, `mysql`, `information_schema`, `information_schema`, and `sys` will not be migrated using this feature. - When you delete a cluster in TiDB Cloud, all migration jobs in that cluster are automatically deleted and not recoverable. -### Supported databases and size - -- Amazon Aurora MySQL writer instances support both existing data and incremental data migration. Amazon Aurora MySQL reader instances only support existing data migration and do not support incremental data migration. - -- When you use Data Migration, it is recommended to keep the size of your dataset smaller than 1 TiB. If the dataset size is larger than 1 TiB, the existing data migration will take a long time due to limited specifications. - ### Limitations of existing data migration - During existing data migration, if the table to be migrated already exists in the target database with duplicated keys, the duplicate keys will be replaced. From fc91cab4c319d447d1e1c7eefedb3444b3eb5900 Mon Sep 17 00:00:00 2001 From: xixirangrang <35301108+hfxsd@users.noreply.github.com> Date: Tue, 10 Oct 2023 09:26:12 +0800 Subject: [PATCH 07/11] Update migrate-from-mysql-using-data-migration.md --- tidb-cloud/migrate-from-mysql-using-data-migration.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/tidb-cloud/migrate-from-mysql-using-data-migration.md b/tidb-cloud/migrate-from-mysql-using-data-migration.md index 5f5b1b3d97232..870abbc04df4d 100644 --- a/tidb-cloud/migrate-from-mysql-using-data-migration.md +++ b/tidb-cloud/migrate-from-mysql-using-data-migration.md @@ -22,12 +22,10 @@ If you want to migrate incremental data only, see [Migrate Incremental Data from - Amazon Aurora MySQL writer instances support both existing data and incremental data migration. Amazon Aurora MySQL reader instances only support existing data migration and do not support incremental data migration. -### Supported sizes +### Maximum number of migration jobs - You can create up to 200 migration jobs for each organization. To create more migration jobs, you need to [file a support ticket](/tidb-cloud/tidb-cloud-support.md). -- When you use Data Migration, it is recommended to keep the size of your dataset smaller than 1 TiB. If the dataset size is larger than 1 TiB, the existing data migration will take a long time due to limited specifications. - ### Filtered out and deleted databases - The system databases will be filtered out and not migrated to TiDB Cloud even if you select all of the databases to migrate. That is, `mysql`, `information_schema`, `information_schema`, and `sys` will not be migrated using this feature. @@ -38,6 +36,8 @@ If you want to migrate incremental data only, see [Migrate Incremental Data from - During existing data migration, if the table to be migrated already exists in the target database with duplicated keys, the duplicate keys will be replaced. +- If your dataset size is smaller than 1 TiB, it is recommended that you use logical mode (the default mode). If your dataset size is larger than 1 TiB, you can use physical mode. For more information, see [Migrate existing data and incremental data](#migrate-existing-data-and-incremental-data). + ### Limitations of incremental data migration - During incremental data migration, if the table to be migrated already exists in the target database with duplicated keys, an error is reported and the migration is interrupted. In this situation, you need to make sure whether the upstream data is accurate. If yes, click the "Restart" button of the migration job and the migration job will replace the downstream conflicting records with the upstream records. @@ -205,11 +205,11 @@ In the **Choose the objects to be migrated** step, you can choose existing data To migrate data to TiDB Cloud once and for all, choose both **Existing data migration** and **Incremental data migration**, which ensures data consistency between the source and target databases. -You can use [physical mode](https://docs.pingcap.com/tidb/stable/tidb-lightning-physical-import-mode) or [logical mode](https://docs.pingcap.com/tidb/stable/tidb-lightning-logical-import-mode) to migrate existing data. +You can use [physical mode](https://docs.pingcap.com/tidb/stable/tidb-lightning-physical-import-mode) or [logical mode](https://docs.pingcap.com/tidb/stable/tidb-lightning-logical-import-mode) to migrate **existing data**. - The default mode is **Logical mode**. This mode supports migrating data into a table with existing data. But the performance is slower than physical mode. -- It is recommended to use **Physical mode** for large datasets. When you use this mode, the target table must be empty. The performance is about 2.5 times faster than logical mode. +- It is recommended to use **Physical mode** for large datasets. When you use this mode, the target table must be empty. For the specification of 16RCU, the performance is about 2.5 times faster than logical mode. The performance of other specifications can also increase by more than 100% compared with logcial mode. There are limitations for using physical mode: From 167522c151558699ffd4242953e341f708404546 Mon Sep 17 00:00:00 2001 From: xixirangrang Date: Tue, 10 Oct 2023 10:14:47 +0800 Subject: [PATCH 08/11] Apply suggestions from code review Co-authored-by: Frank945946 <108602632+Frank945946@users.noreply.github.com> --- .../migrate-from-mysql-using-data-migration.md | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/tidb-cloud/migrate-from-mysql-using-data-migration.md b/tidb-cloud/migrate-from-mysql-using-data-migration.md index 870abbc04df4d..68b2f9084d70c 100644 --- a/tidb-cloud/migrate-from-mysql-using-data-migration.md +++ b/tidb-cloud/migrate-from-mysql-using-data-migration.md @@ -36,7 +36,7 @@ If you want to migrate incremental data only, see [Migrate Incremental Data from - During existing data migration, if the table to be migrated already exists in the target database with duplicated keys, the duplicate keys will be replaced. -- If your dataset size is smaller than 1 TiB, it is recommended that you use logical mode (the default mode). If your dataset size is larger than 1 TiB, you can use physical mode. For more information, see [Migrate existing data and incremental data](#migrate-existing-data-and-incremental-data). +- If your dataset size is smaller than 1 TiB, it is recommended that you use logical mode (the default mode). If your dataset size is larger than 1 TiB, or you want to migrate existing data faster, you can use physical mode. For more information, see [Migrate existing data and incremental data](#migrate-existing-data-and-incremental-data). ### Limitations of incremental data migration @@ -209,21 +209,21 @@ You can use [physical mode](https://docs.pingcap.com/tidb/stable/tidb-lightning- - The default mode is **Logical mode**. This mode supports migrating data into a table with existing data. But the performance is slower than physical mode. -- It is recommended to use **Physical mode** for large datasets. When you use this mode, the target table must be empty. For the specification of 16RCU, the performance is about 2.5 times faster than logical mode. The performance of other specifications can also increase by more than 100% compared with logcial mode. +- It is recommended to use **Physical mode** for large datasets. When you use this mode, the target table must be empty. For the specification of 16RCU, the performance is about 2.5 times faster than logical mode. The performance of other specifications can also increase by 20% to 50% compared with logical mode. Note that the performance data is for reference only and might vary in different scenarios. There are limitations for using physical mode: -- When you use physical mode, you cannot create a second migration job before the existing data migration is completed. +- When you use physical mode, you cannot create a second migration job or import task for the TiDB cluster before the existing data migration is completed. - Physical mode is available only for TiDB clusters in AWS regions. Physical mode exports the upstream data as fast as possible, so [different specifications](/tidb-cloud/tidb-cloud-billing-dm.md#specifications-for-data-migration) have different performance impacts on QPS and TPS of the upstream database during data export. The following table shows the performance regression of each specification. -| Migration specifications | Performance regression of the upstream database | Maximum Export speed | -|-------|-------|----------| -| 2RCU | 15.6% | 80.84M/s | -| 4RCU | 20.0% | 214.2M/s | -| 8RCU | 28.9% | 365.5M/s | -| 16RCU | 46.7% | 424.6M/s | +| Migration specifications | Maximum Export speed | Performance regression of the upstream database | +|---------|-------------|--------| +| 2RCU | 80.84 MB/s | 15.6% | +| 4RCU | 214.2 MB/s | 20.0% | +| 8RCU | 365.5 MB/s | 28.9% | +| 16RCU | 424.6 MB/s | 46.7% | ### Migrate only existing data From 0d84f51a9220e70738371ab24afcaf11022dfd97 Mon Sep 17 00:00:00 2001 From: xixirangrang Date: Tue, 10 Oct 2023 10:18:47 +0800 Subject: [PATCH 09/11] Update tidb-cloud/migrate-from-mysql-using-data-migration.md --- tidb-cloud/migrate-from-mysql-using-data-migration.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tidb-cloud/migrate-from-mysql-using-data-migration.md b/tidb-cloud/migrate-from-mysql-using-data-migration.md index 68b2f9084d70c..9247dc5664fd7 100644 --- a/tidb-cloud/migrate-from-mysql-using-data-migration.md +++ b/tidb-cloud/migrate-from-mysql-using-data-migration.md @@ -209,7 +209,7 @@ You can use [physical mode](https://docs.pingcap.com/tidb/stable/tidb-lightning- - The default mode is **Logical mode**. This mode supports migrating data into a table with existing data. But the performance is slower than physical mode. -- It is recommended to use **Physical mode** for large datasets. When you use this mode, the target table must be empty. For the specification of 16RCU, the performance is about 2.5 times faster than logical mode. The performance of other specifications can also increase by 20% to 50% compared with logical mode. Note that the performance data is for reference only and might vary in different scenarios. +- It is recommended to use **Physical mode** for large datasets. When you use this mode, the target table must be empty. For the specification of 16RCU, the performance is about 2.5 times faster than logical mode. The performance of other specifications can also increase by 20% to 50% compared with logical mode. Note that the performance data is for reference only and might vary in different scenarios. There are limitations for using physical mode: From fa3a1424b0e3a4e8c2b9510551618bcfc1920dd8 Mon Sep 17 00:00:00 2001 From: xixirangrang Date: Tue, 10 Oct 2023 10:27:30 +0800 Subject: [PATCH 10/11] Update tidb-cloud/migrate-from-mysql-using-data-migration.md --- tidb-cloud/migrate-from-mysql-using-data-migration.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/tidb-cloud/migrate-from-mysql-using-data-migration.md b/tidb-cloud/migrate-from-mysql-using-data-migration.md index 9247dc5664fd7..3a8820335aac2 100644 --- a/tidb-cloud/migrate-from-mysql-using-data-migration.md +++ b/tidb-cloud/migrate-from-mysql-using-data-migration.md @@ -220,10 +220,10 @@ Physical mode exports the upstream data as fast as possible, so [different speci | Migration specifications | Maximum Export speed | Performance regression of the upstream database | |---------|-------------|--------| -| 2RCU | 80.84 MB/s | 15.6% | -| 4RCU | 214.2 MB/s | 20.0% | -| 8RCU | 365.5 MB/s | 28.9% | -| 16RCU | 424.6 MB/s | 46.7% | +| 2RCU | 80.84 MiB/s | 15.6% | +| 4RCU | 214.2 MiB/s | 20.0% | +| 8RCU | 365.5 MiB/s | 28.9% | +| 16RCU | 424.6 MiB/s | 46.7% | ### Migrate only existing data From b84f6e53c7d7705f7ded73dc23a7efab33e4f429 Mon Sep 17 00:00:00 2001 From: xixirangrang Date: Tue, 10 Oct 2023 14:16:07 +0800 Subject: [PATCH 11/11] Apply suggestions from code review Co-authored-by: Aolin --- ...migrate-from-mysql-using-data-migration.md | 22 +++++++++---------- 1 file changed, 11 insertions(+), 11 deletions(-) diff --git a/tidb-cloud/migrate-from-mysql-using-data-migration.md b/tidb-cloud/migrate-from-mysql-using-data-migration.md index 3a8820335aac2..223665ae6a6a6 100644 --- a/tidb-cloud/migrate-from-mysql-using-data-migration.md +++ b/tidb-cloud/migrate-from-mysql-using-data-migration.md @@ -24,7 +24,7 @@ If you want to migrate incremental data only, see [Migrate Incremental Data from ### Maximum number of migration jobs -- You can create up to 200 migration jobs for each organization. To create more migration jobs, you need to [file a support ticket](/tidb-cloud/tidb-cloud-support.md). +You can create up to 200 migration jobs for each organization. To create more migration jobs, you need to [file a support ticket](/tidb-cloud/tidb-cloud-support.md). ### Filtered out and deleted databases @@ -36,7 +36,7 @@ If you want to migrate incremental data only, see [Migrate Incremental Data from - During existing data migration, if the table to be migrated already exists in the target database with duplicated keys, the duplicate keys will be replaced. -- If your dataset size is smaller than 1 TiB, it is recommended that you use logical mode (the default mode). If your dataset size is larger than 1 TiB, or you want to migrate existing data faster, you can use physical mode. For more information, see [Migrate existing data and incremental data](#migrate-existing-data-and-incremental-data). +- If your dataset size is smaller than 1 TiB, it is recommended that you use [logical mode](https://docs.pingcap.com/tidb/stable/tidb-lightning-logical-import-mode) (the default mode). If your dataset size is larger than 1 TiB, or you want to migrate existing data faster, you can use [physical mode](https://docs.pingcap.com/tidb/stable/tidb-lightning-physical-import-mode). For more information, see [Migrate existing data and incremental data](#migrate-existing-data-and-incremental-data). ### Limitations of incremental data migration @@ -207,23 +207,23 @@ To migrate data to TiDB Cloud once and for all, choose both **Existing data migr You can use [physical mode](https://docs.pingcap.com/tidb/stable/tidb-lightning-physical-import-mode) or [logical mode](https://docs.pingcap.com/tidb/stable/tidb-lightning-logical-import-mode) to migrate **existing data**. -- The default mode is **Logical mode**. This mode supports migrating data into a table with existing data. But the performance is slower than physical mode. +- The default mode is **logical mode**. This mode supports migrating data into a table with existing data. But the performance is slower than physical mode. -- It is recommended to use **Physical mode** for large datasets. When you use this mode, the target table must be empty. For the specification of 16RCU, the performance is about 2.5 times faster than logical mode. The performance of other specifications can also increase by 20% to 50% compared with logical mode. Note that the performance data is for reference only and might vary in different scenarios. +- It is recommended to use **physical mode** for large datasets. When you use this mode, the target table must be empty. For the specification of 16 RCUs (Replication Capacity Units), the performance is about 2.5 times faster than logical mode. The performance of other specifications can increase by 20% to 50% compared with logical mode. Note that the performance data is for reference only and might vary in different scenarios. -There are limitations for using physical mode: +Physical mode has the following limitations: - When you use physical mode, you cannot create a second migration job or import task for the TiDB cluster before the existing data migration is completed. -- Physical mode is available only for TiDB clusters in AWS regions. +- Physical mode is only available for TiDB clusters deployed on AWS. Physical mode exports the upstream data as fast as possible, so [different specifications](/tidb-cloud/tidb-cloud-billing-dm.md#specifications-for-data-migration) have different performance impacts on QPS and TPS of the upstream database during data export. The following table shows the performance regression of each specification. -| Migration specifications | Maximum Export speed | Performance regression of the upstream database | +| Migration specification | Maximum export speed | Performance regression of the upstream database | |---------|-------------|--------| -| 2RCU | 80.84 MiB/s | 15.6% | -| 4RCU | 214.2 MiB/s | 20.0% | -| 8RCU | 365.5 MiB/s | 28.9% | -| 16RCU | 424.6 MiB/s | 46.7% | +| 2 RCUs | 80.84 MiB/s | 15.6% | +| 4 RCUs | 214.2 MiB/s | 20.0% | +| 8 RCUs | 365.5 MiB/s | 28.9% | +| 16 RCUs | 424.6 MiB/s | 46.7% | ### Migrate only existing data