Skip to content

Commit

Permalink
migration: Add information about Vitess to TiDB migration (pingcap#5783)
Browse files Browse the repository at this point in the history
  • Loading branch information
dveeden authored Nov 18, 2024
1 parent e2f1a36 commit 61cd80f
Show file tree
Hide file tree
Showing 6 changed files with 49 additions and 1 deletion.
1 change: 1 addition & 0 deletions TOC.md
Original file line number Diff line number Diff line change
Expand Up @@ -157,6 +157,7 @@
- [Migrate Large Datasets from MySQL](/migrate-large-mysql-to-tidb.md)
- [Migrate and Merge MySQL Shards of Small Datasets](/migrate-small-mysql-shards-to-tidb.md)
- [Migrate and Merge MySQL Shards of Large Datasets](/migrate-large-mysql-shards-to-tidb.md)
- [Migrate from Vitess](/migrate-from-vitess.md)
- [Migrate from MariaDB](/migrate-from-mariadb.md)
- [Migrate from CSV Files](/migrate-from-csv-files-to-tidb.md)
- [Migrate from SQL Files](/migrate-from-sql-files-to-tidb.md)
Expand Down
Binary file added media/vitess_to_tidb.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added media/vitess_to_tidb_dm.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added media/vitess_to_tidb_dumpling_local.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
41 changes: 41 additions & 0 deletions migrate-from-vitess.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
---
title: Migrate Data from Vitess to TiDB
summary: Learn about the tools to migrate data from Vitess to TiDB.
---

# Migrate Data from Vitess to TiDB

This document describes the tools that you can use to migrate data from [Vitess](https://vitess.io/) to TiDB.

Because the backend of Vitess is based on MySQL, when migrating data from Vitess to TiDB, you can use the same migration tools that apply to MySQL, such as [Dumpling](/dumpling-overview.md), [TiDB Lightning](/tidb-lightning/tidb-lightning-overview.md), and [TiDB Data Migration (DM)](/dm/dm-overview.md). Note that these tools should be set up for each shard in Vitess for data migration.

Generally, before data migration, it is recommended to configure the DM task to set `task-mode` to `all` and `import-mode` to `physical`. For more information, see [Task configuration file template (advanced)](/dm/task-configuration-file-full.md#task-configuration-file-template-advanced).

If your data size exceeds 10 TiB, it is recommended to do the import in two steps:

1. Use Dumpling and TiDB Lightning to import existing data.
2. Use DM to import incremental data.

In addition to these tools, you can also use [Debezium connector for Vitess](https://debezium.io/documentation/reference/connectors/vitess.html). This connector enables you to use [Kafka Connect](https://kafka.apache.org/documentation/#connect) or [Apache Flink](https://nightlies.apache.org/flink/flink-docs-stable/) to stream changes from Vitess to TiDB.

Because both Vitess and TiDB support the MySQL protocol and SQL dialect, changes at the application level are expected to be small. For tasks directly managing sharding or other implementation-specific aspects, however, the changes might be larger. To facilitate the data migration from Vitess to TiDB, TiDB introduces the [`VITESS_HASH()`](/functions-and-operators/tidb-functions.md) function, which returns the hash of a string that is compatible with Vitess' HASH function.

## Examples

### Dumpling and TiDB Lightning

The following two examples show how Dumpling and TiDB Lightning work together to migrate data from Vitess to TiDB.

- In this example, TiDB Lightning uses the [logical import mode](/tidb-lightning/tidb-lightning-logical-import-mode.md), which first encodes data into SQL statements and then runs the SQL statements to import data.

![Vitess to TiDB Migration with TiDB backend](/media/vitess_to_tidb.png)

- In this example, TiDB Lightning uses the [physical import mode](/tidb-lightning/tidb-lightning-physical-import-mode.md) to directly ingest data into TiKV.

![Vitess to TiDB Migration with local backend](/media/vitess_to_tidb_dumpling_local.png)

### DM

The following example shows how [DM](/dm/dm-overview.md) migrates data from Vitess to TiDB.

![Vitess to TiDB with DM](/media/vitess_to_tidb_dm.png)
8 changes: 7 additions & 1 deletion migration-overview.md
Original file line number Diff line number Diff line change
Expand Up @@ -44,6 +44,12 @@ If the data size of the sharded tables is large (for example, larger than 1 TiB)

- [Migrate and Merge MySQL Shards of Large Datasets to TiDB](/migrate-large-mysql-shards-to-tidb.md)

## Migrate data from Vitess to TiDB

To migrate data from Vitess to TiDB, the following guide is available:

- [Migrate Data from Vitess to TiDB](/migrate-from-vitess.md)

## Migrate data from files to TiDB

- [Migrate data from CSV files to TiDB](/migrate-from-csv-files-to-tidb.md)
Expand All @@ -61,4 +67,4 @@ The following features can improve the migration process and might meet more nee
- [Continuous Replication from Databases that Use gh-ost or pt-osc](/migrate-with-pt-ghost.md)
- [Migrate Data to a Downstream TiDB Table with More Columns](/migrate-with-more-columns-downstream.md)
- [Filter Binlog Events](/filter-binlog-event.md)
- [Filter DML Events Using SQL Expressions](/filter-dml-event.md)
- [Filter DML Events Using SQL Expressions](/filter-dml-event.md)

0 comments on commit 61cd80f

Please sign in to comment.