diff --git a/media/ticdc/cdc-architecture.png b/media/ticdc/cdc-architecture.png index 22e7b924f65e6..acb3a1a9cdc87 100644 Binary files a/media/ticdc/cdc-architecture.png and b/media/ticdc/cdc-architecture.png differ diff --git a/migration-tools.md b/migration-tools.md index e1ee7d14cbdf0..16d51ee8b5a2c 100644 --- a/migration-tools.md +++ b/migration-tools.md @@ -45,7 +45,7 @@ This document introduces the user scenarios, supported upstreams and downstreams | User scenario | This tool is implemented by pulling TiKV change logs. It can restore cluster data to a consistent state with any upstream TSO, and support other systems to subscribe to data changes. | |---|---| | **Upstream** | TiDB | -| **Downstream** | TiDB, MySQL, Kafka, Confluent | +| **Downstream** | TiDB, MySQL, Kafka, MQ, Confluent, storage services such as Amazon S3, GCS, Azure Blob Storage, and NFS. | | **Advantages** | Provide TiCDC Open Protocol | | **Limitation** | TiCDC only replicates tables that have at least one valid index. The following scenarios are not supported: | diff --git a/production-deployment-using-tiup.md b/production-deployment-using-tiup.md index c4541f5dd6a49..194a43edc19d3 100644 --- a/production-deployment-using-tiup.md +++ b/production-deployment-using-tiup.md @@ -275,7 +275,7 @@ The following examples cover seven common scenarios. You need to modify the conf | :-- | :-- | :-- | :-- | | OLTP | [Deploy minimal topology](/minimal-deployment-topology.md) | [Simple minimal configuration template](https://github.com/pingcap/docs/blob/master/config-templates/simple-mini.yaml)
[Full minimal configuration template](https://github.com/pingcap/docs/blob/master/config-templates/complex-mini.yaml) | This is the basic cluster topology, including tidb-server, tikv-server, and pd-server. | | HTAP | [Deploy the TiFlash topology](/tiflash-deployment-topology.md) | [Simple TiFlash configuration template](https://github.com/pingcap/docs/blob/master/config-templates/simple-tiflash.yaml)
[Full TiFlash configuration template](https://github.com/pingcap/docs/blob/master/config-templates/complex-tiflash.yaml) | This is to deploy TiFlash along with the minimal cluster topology. TiFlash is a columnar storage engine, and gradually becomes a standard cluster topology. | -| Replicate incremental data using [TiCDC](/ticdc/ticdc-overview.md) | [Deploy the TiCDC topology](/ticdc-deployment-topology.md) | [Simple TiCDC configuration template](https://github.com/pingcap/docs/blob/master/config-templates/simple-cdc.yaml)
[Full TiCDC configuration template](https://github.com/pingcap/docs/blob/master/config-templates/complex-cdc.yaml) | This is to deploy TiCDC along with the minimal cluster topology. TiCDC supports multiple downstream platforms, such as TiDB, MySQL, and MQ. | +| Replicate incremental data using [TiCDC](/ticdc/ticdc-overview.md) | [Deploy the TiCDC topology](/ticdc-deployment-topology.md) | [Simple TiCDC configuration template](https://github.com/pingcap/docs/blob/master/config-templates/simple-cdc.yaml)
[Full TiCDC configuration template](https://github.com/pingcap/docs/blob/master/config-templates/complex-cdc.yaml) | This is to deploy TiCDC along with the minimal cluster topology. TiCDC supports multiple downstream platforms, such as TiDB, MySQL, Kafka, MQ, and storage services. | | Replicate incremental data using [TiDB Binlog](/tidb-binlog/tidb-binlog-overview.md) | [Deploy the TiDB Binlog topology](/tidb-binlog-deployment-topology.md) | [Simple TiDB Binlog configuration template (MySQL as downstream)](https://github.com/pingcap/docs/blob/master/config-templates/simple-tidb-binlog.yaml)
[Simple TiDB Binlog configuration template (Files as downstream)](https://github.com/pingcap/docs/blob/master/config-templates/simple-file-binlog.yaml)
[Full TiDB Binlog configuration template](https://github.com/pingcap/docs/blob/master/config-templates/complex-tidb-binlog.yaml) | This is to deploy TiDB Binlog along with the minimal cluster topology. | | Use OLAP on Spark | [Deploy the TiSpark topology](/tispark-deployment-topology.md) | [Simple TiSpark configuration template](https://github.com/pingcap/docs/blob/master/config-templates/simple-tispark.yaml)
[Full TiSpark configuration template](https://github.com/pingcap/docs/blob/master/config-templates/complex-tispark.yaml) | This is to deploy TiSpark along with the minimal cluster topology. TiSpark is a component built for running Apache Spark on top of TiDB/TiKV to answer the OLAP queries. Currently, TiUP cluster's support for TiSpark is still **experimental**. | | Deploy multiple instances on a single machine | [Deploy a hybrid topology](/hybrid-deployment-topology.md) | [Simple configuration template for hybrid deployment](https://github.com/pingcap/docs/blob/master/config-templates/simple-multi-instance.yaml)
[Full configuration template for hybrid deployment](https://github.com/pingcap/docs/blob/master/config-templates/complex-multi-instance.yaml) | The deployment topologies also apply when you need to add extra configurations for the directory, port, resource ratio, and label. | diff --git a/ticdc-deployment-topology.md b/ticdc-deployment-topology.md index a07591ba4efaf..579b8f6cd2438 100644 --- a/ticdc-deployment-topology.md +++ b/ticdc-deployment-topology.md @@ -11,7 +11,7 @@ summary: Learn the deployment topology of TiCDC based on the minimal TiDB topolo This document describes the deployment topology of [TiCDC](/ticdc/ticdc-overview.md) based on the minimal cluster topology. -TiCDC is a tool for replicating the incremental data of TiDB, introduced in TiDB 4.0. It supports multiple downstream platforms, such as TiDB, MySQL, and MQ. Compared with TiDB Binlog, TiCDC has lower latency and native high availability. +TiCDC is a tool for replicating the incremental data of TiDB, introduced in TiDB 4.0. It supports multiple downstream platforms, such as TiDB, MySQL, Kafka, MQ, and storage services. Compared with TiDB Binlog, TiCDC has lower latency and native high availability. ## Topology information diff --git a/ticdc/ticdc-overview.md b/ticdc/ticdc-overview.md index 8ea70a5dde381..11036b2ca8f84 100644 --- a/ticdc/ticdc-overview.md +++ b/ticdc/ticdc-overview.md @@ -24,6 +24,7 @@ TiCDC has the following key capabilities: - Bidirectional replication between TiDB clusters, allowing the creation of a multi-active TiDB solution using TiCDC. - Replicating incremental data from a TiDB cluster to a MySQL database or other MySQL-compatible databases with low latency. - Replicating incremental data from a TiDB cluster to a Kafka cluster. The recommended data format includes [Canal-JSON](/ticdc/ticdc-canal-json.md) and [Avro](/ticdc/ticdc-avro-protocol.md). +- Replicating incremental data from a TiDB cluster to storage services, such as Amazon S3, GCS, Azure Blob Storage, and NFS. - Replicating tables with the ability to filter databases, tables, DMLs, and DDLs. - High availability with no single point of failure, supporting dynamically adding and deleting TiCDC nodes. - Cluster management through [Open API](/ticdc/ticdc-open-api.md), including querying task status, dynamically modifying task configuration, and creating or deleting tasks. @@ -69,7 +70,7 @@ The components in the architecture diagram are described as follows: - TiCDC: TiCDC nodes where TiCDC processes run. Each node runs a TiCDC process. Each process pulls data changes from one or more tables in TiKV nodes and replicates the changes to the downstream system through the sink component. - PD: The scheduling module in a TiDB cluster. This module is responsible for scheduling cluster data and usually consists of three PD nodes. PD provides high availability through the etcd cluster. In the etcd cluster, TiCDC stores its metadata, such as node status information and changefeed configurations. -As shown in the architecture diagram, TiCDC supports replicating data to TiDB, MySQL, and Kafka databases. +As shown in the architecture diagram, TiCDC supports replicating data to TiDB, MySQL, Kafka, and storage services. ## Best practices