Skip to content

Commit d47a249

Browse files
committed
Use simpletab in Step 2
1 parent 9b44054 commit d47a249

File tree

1 file changed

+77
-40
lines changed

1 file changed

+77
-40
lines changed

tidb-cloud/changefeed-sink-to-apache-kafka.md

Lines changed: 77 additions & 40 deletions
Original file line numberDiff line numberDiff line change
@@ -40,25 +40,27 @@ Ensure that your TiDB cluster can connect to the Apache Kafka service. You can c
4040
- VPC Peering: suitable as a cost-effective option, but requires managing potential VPC CIDR conflicts and security considerations.
4141
- Public IP: suitable for a quick setup.
4242

43-
#### Private Connect
43+
<SimpleTab>
44+
<div label="Private Connect">
4445

4546
Private Connect leverages **Private Link** or **Private Service Connect** technologies from cloud providers to enable resources in your VPC to connect to services in other VPCs using private IP addresses, as if those services were hosted directly within your VPC.
4647

4748
Currently, TiDB Cloud supports Private Connect for generic Kafka only. It does not include special integration with MSK, Confluent Kafka, or other services.
4849

4950
- If your Apache Kafka service is hosted on AWS, follow [Set Up Self Hosted Kafka Private Link Service in AWS](/tidb-cloud/setup-self-hosted-kafka-pls.md) to ensure that the network connection is properly configured. After setup, provide the following information in the TiDB Cloud console to create the changefeed:
5051

51-
- ID in Kafka Advertised Listener Pattern
52-
- Endpoint Service Name
53-
- Bootstrap Ports
52+
- The ID in Kafka Advertised Listener Pattern
53+
- The Endpoint Service Name
54+
- The Bootstrap Ports
5455

5556
- If your Apache Kafka service is hosted on Google Cloud, follow [Set Up Self Hosted Kafka Private Service Connect in Google Cloud](/tidb-cloud/setup-self-hosted-kafka-psc.md) to ensure that the network connection is properly configured. After setup, provide the following information in the TiDB Cloud console to create the changefeed:
5657

57-
- ID in Kafka Advertised Listener Pattern
58-
- Service Attachment
59-
- Bootstrap Ports
58+
- The ID in Kafka Advertised Listener Pattern
59+
- The Service Attachment
60+
- The Bootstrap Ports
6061

61-
#### VPC Peering
62+
</div>
63+
<div label="VPC Peering">
6264

6365
If your Apache Kafka service is in an AWS VPC that has no internet access, take the following steps:
6466

@@ -79,9 +81,15 @@ If your Apache Kafka service is in a Google Cloud VPC that has no internet acces
7981

8082
You must add the CIDR of the region where your TiDB Cloud cluster is located to the ingress firewall rules. The CIDR can be found on the **VPC Peering** page. Doing so allows the traffic to flow from your TiDB cluster to the Kafka brokers.
8183

82-
#### Public IP
84+
</div>
85+
<div label="Public IP">
8386

84-
If you want to provide Public IP access to your Apache Kafka service, assign Public IPs to all your Kafka brokers. However, using Public IP in a production environment is strongly discouraged.
87+
If you want to provide Public IP access to your Apache Kafka service, assign Public IP addresses to all your Kafka brokers.
88+
89+
It is **NOT** recommended to use Public IP in a production environment.
90+
91+
</div>
92+
</SimpleTab>
8593

8694
### Kafka ACL authorization
8795

@@ -94,42 +102,71 @@ For example, if your Kafka cluster is in Confluent Cloud, you can see [Resources
94102

95103
## Step 1. Open the changefeed page for Apache Kafka
96104

97-
1. In the [TiDB Cloud console](https://tidbcloud.com), navigate to the cluster overview page of the target TiDB cluster, and then click **Changefeed** in the left navigation pane.
98-
2. Click **Create Changefeed**, and select **Kafka** as **Target Type**.
105+
1. Log in to the [TiDB Cloud console](https://tidbcloud.com).
106+
2. Navigate to the cluster overview page of the target TiDB cluster, and then click **Changefeed** in the left navigation pane.
107+
3. Click **Create Changefeed**, and select **Kafka** as **Target Type**.
99108

100109
## Step 2. Configure the changefeed target
101110

102-
1. Select **Connectivity Method** by your Apache Kafka Service setup.
103-
104-
- If you select **VPC Peering** or **Public IP**, fill in your Kafka brokers endpoints. You can use commas `,` to separate multiple endpoints.
105-
106-
- If you select **Private Link**, do the following:
111+
The steps vary depending on the connectivity method you select.
107112

108-
1. Authorize the TiDB Cloud AWS account to create an endpoint for your endpoint service. The TiDB Cloud AWS account ID is provided in the tip on the web page.
109-
2. Make sure you select the same **Number of AZs** and **Suggested Kafka Endpoint Service AZs**, and fill the same unique ID in **Kafka Advertised Listener Pattern** when you [set up self hosted Kafka Private Link service in AWS](/tidb-cloud/setup-self-hosted-kafka-pls.md) in the **Network** section.
110-
3. Fill the **Endpoint Service Name** which is configured in [Setup Self Hosted Kafka Private Link Service in AWS](/tidb-cloud/setup-self-hosted-kafka-pls.md).
111-
4. Fill the **Bootstrap Ports**. It is recommended that you set at least one port for one AZ. You can use commas `,` to separate multiple ports.
113+
<SimpleTab>
114+
<div label="VPC Peering or Public IP">
112115

113-
- If you select **Private Service Connect**, do the following:
114-
115-
1. Ensure that you fill the same unique ID in **Kafka Advertised Listener Pattern** when you [Setup Self Hosted Kafka Private Service Connect in Google Cloud](/tidb-cloud/setup-self-hosted-kafka-psc.md) in **Network** section.
116-
2. Fill the **Service Attachment** that you have configured in [Setup Self Hosted Kafka Private Service Connect in Google Cloud](/tidb-cloud/setup-self-hosted-kafka-psc.md)
117-
3. Fill the **Bootstrap Ports**. It is recommended that you provide more than one port. You can use commas `,` to separate multiple ports.
118-
116+
1. In **Connectivity Method**, select **VPC Peering** or **Public IP**, fill in your Kafka brokers endpoints. You can use commas `,` to separate multiple endpoints.
119117
2. Select an **Authentication** option according to your Kafka authentication configuration.
118+
120119
- If your Kafka does not require authentication, keep the default option **Disable**.
121120
- If your Kafka requires authentication, select the corresponding authentication type, and then fill in the **user name** and **password** of your Kafka account for authentication.
122121

123-
3. Select your **Kafka Version**. If you do not know that, use Kafka V2.
122+
3. Select your **Kafka Version**. If you do not know which one to use, use **Kafka v2**.
124123
4. Select a **Compression** type for the data in this changefeed.
125124
5. Enable the **TLS Encryption** option if your Kafka has enabled TLS encryption and you want to use TLS encryption for the Kafka connection.
126-
6. Click **Validate Connection and Next** to test the network connection. If the test is successful, you will be directed to the next page.
125+
6. Click **Next** to test the network connection. If the test succeeds, you will be directed to the next page.
126+
127+
</div>
128+
<div label="Private Link">
129+
130+
1. In **Connectivity Method**, select **Private Link**.
131+
2. Authorize the [AWS Principal](https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_policies_elements_principal.html#principal-accounts) of TiDB Cloud to create an endpoint for your endpoint service. The AWS Principal is provided in the tip on the web page.
132+
3. Make sure you select the same **Number of AZs** and **Suggested Kafka Endpoint Service AZs**, and fill the same unique ID in **Kafka Advertised Listener Pattern** when you [set up self hosted Kafka Private Link service in AWS](/tidb-cloud/setup-self-hosted-kafka-pls.md) in the **Network** section.
133+
4. Fill the **Endpoint Service Name** which is configured in [Setup Self Hosted Kafka Private Link Service in AWS](/tidb-cloud/setup-self-hosted-kafka-pls.md).
134+
5. Fill the **Bootstrap Ports**. It is recommended that you set at least one port for one AZ. You can use commas `,` to separate multiple ports.
135+
6. Select an **Authentication** option according to your Kafka authentication configuration.
136+
137+
- If your Kafka does not require authentication, keep the default option **Disable**.
138+
- If your Kafka requires authentication, select the corresponding authentication type, and then fill in the **user name** and **password** of your Kafka account for authentication.
139+
140+
7. Select your **Kafka Version**. If you do not know which one to use, use **Kafka v2**.
141+
8. Select a **Compression** type for the data in this changefeed.
142+
9. Enable the **TLS Encryption** option if your Kafka has enabled TLS encryption and you want to use TLS encryption for the Kafka connection.
143+
10. Click **Next** to test the network connection. If the test succeeds, you will be directed to the next page.
144+
11. TiDB Cloud creates the endpoint for **Private Link** or **Private Service Connect**, which might take several minutes.
145+
12. Once the endpoint is created, log in to your cloud provider console and accept the connection request.
146+
13. Return to the [TiDB Cloud console](https://tidbcloud.com) to confirm that you have accepted the connection request. TiDB Cloud will test the connection and proceed to the next page if the test succeeds.
147+
148+
</div>
149+
<div label="Private Service Connect">
150+
151+
1. In **Connectivity Method**, select **Private Service Connect**.
152+
2. Ensure that you fill the same unique ID in **Kafka Advertised Listener Pattern** when you [Setup Self Hosted Kafka Private Service Connect in Google Cloud](/tidb-cloud/setup-self-hosted-kafka-psc.md) in **Network** section.
153+
3. Fill the **Service Attachment** that you have configured in [Setup Self Hosted Kafka Private Service Connect in Google Cloud](/tidb-cloud/setup-self-hosted-kafka-psc.md)
154+
4. Fill the **Bootstrap Ports**. It is recommended that you provide more than one port. You can use commas `,` to separate multiple ports.
155+
5. Select an **Authentication** option according to your Kafka authentication configuration.
156+
157+
- If your Kafka does not require authentication, keep the default option **Disable**.
158+
- If your Kafka requires authentication, select the corresponding authentication type, and then fill in the **user name** and **password** of your Kafka account for authentication.
127159

128-
If you select **Private Link** or **Private Service Connect** as the network connectivity method, follow these additional steps:
160+
6. Select your **Kafka Version**. If you do not know which one to use, use **Kafka v2**.
161+
7. Select a **Compression** type for the data in this changefeed.
162+
8. Enable the **TLS Encryption** option if your Kafka has enabled TLS encryption and you want to use TLS encryption for the Kafka connection.
163+
9. Click **Next** to test the network connection. If the test succeeds, you will be directed to the next page.
164+
10. TiDB Cloud creates the endpoint for **Private Link** or **Private Service Connect**, which might take several minutes.
165+
11. Once the endpoint is created, log in to your cloud provider console and accept the connection request.
166+
12. Return to the [TiDB Cloud console](https://tidbcloud.com) to confirm that you have accepted the connection request. TiDB Cloud will test the connection and proceed to the next page if the test succeeds.
129167

130-
1. After clicking **Next**, TiDB Cloud creates the endpoint for **Private Link** or **Private Service Connect**, which might take several minutes.
131-
2. Once the endpoint is created, log in to your cloud provider console and accept the connection request.
132-
3. Return to the [TiDB Cloud console](https://tidbcloud.com) to confirm that you have accepted the connection request. TiDB Cloud will test the connection and proceed to the next page if the test is successful.
168+
</div>
169+
</SimpleTab>
133170

134171
## Step 3. Set the changefeed
135172

@@ -146,8 +183,8 @@ If you select **Private Link** or **Private Service Connect** as the network con
146183

147184
3. In the **Data Format** area, select your desired format of Kafka messages.
148185

149-
- Avro is a compact, fast, and binary data format with rich data structures, which is widely used in various flow systems. For more information, see [Avro data format](https://docs.pingcap.com/tidb/stable/ticdc-avro-protocol).
150-
- Canal-JSON is a plain JSON text format, which is easy to parse. For more information, see [Canal-JSON data format](https://docs.pingcap.com/tidb/stable/ticdc-canal-json).
186+
- Avro is a compact, fast, and binary data format with rich data structures, which is widely used in various flow systems. For more information, see [Avro data format](https://docs.pingcap.com/tidb/stable/ticdc-avro-protocol).
187+
- Canal-JSON is a plain JSON text format, which is easy to parse. For more information, see [Canal-JSON data format](https://docs.pingcap.com/tidb/stable/ticdc-canal-json).
151188

152189
4. Enable the **TiDB Extension** option if you want to add TiDB-extension fields to the Kafka message body.
153190

@@ -164,29 +201,29 @@ If you select **Private Link** or **Private Service Connect** as the network con
164201

165202
The distribution mode controls how the changefeed creates Kafka topics, by table, by database, or creating one topic for all changelogs.
166203

167-
- **Distribute changelogs by table to Kafka Topics**
204+
- **Distribute changelogs by table to Kafka Topics**
168205

169206
If you want the changefeed to create a dedicated Kafka topic for each table, choose this mode. Then, all Kafka messages of a table are sent to a dedicated Kafka topic. You can customize topic names for tables by setting a topic prefix, a separator between a database name and table name, and a suffix. For example, if you set the separator as `_`, the topic names are in the format of `<Prefix><DatabaseName>_<TableName><Suffix>`.
170207

171208
For changelogs of non-row events, such as Create Schema Event, you can specify a topic name in the **Default Topic Name** field. The changefeed will create a topic accordingly to collect such changelogs.
172209

173-
- **Distribute changelogs by database to Kafka Topics**
210+
- **Distribute changelogs by database to Kafka Topics**
174211

175212
If you want the changefeed to create a dedicated Kafka topic for each database, choose this mode. Then, all Kafka messages of a database are sent to a dedicated Kafka topic. You can customize topic names of databases by setting a topic prefix and a suffix.
176213

177214
For changelogs of non-row events, such as Resolved Ts Event, you can specify a topic name in the **Default Topic Name** field. The changefeed will create a topic accordingly to collect such changelogs.
178215

179-
- **Send all changelogs to one specified Kafka Topic**
216+
- **Send all changelogs to one specified Kafka Topic**
180217

181218
If you want the changefeed to create one Kafka topic for all changelogs, choose this mode. Then, all Kafka messages in the changefeed will be sent to one Kafka topic. You can define the topic name in the **Topic Name** field.
182219

183220
7. In the **Partition Distribution** area, you can decide which partition a Kafka message will be sent to:
184221

185-
- **Distribute changelogs by index value to Kafka partition**
222+
- **Distribute changelogs by index value to Kafka partition**
186223

187224
If you want the changefeed to send Kafka messages of a table to different partitions, choose this distribution method. The index value of a row changelog will determine which partition the changelog is sent to. This distribution method provides a better partition balance and ensures row-level orderliness.
188225

189-
- **Distribute changelogs by table to Kafka partition**
226+
- **Distribute changelogs by table to Kafka partition**
190227

191228
If you want the changefeed to send Kafka messages of a table to one Kafka partition, choose this distribution method. The table name of a row changelog will determine which partition the changelog is sent to. This distribution method ensures table orderliness but might cause unbalanced partitions.
192229

0 commit comments

Comments
 (0)