Skip to content

Commit

Permalink
Discard changes to ticdc/ticdc-sink-to-kafka.md
Browse files Browse the repository at this point in the history
  • Loading branch information
Oreoxmt authored Sep 27, 2024
1 parent ac57ce5 commit 84cc6ba
Showing 1 changed file with 1 addition and 18 deletions.
19 changes: 1 addition & 18 deletions ticdc/ticdc-sink-to-kafka.md
Original file line number Diff line number Diff line change
Expand Up @@ -531,21 +531,4 @@ If the message contains the `claimCheckLocation` field, the Kafka consumer reads
}
```
The `key` and `value` fields corresponds to the same fields in the Kafka message. Consumers can parse the data in these two parts to restore the content of the large message, by encoding the `key` and `value` into one JSON object to deliver a complete message. Only Open-protocol encode the `key` field, it's empty for other protocols.
#### Only send the value to external storage
Starting from version v8.4.0, the `claim-check-raw-value` parameter is supported, and it defaults to false. It can be set to true if not using Open-protocol, otherwise error occurs.
An example configuration is as follows:
```toml
protocol = "simple"
[sink.kafka-config.large-message-handle]
large-message-handle-option = "claim-check"
claim-check-storage-uri = "s3://claim-check-bucket"
claim-check-raw-value = true
```
When this parameter is set to true, the changefeed directly sends the Value portion of Kafka messages to external storage, on the consumer side, data can be read directly from external storage and consumed. This reduce CPU overhead introduced by the JSON serialization and deserialization.
The `key` and `value` fields contain the encoded large message, which should have been sent to the corresponding field in the Kafka message. Consumers can parse the data in these two parts to restore the content of the large message.

0 comments on commit 84cc6ba

Please sign in to comment.