From 849683af130a77f880482b12014fd79cc12f345c Mon Sep 17 00:00:00 2001 From: Kathryn Hancox Date: Mon, 19 Aug 2024 11:46:31 -0400 Subject: [PATCH 1/3] Add TA 122372 --- src/current/advisories/a122372.md | 42 +++++++++++++++++++++++++++ src/current/v23.2/changefeed-sinks.md | 9 ++++++ src/current/v24.1/changefeed-sinks.md | 9 ++++++ 3 files changed, 60 insertions(+) create mode 100644 src/current/advisories/a122372.md diff --git a/src/current/advisories/a122372.md b/src/current/advisories/a122372.md new file mode 100644 index 00000000000..08e15c67cd9 --- /dev/null +++ b/src/current/advisories/a122372.md @@ -0,0 +1,42 @@ +--- +title: Technical Advisory 122372 +advisory: A-122372 +summary: Changefeeds could emit events on the same row out of order in some cases. +toc: true +affected_versions: v23.1, v23.2.0 to v23.2.9, v24.1.0 to v24.1.3, and testing versions of v24.2 through v24.2.0-beta.3 +advisory_date: 2024-09-02 +docs_area: releases +--- + +Publication date: {{ page.advisory_date | date: "%B %e, %Y" }} + +## Description + +In all versions of CockroachDB v23.1, v23.2.0 to v23.2.9, v24.1.0 to v24.1.3, and testing versions of v24.2 through v24.2.0-beta.3, [changefeeds]({% link v24.2/change-data-capture-overview.md %}) could emit events on the same row out of order in some cases, which violates [changefeed ordering guarantees]({% link v24.2/changefeed-messages.md %}#per-key-ordering). This issue was caused by a [bug in the sarama Kafka client library](https://github.com/IBM/sarama/issues/2619), a third-party library that CockroachDB uses to talk to Kafka clusters. The bug manifested when a workload had mutations to the same key in rapid succession, because it was possible for [sarama](https://github.com/IBM/sarama) to re-order the messages if it encountered a retryable Kafka error. + +To resolve this issue, CockroachDB now uses a new version of the Kafka sink that uses [franz-go](https://github.com/twmb/franz-go), a different third-party Kafka client library that does not have this bug. The new Kafka sink can be enabled using the cluster setting [`changefeed.new_kafka_sink.enabled`]({% link v24.2/show-cluster-setting.md %}). The new Kafka sink was added in PR [#126213](https://github.com/cockroachdb/cockroach/pull/126213). + +## Statement + +This is resolved in CockroachDB by PR [#126213](https://github.com/cockroachdb/cockroach/pull/126213), which replaces the Kafka client library [sarama](https://github.com/IBM/sarama) with [franz-go](https://github.com/twmb/franz-go). + +The fix has been applied to maintenance releases of CockroachDB v23.2.10, v24.1.4, and v24.2.0-rc.1. + +This public issue is tracked by issue [#122372](https://github.com/cockroachdb/cockroach/issues/122372). + +## Mitigation + +Users of CockroachDB v23.1, v23.2.0 to v23.2.9, v24.1.0 to v24.1.3, and testing versions of v24.2 through v24.2.0-beta.3 are encouraged to upgrade to [v23.2.10]({% link releases/v23.2.md %}#v23-2-10), [v24.1.4]({% link releases/v24.1.md %}#v24-1-4), [v24.2.0]({% link releases/v24.2.md %}#v24-2-0), or a later version, and enable the cluster setting [`changefeed.new_kafka_sink.enabled`]({% link v24.2/show-cluster-setting.md %}): + +{% include_cached copy-clipboard.html %} +~~~ sql +SET CLUSTER SETTING changefeed.new_kafka_sink.enabled = true; +~~~ + +`changefeed.new_kafka_sink.enabled` will be enabled by default in versions v24.2.1, v24.3, and all later versions. + +## Impact + +Changefeeds could emit events on the same row out of order in some cases, violating the [changefeed ordering guarantee]({% link v24.2/changefeed-messages.md %}#per-key-ordering). Versions affected include CockroachDB v23.1, v23.2.0 to v23.2.9, v24.1.0 to v24.1.3, and testing versions of v24.2 through v24.2.0-beta.3. + +Questions about any technical alert can be directed to our [support team](https://support.cockroachlabs.com/). \ No newline at end of file diff --git a/src/current/v23.2/changefeed-sinks.md b/src/current/v23.2/changefeed-sinks.md index b1ba45a3fcd..7857dcb9b5e 100644 --- a/src/current/v23.2/changefeed-sinks.md +++ b/src/current/v23.2/changefeed-sinks.md @@ -42,6 +42,15 @@ To set a different sink URI to an existing changefeed, use the [`sink` option]({ ## Kafka +{{site.data.alerts.callout_info}} +{% include_cached new-in.html version="v23.2.10" %} CockroachDB uses a different version of the Kafka sink that is implemented with the [franz-go](https://github.com/twmb/franz-go) Kafka client library. We recommend that you enable this updated version of the Kafka sink to avoid a potential bug in the previous version of the CockroachDB Kafka sink; for more details, refer to the [technical advisory 122372]({% link advisories/a122372.md %}). You can enable this Kafka sink with the cluster setting [`changefeed.new_kafka_sink.enabled`]({% link v24.2/show-cluster-setting.md %}). + +{% include_cached copy-clipboard.html %} +~~~ sql +SET CLUSTER SETTING changefeed.new_kafka_sink.enabled = true; +~~~ +{{site.data.alerts.end}} + ### Kafka sink connection Example of a Kafka sink URI using `SCRAM-SHA-256` authentication: diff --git a/src/current/v24.1/changefeed-sinks.md b/src/current/v24.1/changefeed-sinks.md index 633f8bffd8d..a1c94e79cd8 100644 --- a/src/current/v24.1/changefeed-sinks.md +++ b/src/current/v24.1/changefeed-sinks.md @@ -44,6 +44,15 @@ To set a different sink URI to an existing changefeed, use the [`sink` option]({ ## Kafka +{{site.data.alerts.callout_info}} +{% include_cached new-in.html version="v24.1.4" %} CockroachDB uses a different version of the Kafka sink that is implemented with the [franz-go](https://github.com/twmb/franz-go) Kafka client library. We recommend that you enable this updated version of the Kafka sink to avoid a potential bug in the previous version of the CockroachDB Kafka sink; for more details, refer to the [technical advisory 122372]({% link advisories/a122372.md %}). You can enable this Kafka sink with the cluster setting [`changefeed.new_kafka_sink.enabled`]({% link v24.2/show-cluster-setting.md %}). + +{% include_cached copy-clipboard.html %} +~~~ sql +SET CLUSTER SETTING changefeed.new_kafka_sink.enabled = true; +~~~ +{{site.data.alerts.end}} + ### Kafka sink connection Example of a Kafka sink URI using `SCRAM-SHA-256` authentication: From 6fb6734920c918fde020baaa185dfb6c54fa8ccc Mon Sep 17 00:00:00 2001 From: Kathryn Hancox Date: Thu, 22 Aug 2024 16:39:35 -0400 Subject: [PATCH 2/3] Add note to v24.2 --- src/current/v24.2/changefeed-sinks.md | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/src/current/v24.2/changefeed-sinks.md b/src/current/v24.2/changefeed-sinks.md index 699cf31b605..68af00bc6d5 100644 --- a/src/current/v24.2/changefeed-sinks.md +++ b/src/current/v24.2/changefeed-sinks.md @@ -38,6 +38,17 @@ To set a different sink URI to an existing changefeed, use the [`sink` option]({ ## Kafka +{{site.data.alerts.callout_info}} +CockroachDB uses a different version of the Kafka sink that is implemented with the [franz-go](https://github.com/twmb/franz-go) Kafka client library. If you are using a [testing release]({% link releases/index.md %}#patch-releases) of v24.2 or v24.2.0, we recommend that you enable this updated version of the Kafka sink to avoid a potential bug in the previous version of the CockroachDB Kafka sink; for more details, refer to the [technical advisory 122372]({% link advisories/a122372.md %}). You can enable this Kafka sink with the cluster setting [`changefeed.new_kafka_sink.enabled`]({% link v24.2/show-cluster-setting.md %}). + +{% include_cached copy-clipboard.html %} +~~~ sql +SET CLUSTER SETTING changefeed.new_kafka_sink.enabled = true; +~~~ + +If you are running v24.2.1 and later, the `changefeed.new_kafka_sink.enabled` cluster setting is enabled by default. +{{site.data.alerts.end}} + ### Kafka sink connection Example of a Kafka sink URI using `SCRAM-SHA-256` authentication: From 6f544c83f65358a6a8933d9f3909603c88f991ec Mon Sep 17 00:00:00 2001 From: Rich Loveland Date: Tue, 3 Sep 2024 16:20:07 -0400 Subject: [PATCH 3/3] Update with date of TA publish --- src/current/advisories/a122372.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/src/current/advisories/a122372.md b/src/current/advisories/a122372.md index 08e15c67cd9..4bc217b89c4 100644 --- a/src/current/advisories/a122372.md +++ b/src/current/advisories/a122372.md @@ -4,7 +4,7 @@ advisory: A-122372 summary: Changefeeds could emit events on the same row out of order in some cases. toc: true affected_versions: v23.1, v23.2.0 to v23.2.9, v24.1.0 to v24.1.3, and testing versions of v24.2 through v24.2.0-beta.3 -advisory_date: 2024-09-02 +advisory_date: 2024-09-03 docs_area: releases --- @@ -39,4 +39,4 @@ SET CLUSTER SETTING changefeed.new_kafka_sink.enabled = true; Changefeeds could emit events on the same row out of order in some cases, violating the [changefeed ordering guarantee]({% link v24.2/changefeed-messages.md %}#per-key-ordering). Versions affected include CockroachDB v23.1, v23.2.0 to v23.2.9, v24.1.0 to v24.1.3, and testing versions of v24.2 through v24.2.0-beta.3. -Questions about any technical alert can be directed to our [support team](https://support.cockroachlabs.com/). \ No newline at end of file +Questions about any technical alert can be directed to our [support team](https://support.cockroachlabs.com/).