Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add TA 122372 #18837

Merged
merged 6 commits into from
Sep 3, 2024
Merged

Add TA 122372 #18837

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
849683a
Add TA 122372
kathancox Aug 19, 2024
6fb6734
Add note to v24.2
kathancox Aug 22, 2024
ad5f3d0
Merge branch 'main' into ta-122372
rmloveland Sep 3, 2024
2136a13
Merge branch 'main' into ta-122372
rmloveland Sep 3, 2024
ace7708
Merge branch 'main' into ta-122372
rmloveland Sep 3, 2024
6f544c8
Update with date of TA publish
rmloveland Sep 3, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
42 changes: 42 additions & 0 deletions src/current/advisories/a122372.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,42 @@
---
title: Technical Advisory 122372
advisory: A-122372
summary: Changefeeds could emit events on the same row out of order in some cases.
toc: true
affected_versions: v23.1, v23.2.0 to v23.2.9, v24.1.0 to v24.1.3, and testing versions of v24.2 through v24.2.0-beta.3
advisory_date: 2024-09-03
docs_area: releases
---

Publication date: {{ page.advisory_date | date: "%B %e, %Y" }}

## Description

In all versions of CockroachDB v23.1, v23.2.0 to v23.2.9, v24.1.0 to v24.1.3, and testing versions of v24.2 through v24.2.0-beta.3, [changefeeds]({% link v24.2/change-data-capture-overview.md %}) could emit events on the same row out of order in some cases, which violates [changefeed ordering guarantees]({% link v24.2/changefeed-messages.md %}#per-key-ordering). This issue was caused by a [bug in the sarama Kafka client library](https://github.com/IBM/sarama/issues/2619), a third-party library that CockroachDB uses to talk to Kafka clusters. The bug manifested when a workload had mutations to the same key in rapid succession, because it was possible for [sarama](https://github.com/IBM/sarama) to re-order the messages if it encountered a retryable Kafka error.

To resolve this issue, CockroachDB now uses a new version of the Kafka sink that uses [franz-go](https://github.com/twmb/franz-go), a different third-party Kafka client library that does not have this bug. The new Kafka sink can be enabled using the cluster setting [`changefeed.new_kafka_sink.enabled`]({% link v24.2/show-cluster-setting.md %}). The new Kafka sink was added in PR [#126213](https://github.com/cockroachdb/cockroach/pull/126213).

## Statement

This is resolved in CockroachDB by PR [#126213](https://github.com/cockroachdb/cockroach/pull/126213), which replaces the Kafka client library [sarama](https://github.com/IBM/sarama) with [franz-go](https://github.com/twmb/franz-go).

The fix has been applied to maintenance releases of CockroachDB v23.2.10, v24.1.4, and v24.2.0-rc.1.

This public issue is tracked by issue [#122372](https://github.com/cockroachdb/cockroach/issues/122372).

## Mitigation

Users of CockroachDB v23.1, v23.2.0 to v23.2.9, v24.1.0 to v24.1.3, and testing versions of v24.2 through v24.2.0-beta.3 are encouraged to upgrade to [v23.2.10]({% link releases/v23.2.md %}#v23-2-10), [v24.1.4]({% link releases/v24.1.md %}#v24-1-4), [v24.2.0]({% link releases/v24.2.md %}#v24-2-0), or a later version, and enable the cluster setting [`changefeed.new_kafka_sink.enabled`]({% link v24.2/show-cluster-setting.md %}):

{% include_cached copy-clipboard.html %}
~~~ sql
SET CLUSTER SETTING changefeed.new_kafka_sink.enabled = true;
~~~

`changefeed.new_kafka_sink.enabled` will be enabled by default in versions v24.2.1, v24.3, and all later versions.

## Impact

Changefeeds could emit events on the same row out of order in some cases, violating the [changefeed ordering guarantee]({% link v24.2/changefeed-messages.md %}#per-key-ordering). Versions affected include CockroachDB v23.1, v23.2.0 to v23.2.9, v24.1.0 to v24.1.3, and testing versions of v24.2 through v24.2.0-beta.3.

Questions about any technical alert can be directed to our [support team](https://support.cockroachlabs.com/).
9 changes: 9 additions & 0 deletions src/current/v23.2/changefeed-sinks.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,15 @@ To set a different sink URI to an existing changefeed, use the [`sink` option]({

## Kafka

{{site.data.alerts.callout_info}}
{% include_cached new-in.html version="v23.2.10" %} CockroachDB uses a different version of the Kafka sink that is implemented with the [franz-go](https://github.com/twmb/franz-go) Kafka client library. We recommend that you enable this updated version of the Kafka sink to avoid a potential bug in the previous version of the CockroachDB Kafka sink; for more details, refer to the [technical advisory 122372]({% link advisories/a122372.md %}). You can enable this Kafka sink with the cluster setting [`changefeed.new_kafka_sink.enabled`]({% link v24.2/show-cluster-setting.md %}).

{% include_cached copy-clipboard.html %}
~~~ sql
SET CLUSTER SETTING changefeed.new_kafka_sink.enabled = true;
~~~
{{site.data.alerts.end}}

### Kafka sink connection

Example of a Kafka sink URI using `SCRAM-SHA-256` authentication:
Expand Down
9 changes: 9 additions & 0 deletions src/current/v24.1/changefeed-sinks.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -44,6 +44,15 @@ To set a different sink URI to an existing changefeed, use the [`sink` option]({

## Kafka

{{site.data.alerts.callout_info}}
{% include_cached new-in.html version="v24.1.4" %} CockroachDB uses a different version of the Kafka sink that is implemented with the [franz-go](https://github.com/twmb/franz-go) Kafka client library. We recommend that you enable this updated version of the Kafka sink to avoid a potential bug in the previous version of the CockroachDB Kafka sink; for more details, refer to the [technical advisory 122372]({% link advisories/a122372.md %}). You can enable this Kafka sink with the cluster setting [`changefeed.new_kafka_sink.enabled`]({% link v24.2/show-cluster-setting.md %}).

{% include_cached copy-clipboard.html %}
~~~ sql
SET CLUSTER SETTING changefeed.new_kafka_sink.enabled = true;
~~~
{{site.data.alerts.end}}

### Kafka sink connection

Example of a Kafka sink URI using `SCRAM-SHA-256` authentication:
Expand Down
11 changes: 11 additions & 0 deletions src/current/v24.2/changefeed-sinks.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,17 @@ To set a different sink URI to an existing changefeed, use the [`sink` option]({

## Kafka

{{site.data.alerts.callout_info}}
CockroachDB uses a different version of the Kafka sink that is implemented with the [franz-go](https://github.com/twmb/franz-go) Kafka client library. If you are using a [testing release]({% link releases/index.md %}#patch-releases) of v24.2 or v24.2.0, we recommend that you enable this updated version of the Kafka sink to avoid a potential bug in the previous version of the CockroachDB Kafka sink; for more details, refer to the [technical advisory 122372]({% link advisories/a122372.md %}). You can enable this Kafka sink with the cluster setting [`changefeed.new_kafka_sink.enabled`]({% link v24.2/show-cluster-setting.md %}).

{% include_cached copy-clipboard.html %}
~~~ sql
SET CLUSTER SETTING changefeed.new_kafka_sink.enabled = true;
~~~

If you are running v24.2.1 and later, the `changefeed.new_kafka_sink.enabled` cluster setting is enabled by default.
{{site.data.alerts.end}}

### Kafka sink connection

Example of a Kafka sink URI using `SCRAM-SHA-256` authentication:
Expand Down
Loading