fs2-kafka v3.0.0

This is the first stable fs2-kafka release to be based on the 3.x series of the Apache Kafka client library. It is available for Scala 2.12, 2.13, and 3.2; all users are encouraged to upgrade. There are some backwards-incompatible API changes, set out below; for further details, see release notes for 3.0.0-RC1 and previous milestone versions.

Development of the 2.x series will now start to wind down, and the 1.x series will be discontinued. New pull requests that make substantial changes should target the 3.x branch.

Apologies to those whose outstanding pull requests weren't merged in time for this release - it was already very overdue. The intention is to start a 4.x branch in the near future to make further backwards-incompatible improvements.

Backwards-incompatible changes

Serializers and Deserializers

Some changes have been made to how fs2-kafka represents Serializers and Deserializers. The following references Serializers but the same changes have been made to Deserializers.

Typed KeySerializer and ValueSerializer

In some cases, such as when using a schema registry with the fs2-kafka-vulcan module, a given serializer will only be valid either for keys or for values, but not both. This is now represented by the KeySerializer type, representing the ability to serialize a key, and the ValueSerializer type, representing the ability to serialize a value. These are supertypes of the plain Serializer type, which can serialize either a key or a value.

Implementation note: KeySerializer, ValueSerializer and Serializer are aliased subtypes of a new GenericSerializer class, which is contravariantly parameterized with a marker trait KeyOrValue with subtypes Key and Value. User code usually won't need to be concerned with these details.

Replacement of RecordSerializer (and RecordDeserializer) with Resource

RecordSerializer served two purposes: to distinguish between key and value serializers when summoning them implicitly, and to wrap effectful instantiation of serializers. The former purpose is now served by the KeySerializer and ValueSerializer types; the latter, by cats-effect's Resource type. Usages of RecordSerializer[F, A] are thus now replaced by Resource[F, KeySerializer[F, A]] or Resource[F, ValueSerializer[F, A]].

Change to Vulcan module's API

The AvroSerializer class no longer has a using method that returns RecordSerializer; instead it has two methods, forKey and forValue, that respectively return KeySerializer and ValueSerializer inside Resource. (Equivalent changes have been made to AvroDeserializer.)

Simplified representation of producer records (#901)

The ProducerRecords, TransactionalProducerRecords and ProducerResult classes have been replaced with type aliases for different kinds of fs2.Chunk, which they previously wrapped. This was enabled by removal of "passthrough" functionality, which can instead be implemented in user code if needed.

What's Changed since v3.0.0-RC1

• Update kafka-clients to 3.4.0 in series/3.x by @scala-steward in #1186
• Update kafka-avro-serializer to 7.3.3 in series/3.x by @scala-steward in #1185

Full Changelog since v2.6.0: v2.6.0...v3.0.0

fs2-kafka v1.11.0 - final 1.x release

This is likely to be the final release of fs2-kafka v1.x, unless a serious security vulnerability emerges; please upgrade to v3.0.0.

What's Changed

• Update sbt-mdoc to 2.3.2 in series/1.x by @scala-steward in #937
• Update vulcan to 1.8.3 in series/1.x by @scala-steward in #952
• Update sbt-mima-plugin to 1.1.0 in series/1.x by @scala-steward in #953
• Update fs2-core to 2.5.11 in series/1.x by @scala-steward in #967
• Update cats-effect, cats-effect-laws to 2.5.5 in series/1.x by @scala-steward in #980
• Pin kafka-clients to 2.x in series/1.x by @bplommer in #1013
• Update kafka-avro-serializer to 6.2.5 in series/1.x by @scala-steward in #1002
• Update sbt-header to 5.7.0 in series/1.x by @scala-steward in #969
• Update scala-library to 2.12.16 in series/1.x by @scala-steward in #1004
• Add support for CooperativeStickyAssignor with KafkaConsumer on 1.x by @nakarinh14 in #1063
• Update kafka-clients to 2.8.2 in series/1.x by @scala-steward in #1060
• Update sbt to 1.7.1 in series/1.x by @scala-steward in #1033
• Update kafka-avro-serializer to 6.2.7 in series/1.x by @scala-steward in #1079
• Update sbt-mdoc to 2.3.6 in series/1.x by @scala-steward in #1083
• Update sbt-ci-release to 1.5.11 in series/1.x by @scala-steward in #1088
• Update scala-library to 2.12.17 in series/1.x by @scala-steward in #1080
• Update sbt-scalafmt to 2.5.0 in series/1.x by @scala-steward in #1106
• Update sbt-header to 5.9.0 in series/1.x by @scala-steward in #1105
• Update scala-library to 2.13.10 in series/1.x by @scala-steward in #1081
• series/1.x: Pin kafka-avro-serializer to 6.x by @bplommer in #1113
• Update sbt-mima-plugin to 1.1.1 in series/1.x by @scala-steward in #1078
• Update sbt to 1.7.3 in series/1.x by @scala-steward in #1090
• 1.x - update headers for 2023 by @bplommer in #1161
• Update kafka-avro-serializer to 6.2.9 in series/1.x by @scala-steward in #1154
• Update sbt-mdoc to 2.3.7 in series/1.x by @scala-steward in #1145
• Update sbt to 1.8.2 in series/1.x by @scala-steward in #1138
• 1.x: pin logback to 1.3.x by @bplommer in #1163
• Update vulcan to 1.8.4 in series/1.x by @scala-steward in #1120
• Update vulcan to 1.9.0 in series/1.x by @scala-steward in #1178
• Update kafka-avro-serializer to 6.2.10 in series/1.x by @scala-steward in #1180
• Update sbt-mima-plugin to 1.1.2 in series/1.x by @scala-steward in #1179

New Contributors

• @nakarinh14 made their first contribution in #1063

Full Changelog: v1.10.0...v1.11.0

fs2-kafka v3.0.0-RC1

This release candidate builds on previous milestone releases of fs2-kafka 3.0. It includes all changes from v2.6.0 and also introduces the breaking change of removing the RecordDeserializer and RecordSerializer types, in favour of simply using Resource for allocating KeySerializer, ValueSerializer etc. It also reintroduces support for Scala 2.12, to encourage all users to migrate quickly to the new version.

What's Changed

Breaking changes

• Allocate Serdes in Resource by @bplommer in #1174
• Replace RecordSerializer/RecordDeserializer with Resource by @bplommer in #1175

Enhancements

• Add partitionFor for given topic with KafkaProducer by @sergio-margale in #1111
• 1099 Fix produce method of TransactionalKafkaProducer, when method is invoked with empty records and non-empty offsets by @chernovsa in #1100
• Change transactional producer to await produce before ending transaction by @bastewart in #1127

Dependency updates

• Update discipline-scalatest to 2.2.0 by @scala-steward in #1030
• Update sbt-typelevel to 0.4.17 in series/3.x by @scala-steward in #1110
• Update sbt-header to 5.9.0 by @scala-steward in #1101
• Update sbt-scalafmt to 2.5.0 by @scala-steward in #1103
• Update sbt-mima-plugin to 1.1.1 in series/3.x by @scala-steward in #1091
• Update scala-library to 2.13.10 by @scala-steward in #1071
• Update scala-library to 2.12.17 by @scala-steward in #1070
• Update sbt-mima-plugin to 1.1.1 by @scala-steward in #1067
• Update sbt-typelevel to 0.4.19 in series/3.x by @scala-steward in #1158
• Update sbt-mdoc to 2.3.7 by @scala-steward in #1148
• Update testcontainers-scala-kafka, ... to 0.40.12 by @scala-steward in #1116
• Update kafka-avro-serializer to 7.3.0 in series/3.x by @scala-steward in #1123
• Update cats-effect, cats-effect-laws, ... to 3.4.8 by @scala-steward in #1153
• Update sbt-mdoc to 2.3.7 in series/3.x by @scala-steward in #1146
• Update scala3-library to 3.2.2 by @scala-steward in #1147
• Update sbt to 1.8.2 by @scala-steward in #1135
• Update fs2-core to 3.6.1 by @scala-steward in #1151
• Update kafka-clients to 3.3.1 in series/3.x by @scala-steward in #1094
• Update vulcan to 1.9.0 by @bplommer in #1168
• Update kafka-clients to 3.3.2 in series/3.x by @scala-steward in #1172
• Update kafka-avro-serializer to 7.3.2 in series/3.x by @scala-steward in #1171
• Update testcontainers-scala-kafka, ... to 0.40.14 by @scala-steward in #1170

Misc

• 3.x - Restore support for Scala 2.12 by @bplommer in #1167
• 3.x - pin some dependencies by @bplommer in #1165
• scala-steward (3.x series): ignore dependencies that should be merged… by @bplommer in #1112
• remove broken link by @EstebanMarin in #1134

New Contributors

• @sergio-margale made their first contribution in #1111
• @chernovsa made their first contribution in #1100
• @EstebanMarin made their first contribution in #1134

Full Changelog: v3.0.0-M9...v3.0.0-RC1

fs2-kafka v2.6.0

This release previously failed to publish but has now been published successfully.

What's Changed

Enhancements

• Change transactional producer to await produce before ending transaction by @bastewart in #1127
• 1099 Fix produce method of TransactionalKafkaProducer, when method is invoked with empty records and non-empty offsets by @chernovsa in #1100
• #1143 - Add transform and option methods on RecordSerializer and RecordDeserializer by @geirolz in #1160
• Add partitionFor for given topic with KafkaProducer by @sergio-margale in #1111

Dependency updates

• Update discipline-scalatest to 2.2.0 by @scala-steward in #1030
• Update sbt-header to 5.9.0 by @scala-steward in #1101
• Update sbt-scalafmt to 2.5.0 by @scala-steward in #1103
• Update scala-library to 2.13.10 by @scala-steward in #1071
• Update scala-library to 2.12.17 by @scala-steward in #1070
• Update sbt-mima-plugin to 1.1.1 by @scala-steward in #1067
• Update sbt-mdoc to 2.3.6 by @scala-steward in #1074
• Update sbt-typelevel to 0.4.19 by @scala-steward in #1157
• Update kafka-avro-serializer to 6.2.9 by @scala-steward in #1152
• Update sbt-mdoc to 2.3.7 by @scala-steward in #1148
• Update cats-effect, cats-effect-laws, ... to 3.4.8 by @scala-steward in #1153
• Update scala3-library to 3.2.2 by @scala-steward in #1147
• Update sbt to 1.8.2 by @scala-steward in #1135
• Update fs2-core to 3.6.1 by @scala-steward in #1151
• Update vulcan to 1.9.0 by @bplommer in #1168
• Update testcontainers-scala-kafka, ... to 0.40.14 by @scala-steward in #1170

Misc

• Refactor future cancellation test to avoid dependence on KakfaFuture internals by @bplommer in #1023
• pin logback to 1.3.x by @bplommer in #1164
• remove broken link by @EstebanMarin in #1134

New Contributors

• @sergio-margale made their first contribution in #1111
• @chernovsa made their first contribution in #1100
• @EstebanMarin made their first contribution in #1134

Full Changelog: v2.5.0...v2.6.0

v3.0.0-M9

This milestone release replaces v3.0.0-M8, which was released from the wrong branch and should not be used.

What's Changed

• Update testcontainers-scala-kafka, ... to 0.40.6 in series/3.x by @scala-steward in #957
• Update kafka-avro-serializer to 7.0.3 in series/3.x by @scala-steward in #959
• Update sbt-header to 5.7.0 in series/3.x by @scala-steward in #958
• Update scala3-library to 3.1.2 in series/3.x by @scala-steward in #961
• Update scala3-library to 3.1.2 by @scala-steward in #949
• Update vulcan to 1.8.3 by @scala-steward in #945
• Update cats-effect-laws, ... to 3.3.11 by @scala-steward in #950
• Update sbt-mima-plugin to 1.1.0 by @scala-steward in #946
• Update sbt-header to 5.7.0 by @scala-steward in #964
• Update kafka-avro-serializer to 7.1.1 in series/3.x by @scala-steward in #974
• Update testcontainers-scala-kafka, ... to 0.40.7 by @scala-steward in #975
• Revert #902 by @bplommer in #976
• Update kafka-avro-serializer to 7.1.2 in series/3.x by @scala-steward in #1006
• Update scala3-library to 3.1.3 by @scala-steward in #999
• Adding sbt-typelevel (second try) by @bplommer in #1012
• Pin kafka-clients to 2.x in series/2.x by @bplommer in #1014
• Update vulcan to 1.8.3 in series/3.x by @scala-steward in #939
• Update testcontainers-scala-kafka, ... to 0.40.8 by @scala-steward in #995
• Update cats-effect-laws, ... to 3.3.13 in series/3.x by @scala-steward in #1008
• Update scala3-library to 3.1.3 in series/3.x by @scala-steward in #1007
• Update cats-effect-laws, ... to 3.3.13 by @scala-steward in #1000
• Ignore cats-effect/fs2 updates in series/3.x, pin kafka-client to 3 by @bplommer in #1015
• Ignore testcontainers updates in 3.x by @bplommer in #1016
• Update kafka-clients to 3.1.1 in series/3.x by @scala-steward in #984
• Update kafka-avro-serializer to 6.2.5 by @scala-steward in #996
• Update sbt-mima-plugin to 1.1.0 in series/3.x by @scala-steward in #940
• Remove ConsumerSettings.unit and ProducerSettings.nothing by @bplommer in #977
• Optimise {Committable,}ProducerRecords apply methods by @bastewart in #978
• Update scala-library to 2.12.16 by @scala-steward in #998
• Fix for #992 A NPE is thrown from AvroDeserializer if the value is empty by @channingwalton in #993
• Add chunk constructors to {Committable,}ProducerRecords by @bastewart in #979
• Add implicit Deserializer attempt by @geirolz in #983
• Fix cancellation behaviour in KafkaAdminClient by @bplommer in #1017
• Remove sbt-ci-release plugin, use sbt-typelevel for latest version by @bplommer in #1018
• Fix release command by @bplommer in #1019
• Update github workflow by @bplommer in #1021
• Removing references to commitBatch and commitBatchOption in Documentation by @ASRagab in #1022
• Refactor future cancellation test to avoid dependence on KakfaFuture internals by @bplommer in #1023
• Merge 2.x into 3.x by @bplommer in #1020

New Contributors

• @bastewart made their first contribution in #978
• @channingwalton made their first contribution in #993
• @geirolz made their first contribution in #983
• @ASRagab made their first contribution in #1022

Full Changelog: v3.0.0-M7...v3.0.0-M9

fs2-kakfa v3.0.0-M8

This is a milestone for the forthcoming v3.0.0, based on the v3.x series of the Java kafka-clients library. It contains the same changes as v2.5.0; for differences from the v2.x series, see release notes for previous milestones.

What's Changed

• Update scala3-library to 3.1.2 by @scala-steward in #949
• Update vulcan to 1.8.3 by @scala-steward in #945
• Update cats-effect-laws, ... to 3.3.11 by @scala-steward in #950
• Update sbt-mima-plugin to 1.1.0 by @scala-steward in #946
• Update sbt-header to 5.7.0 by @scala-steward in #964
• Update testcontainers-scala-kafka, ... to 0.40.7 by @scala-steward in #975
• Revert #902 by @bplommer in #976
• Update scala3-library to 3.1.3 by @scala-steward in #999
• Adding sbt-typelevel (second try) by @bplommer in #1012
• Pin kafka-clients to 2.x in series/2.x by @bplommer in #1014
• Update testcontainers-scala-kafka, ... to 0.40.8 by @scala-steward in #995
• Update cats-effect-laws, ... to 3.3.13 by @scala-steward in #1000
• Update kafka-avro-serializer to 6.2.5 by @scala-steward in #996
• Remove ConsumerSettings.unit and ProducerSettings.nothing by @bplommer in #977
• Optimise {Committable,}ProducerRecords apply methods by @bastewart in #978
• Update scala-library to 2.12.16 by @scala-steward in #998
• Fix for #992 A NPE is thrown from AvroDeserializer if the value is empty by @channingwalton in #993
• Add chunk constructors to {Committable,}ProducerRecords by @bastewart in #979
• Add implicit Deserializer attempt by @geirolz in #983
• Fix cancellation behaviour in KafkaAdminClient by @bplommer in #1017
• Remove sbt-ci-release plugin, use sbt-typelevel for latest version by @bplommer in #1018
• Fix release command by @bplommer in #1019
• Update github workflow by @bplommer in #1021
• Removing references to commitBatch and commitBatchOption in Documentation by @ASRagab in #1022
• Refactor future cancellation test to avoid dependence on KakfaFuture internals by @bplommer in #1023

New Contributors

• @bastewart made their first contribution in #978
• @channingwalton made their first contribution in #993
• @geirolz made their first contribution in #983
• @ASRagab made their first contribution in #1022

Full Changelog: v3.0.0-M7...v3.0.0-M8

v2.5.0

This release is identical to v2.5.0-RC1. It is backwards binary compatible with previous stable releases in the 2.x series, but not with milestones for v2.5.0. It includes several enhancements and bug fixes - for full details see notes for v2.5.0-M2, v2.5.0-M3 and v2.5.0-RC1.

What's Changed

• Move deserialization from KafkaConsumerActor to KafkaConsumer by @bplommer in #902
• withParallelism default 100 -> 10000 in docs by @jbwheatley in #907
• Use IOApp.Simple in docs by @bplommer in #908
• Add produce method to KafkaProducerConnection by @bplommer in #900
• Update cats-effect-laws, ... to 3.3.7 by @scala-steward in #910
• Update logback-classic to 1.2.11 by @scala-steward in #909
• Use semaphore-like permits for some actor operations by @bplommer in #906
• Allow creating settings with placeholder serdes, changing serdes in settings by @bplommer in #911
• Move derived consumer methods to traits by @bplommer in #916
• Update testcontainers version, set image name and version explicitly by @bplommer in #923
• Use permit for fetch, rework permits by @bplommer in #918
• Use a shared test container for each suite by @bplommer in #925
• Update testcontainers-scala-kafka, ... to 0.40.4 by @scala-steward in #928
• Update sbt-mdoc to 2.3.2 by @scala-steward in #929
• Update cats-effect-laws, ... to 3.3.9 by @scala-steward in #930
• Additional consumer functionality exposed by @dmedser in #919
• Fix auto-registration of Avro union (etc) types by @bplommer in #932
• Update sbt-gha, add JDK 17 to pipeline by @bplommer in #927
• Update scala3-library to 3.1.2 by @scala-steward in #949
• Update vulcan to 1.8.3 by @scala-steward in #945
• Update cats-effect-laws, ... to 3.3.11 by @scala-steward in #950
• Update sbt-mima-plugin to 1.1.0 by @scala-steward in #946
• Update sbt-header to 5.7.0 by @scala-steward in #964
• Update testcontainers-scala-kafka, ... to 0.40.7 by @scala-steward in #975
• Revert #902 by @bplommer in #976
• Update scala3-library to 3.1.3 by @scala-steward in #999
• Adding sbt-typelevel (second try) by @bplommer in #1012
• Pin kafka-clients to 2.x in series/2.x by @bplommer in #1014
• Update testcontainers-scala-kafka, ... to 0.40.8 by @scala-steward in #995
• Update cats-effect-laws, ... to 3.3.13 by @scala-steward in #1000
• Update kafka-avro-serializer to 6.2.5 by @scala-steward in #996
• Remove ConsumerSettings.unit and ProducerSettings.nothing by @bplommer in #977
• Optimise {Committable,}ProducerRecords apply methods by @bastewart in #978
• Update scala-library to 2.12.16 by @scala-steward in #998
• Fix for #992 A NPE is thrown from AvroDeserializer if the value is empty by @channingwalton in #993
• Add chunk constructors to {Committable,}ProducerRecords by @bastewart in #979
• Add implicit Deserializer attempt by @geirolz in #983
• Fix cancellation behaviour in KafkaAdminClient by @bplommer in #1017
• Remove sbt-ci-release plugin, use sbt-typelevel for latest version by @bplommer in #1018
• Fix release command by @bplommer in #1019
• Update github workflow by @bplommer in #1021
• Removing references to commitBatch and commitBatchOption in Documentation by @ASRagab in #1022

New Contributors

• @jbwheatley made their first contribution in #907
• @dmedser made their first contribution in #919
• @bastewart made their first contribution in #978
• @channingwalton made their first contribution in #993
• @geirolz made their first contribution in #983
• @ASRagab made their first contribution in #1022

Full Changelog: v2.4.0...v2.5.0

fs2-kafka v2.5.0-RC1

This release contains several enhancements and bug fixes. It also reverts two changes from v2.5.0-M2: see below.

What's Changed

Enhancements

• Optimise {Committable,}ProducerRecords apply methods by @bastewart in #978
• Add chunk constructors to {Committable,}ProducerRecords by @bastewart in #979
• Add implicit Deserializer attempt by @geirolz in #983

Bug fixes

• Fix for #992 A NPE is thrown from AvroDeserializer if the value is empty by @channingwalton in #993
• Fix cancellation behaviour in KafkaAdminClient by @bplommer in #1017

Reverted changes

• Remove ConsumerSettings.unit and ProducerSettings.nothing by @bplommer in #977
  • These were added in #911 to serve as placeholders to use with the newly added withSerializers/withDeserializers method, but the same purpose can be served by using Array[Byte] for key and value types.
• Revert #902 by @bplommer in #976
  • Restores previous behaviour whereby deserialization happens inside KafkaConsumerActor - see #962 for rationale.

Updates

• Update scala3-library to 3.1.2 by @scala-steward in #949
• Update vulcan to 1.8.3 by @scala-steward in #945
• Update cats-effect-laws, ... to 3.3.11 by @scala-steward in #950
• Update sbt-mima-plugin to 1.1.0 by @scala-steward in #946
• Update sbt-header to 5.7.0 by @scala-steward in #964
• Update testcontainers-scala-kafka, ... to 0.40.7 by @scala-steward in #975
• Update scala3-library to 3.1.3 by @scala-steward in #999
• Update testcontainers-scala-kafka, ... to 0.40.8 by @scala-steward in #995
• Update cats-effect-laws, ... to 3.3.13 by @scala-steward in #1000
• Update kafka-avro-serializer to 6.2.5 by @scala-steward in #996
• Update scala-library to 2.12.16 by @scala-steward in #998

CI

• Adding sbt-typelevel (second try) by @bplommer in #1012
• Pin kafka-clients to 2.x in series/2.x by @bplommer in #1014
• Remove sbt-ci-release plugin, use sbt-typelevel for latest version by @bplommer in #1018
• Fix release command by @bplommer in #1019
• Update github workflow by @bplommer in #1021

New Contributors

• @bastewart made their first contribution in #978
• @channingwalton made their first contribution in #993
• @geirolz made their first contribution in #983

Full Changelog: v2.5.0-M3...v2.5.0-RC1

fs2-kafka v3.0.0-M7

This milestone release for fs2-kafka 3.x contains the same improvements as v2.5.0-M3.

fs2-kafka v2.5.0-M3

This milestone release exposes some additional consumer functionality and fixes a long-standing bug/shortcoming in the Vulcan integration module, inherited from the Confluent Kafka Avro serializer, whereby auto-registration of Union and some other schemas would not work as expected.

What's Changed

Enhancements

• Additional consumer functionality exposed by @dmedser in #919
• Fix auto-registration of Avro union (etc) types by @bplommer in #932

Internals and updates

• Move derived consumer methods to traits by @bplommer in #916
• Update testcontainers version, set image name and version explicitly by @bplommer in #923
• Use permit for fetch, rework permits by @bplommer in #918
• Use a shared test container for each suite by @bplommer in #925
• Update testcontainers-scala-kafka, ... to 0.40.4 by @scala-steward in #928
• Update sbt-mdoc to 2.3.2 by @scala-steward in #929
• Update cats-effect-laws, ... to 3.3.9 by @scala-steward in #930
• Update sbt-gha, add JDK 17 to pipeline by @bplommer in #927

New Contributors

• @dmedser made their first contribution in #919

Full Changelog: 2.5.0-M1...v2.5.0-M3