fs2-kafka v2.5.0-M2

This release is identical to v2.5.0-M1, which failed to publish.

This release adds some new features and significantly simplifies the internals of the consumer. Although we don't expect these changes to cause any problems, we're releasing this as a milestone to allow for some extra battle testing before the next stable release.

Note: there is a small change to the behaviour of consumers when using partitionedStream or partitionsMapStream. Previously, failure to deserialize a record would cause the entire consumer to fail; now, it will only cause a single partition stream to fail.

What's Changed

New features

• Add produce method to KafkaProducerConnection by @bplommer in #900
  • KafkaProducerConnection allocates a producer without being tied to particular serializers, allowing creation of multiple KafkaProducer instances that share a single set of connections to the broker. With this change, it is possible to produce records directly from KafkaProducerConnection by providing serializers when calling produce.
• Allow creating settings with placeholder serdes, changing serdes in settings by @bplommer in #911
  • This allows creation of ProducerSettings and ConsumerSettings to be decoupled from provision of serializers and deserializers, which is particularly useful when creating multiple producers or consumers with different types.

Docs improvements

• withParallelism default 100 -> 10000 in docs by @jbwheatley in #907
• Use IOApp.Simple in docs by @bplommer in #908

Simplification of internals

• Move deserialization from KafkaConsumerActor to KafkaConsumer by @bplommer in #902
• Use semaphore-like permits for some actor operations by @bplommer in #906

Dependency updates

• Update cats-effect-laws, ... to 3.3.7 by @scala-steward in #910
• Update logback-classic to 1.2.11 by @scala-steward in #909

New Contributors

• @jbwheatley made their first contribution in #907

Full Changelog: v2.4.0...2.5.0-M1

fs2-kafka v2.5.0-M1

This release adds some new features and significantly simplifies the internals of the consumer. Although we don't expect these changes to cause any problems, we're releasing this as a milestone to allow for some extra battle testing before the next stable release.

Note: there is a small change to the behaviour of consumers when using partitionedStream or partitionsMapStream. Previously, failure to deserialize a record would cause the entire consumer to fail; now, it will only cause a single partition stream to fail.

What's Changed

New features

• Add produce method to KafkaProducerConnection by @bplommer in #900
  • KafkaProducerConnection allocates a producer without being tied to particular serializers, allowing creation of multiple KafkaProducer instances that share a single set of connections to the broker. With this change, it is possible to produce records directly from KafkaProducerConnection by providing serializers when calling produce.
• Allow creating settings with placeholder serdes, changing serdes in settings by @bplommer in #911
  • This allows creation of ProducerSettings and ConsumerSettings to be decoupled from provision of serializers and deserializers, which is particularly useful when creating multiple producers or consumers with different types.

Docs improvements

• withParallelism default 100 -> 10000 in docs by @jbwheatley in #907
• Use IOApp.Simple in docs by @bplommer in #908

Simplification of internals

• Move deserialization from KafkaConsumerActor to KafkaConsumer by @bplommer in #902
• Use semaphore-like permits for some actor operations by @bplommer in #906

Dependency updates

• Update cats-effect-laws, ... to 3.3.7 by @scala-steward in #910
• Update logback-classic to 1.2.11 by @scala-steward in #909

New Contributors

• @jbwheatley made their first contribution in #907

Full Changelog: v2.4.0...2.5.0-M1

fs2-kafka v3.0.0-M5

This is the latest milestone release for the fs2-kafka v3.x series, which uses Kafka 3. It updates the kafka-clients library to 3.1 and includes the same changes as v2.4.0.

fs2-kafka v2.4.0

This release adds several new features to the fs2-kafka 2.x line, which uses cats-effect 3 and Kafka 2. It also fixes a concurrency-related bug in TransactionalKafkaProducer.

What's Changed

Enhancements

• Add support for using CooperativeStickyAssignor with KafkaConsumer by @biochimia in #844
• Allow transaction without upstream offsets by @janstenpickle in #883
• Add attempt method to RecordDeserializer by @bplommer in #891

Bugfixes

• Threadsafe transactions by @janstenpickle in #892

Doc improvements

• Add a basic version matrix to the readme by @bplommer in #841
• Slightly improve docs for stream methods by @LMnet in #872
• Publish certs docs by @bplommer in #881
• Update transactional producer docs by @janstenpickle in #890

Dependency updates

• Update cats-effect, cats-effect-laws, ... to 3.3.4 by @scala-steward in #835
• Update scala-library to 2.13.8 by @scala-steward in #838
• Update testcontainers-scala-kafka, ... to 0.40.0 in series/1.x by @scala-steward in #846
• Update fs2-core to 3.2.5 by @scala-steward in #868
• Update sbt-buildinfo to 0.11.0 by @scala-steward in #865
• Update sbt-header to 5.6.5 by @scala-steward in #862
• Update sbt to 1.6.2 by @scala-steward in #854
• Update scala3-library to 3.1.1 by @scala-steward in #857
• Update cats-effect-laws, ... to 3.3.6 by @scala-steward in #882
• Update vulcan version to 1.8.0 by @bplommer in #896

Housekeeping

• Update copyright notices by @bplommer in #833

New Contributors

• @biochimia made their first contribution in #844

Full Changelog: v2.3.0...v2.4.0

v1.10.0

This is a maintenance release for the 1.x series, which is based on cats-effect 2 and is no longer receiving new features. It fixes a concurrency-related bug in TransactionalKafkaProducer (see #888) and updates various dependencies.

What's Changed

• Update scala-library to 2.13.8 in series/1.x by @scala-steward in #839
• Update testcontainers-scala-kafka, ... to 0.40.0 in series/1.x by @scala-steward in #846
• Update sbt-buildinfo to 0.11.0 in series/1.x by @scala-steward in #866
• Update sbt-header to 5.6.5 in series/1.x by @scala-steward in #863
• Update sbt to 1.6.2 in series/1.x by @scala-steward in #855
• Improve scala-steward config for 1.x series by @LMnet in #871
• Publish certs docs by @bplommer in #881
• Update testcontainers-scala-kafka, ... to 0.40.1 in series/1.x by @scala-steward in #860
• Update logback-classic to 1.2.11 in series/1.x by @scala-steward in #884
• Update sbt-mima-plugin to 1.0.1 in series/1.x by @scala-steward in #711
• Threadsafe transactions by @janstenpickle in #888
• Update vulcan version to 1.8.0 in series/1.x by @bplommer in #895
• Don't publish docs when releasing 1.x by @bplommer in #893

Full Changelog: v1.9.0...v1.10.0

fs2-kafka v3.0.0-M4

This is the first milestone release of fs2-kafka following the release of 2.0.0. (We have previously released v3.0.0-M1 through v3.0.0-M3 but these became the 2.x series following a change in our roadmap.)

This is the first release of fs2-kafka to use version 3.0.0 of the Apache Kafka client library, which is not binary-compatible with previous versions. It is available for Scala 2.13 and Scala 3.1; Scala 2.12 has been dropped for the 3.x series but will continue to be supported in the 1.x and 2.x series.

fs2-kafka v2.3.0

This release brings new functionality and dependency updates to the fs2-kafka 2.x series, which is based on cats-effect 2. These include an update to kafka-clients that resolves a security vulnerability (see #821)

What's Changed

Functionality

• Add deleteConsumerGroups method by @keirlawson in #730
• Add avro settings register schema method by @agustafson in #687
• Add producer convenience methods by @keirlawson in #632

Updates

• Update sbt-mima-plugin to 1.0.1 by @scala-steward in #698
• Update kind-projector to 0.13.2 by @scala-steward in #682
• Update kafka-avro-serializer to 6.2.1 by @scala-steward in #695
• Update munit to 0.7.29 by @scala-steward in #679
• Update kafka-clients to 2.8.1 by @scala-steward in #690
• Update sbt-mdoc to 2.2.23 by @scala-steward in #675
• Update scala-library to 2.12.15 by @scala-steward in #688
• Update sbt-ci-release to 1.5.10 by @scala-steward in #700
• Update fs2-core to 3.1.6 by @scala-steward in #704
• Update logback-classic to 1.2.10 by @scala-steward in #804
• Update sbt to 1.6.1 by @scala-steward in #814
• Update scala-library to 2.13.7 by @scala-steward in #746
• Update sbt-unidoc to 0.5.0 by @scala-steward in #736
• Update sbt-scalafmt to 2.4.6 by @scala-steward in #807
• Update kafka-avro-serializer to 6.2.2 by @scala-steward in #781
• Update testcontainers-scala-kafka, ... to 0.39.12 by @scala-steward in #759
• Update scala3-library to 3.1.0 by @scala-steward in #705
• Update cats-effect, cats-effect-laws, ... to 3.3.3 by @scala-steward in #819
• Update fs2-core to 3.2.4 by @scala-steward in #810
• Update sbt-mdoc to 2.2.24 by @scala-steward in #727

Other

• Update code to use modern cats import by @Daenyth in #765

New Contributors

• @keirlawson made their first contribution in #632

Full Changelog: v2.2.0...v2.3.0

fs2-kafka v1.9.0

This is a maintenance release for the fs2-kafka 1.x series, which is based on cats-effect 2. It includes various dependency updates, including an update to kafka-clients that resolves a security vulnerability (see #821), and adds docs for the munit module. There are no other user-facing changes.

An update to the fs2-kafka 2.x series, which is based on cats-effect 3, will be released soon.

What's Changed

Docs

• Add docs for munit module by @keirlawson in #672

Dependency updates

• Update munit to 0.7.29 in series/1.x by @scala-steward in #716
• Update sbt-ci-release to 1.5.9 in series/1.x by @scala-steward in #710
• Update sbt-mdoc to 2.2.24 in series/1.x by @scala-steward in #728
• Update logback-classic to 1.2.10 in series/1.x by @scala-steward in #805
• Update sbt to 1.6.1 in series/1.x by @scala-steward in #815
• Update kind-projector to 0.13.2 in series/1.x by @scala-steward in #720
• Update kafka-avro-serializer to 6.2.2 in series/1.x by @scala-steward in #782
• Update sbt-scalafmt to 2.4.6 in series/1.x by @scala-steward in #808
• Update testcontainers-scala-kafka, ... to 0.39.12 in series/1.x by @scala-steward in #760
• Update scala-library to 2.12.15 in series/1.x by @scala-steward in #714
• Update kafka-clients to 2.8.1 in series/1.x by @scala-steward in #713
• Update sbt-unidoc to 0.5.0 in series/1.x by @scala-steward in #737
• Update cats-effect, cats-effect-laws to 2.5.4 in series/1.x by @scala-steward in #719
• Update scala-library to 2.13.7 in series/1.x by @scala-steward in #748
• Update fs2-core to 2.5.10 in series/1.x by @scala-steward in #707

Housekeeping

• Update copyright notices by @bplommer in #833

Full Changelog: v1.8.0...v1.9.0

fs2-kafka v2.2.0

This is the third release in the fs2-kafka 2.x series, based on cats-effect 3. If focuses on usability, bringing the same improvements as v1.8.0. It also includes various dependency updates.

fs2-kafka v1.8.0

This release focuses on improving usability, with new helper methods for working with Kafka producers and consumers. This release is for users stuck on cats-effect 2. For cats-effect 3 users, v2.2.0 will be coming shortly.

New KafkaProducer syntax (#632 @keirlawson)

We've added several new helper methods for KafkaProducer to simplify common use cases.

Producing one record and getting the resulting record metadata - before:

producer.produce(ProducerRecords.one(ProducerRecord(topic, key, value))).map(_.map(_.records.head.get._2))

After:

producer.produceOne_(topic, key, value)

New KafkaConsumer syntax (#662 #664 @bplommer )

We've also added new extension methods for KafkaConsumer in the Stream context (no new imports needed!), so the most common operations no longer require explicit use of flatMap and evalTap. The stream and partitionedStream methods on KafkaConsumer can now be invoked as records and partitionedRecords respectively, for greater clarity.

Allocating a consumer, subscribing it to a topic, and streaming records - before:

KafkaConsumer.stream(consumerSettings)
  .evalTap(_.subscribeTo("topic"))
  .flatMap(_.stream)

After:

KafkaConsumer.stream(consumerSettings)
  .subscribeTo("topic")
  .records

Vulcan testkit (#629 @keirlawson)

A new module, fs2-kafka-vulcan-testkit-munit, makes it easier to test the schema compatibility of Vulcan codecs against a live schema registry instance.

Other changes

Various dependency updates