Dataplane migration for Apache Kafka communications: From Vert.x to Project Loom

[matzew]

Problem

The Knative Broker's data-plane communication with Apache Kafka for consuming and producing records is currently done via the Vertx Kafka client library (see [1]). The libary is basically a wrapper for communications with Apache Kafka inside of the vertx threading model.

The project requires an evaluation the OpenJDK 19 "Project Loom" ([2], [3]) itself and how to leverage it for efficient communications with an Apache Kafka cluster.

The goal of the project is to migrate the usage of vert.x for any communications with Apache Kafka to pure OpenJDK 19's Loom, and reduce the number of 3rd party frameworks, such as vert.x for Apache Kafka communication.

Exit Criteria

A Knative Kafka Broker data plane implementation working on Java 19, leveraging the Project Loom APIs while communicating with Apache Kafka. No usage of the vertx-kafka-client, since all Kafka communications are done via Loom.

Additional context (optional)

[1] https://github.com/vert-x3/vertx-kafka-client
[2] https://wiki.openjdk.org/display/loom
[3] https://openjdk.org/projects/jdk/19/

[ppatierno]

I am not sure what do you mean by all Kafka communications are done via Loom, how do you envisage the usage of virtual threads as a replacement for the Vert.x Kafka client? I mean, isn't it something you can already do without Loom.

This is something we have already done in our Strimzi HTTP bridge, moving to the usage of CompletableFuture(s) to run the async Kafka communication.
Following a couple of PRs where it was made replacing the producer (called source endpoint) and consumer (called sink endpoint):

strimzi/strimzi-kafka-bridge#724
strimzi/strimzi-kafka-bridge#734

[debasishbsws]

/assign