How would the console integrate with a Kafka proxy like Kroxylicious

[katheris]

It is possible that the Apache Kafka cluster console is facilitating interactions with has a proxy deployed that sits between Kafka clients and Kafka. We should think about what features the console might provide for that kind of setup.

One such proxy we could likely consider is Kroxylicious since it is also an open source project.

Some ideas so far from chatting to people involved in that project:

• Configuring the proxy, e.g. enabling/disabling filters
• Configuring specific filters, e.g. determining which topics should have encryption enabled
• Showing "virtual clusters" to the user

[tombentley]

Currently Kroxylicious only really has its configuration file as an API for changing the behaviour of a proxy instance. That imposes a lot of restrictions that I'm not sure should end up in the console, because they're very limiting in terms of the use cases that can be addressed by the combined system.

I would like to ensure we can support a operational model that supports a separation of concerns between:

• Kafka infrastructure owners (who we might assume are also the people responsible for running a reverse proxy)
• Application developers (who might own topics)
• Data owners (who might be responsible for the data in some of the topics, especially where that data is shared between parts of a business or between businesses)
• Data scientists and engineers (who need access to the data, but aren't themselves responsible for any Kafka applications)
• Compliance people (who might be responsible for the security of the data within topics).

Depending on the use cases for deploying a proxy those different personas might each need to have control over the configuration of different filters. For example, in an org which has an internal central KaaS it could be that a multitenancy filter is used, and owned by the Kafka infra team. Each tenant might be a different siloed line-of business application team. Some of them might want to be using an encryption filter on some topics, but not all, according to a security policy owned by another team, and so on.

In other words the proxy needs a more clearly-defined control plane that the console can be a client of. Some properties which we'd need that control plane to have include:

• Addressability of individual filters (e.g. by the configuration given each filter a unique id)
• Dynamic reconfiguration of individual filters
• Authentication and authorization for requests to change filter configuration

Building this on top of the Kubernetes API (i.e. a "Kubernetes' API is the control plane" model) is one possibility. But I'm not sure we should fixate on that immediately. That's because in many organisations the Kube clusters (and specifically the resources they contain) are locked down within an operations team. I think the console seeks to broaden access beyond that kind of "operations" scope, into a company more broadly. So we'd need to decide if the Kube API is actually a good fit, or whether we need something else.

[MikeEdgar]

Building this on top of the Kubernetes API (i.e. a "Kubernetes' API is the control plane" model) is one possibility. But I'm not sure we should fixate on that immediately. That's because in many organisations the Kube clusters (and specifically the resources they contain) are locked down within an operations team. I think the console seeks to broaden access beyond that kind of "operations" scope, into a company more broadly. So we'd need to decide if the Kube API is actually a good fit, or whether we need something else.

What would you say is the main difference between the Kroxylicious and Strimzi's Kafka* resources as it relates to using the Kubernetes API (aside from one pre-existing)? Since we already have it in our roadmap/future plan to support modifications to several of the Strimzi CRs, I'm curious on your thinking about how having Kroxy do the same may interfere with broadening access beyond operations.

[k-wall]

I think it is worth making a distinction between presenting a cluster and managing Kroxylicious. One of the aims of Kroxylious is to present a cluster to the Kafka Client that its indistinguishable from that a real kafka cluster. If the Console makes no assumptions about the nature of the Kafka Cluster (and is prepared to degrade gracefully if some optional ops fail) then it should be able to present a Kroyxlicious virtual kafka cluster as it would a 'real' kafka cluster. If that's true, then I would expect the Console to also function with Kafka Clusters from other providers such as Confluent Kafka, Redpanda etc.