Help Needed with Tiered Storage Setup on Strimzi 0.43.0 using Azure Blob and Custom Kafka Image

[Ashok13193]

Hi Team,

I am currently setting up tiered storage with Azure Blob using Strimzi 0.43.0 and a custom-built Kafka image. The goal is to enable tiered storage in my Kafka cluster with Azure Blob as the storage backend. For local testing, I am using Azurite as a mock Azure Blob storage.

I’ve built a custom Kafka image that includes a JAR file from the following repository:

linkedin/kafka-remote-storage-azure

Here’s the Dockerfile I used:

Dockerfile:

FROM quay.io/strimzi/kafka:0.43.0-kafka-3.7.0
COPY ./kafka-azure-rsm/build/libs/kafka-azure-rsm-0.1.0-SNAPSHOT-shadow.jar /opt/kafka/plugins/
ENV CLASSPATH=/opt/kafka/plugins/kafka-azure-rsm-0.1.0-SNAPSHOT-shadow.jar

However, I’m encountering an error when starting the Kafka pod, as shown in the logs:

`2024-09-18 09:01:57,871 ERROR Exiting Kafka due to fatal exception during startup. (kafka.Kafka$) [main]
org.apache.kafka.common.config.ConfigException: Invalid value PLAIN for configuration remote.log.metadata.manager.listener.name: Should be set as a listener name within valid broker listener name list: ListenerName(CONTROLPLANE-9090),ListenerName(REPLICATION-9091),ListenerName(PLAIN-9092)
	at kafka.server.KafkaServer.$anonfun$startup$24(KafkaServer.scala:551)
	at scala.Option.orElse(Option.scala:477)
	at kafka.server.KafkaServer.$anonfun$startup$22(KafkaServer.scala:549)
	at kafka.server.KafkaServer.$anonfun$startup$22$adapted(KafkaServer.scala:544)
	at scala.Option.foreach(Option.scala:437)
	at kafka.server.KafkaServer.startup(KafkaServer.scala:544)
	at kafka.Kafka$.main(Kafka.scala:112)
	at kafka.Kafka.main(Kafka.scala)
`

It seems related to the listener name configuration for remote.log.metadata.manager.listener.name. I’ve ensured that the broker listeners (CONTROLPLANE, REPLICATION, PLAIN) are configured, but the error persists.

Questions:

1. Has anyone encountered this configuration issue when setting up tiered storage with Azure Blob?
2. Any advice on correctly configuring the remote.log.metadata.manager.listener.name property?
3. Could this be related to Azurite, or do I need additional steps when using Strimzi for tiered storage?

Due to file size constraints, I’m unable to attach the JAR file directly, but I’ve uploaded the configuration and relevant files here.

Any insights or suggestions would be greatly appreciated!

Thanks in advance!