Strimzi
Transitioning to Strimzi #10601
-
|
Hi there, first-time here and hoping to get some guidance! At my company, we're currently running Kafka on EKS with separate clusters for both Zookeeper and Kafka brokers. Each of the clusters runs as a StatefulSet with each pod and it's own PV. We're exploring the transition to using the Strimzi Kafka Operator, but most of the resources I've found focus on creating a new Kafka cluster from scratch or assume Strimzi is already in place. Since we have an existing setup with significant production traffic, I'm trying to understand the best way to migrate our Kafka brokers to Strimzi without causing any downtime. Any advice or resources for handling this kind of transition would be greatly appreciated! Each of the clusters runs as a StatefulSet with each pod and it's own PV. |
Beta Was this translation helpful? Give feedback.
Replies: 1 comment 1 reply
-
|
If minimal downtime is important for you, then I think the only option is to set up a new Strimzi-based cluster and use mirroring to mirror the data from the old cluster to the new one. And once they are mirrored, you can switch the consumers / producers to the new cluster as well. It is not completely simple and quick. And possibly not without any outage, but the outage should be small -> the mirroring is async, so you at some point need to stop the consumer / producer, wait for Mirror Maker to finish the last part of the mirroring and than start them again with the new cluster configured in them. You might be able to reuse the data from the PVCs. Whether that would work and how to do it would depend on how your current clusters looks like and is configured. But in any case it would require you to stop the whole cluster, transfer the data to new PVs/PVCs or prepare the PVs/PVCs to be used by Strimzi. This would in general be faster than using mirroring. But you might have the cluster completely down for when the transfer happens. What is definitely not possible is for Strimzi to somehow take over your existing cluster and just start operating it. That is not how it works or not what we support I'm afraid. |
Beta Was this translation helpful? Give feedback.
-
|
Thank you for the quick response @scholzj , we'll dive in to the Mirroring solution and try to see if we can use it 🙏🏻 |
Beta Was this translation helpful? Give feedback.
If minimal downtime is important for you, then I think the only option is to set up a new Strimzi-based cluster and use mirroring to mirror the data from the old cluster to the new one. And once they are mirrored, you can switch the consumers / producers to the new cluster as well.
It is not completely simple and quick. And possibly not without any outage, but the outage should be small -> the mirroring is async, so you at some point need to stop the consumer / producer, wait for Mirror Maker to finish the last part of the mirroring and than start them again with the new cluster configured in them.
You might be able to reuse the data from the PVCs. Whether that would work and how to do it…