Kafka running in two data centers

[Casper-dss]

Hi

I don't know if is the right place to ask this. But here I go. The company i am working for, is planning to have two data centers (unfortunately not 3). So the solution for running Kafka is creating a stretched cluster, with 6 brokers and 5 zookeeper nodes (3 in primary, and 2 in secondary). If the primary cluster goes down, would we be able to make the secondary cluster run, by increasing the number of zookeeper nodes, in the secondary cluster by deploying an update to the Kafka resource, with a new amount of zookeeper replica's? And then how many, if it is set to 5, but 3 is missing?

Casper

[scholzj]

If you have some nodes insync in the secondary DC, you should be able to recover eventually by deleting the old nodes and creating the new nodes in the secondary DC and having them resync from the existing nodes in the secondary DC. However:

• You need to have the resources for it in the secondary DC
• This would likely mean some kind of outage
• It is not trivial to make sure you have an in-sync replica in the secondary DC

For example, to make sure you have a in-sync replica in both DCs on the Kafka side, you have to use something like replication factor 4 and min.insync.replicas=3 for every topic and have 2 replicas in each DC. But that means you will have outage if any of your DC fails and will need to recover from it through reconcifiguration. For ZooKeeper, I do not think there is any way to guarantee this at all.