Application pod starting before kafka pod is causing leader election for the topic

[shreyasarani23]

Hi @scholzj we have application pod that writes to kafka. And we are using 3 kafka broker pods. since daily we are starting and stopping AKS cluster, after starting the AKS cluster first the application pod is in ready state. whereas the kafka pods are not in ready state. usually one by one kafka pods will come to ready state from running state. Since all the 3 kafka pods are not in ready state, the application pods is trying to connect to all the 3 kafka brokers since one of the broker might not be in ready state and this broker might be the partition leader for that topic it is causing leader election. below is the logs of application pod.

{"level":"ERROR","timestamp":"2021-09-29T03:40:55.985Z","logger":"kafkajs","message":"[Connection] Response Metadata(key: 3, version: 6)","broker":"my-kafka-cluster-kafka-bootstrap.kafka.svc:9092","clientId":"cms-kafka-service","error":"There is no leader for this topic-partition as we are in the middle of a leadership election","correlationId":1,"size":10518}
{"level":"ERROR","timestamp":"2021-09-29T03:40:55.992Z","logger":"kafkajs","message":"[Connection] Response Metadata(key: 3, version: 6)","broker":"my-kafka-cluster-kafka-0.my-kafka-cluster-kafka-brokers.kafka.svc:9092","clientId":"cms-kafka-service","error":"There is no leader for this topic-partition as we are in the middle of a leadership election","correlationId":0,"size":318}
{"level":"ERROR","timestamp":"2021-09-29T03:41:23.469Z","logger":"kafkajs","message":"[Connection] Response Metadata(key: 3, version: 6)","broker":"my-kafka-cluster-kafka-0.my-kafka-cluster-kafka-brokers.kafka.svc:9092","clientId":"cms-kafka-service","error":"There is no leader for this topic-partition as we are in the middle of a leadership election","correlationId":1,"size":318}
{"level":"ERROR","timestamp":"2021-09-29T03:41:25.972Z","logger":"kafkajs","message":"Timeout while acquiring lock (75 waiting locks): \"updating target topics\"","stack":"KafkaJSLockTimeout: Timeout while acquiring lock (75 waiting locks): \"updating target topics\"\n    at Timeout._onTimeout (/cms-api/node_modules/kafkajs/src/utils/lock.js:48:23)\n    at listOnTimeout (internal/timers.js:557:17)\n    at processTimers (internal/timers.js:500:7)"}
{"level":"ERROR","timestamp":"2021-09-29T03:41:25.973Z","logger":"kafkajs","message":"Timeout while acquiring lock (74 waiting locks): \"updating target topics\"","stack":"KafkaJSLockTimeout: Timeout while acquiring lock (74 waiting locks): \"updating target topics\"\n    at Timeout._onTimeout (/cms-api/node_modules/kafkajs/src/utils/lock.js:48:23)\n    at listOnTimeout (internal/timers.js:557:17)\n    at processTimers (internal/timers.js:500:7)"}
{"level":"ERROR","timestamp":"2021-09-29T03:41:25.973Z","logger":"kafkajs","message":"Timeout while acquiring lock (73 waiting locks): \"updating target topics\"","stack":"KafkaJSLockTimeout: Timeout while acquiring lock (73 waiting locks): \"updating target topics\"\n    at Timeout._onTimeout (/cms-api/node_modules/kafkajs/src/utils/lock.js:48:23)\n    at listOnTimeout (internal/timers.js:557:17)\n    at processTimers (internal/timers.js:500:7)"}

And also getting the error timeout while acquiring locks.

Please help me to resolve this issue.

[scholzj]

I'm not sure I follow what do you expect from Strimzi. Your application IMHO should be able to handle errors like this - either by recovering from them or for example by exiting and letting the Kubernetes to restart the pod with back-off until the Kafka cluster is ready.

[shreyasarani23]

Does strimzi provide any init container template that can be used in the application pod so that init container checks whether kafka cluster is ready or not. if it is ready state then only the application pod starts.

[scholzj]

No we do not provide anything like that. Your application needs to be able to handle this anyway, because the cluster might for various reasons become unavailable even while your app is running.

[shreyasarani23]

Oh ok any idea why this error is occuring?

{"level":"ERROR","timestamp":"2021-09-29T03:41:25.973Z","logger":"kafkajs","message":"Timeout while acquiring lock (73 waiting locks): \"updating target topics\"","stack":"KafkaJSLockTimeout: Timeout while acquiring lock (73 waiting locks): \"updating target topics\"\n    at Timeout._onTimeout (/cms-api/node_modules/kafkajs/src/utils/lock.js:48:23)\n    at listOnTimeout (internal/timers.js:557:17)\n    at processTimers (internal/timers.js:500:7)"}

[scholzj]

You probably have to ask the authors of your Kafka client what does it mean.