Kafkaconnect replicas demystify amount of consumers

[verbotenj]

I want to demystify the number of consumers to match 6 partitions.

KafkaConnect
replica:1
KafkaConnector
tasksMax: 6

Replica * tasksMax = consumers (Partitions)
1 * 6 = 6

I understand that this will create 6 consumers (tasks) in one worker pod.

Do I understand it right if I configure?

KafkaConnect
replica:6
KafkaConnector
tasksMax: 1

Replica * tasksMax = consumers (Partitions)
6 * 1 = 6

Or I can do

KafkaConnect
replica:6
KafkaConnector
tasksMax: 6

6*6 = ?

6 or 36 consumers based on max tasks

Or replicas are meant only for fault-tolerance

Thank you

[scholzj]

I'm not sure it is as simple as that. spec.replicas in KafkaConnect indicates how many worker pods your Connect cluster will have. The spec.taskMax in KafkaConnector resource should define the maximum number of tasks a Connector will have. But as far as I know, not every connector always creates the maximum number of connectors. And some might even create more tasks than the configured maximum. That depends on the connector.

Some of the tasks might end up running on the same worker Pod (no idea if the worker Pod will open a connection per task or not - that would be more a Kafka question than Strimzi one). Additionally:

• Some connectors (e.g. Debezium as far as I know) open also their own connections to the Kafka cluster.
• The Connect framework has its own connections to manage the offsets, configs, sync the nodes etc.

I'm afraid I do not have the exact answer for you ... but I think the idea that ConnectConnectionCount = Replicas * TaskMax is a bit oversimplified and will probably not apply in all cases.

[verbotenj]

Thank you