Weird behaviour whilst pausing and resuming connectors via REST API

[alessio-gastaldo]

Hi there, I am noticing a weird behaviour when pausing and resuming connectors. Here's the details:

• I have created two sink connectors: connector_1, state: running and connector_2, state: paused
• Using the kafka connect rest api, I changed the state of the 2 connectors such as connector_1 running --> paused and connector_2 paused --> running. The API works fine and the statuses are updated for both
• After N seconds, when I query the statuses of the 2 connectors, I see that both reverted back to the state they were originally created in.

The last part of the process is what I define "weird behaviour". I don't understand why this is happening, can anyone help me understanding this?

[ppatierno]

Which version of Strimzi are you using?
Are you using the KafkaConnector custom resource to deploy your connectors?
Are you exposing the Kafka Connect REST API and interacting directly with them in order to change the connectors state?
I am guessing but it's possible that if you are using the REST API, the operator is reverting back the state you have in the KafkaConnector custom resource (you can read more about that here https://strimzi.io/docs/operators/latest/deploying#con-kafka-connector-config-str) if you are using the Strimzi version supporting it.

[alessio-gastaldo]

It looks like you got it:

2025-02-18 09:11:20 INFO  KafkaConnectAssemblyOperator:487 - Reconciliation #286573(timer) KafkaConnect(kafka-connect/rc-strimzi-production): creating/updating connector: xx-strimzi-xx
2025-02-18 09:11:20 INFO  AbstractConnectOperator:471 - Reconciliation #286573(timer) KafkaConnect(kafka-connect/rc-strimzi-production): Pausing connector xx-strimzi-xx
2025-02-18 09:11:20 INFO  CrdOperator:123 - Reconciliation #286573(timer) KafkaConnect(kafka-connect/rc-strimzi-production): Status of KafkaConnector xx-strimzi-xx has been updated

I wonder if there's a way to prevent this, without going through the link you suggested

[ppatierno]

So as I thought, the operator is reconciling the KafkaConnector custom resource and its state is pause but in reality you changed that by interacting with the REST API but of course on the next reconciliation, the operator is doing what you are asking through the custom resource ... so pausing.
It is supposed to be used this way and not going through the Kafka REST API otherwise you could just deploy connectors without using the KafkaConnector custom resource and then interacting via the REST API.
@mimaison wdyt? you contributed the most on this feature.

[mimaison]

That's my understanding too.

[ppatierno]

@alessio-gastaldo so if you are relying on the Kube-native way by using the KafkaConnector resources to deploy your connectors you should stick by using them for all the operations. If you need to use the REST API, you should also use them to deploy the connectors and not involving the KafkaConnector resources at all.
A real hack would be pausing the KafkaConnector reconciliation https://strimzi.io/docs/operators/latest/deploying#proc-pausing-reconciliation-str to avoid operator to reconcile when you interact with the REST API. it but then you would lost the support from the operator so it would make no sense. Also, when unpausing, the operator will reconcile doing what's requested by the resource.

[alessio-gastaldo]

Thanks both. At this point then, the cleanest way to implement this pause / resume is to use the pattern described in your first link https://strimzi.io/docs/operators/latest/deploying#con-kafka-connector-config-str. It's def less hacky IMO