Related problem

Currently, we restart controller-only nodes if there is any controller relevant configuration changes detected via Pod annotation. The reason for this was previously controller-only nodes did not provide API access. After Kafka 3.9.0 release and #10016, the operator can now talk to controller-only nodes directly via Admin API.

Suggested solution

We should apply dynamic configurations to controller-only nodes (same as how it's done for broker and mixed nodes) instead of restarting them.

However, we should take into an account that this can be done only for controller nodes with Kafka version 3.9 or later and impact on the upgrade from an older version.

Alternatives

No response

Additional context

No response