Connecting to Strrimzi from outside the k8s cluster using NodePorts

[Hen0k]

I have a working cluster inside k8s with clients like kafka connect, schema registry and a few python script. This setup uses mTLS on all listeners. The idea is to have a new connect worker in a different place connect to my k8s cluster. For this, I created a KafkaUser with TLS as auth and took this users secrets the user.p12 and user.password files and the -cluster-ca-cert ca.crt. I am trying to use the first two as keystore and the ca.crt as my trust store..

But, the client all crash siting a timeout. And there is no error on the brokers end.

Broker listeners

listeners:
      - name: plain
        port: 9092
        type: internal
        tls: true
        configuration:
          useServiceDnsDomain: true
      - name: externaltls
        port: 9094
        type: nodeport
        tls: true
        authentication:
          type: tls
        configuration:
          bootstrap:
            nodePort: <port 0>
            alternativeNames:
              - <broker external host>
         brokers:
          - broker: 0
            nodePort: <port 1>
          - broker: 1
            nodePort: <port 2>
          - broker: 2
            nodePort: <port 3>
          - broker: 3
            nodePort: <port 4>
          - broker: 4
            nodePort: <port 5>
          - broker: 5
            nodePort: <port 6>

connect worker broker config

spec:
  version: 3.9.0
  replicas: 1
  bootstrapServers: <bootstrap-host:port>
  tls:
    trustedCertificates:
      - secretName: hansel-kafka-cluster-cluster-ca-cert
        certificate: ca.crt
  authentication:
    type: tls
    certificateAndKey:
      secretName: external-schema-registry-client-secret
      certificate: user.crt
      key: user.key

Connect worker log

2025-02-19 04:44:45,551 INFO Kafka version: 3.9.0 (org.apache.kafka.common.utils.AppInfoParser) [main]
2025-02-19 04:44:45,553 INFO Kafka commitId: a60e31147e6b01ee (org.apache.kafka.common.utils.AppInfoParser) [main]
2025-02-19 04:44:45,553 INFO Kafka startTimeMs: 1739940285550 (org.apache.kafka.common.utils.AppInfoParser) [main]
2025-02-19 04:44:58,845 INFO [AdminClient clientId=adminclient-1] Disconnecting from node 5 due to socket connection setup timeout. The timeout value is 11783 ms. (org.apache.kafka.clients.NetworkClient) [kafka-admin-client-thread | adminclient-1]
2025-02-19 04:45:09,061 INFO [AdminClient clientId=adminclient-1] Disconnecting from node 3 due to socket connection setup timeout. The timeout value is 10204 ms. (org.apache.kafka.clients.NetworkClient) [kafka-admin-client-thread | adminclient-1]
2025-02-19 04:45:18,641 INFO [AdminClient clientId=adminclient-1] Disconnecting from node 2 due to socket connection setup timeout. The timeout value is 9575 ms. (org.apache.kafka.clients.NetworkClient) [kafka-admin-client-thread | adminclient-1]
2025-02-19 04:45:28,857 INFO [AdminClient clientId=adminclient-1] Disconnecting from node 1 due to socket connection setup timeout. The timeout value is 11762 ms. (org.apache.kafka.clients.NetworkClient) [kafka-admin-client-thread | adminclient-1]
2025-02-19 04:45:28,858 INFO [AdminClient clientId=adminclient-1] Metadata update failed (org.apache.kafka.clients.admin.internals.AdminMetadataManager) [kafka-admin-client-thread | adminclient-1]
org.apache.kafka.common.errors.TimeoutException: Timed out waiting for a node assignment. Call: fetchMetadata
2025-02-19 04:45:36,444 INFO [AdminClient clientId=adminclient-1] Node -1 disconnected. (org.apache.kafka.clients.NetworkClient) [kafka-admin-client-thread | adminclient-1]
2025-02-19 04:45:44,688 INFO [AdminClient clientId=adminclient-1] Disconnecting from node 4 due to socket connection setup timeout. The timeout value is 8238 ms. (org.apache.kafka.clients.NetworkClient) [kafka-admin-client-thread | adminclient-1]
2025-02-19 04:45:45,562 INFO App info kafka.admin.client for adminclient-1 unregistered (org.apache.kafka.common.utils.AppInfoParser) [kafka-admin-client-thread | adminclient-1]
2025-02-19 04:45:45,562 INFO [AdminClient clientId=adminclient-1] Metadata update failed (org.apache.kafka.clients.admin.internals.AdminMetadataManager) [kafka-admin-client-thread | adminclient-1]
org.apache.kafka.common.errors.TimeoutException: Timed out waiting to send the call. Call: fetchMetadata
2025-02-19 04:45:45,562 INFO [AdminClient clientId=adminclient-1] Timed out 1 remaining operation(s) during close. (org.apache.kafka.clients.admin.KafkaAdminClient) [kafka-admin-client-thread | adminclient-1]
2025-02-19 04:45:45,570 INFO Metrics scheduler closed (org.apache.kafka.common.metrics.Metrics) [kafka-admin-client-thread | adminclient-1]
2025-02-19 04:45:45,570 INFO Closing reporter org.apache.kafka.common.metrics.JmxReporter (org.apache.kafka.common.metrics.Metrics) [kafka-admin-client-thread | adminclient-1]
2025-02-19 04:45:45,571 INFO Metrics reporters closed (org.apache.kafka.common.metrics.Metrics) [kafka-admin-client-thread | adminclient-1]
2025-02-19 04:45:45,571 ERROR Stopping due to error (org.apache.kafka.connect.cli.AbstractConnectCli) [main]
org.apache.kafka.connect.errors.ConnectException: Failed to connect to and describe Kafka cluster. Check worker's broker connection and security properties.
        at org.apache.kafka.connect.runtime.WorkerConfig.lookupKafkaClusterId(WorkerConfig.java:303)
        at org.apache.kafka.connect.runtime.WorkerConfig.lookupKafkaClusterId(WorkerConfig.java:283)
        at org.apache.kafka.connect.runtime.WorkerConfig.kafkaClusterId(WorkerConfig.java:413)
        at org.apache.kafka.connect.cli.AbstractConnectCli.startConnect(AbstractConnectCli.java:124)
        at org.apache.kafka.connect.cli.AbstractConnectCli.run(AbstractConnectCli.java:95)
        at org.apache.kafka.connect.cli.ConnectDistributed.main(ConnectDistributed.java:112)
Caused by: java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.TimeoutException: Timed out waiting to send the call. Call: listNodes
        at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:396)
        at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2073)
        at org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:165)
        at org.apache.kafka.connect.runtime.WorkerConfig.lookupKafkaClusterId(WorkerConfig.java:297)
        ... 5 more
Caused by: org.apache.kafka.common.errors.TimeoutException: Timed out waiting to send the call. Call: listNodes

I have tried using the certificate for test connection from within the client using openssl and it does verify. And firewall is open. when I misconfigure something, I get an authentication error log on the browsers.

So, my question is what am missing in my setup?

[scholzj]

TBH, it is hard to help without the full logs and the full steps how did you created the Secrets, what addresses you are using, understanding where things are running etc. That said, the parts of the configs you shared look correct and my best guess would be some networking issue such as firewals etc. that is blocking the connection.