Kafka broker pod never created if a listener of type ingress is present

[somejfn]

Hi,

I remember having no issue creating a cluster with a listener of type ingress on 0.19 but now on 0.20, I never see the kafka broker pod created and the operator logs that the ingress resource is not "addressable". I wonder what I'm doing wrong here. The kafka resource:

apiVersion: kafka.strimzi.io/v1beta1
kind: Kafka
metadata:
  name: my-cluster
spec:
  kafka:
    version: 2.6.0
    replicas: 1
    listeners:
      - name: plain
        port: 9092
        type: internal
        tls: false
      - name: external
        port: 9094
        type: ingress
        tls: true
        configuration:
          class: a1-nginx-ingress
          bootstrap:
            host: mykafka.a1-nginx-ingress.lab.masked.io
          brokers:
          - broker: 0
            host: mykafka-b0.a1-nginx-ingress.lab.masked.io
    resources:
      requests:
        memory: 512Mi
        cpu: "100m"
      limits:
        memory: 1500Mi
        cpu: "1"
    jvmOptions:
      -Xms: 512m
      -Xmx: 1024m
    config:
      offsets.topic.replication.factor: 1
      transaction.state.log.replication.factor: 1
      transaction.state.log.min.isr: 1
      log.message.format.version: "2.6"
      inter.broker.protocol.version: "2.6"
    storage:
      type: jbod
      volumes:
      - id: 0
        type: persistent-claim
        size: 10Gi
        deleteClaim: true
        class: a1-tier1-nfs
  zookeeper:
    replicas: 1
    storage:
      type: persistent-claim
      size: 10Gi
      deleteClaim: true
      class: a1-tier1-nfs
    resources:
      requests:
        memory: 128Mi
        cpu: "100m"
      limits:
        memory: 512Mi
        cpu: "1"
  entityOperator:
    topicOperator: {}
    userOperator: {}

The cluster operator logs:

2021-01-20 18:24:05 WARN  AbstractOperator:470 - Reconciliation #117(timer) Kafka(saas-platform/my-cluster): Failed to reconcile
io.strimzi.operator.common.operator.resource.TimeoutException: Exceeded timeout of 300000ms while waiting for Ingress resource my-cluster-kafka-bootstrap in namespace saas-platform to be addressable
	at io.strimzi.operator.common.Util$1.lambda$handle$1(Util.java:125) ~[io.strimzi.operator-common-0.20.0.jar:0.20.0]
	at io.vertx.core.impl.ContextImpl.lambda$null$0(ContextImpl.java:327) ~[io.vertx.vertx-core-3.9.1.jar:3.9.1]
	at io.vertx.core.impl.ContextImpl.executeTask(ContextImpl.java:366) ~[io.vertx.vertx-core-3.9.1.jar:3.9.1]
	at io.vertx.core.impl.EventLoopContext.lambda$executeAsync$0(EventLoopContext.java:38) ~[io.vertx.vertx-core-3.9.1.jar:3.9.1]
	at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) [io.netty.netty-common-4.1.50.Final.jar:4.1.50.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) [io.netty.netty-common-4.1.50.Final.jar:4.1.50.Final]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500) [io.netty.netty-transport-4.1.50.Final.jar:4.1.50.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [io.netty.netty-common-4.1.50.Final.jar:4.1.50.Final]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [io.netty.netty-common-4.1.50.Final.jar:4.1.50.Final]
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty.netty-common-4.1.50.Final.jar:4.1.50.Final]
	at java.lang.Thread.run(Thread.java:834) [?:?]

It looks like the operator tries to reach the ingress URL but how could that be reachable if there no pod endpoint yet ?

If I drop that "external" listener to keep only the plain internal one, the cluster gets created as expected.

Please help ! :)

[scholzj]

It is not trying to reach the URL. It creates the Ingress resources and then waits for the Ingress controller to confirm their address in the Status section. And that seems to not happen in your case. That indicates that the Ingress resources might not work. So I guess this suggests that your Ingress controller did not see these resources.

If you want, you can do kubectl get ingress -o yaml an share the YAMLs. I can compare them to what the operator expects. But in most cases like these the status is just empty which means that the Ingress controller did not saw them / use them etc.

[somejfn]

Thanks for the quick reply and the hints on the ingress status IP !

[scholzj]

Great, glad it works now. And thanks for sharing back the config - that might be useful for other users.

[oana-s]

@somejfn could you share how you created the Ingress Controller? thanks

[fuhuasheng]

@somejfn I add the update-status & publish-status-address, but the strimzi still not create kafka pod and report the same error. Could you share how to let it work? thanks

[somejfn]

See the args to the nginx controller below, but before testing with Strimzi, I'd suggest you test an ingress resource in a simpler context (say just a web app, service and ingress) to confirm the ingress resource gets updated after a few seconds (look for the ADDRESS colum when listing ingresses)

      - args:
        - /nginx-ingress-controller
        - --disable-catch-all
        - --configmap=$(POD_NAMESPACE)/my-nginx-config
        - --ingress-class=a1-nginx-ingress
        - --tcp-services-configmap=$(POD_NAMESPACE)/my-nginx-config-tcp
        - --enable-ssl-passthrough
        - --publish-status-address=10.0.5.160
        - --update-status

Note: you likely don't need the publish-status-address parameter like I did... this is because I was not exposing the ingress controller the usual way with a service type=LoadBalancer (use the --publish-service "namespace/name" argument if that is your case). On another note, also don't forget to set the right ingress class in your strimzi cluster CR to match your ingress controller.