Kafka-Zookeeper Cluster on a single node not running?

[bitvijays]

o/ Strimzi Team, Hope you are doing. Thank you for the awesome support you have been providing. 👍

We are trying to deploy a kafka cluster with Single Kafka and Single Zookeeper on a particular single node using node affinity.

While deploying the cluster we are getting the below error. While debugging kubectl describe kafka -n my-kafka-project

Failure executing: GET at: https://10.96.0.1/apis/networking.k8s.io/v1/namespaces/my-kafka-project/ingresses?labelSelector=strimzi.io%2Fcluster%3Dmy-cluster2%2Cstrimzi.io%2Fname%3Dmy-cluster2-kafka%2Cstrimzi.io%2Fkind%3DKafka. Message: Forbidden!Configured service account doesn't have access. Service account may have been revoked. ingresses.networking.k8s.io is forbidden: User "system:serviceaccount:kafka:strimzi-cluster-operator" cannot list resource "ingresses" in API group "networking.k8s.io" in the namespace "my-kafka-project".` 

The YAML is present below:

apiVersion: kafka.strimzi.io/v1beta1
kind: Kafka
metadata:
  name: my-cluster2
spec:
  kafka:
    version: 2.7.0
    replicas: 1
    listeners:
      - name: plain
        port: 9092
        type: nodeport
        tls: false
      - name: tls
        port: 9093
        type: nodeport
        tls: true
        configuration:
          bootstrap:
            nodePort: 32100
          brokers:
          - broker: 0
            nodePort: 32000

    config:
      offsets.topic.replication.factor: 1
      transaction.state.log.replication.factor: 1
      transaction.state.log.min.isr: 1
      log.message.format.version: "2.7"
      inter.broker.protocol.version: "2.7"
    storage:
      type: ephemeral
    template:
      pod:
        affinity:
          nodeAffinity:
            requiredDuringSchedulingIgnoredDuringExecution:
              nodeSelectorTerms:
                - matchExpressions:
                  - key: kubernetes.io/hostname
                    operator: In
                    values:
                    - kafkaserver
  zookeeper:
    replicas: 1
    storage:
      type: ephemeral
    template:
      pod:
        affinity:
          nodeAffinity:
            requiredDuringSchedulingIgnoredDuringExecution:
              nodeSelectorTerms:
                - matchExpressions:
                  - key: kubernetes.io/hostname
                    operator: In
                    values:
                    - kafkaserver
  entityOperator:
    topicOperator: {}
    userOperator: {}

Performing kubectl describe kafka -n my-kafka-project

Status:
  Conditions:
    Last Transition Time:  2021-03-16T01:33:09.017769Z
    Message:               A Kafka cluster with a single replica and ephemeral storage will lose topic messages after any restart or rolling update.
    Reason:                KafkaStorage
    Status:                True
    Type:                  Warning
    Last Transition Time:  2021-03-16T01:33:09.017839Z
    Message:               A ZooKeeper cluster with a single replica and ephemeral storage will be in a defective state after any restart or rolling update. It is recommended that a minimum of three replicas are used.
    Reason:                ZooKeeperStorage
    Status:                True
    Type:                  Warning
    Last Transition Time:  2021-03-16T01:33:10.435Z
    Message:               Failure executing: GET at: https://10.96.0.1/apis/networking.k8s.io/v1/namespaces/my-kafka-project/ingresses?labelSelector=strimzi.io%2Fcluster%3Dmy-cluster2%2Cstrimzi.io%2Fname%3Dmy-cluster2-kafka%2Cstrimzi.io%2Fkind%3DKafka. Message: Forbidden!Configured service account doesn't have access. Service account may have been revoked. ingresses.networking.k8s.io is forbidden: User "system:serviceaccount:kafka:strimzi-cluster-operator" cannot list resource "ingresses" in API group "networking.k8s.io" in the namespace "my-kafka-project".
    Reason:                KubernetesClientException
    Status:                True
    Type:                  NotReady
  Observed Generation:     1
Events:                    <none>

We did followed Strimzi Quickstart, we did performed the below steps

Give permission to the Cluster Operator to watch the my-kafka-project namespace.

kubectl apply -f install/cluster-operator/020-RoleBinding-strimzi-cluster-operator.yaml -n my-kafka-project
kubectl apply -f install/cluster-operator/032-RoleBinding-strimzi-cluster-operator-topic-operator-delegation.yaml -n my-kafka-project
kubectl apply -f install/cluster-operator/031-RoleBinding-strimzi-cluster-operator-entity-operator-delegation.yaml -n my-kafka-project

Currently, we have the below namespaces

kubectl get namespace
NAME               STATUS   AGE
default            Active   11d
kafka              Active   10h
kube-node-lease    Active   11d
kube-public        Active   11d
kube-system        Active   11d
kubeedge           Active   10d
my-kafka-project   Active   10h

[scholzj]

That seems to be the same issue as #4576 ... you built your images from master but you did not used the latest RBAC files for master from packaging/install/cluster-operator to deploy it.

[scholzj]

You should use kubectl create ... instead of kubectl apply .... This is because kubectl apply cannot handle big files.

[bitvijays]

Thanks @scholzj Currently, when deploying the Kakfa cluster, getting the below error for Kafka Pod

{"log":"2021-03-16 15:48:06,855 ERROR Exiting Kafka due to fatal exception (kafka.Kafka$) [main]\n","stream":"stdout","time":"2021-03-16T15:48:06.856877176Z"}
{"log":"org.apache.kafka.common.config.ConfigException: Invalid value 2.7 for configuration log.message.format.version: Version `2.7` is not a valid version\n","stream":"stdout","time":"2021-03-16T15:48:06.856914362Z"}

Should we change the log.message.format.version to 2.6?

[scholzj]

They depends on what Kafka version you use. if You use something older than 2.7 you will need to change it.

[bitvijays]

Q1: When we compile Strimzi from the source using MVN_ARGS="-Dmaven.javadoc.skip=true -DskipITs -DskipTests" make docker_build and it builds the images. What Kafka version does it contain? How do we defined?

Q2: Also, we are using the kafka_version: 2.7 as mentioned in the examples.

[scholzj]

Q1: When we compile Strimzi from the source using MVN_ARGS="-Dmaven.javadoc.skip=true -DskipITs -DskipTests" make docker_build and it builds the images. What Kafka version does it contain? How do we defined?

That depends on what do you have in the code you build. It is more or less based on kafka-versions.yaml.

Q2: Also, we are using the kafka_version: 2.7 as mentioned in the examples.

Then log.message.format.version: 2.7 should not be an issue.