Adding a volume in a active/running cluster

[jriv96]

Does strimzi have the ability to add a disk to a live running cluster?

For example, I patched in the following volume with id:1 to the Kafka CRD.

    storage:
      type: jbod
      volumes:
      - class: aws-ssd
        deleteClaim: false
        id: 0
        size: 500Gi
        type: persistent-claim
      - class: aws-ssd
        deleteClaim: false
        id: 1
        size: 500Gi
        type: persistent-claim

I saw failure events related to the update below:

k get events -n strimzi-dev
LAST SEEN   TYPE      REASON         OBJECT                                              MESSAGE
32m         Normal    Pulled         pod/kafka-cluster-cruise-control-849f899d98-4ht25   Container image "strimzi/kafka:0.20.1-kafka-2.6.0" already present on machine
2m56s       Warning   BackOff        pod/kafka-cluster-cruise-control-849f899d98-4ht25   Back-off restarting failed container
9m37s       Warning   FailedUpdate   statefulset/kafka-cluster-kafka                     update Pod kafka-cluster-kafka-1 in StatefulSet kafka-cluster-kafka failed error: Pod "kafka-cluster-kafka-1" is invalid: spec: Forbidden: pod updates may not change fields other than `spec.containers[*].image`, `spec.initContainers[*].image`, `spec.activeDeadlineSeconds` or `spec.tolerations` (only additions to existing tolerations)
  core.PodSpec{
            Volumes: []core.Volume{
                      {Name: "data-0", VolumeSource: core.VolumeSource{PersistentVolumeClaim: &core.PersistentVolumeClaimVolumeSource{ClaimName: "data-0-kafka-cluster-kafka-1"}}},
-                     {
-                                    Name: "data-1",
-                                    VolumeSource: core.VolumeSource{
-                                                                                        PersistentVolumeClaim: &core.PersistentVolumeClaimVolumeSource{ClaimName: "data-1-kafka-cluster-kafka-1"},
-                                    },
-                     },
                      {Name: "rack-volume", VolumeSource: core.VolumeSource{EmptyDir: &core.EmptyDirVolumeSource{}}},
                      {Name: "cluster-ca", VolumeSource: core.VolumeSource{Secret: &core.SecretVolumeSource{SecretName: "kafka-cluster-cluster-ca-cert", DefaultMode: &292}}},
                      ... // 8 identical elements
            },
            InitContainers: []core.Container{{Name: "kafka-init", Image: "strimzi/operator:0.20.1", Args: []string{"/opt/strimzi/bin/kafka_init_run.sh"}, Env: []core.EnvVar{{Name: "NODE_NAME", ValueFrom: &core.EnvVarSource{FieldRef: &core.ObjectFieldSelector{APIVersion: "v1", FieldPath: "spec.nodeName"}}}, {Name: "RACK_TOPOLOGY_KEY", Value: "failure-domain.beta.kubernetes.io/zone"}}, Resources: core.ResourceRequirements{Limits: core.ResourceList{"cpu": {i: resource.int64Amount{value: 1}, s: "1", Format: "DecimalSI"}, "memory": {i: resource.int64Amount{value: 268435456}, Format: "BinarySI"}}, Requests: core.ResourceList{"cpu": {i: resource.int64Amount{value: 100, scale: -3}, s: "100m", Format: "DecimalSI"}, "memory": {i: resource.int64Amount{value: 134217728}, Format: "BinarySI"}}}, VolumeMounts: []core.VolumeMount{{Name: "rack-volume", MountPath: "/opt/kafka/init"}, {Name: "kafka-cluster-kafka-token-c6rgh", ReadOnly: true, MountPath: "/var/run/secrets/kubernetes.io/serviceaccount"}}, TerminationMessagePath: "/dev/termination-log", TerminationMessagePolicy: "File", ImagePullPolicy: "IfNotPresent"}},
            Containers:     []core.Container{{Name: "kafka", Image: "strimzi/kafka:0.20.1-kafka-2.6.0", Command: []string{"/opt/kafka/kafka_run.sh"}, Ports: []core.ContainerPort{{Name: "tcp-replication", ContainerPort: 9091, Protocol: "TCP"}, {Name: "tcp-clients", ContainerPort: 9092, Protocol: "TCP"}, {Name: "tcp-clientstls", ContainerPort: 9093, Protocol: "TCP"}, {Name: "tcp-external1", ContainerPort: 9094, Protocol: "TCP"}, {Name: "tcp-prometheus", ContainerPort: 9404, Protocol: "TCP"}}, Env: []core.EnvVar{{Name: "KAFKA_METRICS_ENABLED", Value: "true"}, {Name: "STRIMZI_KAFKA_GC_LOG_ENABLED", Value: "false"}, {Name: "DYNAMIC_HEAP_FRACTION", Value: "0.5"}, {Name: "DYNAMIC_HEAP_MAX", Value: "5368709120"}, {Name: "STRIMZI_TLS_9093_OAUTH_CLIENT_SECRET", ValueFrom: &core.EnvVarSource{SecretKeyRef: &core.SecretKeySelector{LocalObjectReference: core.LocalObjectReference{Name: "broker-oauth-secret"}, Key: "secret"}}}, {Name: "STRIMZI_EXTERNAL1_9094_OAUTH_CLIENT_SECRET", ValueFrom: &core.EnvVarSource{SecretKeyRef: &core.SecretKeySelector{LocalObjectReference: core.LocalObjectReference{Name: "broker-oauth-secret"}, Key: "secret"}}}}, Resources: core.ResourceRequirements{Limits: core.ResourceList{"cpu": {i: resource.int64Amount{value: 5}, s: "5", Format: "DecimalSI"}, "memory": {i: resource.int64Amount{value: 38654705664}, s: "36Gi", Format: "BinarySI"}}, Requests: core.ResourceList{"cpu": {i: resource.int64Amount{value: 5}, s: "5", Format: "DecimalSI"}, "memory": {i: resource.int64Amount{value: 38654705664}, s: "36Gi", Format: "BinarySI"}}}, VolumeMounts: []core.VolumeMount{{Name: "data-0", MountPath: "/var/lib/kafka/data-0"}, {Name: "cluster-ca", MountPath: "/opt/kafka/cluster-ca-certs"}, {Name: "broker-certs", MountPath: "/opt/kafka/broker-certs"}, {Name: "client-ca-cert", MountPath: "/opt/kafka/client-ca-certs"}, {Name: "kafka-metrics-and-logging", MountPath: "/opt/kafka/custom-config/"}, {Name: "ready-files", MountPath: "/var/opt/kafka"}, {Name: "rack-volume", MountPath: "/opt/kafka/init"}, {Name: "oauth-tls-9093-0", MountPath: "/opt/kafka/certificates/oauth-tls-9093-certs/ca-truststore-0"}, {Name: "custom-external1-9094-certs", MountPath: "/opt/kafka/certificates/custom-external1-9094-certs"}, {Name: "oauth-external1-9094-0", MountPath: "/opt/kafka/certificates/oauth-external1-9094-certs/ca-truststore-0"}, {Name: "kafka-cluster-kafka-token-c6rgh", ReadOnly: true, MountPath: "/var/run/secrets/kubernetes.io/serviceaccount"}}, LivenessProbe: &core.Probe{Handler: core.Handler{Exec: &core.ExecAction{Command: []string{"/opt/kafka/kafka_liveness.sh"}}}, InitialDelaySeconds: 15, TimeoutSeconds: 5, PeriodSeconds: 10, SuccessThreshold: 1, FailureThreshold: 3}, ReadinessProbe: &core.Probe{Handler: core.Handler{Exec: &core.ExecAction{Command: []string{"test", "-f", "/var/opt/kafka/kafka-ready"}}}, InitialDelaySeconds: 15, TimeoutSeconds: 5, PeriodSeconds: 10, SuccessThreshold: 1, FailureThreshold: 3}, TerminationMessagePath: "/dev/termination-log", TerminationMessagePolicy: "File", ImagePullPolicy: "IfNotPresent"}},
            ... // 26 identical fields
  }
26m         Warning   FailedUpdate   statefulset/kafka-cluster-kafka    

[scholzj]

What version do you use? There was a bug which was fixed in #4153.

[jriv96]

0.20.1 Looks like I need to upgrade. Thank you.

…

On Thu, Mar 18, 2021 at 4:30 PM Jakub Scholz ***@***.***> wrote: What version do you use? There was a bug which was fixed in #4153 <#4153>. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#4602 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAH7OUINQQFNJTQWSM7DUTDTEJ5IBANCNFSM4ZNCZJDA> .

[scholzj]

Yeah, that that would be that bug then. Unfortunately fixed only in 0.21 :-/.

[jriv96]

No worries. Thanks for the help!