The gateway Pod stuck in CrashLoopBackOff state

[0x6675636b796f75676974687562]

Commit range fa364ff..0e19690 (subsequent commits affected by either GH-2974 and/or GH-2975):

$ kubectl get pods
NAME                                                 READY   STATUS             RESTARTS   AGE
backend-7d4f996496-5ktdd                             1/1     Running            0          54m
backend-cosv-65c58dc899-wlr7b                        0/1     Init:0/2           0          54m
demo-5985955c49-8ct2z                                1/1     Running            0          54m
demo-cpg-66d9f959f-vrbl2                             1/1     Running            0          54m
frontend-cosv-565774f969-tp65l                       1/1     Running            0          54m
frontend-df5ccd8fc-8wfnj                             1/1     Running            0          54m
gateway-7ccfdb8788-kbcwg                             0/1     CrashLoopBackOff   15         54m
loki-canary-4v8q8                                    1/1     Running            0          54m
loki-canary-qcttf                                    1/1     Running            0          54m
loki-canary-xv6zm                                    1/1     Running            0          54m
mysql-tunnel                                         1/1     Running            0          45h
orchestrator-56568c8cdf-drxxl                        1/1     Running            0          54m
preprocessor-5cd447949b-9ngw4                        1/1     Running            0          54m
save-cloud-0                                         1/1     Running            0          54m
save-cloud-grafana-agent-operator-6b9d4f9d8d-qsrxr   1/1     Running            0          54m
save-cloud-grafana-c8c645d67-shc6j                   1/1     Running            0          54m
save-cloud-loki-0                                    1/1     Running            0          54m
save-cloud-prometheus-server-6f8577bbf6-9sgjn        1/1     Running            0          54m
save-cloud-promtail-df95d                            1/1     Running            0          54m
save-cloud-promtail-fzp28                            1/1     Running            0          54m
save-cloud-promtail-kvg6q                            1/1     Running            0          54m

kubectl describe for backend-cosv-65c58dc899-wlr7b shows the following events:

  Type     Reason       Age                 From               Message
  ----     ------       ----                ----               -------
  Normal   Scheduled    56m                 default-scheduler  Successfully assigned save-cloud/backend-cosv-65c58dc899-wlr7b to 172.16.0.55
  Warning  FailedMount  49m                 kubelet            Unable to attach or mount volumes: unmounted volumes=[ca-certs], unattached volumes=[database-secret ca-certs config-volume s3-secrets migrations-data kube-api-access-ztcpw]: timed out waiting for the condition
  Warning  FailedMount  44m (x3 over 53m)   kubelet            Unable to attach or mount volumes: unmounted volumes=[ca-certs], unattached volumes=[migrations-data kube-api-access-ztcpw database-secret ca-certs config-volume s3-secrets]: timed out waiting for the condition
  Warning  FailedMount  38m                 kubelet            Unable to attach or mount volumes: unmounted volumes=[ca-certs], unattached volumes=[ca-certs config-volume s3-secrets migrations-data kube-api-access-ztcpw database-secret]: timed out waiting for the condition
  Warning  FailedMount  20m (x3 over 47m)   kubelet            Unable to attach or mount volumes: unmounted volumes=[ca-certs], unattached volumes=[s3-secrets migrations-data kube-api-access-ztcpw database-secret ca-certs config-volume]: timed out waiting for the condition
  Warning  FailedMount  15m (x3 over 40m)   kubelet            Unable to attach or mount volumes: unmounted volumes=[ca-certs], unattached volumes=[kube-api-access-ztcpw database-secret ca-certs config-volume s3-secrets migrations-data]: timed out waiting for the condition
  Warning  FailedMount  10m (x5 over 42m)   kubelet            Unable to attach or mount volumes: unmounted volumes=[ca-certs], unattached volumes=[config-volume s3-secrets migrations-data kube-api-access-ztcpw database-secret ca-certs]: timed out waiting for the condition
  Warning  FailedMount  61s (x35 over 56m)  kubelet            MountVolume.SetUp failed for volume "cacerts" : configmap "ca-certs" not found

Events for gateway-7ccfdb8788-kbcwg (0.4.0-alpha.0.429-0e19690):

  Type     Reason                 Age                    From               Message
  ----     ------                 ----                   ----               -------
  Normal   Scheduled              58m                    default-scheduler  Successfully assigned save-cloud/gateway-7ccfdb8788-kbcwg to 172.16.0.87
  Normal   Pulled                 58m                    kubelet            Successfully pulled image "ghcr.io/saveourtool/api-gateway:0.4.0-alpha.0.429-0e19690" in 3.485966887s
  Normal   SuccessfulMountVolume  58m (x2 over 58m)      kubelet            Successfully mounted volumes for pod "gateway-7ccfdb8788-kbcwg_save-cloud(d2bb10f3-1375-45ee-b550-fbf3a69661a6)"
  Normal   Pulled                 58m                    kubelet            Successfully pulled image "ghcr.io/saveourtool/api-gateway:0.4.0-alpha.0.429-0e19690" in 984.39591ms
  Normal   Pulling                58m (x3 over 58m)      kubelet            Pulling image "ghcr.io/saveourtool/api-gateway:0.4.0-alpha.0.429-0e19690"
  Normal   SuccessfulCreate       58m (x3 over 58m)      kubelet            Created container gateway
  Normal   Started                58m (x3 over 58m)      kubelet            Started container gateway
  Warning  BackOffStart           58m (x5 over 58m)      kubelet            Back-off restarting failed container
  Normal   Pulled                 58m                    kubelet            Successfully pulled image "ghcr.io/saveourtool/api-gateway:0.4.0-alpha.0.429-0e19690" in 943.128249ms
  Warning  BackOffStart           3m20s (x278 over 58m)  kubelet            the failed container exited with ExitCode: 82

Events for gateway-8454c8bcb6-hbrhs (0.4.0-alpha.0.414-fa364ff):

  Type     Reason       Age                   From               Message
  ----     ------       ----                  ----               -------
  Normal   Scheduled    4m57s                 default-scheduler  Successfully assigned save-cloud/gateway-8454c8bcb6-hbrhs to 172.16.0.52
  Warning  FailedMount  2m54s                 kubelet            Unable to attach or mount volumes: unmounted volumes=[his-certs], unattached volumes=[cacerts his-certs kube-api-access-4fv7f config-volume oauth-credentials]: timed out waiting for the condition
  Warning  FailedMount  47s (x10 over 4m57s)  kubelet            MountVolume.SetUp failed for volume "his-certs" : secret "his-certs" not found
  Warning  FailedMount  38s                   kubelet            Unable to attach or mount volumes: unmounted volumes=[his-certs], unattached volumes=[his-certs kube-api-access-4fv7f config-volume oauth-credentials cacerts]: timed out waiting for the condition

Logs for backend-cosv-65c58dc899-wlr7b:

Defaulted container "backend-cosv" out of: backend-cosv, git-cloner (init), liquibase-runner (init)
Error from server (BadRequest): container "backend-cosv" in pod "backend-cosv-65c58dc899-wlr7b" is waiting to start: PodInitializing

Logs for gateway-7ccfdb8788-kbcwg (0.4.0-alpha.0.429-0e19690):

Setting Active Processor Count to 4
Calculated JVM Memory Configuration: -XX:MaxDirectMemorySize=10M -Xmx91737K -XX:MaxMetaspaceSize=188294K -Xss1M (Total Memory: 390624K, Thread Count: 50, Loaded Class Count: 30830, Headroom: 0
%)
Enabling Java Native Memory Tracking
unable to load certificates
Using readonly truststore: /tmp/truststore
unable to read keystore
unable to open /layers/paketo-buildpacks_bellsoft-liberica/jre/lib/security/cacerts
open /layers/paketo-buildpacks_bellsoft-liberica/jre/lib/security/cacerts: no such file or directory
ERROR: failed to launch: exec.d: failed to execute exec.d file at path '/layers/paketo-buildpacks_bellsoft-liberica/helper/exec.d/openssl-certificate-loader': exit status 1

• Caused by: fa364ff, updated deployments config for HIS usecase #2898
• Helm chart version range: 0.2.1+1181..0.2.1+1196
• Date range: 29.12.2023..31.01.2024
• Commit range: fa364ff..0e19690
• Docker tag range: 0.4.0-alpha.0.414-fa364ff..0.4.0-alpha.0.429-0e19690

[0x6675636b796f75676974687562]

Fixed with #2984.