Many pods in "Error" or "Completed" status after reboots / applying machineconfig

[kastl-ars]

Dear all,

I noticed that after most reboots, e.g. when applying a machineconfig with a reboot, I have lots of pods in status Error or Completed on my singlenode talos instance.

For all pods in Error or Completed there seems to be a new pod that is health and Running. But there does not seem to be a "broken" pod for each running pod.

Example:

NAME                                                READY   STATUS      RESTARTS      AGE
csi-attacher-7744ffbff4-7979v                       0/1     Error       6             72m
csi-attacher-7744ffbff4-9df27                       1/1     Running     1 (21m ago)   26m
csi-provisioner-6c6798d8f7-d4j5h                    0/1     Error       6             72m
csi-provisioner-6c6798d8f7-vglmx                    1/1     Running     0             26m
csi-resizer-65bb74cc75-g9vp4                        0/1     Error       6             72m
csi-resizer-65bb74cc75-v6tg2                        1/1     Running     0             26m
csi-snapshotter-874b9f887-sbdj6                     0/1     Error       6             72m
csi-snapshotter-874b9f887-vgnrt                     1/1     Running     0             26m
engine-image-ei-b0369a5d-mtvml                      1/1     Running     2 (25m ago)   77m
engine-image-ei-c2d50bcc-blmqb                      1/1     Running     2 (25m ago)   77m
instance-manager-3affc1b780809ddf55500b0a4ae23ff0   1/1     Running     0             21m
longhorn-csi-plugin-mjv64                           3/3     Running     0             21m
longhorn-driver-deployer-5ff67f744-b45cf            0/1     Completed   0             72m
longhorn-driver-deployer-5ff67f744-lprvm            1/1     Running     0             26m
longhorn-manager-wmzh2                              2/2     Running     4 (25m ago)   77m
longhorn-ui-7749bb466f-l8h4d                        1/1     Running     5 (21m ago)   72m

To me it looks like only Deployments are affected, while DaemonSets and StatefulSets are fine (due to the deterministic naming?).

Any idea, what could cause that? Any idea how to prevent that?

Kind Regards,
Johannes

P.S.: Singlenode instance, currently Talos 1.9.4 with Kubernetes 1.30.2.

[smira]

There were a number of Kubernetes issues related to that. Long story short, it's harmless and Kubernetes will clean it up.

Way better: run a recent Kubernetes version, 1.30.2 is too old, use latest 1.30.x if you're stuck with 1.30.x for any reason. It might have been already fixed.

It's not a Talos issue in any way though.

[kastl-ars]

Thanks for the fast reply, Andrey!

It's not a Talos issue in any way though.

As I only saw this frequently on my talos machine and not on other k8s variants (rke2, k3s, ...) I was not sure.

Anyway, my instance is running 1.9.4 and k8s 1.32.x now, so I will keep an eye on this if it reappears with the next upgrades.