bento/ubuntu-22.04 vagrant box update 2 days prevent calico network to initialise

[weetat]

Hi
After Vagrant box for bento/ubuntu-22.04 was updated https://app.vagrantup.com/bento/boxes/ubuntu-22.04 on 2 days ago, the calico network pods are not in running mode. Refer to below for more details.
Error is "Error: container create failed: pivot_root: Invalid argument".
Please advise. Thank you.

kubectl get po -A
NAMESPACE     NAME                                       READY   STATUS                      RESTARTS   AGE
kube-system   calico-kube-controllers-658d97c59c-78lv4   1/1     Running                     0          91s
kube-system   calico-node-mvqw7                          0/1     Init:CreateContainerError   0          91s
kube-system   coredns-76f75df574-8n6wj                   1/1     Running                     0          91s
kube-system   coredns-76f75df574-96d8g                   1/1     Running                     0          91s
kube-system   etcd-controlplane                          1/1     Running                     0          104s
kube-system   kube-apiserver-controlplane                1/1     Running                     0          104s
kube-system   kube-controller-manager-controlplane       1/1     Running                     0          104s
kube-system   kube-proxy-hxnk9                           1/1     Running                     0          91s
kube-system   kube-scheduler-controlplane                1/1     Running                     0          104s
kube-system   metrics-server-d4dc9c4f-znwdn              0/1     Pending                     0          91s

kubectl describe po calico-node-mvqw7 -n kube-system
Name:                 calico-node-mvqw7
Namespace:            kube-system
Priority:             2000001000
Priority Class Name:  system-node-critical
Service Account:      calico-node
Node:                 controlplane/192.168.96.116
Start Time:           Sat, 10 Aug 2024 00:07:49 +0000
Labels:               controller-revision-hash=574c44bccd
                      k8s-app=calico-node
                      pod-template-generation=1
Annotations:          <none>
Status:               Pending
IP:                   192.168.96.116
IPs:
  IP:           192.168.96.116
Controlled By:  DaemonSet/calico-node
Init Containers:
  upgrade-ipam:
    Container ID:  cri-o://bf6974f1a6c71723bf9f2a70d2c28ccf083eba89aac0e4cdb2b4ca8178aefd2e
    Image:         docker.io/calico/cni:v3.25.0
    Image ID:      docker.io/calico/cni@sha256:a38d53cb8688944eafede2f0eadc478b1b403cefeff7953da57fe9cd2d65e977
    Port:          <none>
    Host Port:     <none>
    Command:
      /opt/cni/bin/calico-ipam
      -upgrade
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Sat, 10 Aug 2024 00:08:02 +0000
      Finished:     Sat, 10 Aug 2024 00:08:02 +0000
    Ready:          True
    Restart Count:  0
    Environment Variables from:
      kubernetes-services-endpoint  ConfigMap  Optional: true
    Environment:
      KUBERNETES_NODE_NAME:        (v1:spec.nodeName)
      CALICO_NETWORKING_BACKEND:  <set to the key 'calico_backend' of config map 'calico-config'>  Optional: false
    Mounts:
      /host/opt/cni/bin from cni-bin-dir (rw)
      /var/lib/cni/networks from host-local-net-dir (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-j5hnn (ro)
  install-cni:
    Container ID:  cri-o://4a3d1b676e31d696a440b41ff038b6c016c050c34804fbf89bdf473942416db4
    Image:         docker.io/calico/cni:v3.25.0
    Image ID:      docker.io/calico/cni@sha256:a38d53cb8688944eafede2f0eadc478b1b403cefeff7953da57fe9cd2d65e977
    Port:          <none>
    Host Port:     <none>
    Command:
      /opt/cni/bin/install
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Sat, 10 Aug 2024 00:08:02 +0000
      Finished:     Sat, 10 Aug 2024 00:08:04 +0000
    Ready:          True
    Restart Count:  0
    Environment Variables from:
      kubernetes-services-endpoint  ConfigMap  Optional: true
    Environment:
      CNI_CONF_NAME:         10-calico.conflist
      CNI_NETWORK_CONFIG:    <set to the key 'cni_network_config' of config map 'calico-config'>  Optional: false
      KUBERNETES_NODE_NAME:   (v1:spec.nodeName)
      CNI_MTU:               <set to the key 'veth_mtu' of config map 'calico-config'>  Optional: false
      SLEEP:                 false
    Mounts:
      /host/etc/cni/net.d from cni-net-dir (rw)
      /host/opt/cni/bin from cni-bin-dir (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-j5hnn (ro)
  mount-bpffs:
    Container ID:
    Image:         docker.io/calico/node:v3.25.0
    Image ID:
    Port:          <none>
    Host Port:     <none>
    Command:
      calico-node
      -init
      -best-effort
    State:          Waiting
      Reason:       CreateContainerError
    Ready:          False
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /nodeproc from nodeproc (ro)
      /sys/fs from sys-fs (rw)
      /var/run/calico from var-run-calico (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-j5hnn (ro)
Containers:
  calico-node:
    Container ID:
    Image:          docker.io/calico/node:v3.25.0
    Image ID:
    Port:           <none>
    Host Port:      <none>
    State:          Waiting
      Reason:       PodInitializing
    Ready:          False
    Restart Count:  0
    Requests:
      cpu:      250m
    Liveness:   exec [/bin/calico-node -felix-live -bird-live] delay=10s timeout=10s period=10s #success=1 #failure=6
    Readiness:  exec [/bin/calico-node -felix-ready -bird-ready] delay=0s timeout=10s period=10s #success=1 #failure=3
    Environment Variables from:
      kubernetes-services-endpoint  ConfigMap  Optional: true
    Environment:
      DATASTORE_TYPE:                     kubernetes
      WAIT_FOR_DATASTORE:                 true
      NODENAME:                            (v1:spec.nodeName)
      CALICO_NETWORKING_BACKEND:          <set to the key 'calico_backend' of config map 'calico-config'>  Optional: false
      CLUSTER_TYPE:                       k8s,bgp
      IP:                                 autodetect
      CALICO_IPV4POOL_IPIP:               Always
      CALICO_IPV4POOL_VXLAN:              Never
      CALICO_IPV6POOL_VXLAN:              Never
      FELIX_IPINIPMTU:                    <set to the key 'veth_mtu' of config map 'calico-config'>  Optional: false
      FELIX_VXLANMTU:                     <set to the key 'veth_mtu' of config map 'calico-config'>  Optional: false
      FELIX_WIREGUARDMTU:                 <set to the key 'veth_mtu' of config map 'calico-config'>  Optional: false
      CALICO_DISABLE_FILE_LOGGING:        true
      FELIX_DEFAULTENDPOINTTOHOSTACTION:  ACCEPT
      FELIX_IPV6SUPPORT:                  false
      FELIX_HEALTHENABLED:                true
    Mounts:
      /host/etc/cni/net.d from cni-net-dir (rw)
      /lib/modules from lib-modules (ro)
      /run/xtables.lock from xtables-lock (rw)
      /sys/fs/bpf from bpffs (rw)
      /var/lib/calico from var-lib-calico (rw)
      /var/log/calico/cni from cni-log-dir (ro)
      /var/run/calico from var-run-calico (rw)
      /var/run/nodeagent from policysync (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-j5hnn (ro)
Conditions:
  Type                        Status
  PodReadyToStartContainers   True
  Initialized                 False
  Ready                       False
  ContainersReady             False
  PodScheduled                True
Volumes:
  lib-modules:
    Type:          HostPath (bare host directory volume)
    Path:          /lib/modules
    HostPathType:
  var-run-calico:
    Type:          HostPath (bare host directory volume)
    Path:          /var/run/calico
    HostPathType:
  var-lib-calico:
    Type:          HostPath (bare host directory volume)
    Path:          /var/lib/calico
    HostPathType:
  xtables-lock:
    Type:          HostPath (bare host directory volume)
    Path:          /run/xtables.lock
    HostPathType:  FileOrCreate
  sys-fs:
    Type:          HostPath (bare host directory volume)
    Path:          /sys/fs/
    HostPathType:  DirectoryOrCreate
  bpffs:
    Type:          HostPath (bare host directory volume)
    Path:          /sys/fs/bpf
    HostPathType:  Directory
  nodeproc:
    Type:          HostPath (bare host directory volume)
    Path:          /proc
    HostPathType:
  cni-bin-dir:
    Type:          HostPath (bare host directory volume)
    Path:          /opt/cni/bin
    HostPathType:
  cni-net-dir:
    Type:          HostPath (bare host directory volume)
    Path:          /etc/cni/net.d
    HostPathType:
  cni-log-dir:
    Type:          HostPath (bare host directory volume)
    Path:          /var/log/calico/cni
    HostPathType:
  host-local-net-dir:
    Type:          HostPath (bare host directory volume)
    Path:          /var/lib/cni/networks
    HostPathType:
  policysync:
    Type:          HostPath (bare host directory volume)
    Path:          /var/run/nodeagent
    HostPathType:  DirectoryOrCreate
  kube-api-access-j5hnn:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              kubernetes.io/os=linux
Tolerations:                 :NoSchedule op=Exists
                             :NoExecute op=Exists
                             CriticalAddonsOnly op=Exists
                             node.kubernetes.io/disk-pressure:NoSchedule op=Exists
                             node.kubernetes.io/memory-pressure:NoSchedule op=Exists
                             node.kubernetes.io/network-unavailable:NoSchedule op=Exists
                             node.kubernetes.io/not-ready:NoExecute op=Exists
                             node.kubernetes.io/pid-pressure:NoSchedule op=Exists
                             node.kubernetes.io/unreachable:NoExecute op=Exists
                             node.kubernetes.io/unschedulable:NoSchedule op=Exists
Events:
  Type     Reason     Age                From               Message
  ----     ------     ----               ----               -------
  Normal   Scheduled  114s               default-scheduler  Successfully assigned kube-system/calico-node-mvqw7 to controlplane
  Normal   Pulling    113s               kubelet            Pulling image "docker.io/calico/cni:v3.25.0"
  Normal   Pulled     101s               kubelet            Successfully pulled image "docker.io/calico/cni:v3.25.0" in 11.461s (11.461s including waiting)
  Normal   Created    101s               kubelet            Created container upgrade-ipam
  Normal   Started    101s               kubelet            Started container upgrade-ipam
  Normal   Pulled     101s               kubelet            Container image "docker.io/calico/cni:v3.25.0" already present on machine
  Normal   Created    101s               kubelet            Created container install-cni
  Normal   Started    101s               kubelet            Started container install-cni
  Normal   Pulling    99s                kubelet            Pulling image "docker.io/calico/node:v3.25.0"
  Normal   Pulled     80s                kubelet            Successfully pulled image "docker.io/calico/node:v3.25.0" in 12.332s (18.852s including waiting)
  Warning  Failed     13s (x7 over 80s)  kubelet            Error: container create failed: pivot_root: Invalid argument
  Normal   Pulled     13s (x6 over 80s)  kubelet            Container image "docker.io/calico/node:v3.25.0" already present on machine

[Adhito]

Just re-created my cluster and confirm this issue persist

vagrant@controlplane:~$ kubectl logs calico-node-pods -n kube-system
Defaulted container "calico-node" out of: calico-node, upgrade-ipam (init), install-cni (init), mount-bpffs (init)
Error from server (BadRequest): container "calico-node" in pod "calico-node-xqlsn" is waiting to start: PodInitializing

[MarosBenadik]

Yeah, same here. Interestingly, I've tried that with older versions of bento/ubuntu-22.04 (202404.23.0 and 202401.31.0) and encountered the same error.

[Adhito]

Just tried to upgrading Calico from 3.26 to 3.28 and it seemed to resolve the issue.