kubelet.service: Failed at step EXEC spawning /usr/bin/hyperkube: No such file or directory #849

tanmoydutta · 2021-09-02T07:56:36Z

tanmoydutta
Sep 2, 2021

Describe the bug
OKD bootstrap node initialization is not successful.
Error in the system log -

Sep 01 18:08:01 bootstrap-test audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=kubelet comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=failed'
Sep 01 18:08:01 bootstrap-test kernel: audit: type=1130 audit(1630519681.676:17996): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=kubelet comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=failed'
Sep 01 18:08:11 bootstrap-test systemd[1]: kubelet.service: Scheduled restart job, restart counter is at 2323.
Sep 01 18:08:11 bootstrap-test audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=kubelet comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Sep 01 18:08:11 bootstrap-test systemd[1]: Stopped Kubernetes Kubelet.
Sep 01 18:08:11 bootstrap-test systemd[1]: Starting Kubernetes Kubelet...
Sep 01 18:08:11 bootstrap-test audit[1]: SERVICE_STOP pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=kubelet comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Sep 01 18:08:11 bootstrap-test kernel: audit: type=1130 audit(1630519691.789:17997): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=kubelet comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Sep 01 18:08:11 bootstrap-test kernel: audit: type=1131 audit(1630519691.789:17998): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=kubelet comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Sep 01 18:08:11 bootstrap-test podman[302173]: 2021-09-01 18:08:11.997598591 +0000 UTC m=+0.056148228 container create b227a0af15e629704c2c6c1d9c013a3877222372c651b345af8ce92e3bd11733 (image=quay.io/openshift/okd@sha256:02c01aaaed9d292376e7808b6365ed38c34856e2a13c99718>
Sep 01 18:08:12 bootstrap-test systemd[1]: Started libcrun container.
Sep 01 18:08:12 bootstrap-test podman[302173]: 2021-09-01 18:08:12.039244503 +0000 UTC m=+0.097794028 container init b227a0af15e629704c2c6c1d9c013a3877222372c651b345af8ce92e3bd11733 (image=quay.io/openshift/okd@sha256:02c01aaaed9d292376e7808b6365ed38c34856e2a13c99718b5>
Sep 01 18:08:12 bootstrap-test podman[302173]: 2021-09-01 18:08:12.044910865 +0000 UTC m=+0.103460388 container start b227a0af15e629704c2c6c1d9c013a3877222372c651b345af8ce92e3bd11733 (image=quay.io/openshift/okd@sha256:02c01aaaed9d292376e7808b6365ed38c34856e2a13c99718b>
Sep 01 18:08:12 bootstrap-test podman[302173]: 2021-09-01 18:08:12.044988448 +0000 UTC m=+0.103537973 container attach b227a0af15e629704c2c6c1d9c013a3877222372c651b345af8ce92e3bd11733 (image=quay.io/openshift/okd@sha256:02c01aaaed9d292376e7808b6365ed38c34856e2a13c99718>
Sep 01 18:08:12 bootstrap-test systemd[1]: libpod-b227a0af15e629704c2c6c1d9c013a3877222372c651b345af8ce92e3bd11733.scope: Deactivated successfully.
Sep 01 18:08:12 bootstrap-test podman[302173]: 2021-09-01 18:08:12.10129015 +0000 UTC m=+0.159839685 container died b227a0af15e629704c2c6c1d9c013a3877222372c651b345af8ce92e3bd11733 (image=quay.io/openshift/okd@sha256:02c01aaaed9d292376e7808b6365ed38c34856e2a13c99718b54>
Sep 01 18:08:12 bootstrap-test podman[302173]: 2021-09-01 18:08:12.119485253 +0000 UTC m=+0.178034778 container remove b227a0af15e629704c2c6c1d9c013a3877222372c651b345af8ce92e3bd11733 (image=quay.io/openshift/okd@sha256:02c01aaaed9d292376e7808b6365ed38c34856e2a13c99718>
Sep 01 18:08:12 bootstrap-test systemd[1]: kubelet.service: Found left-over process 302210 (conmon) in control group while starting unit. Ignoring.
Sep 01 18:08:12 bootstrap-test systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
Sep 01 18:08:12 bootstrap-test systemd[302257]: kubelet.service: Failed to locate executable /usr/bin/hyperkube: No such file or directory
Sep 01 18:08:12 bootstrap-test systemd[302257]: kubelet.service: Failed at step EXEC spawning /usr/bin/hyperkube: No such file or directory
Sep 01 18:08:12 bootstrap-test systemd[1]: kubelet.service: Main process exited, code=exited, status=203/EXEC
Sep 01 18:08:12 bootstrap-test systemd[1]: kubelet.service: Failed with result 'exit-code'.
Sep 01 18:08:12 bootstrap-test systemd[1]: Failed to start Kubernetes Kubelet.

Version
OKD version: 4.7.0-0.okd-2021-03-28-152009
FCOS version: Fedora CoreOS 34.20210626.3.1
Attempting UPI install for the cluster on IBM Cloud VPC

How reproducible
100% reproducible

Log bundle
openshift_install.log

vrutkovs · 2021-09-02T08:03:41Z

vrutkovs
Sep 2, 2021
Maintainer

Please attach log bundle

0 replies

tanmoydutta · 2021-09-02T09:24:08Z

tanmoydutta
Sep 2, 2021
Author

ctr.log.txt.gz
Log generation fails. Manually generated the following -

[core@bootstrap-test ~]$ journalctl -b -f -u bootkube.service
-- Journal begins at Wed 2021-09-01 10:37:54 UTC. --
Sep 01 11:21:36 bootstrap-test systemd[1]: Dependency failed for Bootstrap a Kubernetes cluster.
Sep 01 11:21:36 bootstrap-test systemd[1]: bootkube.service: Job bootkube.service/start failed with result 'dependency'.

[core@bootstrap-test ~]$ journalctl -b -f -u kubelet.service -u crio.service
-- Journal begins at Wed 2021-09-01 10:37:54 UTC. --
Sep 02 09:15:35 bootstrap-test podman[974405]: 2021-09-02 09:15:35.103862505 +0000 UTC m=+0.162732690 container remove b0e6f30c71d1a8df5beea7418faf66889a5249c1b0e7bbc0aa956676599bbe74 (image=quay.io/openshift/okd@sha256:02c01aaaed9d292376e7808b6365ed38c34856e2a13c99718b54a12958f4e6da, name=elated_solomon, io.openshift.release=4.7.0-0.okd-2021-03-28-152009, io.openshift.release.base-image-digest=sha256:69707c5f0c5285128fd7649e2d262fca8b0855dfcd7b3c108e5436290330d4df)
Sep 02 09:15:35 bootstrap-test systemd[1]: kubelet.service: Found left-over process 974442 (conmon) in control group while starting unit. Ignoring.
Sep 02 09:15:35 bootstrap-test systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
Sep 02 09:15:35 bootstrap-test systemd[1]: kubelet.service: Found left-over process 974454 (podman) in control group while starting unit. Ignoring.
Sep 02 09:15:35 bootstrap-test systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
Sep 02 09:15:35 bootstrap-test systemd[974489]: kubelet.service: Failed to locate executable /usr/bin/hyperkube: No such file or directory
Sep 02 09:15:35 bootstrap-test systemd[974489]: kubelet.service: Failed at step EXEC spawning /usr/bin/hyperkube: No such file or directory
Sep 02 09:15:35 bootstrap-test systemd[1]: kubelet.service: Main process exited, code=exited, status=203/EXEC
Sep 02 09:15:35 bootstrap-test systemd[1]: kubelet.service: Failed with result 'exit-code'.
Sep 02 09:15:35 bootstrap-test systemd[1]: Failed to start Kubernetes Kubelet.

1 reply

vrutkovs Sep 2, 2021
Maintainer

Check release-image and release-image-download systemd service logs - if your machine doesn't have kubelet binary it means it didn't pivot itself in OKD ostree commit

tanmoydutta · 2021-09-02T10:16:56Z

tanmoydutta
Sep 2, 2021
Author

[core@bootstrap-test ~]$ journalctl -b -f -u release-image
-- Journal begins at Wed 2021-09-01 10:37:54 UTC. --
Sep 01 11:21:32 bootstrap-test systemd[1]: Starting Download the OpenShift Release Image...
Sep 01 11:21:32 bootstrap-test release-image-download.sh[764]: Pulling quay.io/openshift/okd@sha256:02c01aaaed9d292376e7808b6365ed38c34856e2a13c99718b54a12958f4e6da...
Sep 01 11:21:34 bootstrap-test podman[769]: 2021-09-01 11:21:34.087209148 +0000 UTC m=+1.571760235 system refresh
Sep 01 11:21:34 bootstrap-test podman[769]: 2021-09-01 11:21:34.087947075 +0000 UTC m=+1.572498150 image pull quay.io/openshift/okd@sha256:02c01aaaed9d292376e7808b6365ed38c34856e2a13c99718b54a12958f4e6da
Sep 01 11:21:35 bootstrap-test release-image-download.sh[769]: 3b229e39bc869ea76bc4493ec494610fd132259b2f98fab86e6fc9ae895c9a21
Sep 01 11:21:35 bootstrap-test systemd[1]: Finished Download the OpenShift Release Image.

[core@bootstrap-test ~]$ journalctl -b -f -u release-image-download
-- Journal begins at Wed 2021-09-01 10:37:54 UTC. --

4 replies

vrutkovs Sep 2, 2021
Maintainer

Are you sure you're using OKD installer and not one from OCP?

tanmoydutta Sep 2, 2021
Author

Yes, this is the version output of the installer -

% ./openshift-install version
./openshift-install 4.7.0-0.okd-2021-03-28-152009
built from commit cf35dc04aedf7a80bea5470358025cbe91d232c9
release image quay.io/openshift/okd@sha256:02c01aaaed9d292376e7808b6365ed38c34856e2a13c99718b54a12958f4e6da

Ignition config was created using the same.

vrutkovs Sep 2, 2021
Maintainer

Check bootstrap.ign - it must have release-image-download service

tanmoydutta Sep 2, 2021
Author

Yes I see two occurrences of that. e.g. -
{"overwrite":true,"path":"/usr/local/bin/release-image-download.sh","user":{"name":"root"},"contents":{"source":"data:text/plain;charset=utf-8...

and another -

ExecStart=/usr/local/bin/release-image-download.sh

tanmoydutta · 2021-09-02T13:50:22Z

tanmoydutta
Sep 2, 2021
Author

Boot.log
Here are some more info from boot for the bootstrap node (See attached file)

2 replies

vrutkovs Sep 2, 2021
Maintainer

error copying from container: error during bulk transfer for copier.request{Request:"GET", Root:"/", preservedRoot:"/var/lib/containers/storage/overlay/7a128f642baf45e650e6e1cad241b864bffd15fefc8>

Is this a libvirt install?

tanmoydutta Sep 2, 2021
Author

No, VSI provisioning on IBM Cloud using FCOS qcow2 image

tanmoydutta · 2021-09-06T16:03:39Z

tanmoydutta
Sep 6, 2021
Author

Any update on this ?

5 replies

wupeng606 Mar 13, 2022

I'm facing this error in bootstrap node of OKD 4.9.
ENV:
vSphere 7.0.2,
Restricted network with Private registry Harbor,
FCOS 35.20220213.3.0

pkqsun Jul 22, 2022

Facing same issue of using:

openshift-install 4.10.0-0.okd-2022-07-09-073606
built from commit d63a12ba0ec33d492093a8fc0e268a01a075f5da
release image quay.io/openshift/okd@sha256:5a748561c829c23c7413b0fe0ec8f97b4eabf5cfd1934b9e93a8a5df3f59738d
release architecture amd64
fedora-coreos-35.20220327.3.0-openstack.x86_64.qcow2

Jul 22 16:24:58 okd410-6jfbz-bootstrap podman[30484]: 2022-07-22 16:24:58.719643717 +0000 UTC m=+0.263650635 container died 508e873e676695e66890a75b9e1978df00733776b95438a2334d89cd0258acca (image=quay.io/openshift/okd@sha256:5a748561c829c23c7413b0fe0ec8f97b4eabf5cfd1934b9e93a8a5df3f59738d, name=vibrant_cerf)
Jul 22 16:24:58 okd410-6jfbz-bootstrap podman[30484]: 2022-07-22 16:24:58.782347878 +0000 UTC m=+0.326354796 container remove 508e873e676695e66890a75b9e1978df00733776b95438a2334d89cd0258acca (image=quay.io/openshift/okd@sha256:5a748561c829c23c7413b0fe0ec8f97b4eabf5cfd1934b9e93a8a5df3f59738d, name=vibrant_cerf, io.openshift.release.base-image-digest=sha256:6c1c967d70be9cb1f03ac612bbe0c0a564302e9d0ae16f857a33a298ca55c94a, io.openshift.release=4.10.0-0.okd-2022-07-09-073606)
Jul 22 16:24:58 okd410-6jfbz-bootstrap kubelet.sh[30592]: /usr/local/bin/kubelet.sh: line 7: /usr/bin/hyperkube: No such file or directory
Jul 22 16:24:58 okd410-6jfbz-bootstrap systemd[1]: kubelet.service: Main process exited, code=exited, status=127/n/a
░░ Subject: Unit process exited
░░ Defined-By: systemd
░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
░░ 
░░ An ExecStart= process belonging to unit kubelet.service has exited.
░░ 
░░ The process' exit code is 'exited' and its exit status is 127.
Jul 22 16:24:58 okd410-nuke-6jfbz-bootstrap systemd[1]: kubelet.service: Failed with result 'exit-code'.
░░ Subject: Unit failed
░░ Defined-By: systemd
░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
░░ 
░░ The unit kubelet.service has entered the 'failed' state with result 'exit-code'.
Jul 22 16:24:58 okd410-nuke-6jfbz-bootstrap systemd[1]: Failed to start Kubernetes Kubelet.
░░ Subject: A start job for unit kubelet.service has failed
░░ Defined-By: systemd
░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel

yaroze Mar 15, 2023

/usr/local/bin/kubelet.sh: line 7: /usr/bin/hyperkube: No such file or directory

Getting exactly the same.

openshift-install 4.12.0-0.okd-2023-03-05-022504
built from commit 7c2530226516a12c37f10bc14e070f66c0f27930
release image quay.io/openshift/okd@sha256:fef2d0803e6ee6ec36d1979a0b847441580fbd9ee3ad583c97edb7ff811ee3b6
☝️ I might be looking at the wrong place but I saw that image doesn't have /usr/bin/hyperkube
release architecture amd64
fedora-coreos-37.20230218.3.0-live.x86_64.iso

Should I try a different build?

rbuzatu90 Mar 29, 2023

@tanmoydutta did you manage to have it working ?

tandutta Mar 29, 2023

@rbuzatu90 We did not solve it in the end as we also could not find any solution nor support. We just moved to another hyperscaler.

rbuzatu90 · 2023-03-29T01:05:46Z

rbuzatu90
Mar 29, 2023

I'm also having this exact issue on 4.10. Anyone have found the fix ?

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

kubelet.service: Failed at step EXEC spawning /usr/bin/hyperkube: No such file or directory #849

{{title}}

Replies: 6 comments 12 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

kubelet.service: Failed at step EXEC spawning /usr/bin/hyperkube: No such file or directory #849

tanmoydutta Sep 2, 2021

Replies: 6 comments · 12 replies

vrutkovs Sep 2, 2021 Maintainer

tanmoydutta Sep 2, 2021 Author

vrutkovs Sep 2, 2021 Maintainer

tanmoydutta Sep 2, 2021 Author

vrutkovs Sep 2, 2021 Maintainer

tanmoydutta Sep 2, 2021 Author

vrutkovs Sep 2, 2021 Maintainer

tanmoydutta Sep 2, 2021 Author

tanmoydutta Sep 2, 2021 Author

vrutkovs Sep 2, 2021 Maintainer

tanmoydutta Sep 2, 2021 Author

tanmoydutta Sep 6, 2021 Author

wupeng606 Mar 13, 2022

pkqsun Jul 22, 2022

yaroze Mar 15, 2023

rbuzatu90 Mar 29, 2023

tandutta Mar 29, 2023

rbuzatu90 Mar 29, 2023

tanmoydutta
Sep 2, 2021

Replies: 6 comments 12 replies

vrutkovs
Sep 2, 2021
Maintainer

tanmoydutta
Sep 2, 2021
Author

vrutkovs Sep 2, 2021
Maintainer

tanmoydutta
Sep 2, 2021
Author

vrutkovs Sep 2, 2021
Maintainer

tanmoydutta Sep 2, 2021
Author

vrutkovs Sep 2, 2021
Maintainer

tanmoydutta Sep 2, 2021
Author

tanmoydutta
Sep 2, 2021
Author

vrutkovs Sep 2, 2021
Maintainer

tanmoydutta Sep 2, 2021
Author

tanmoydutta
Sep 6, 2021
Author

rbuzatu90
Mar 29, 2023