image pull: net/http: TLS handshake timeout on **one** node

[ibotty]

Describe the bug
I get an imagePullBackoff starting some pods:

  Warning  Failed          2m28s (x2 over 2m53s)  kubelet            Failed to pull image "image-registry.openshift-image-registry.svc:5000/website/website-with-legacy@sha256:0e6bff4ef387933
d46592a9a6a24831c526318fd20ecbde182c076bf9b798287": rpc error: code = Unknown desc = Get "https://image-registry.openshift-image-registry.svc:5000/openshift/token?account=serviceaccount&scop
e=repository%3Awebsite%2Fwebsite-with-legacy%3Apull": net/http: TLS handshake timeout

Indeed, I cannot pull the image from the node either:

[core@node-012 ~]$ podman pull image-registry.openshift-image-registry.svc:5000/website/website-with-legacy@sha256:0e6bff4ef387933d46592a9a6a24831c526318fd20ecbde182c076bf9b798287
Trying to pull image-registry.openshift-image-registry.svc:5000/website/website-with-legacy@sha256:0e6bff4ef387933d46592a9a6a24831c526318fd20ecbde182c076bf9b798287...
Error: initializing source docker://image-registry.openshift-image-registry.svc:5000/website/website-with-legacy@sha256:0e6bff4ef387933d46592a9a6a24831c526318fd20ecbde182c076bf9b798287: pinging container registry image-registry.openshift-image-registry.svc:5000: Get "https://image-registry.openshift-image-registry.svc:5000/v2/": net/http: TLS handshake timeout

I can curl https://image-registry.openshift-image-registry.svc:5000/ -k though, but not with any path appended to the URL:

[core@node-012 ~]$ curl https://image-registry.openshift-image-registry.svc:5000/ -k
[core@node-012 ~]$ curl https://image-registry.openshift-image-registry.svc:5000/oue -k
[hangs]

Version
UPI-baremetal installation.

Client Version: 4.10.0-0.okd-2022-04-23-131357
Server Version: 4.11.0-0.okd-2022-08-20-022919
Kubernetes Version: v1.24.0-2368+b62823b40c2cb1-dirty

How reproducible
rare.

Log bundle
https://send.vis.ee/download/ec1a700f746bf112/#QrhxFySzWuJJx6c8B_tE_w

[vrutkovs]

Seeing a lot of

2022-09-16T08:13:44.960277215Z 2022/09/16 08:13:44 http2: server: error reading preface from client 172.22.4.1:36342: read tcp 172.23.4.45:5000->172.22.4.1:36342: read: connection reset by peer

Which means it might be an external network problem. Perhaps its some firewall preventing it from connecting to image-registry?

[ibotty]

These nodes are all on the same network without any firewall in between (only a switch). Sometime later the node got unresponsive (to ssh and the kubelet) and we had to reset it. Might there be a resource leak somewhere that can trigger it? Maybe the SDN is not working anymore.

The built-in dashboards did look quiet ordinary unfortunately. If you have a metric that might be interesting we can get a graph.

[ibotty]

Any other Idea? We are facing massive problems with this cluster: not being able to build images, etc.

[ibotty]

The problem was openshift-sdn. Or at least it disappeared when the cluster switched to ovn-kubernetes.