Can't reach a pod IP address on a Worker Node from another Worker Node

[wayne-cec]

After upgrade to 4.9.0-0.okd-2022-02-12-140851, it shows "Error while reconciling 4.9.0-0.okd-2022-02-12-140851: the cluster operator machine-config has not yet successfully rolled out". As result, I can't reach a pod IP address on a worker node from another worker host. In Master node, network is normally. I added 2 new worker nodes, but new worker nodes have the same issue.
Who can help me?

[vrutkovs]

Please attach (or upload to the public file sharing service) must-gather archive

[wayne-cec]

@vrutkovs
Upload to Google drive:
must-gather.tar.gz

Thanks

[vrutkovs]

MCP still lists rendered-worker-57ecd09a7d6b08b6917e18bf1db88194 as required for workers, but all nodes seem to have applied rendered-worker-66a67af19ab9eb32d439456c514432b5 (which is older, from March).

Apparently, the controller never applied it to the worker nodes - but the controller was restarted on May 5 and we have no logs on why this happened. It also throws a confusing error:
2022-05-05T11:22:38.747975498+08:00 E0505 03:22:38.747926 1 template_controller.go:189] couldn't get ControllerConfig on dependency callback &errors.StatusError{ErrStatus:v1.Status{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ListMeta:v1.ListMeta{SelfLink:"", ResourceVersion:"", Continue:"", RemainingItemCount:(*int64)(nil)}, Status:"Failure", Message:"controllerconfig.machineconfiguration.openshift.io \"machine-config-controller\" not found", Reason:"NotFound", Details:(*v1.StatusDetails)(0xc0002d5380), Code:404}}

I suggest restarting machine-config-controller and hopefully it would start applying new config to the nodes. Also, compute-2 has no-network taint so you may want to look into it

[wayne-cec]

@vrutkovs
Thank you for your finding. I forced MCP update to new version as below step:
https://gist.github.com/ikurni/067ea20449588008de4e19a07f0de5f4
The status is strange now.
I have 5 worker nodes(compute-0 ~ compute-4), 2 of them(compute-0 and compute-2) worked well. Other 3 nodes have network issue. From these 3 nodes, I can reach Pod IP on each of them, but can't reach pod IP on compute-0 and compute-2 and 3 control-plane node.

Even I upgraded my cluster to 4.10.0-0.okd-2022-03-07-131213, the issue is still there. So I make taint on these 3 node as you found "no-network".

I tried to restart machine-config-controller, it doesn't work.

[vrutkovs]

but can't reach pod IP on compute-0 and compute-2 and 3 control-plane node

Check steps at https://github.com/openshift/ovn-kubernetes/blob/master/docs/debugging.md

[wayne-cec]

Followed the debug steps

Sanity check cross host ping

From compute-2, I can't ping the ovn-k8s interface ip in compute-1. Loos like there are 2 geneve network, compute-0 and compute-2 and 3 control nodes live in one geneve network, the other 3 compute nodes live in another geneve network.

But, when I try to run ovnkube-trace from Pod in compute-1 to Pod in compute-2, it shows successful.

[wayne-cec]

@vrutkovs
This network issue is resolved now. All VMs lived in 2 Hosts before, I migrated them to the same host. Then all work well. I think maybe it's the compatibility issue between Fedora CoreOS 35 and vSphere ESXi 7.0U2.
Thanks for your help.

[rvanderp3]

@wayne-cec what hw version are your VMs? This sounds like a prior issue with offloading on the vmxnet3 adapter driver.

[wayne-cec]

@rvanderp3
One is hw 15, one is hw 17, others are hw 14. Any docs for this issue?

[youssefbouzaine]

i have the same issue i moved all the nodes into the same host but still don't work (I actually added a new worker node and it's this one that can't reach pods on other nodes)