the new worker nodes are stuck with " Waiting for probes: calico, kubelet " error on the RKE2 Management Cluster.

[DipindasThayyil]

Noticed below message from " systemctl status rke2-agent.service & journalctl -f -u rke2-agent.service " log and rke2-agent.service
service were in " Active: activating (start) " state on the worker nodes end.

level=error msg="failed to get CA certs: Get "https://127.0.0.1:6444/cacerts\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"

Additionally rke2-agent.service services are not allowing to start the " calico-node-fk28z " container or vxlan.calico interface configuration in RKE Worker node end.

~# nc -vz 127.0.0.1 6444
Connection to 127.0.0.1 6444 port [tcp/sge-qmaster] succeeded!

To fix the above error I've tried with restarting agent service ,restarted the nodes , re-register the node again after the " rke2-uninstall.sh " command execution , however there is no luck.

Cluster Detail -
K8s version - v1.28.10+rke2r1
Rancher v2.8.5

Are they any suggestion here to check further on this

[brandond]

level=error msg="failed to get CA certs: Get "[https://127.0.0.1:6444/cacerts\](https://127.0.0.1:6444/cacerts%5C)": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"

Connections to 127.0.0.1:6444 on the agent are proxied to the server via a local loadbalancer. Make sure that you can curl the server address (https://SERVERNAME:9345/cacerts) from the agent. I suspect you do not have the correct ports open between nodes.

[DipindasThayyil]

thanks , look like the new nodes having connectivity issue and I've removed those nodes for time being.