HCCM Controller don't set metadata to the Nodes at all (nodes are connected, using internal network)

[DmitrySkrotski]

TL;DR

I set up cluster on hcloud (controlplanes are cloud, workers are robots + cloud) and I am expecting that the HCCM will populate node objects with metadata.

Expected behavior

HCCM will populate information about the server to the Node object as described here hccm

Observed behavior

Nodes joined using join.yml file

kind: JoinConfiguration
discovery:
  bootstrapToken:
    apiServerEndpoint: 10.195.4.250:6443
    caCertHashes:
    - sha256:xxxxxxxxx
    token: xxxxxxx.xxxxxxxxxxxxxx
nodeRegistration:
  name: prod-k8s-metall01
  criSocket: unix:///var/run/containerd/containerd.sock
  kubeletExtraArgs:
    node-ip: 10.195.10.2

HCCM is setup using helm

helm install hcloud-controller-manager hcloud/hcloud-cloud-controller-manager --set robot.enabled=true --set netwo
rk.enabled=false -n kube-system

secret hcloud persists

HCCM logs:

pod initialization log:

Minimal working example

No response

Log output

plain text
I0731 08:29:07.120000       1 controllermanager.go:169] Version: v0.0.0-master+$Format:%H$
I0731 08:29:07.139351       1 secure_serving.go:213] Serving securely on [::]:10258
I0731 08:29:07.139953       1 tlsconfig.go:240] "Starting DynamicServingCertificateController"
I0731 08:29:07.140307       1 requestheader_controller.go:169] Starting RequestHeaderAuthRequestController
I0731 08:29:07.140913       1 shared_informer.go:313] Waiting for caches to sync for RequestHeaderAuthRequestController
I0731 08:29:07.140361       1 configmap_cafile_content.go:202] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::client-ca-file"
I0731 08:29:07.141232       1 shared_informer.go:313] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::client-ca-file
I0731 08:29:07.140387       1 configmap_cafile_content.go:202] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file"
I0731 08:29:07.142642       1 shared_informer.go:313] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
I0731 08:29:07.172363       1 controllermanager.go:338] Started "cloud-node-controller"
I0731 08:29:07.172570       1 node_controller.go:164] Sending events to api server.
I0731 08:29:07.173069       1 node_controller.go:173] Waiting for informer caches to sync
I0731 08:29:07.183862       1 controllermanager.go:338] Started "cloud-node-lifecycle-controller"
I0731 08:29:07.185314       1 controllermanager.go:338] Started "service-lb-controller"
W0731 08:29:07.185470       1 core.go:111] --configure-cloud-routes is set, but cloud provider does not support routes. Will not configure cloud provider routes.
W0731 08:29:07.185532       1 controllermanager.go:326] Skipping "node-route-controller"
I0731 08:29:07.187368       1 node_lifecycle_controller.go:113] Sending events to api server
I0731 08:29:07.189552       1 controller.go:231] Starting service controller
I0731 08:29:07.189768       1 shared_informer.go:313] Waiting for caches to sync for service
I0731 08:29:07.242063       1 shared_informer.go:320] Caches are synced for RequestHeaderAuthRequestController
I0731 08:29:07.242666       1 shared_informer.go:320] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::client-ca-file
I0731 08:29:07.243406       1 shared_informer.go:320] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
I0731 08:29:07.290553       1 shared_informer.go:320] Caches are synced for service
E0731 08:29:07.446872       1 node_controller.go:389] Failed to update node addresses for node "prod-k8s-controlplane01": failed to get node address from cloud provider that matches ip: 10.195.4.1
E0731 08:29:07.706931       1 node_controller.go:389] Failed to update node addresses for node "prod-k8s-metall01": failed to get node address from cloud provider that matches ip: 10.195.10.2
E0731 08:29:07.831326       1 node_controller.go:389] Failed to update node addresses for node "prod-k8s-metall04": failed to get node address from cloud provider that matches ip: 10.195.10.5
E0731 08:29:08.638471       1 node_controller.go:389] Failed to update node addresses for node "prod-k8s-controlplane03": failed to get node address from cloud provider that matches ip: 10.195.4.3
E0731 08:29:08.721808       1 node_controller.go:389] Failed to update node addresses for node "prod-k8s-metall02": failed to get node address from cloud provider that matches ip: 10.195.10.3
E0731 08:29:08.816775       1 node_controller.go:389] Failed to update node addresses for node "prod-k8s-metall03": failed to get node address from cloud provider that matches ip: 10.195.10.4
E0731 08:29:08.922346       1 node_controller.go:389] Failed to update node addresses for node "prod-k8s-metall05": failed to get node address from cloud provider that matches ip: 10.195.10.6
E0731 08:29:09.079807       1 node_controller.go:389] Failed to update node addresses for node "prod-k8s-rabbit01": failed to get node address from cloud provider that matches ip: 10.195.4.4
E0731 08:29:09.261077       1 node_controller.go:389] Failed to update node addresses for node "prod-k8s-rabbit02": failed to get node address from cloud provider that matches ip: 10.195.4.5
E0731 08:34:10.240322       1 node_controller.go:389] Failed to update node addresses for node "prod-k8s-rabbit02": failed to get node address from cloud provider that matches ip: 10.195.4.5

Additional information

No response

[RoodeyMental]

I am facing the same problem in a talos kubernetes. I use hccm also without letting it manage the routing and cilium as my cni. I use a private network for control plane traffic and planned on using it for ingress as well. The cloud-provider: external kubeletExtraArgs is set in my machine config and new nodes are tainted with Uninitialized as expected.

[RoodeyMental]

I stumbled upon this issue https://github.com/hetznercloud/hcloud-cloud-controller-manager/issues/620 which helped me resolve it. The solution was to set those environment variables in my manifest:

            - name: HCLOUD_NETWORK
              valueFrom:
                secretKeyRef:
                  name: hcloud
                  key: network
            - name: HCLOUD_NETWORK_ROUTES_ENABLED
              value: "false"

I also think you did not set the cloud-provider: external kubeletExtraArgs.
EDIT: Don't forget to add the network name or id to the secret as described in the docs:
kubectl -n kube-system create secret generic hcloud --from-literal=token=<hcloud API token> --from-literal=network=<hcloud Network_ID_or_Name>