How to run CoreDNS with cilium and without kube-proxy ?

[mulder999]

I am using cilium without kube-proxy as explained here.

CoreDNS remains trapped in a never ending waiting condition:

[INFO] plugin/kubernetes: pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:243: failed to list *v1.Namespace: Get "https://10.96.0.1:443/api/v1/namespaces?limit=500&resourceVersion=0": dial tcp 10.96.0.1:443: i/o timeout
[ERROR] plugin/kubernetes: Unhandled Error
[INFO] plugin/ready: Still waiting on: "kubernetes"
[INFO] plugin/kubernetes: pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:243: failed to list *v1.EndpointSlice: Get "https://10.96.0.1:443/apis/discovery.k8s.io/v1/endpointslices?limit=500&resourceVersion=0": dial tcp 10.96.0.1:443: i/o timeout
[ERROR] plugin/kubernetes: Unhandled Error
[INFO] plugin/ready: Still waiting on: "kubernetes"
[INFO] plugin/kubernetes: pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:243: failed to list *v1.Service: Get "https://10.96.0.1:443/api/v1/services?limit=500&resourceVersion=0": dial tcp 10.96.0.1:443: i/o timeout
[ERROR] plugin/kubernetes: Unhandled Error
[INFO] plugin/ready: Still waiting on: "kubernetes"

kubectl get svc --all-namespaces 
NAMESPACE     NAME           TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                  AGE
default       kubernetes     ClusterIP   10.96.0.1        <none>        443/TCP                  55m
default       talos          ClusterIP   10.99.16.27      <none>        50000/TCP                13m
kube-system   cilium-envoy   ClusterIP   None             <none>        9964/TCP                 53m
kube-system   hubble-peer    ClusterIP   10.105.234.105   <none>        443/TCP                  53m
kube-system   kube-dns       ClusterIP   10.96.0.10       <none>        53/UDP,53/TCP,9153/TCP   54m

kubectl -n kube-system exec -it $(kubectl -n kube-system get pods -l k8s-app=cilium -o jsonpath='{.items[0].metadata.name}') -- cilium status
Defaulted container "cilium-agent" out of: cilium-agent, config (init), apply-sysctl-overwrites (init), mount-bpf-fs (init), clean-cilium-state (init), install-cni-binaries (init)
KVStore:                 Ok   Disabled
Kubernetes:              Ok   1.32 (v1.32.1) [linux/amd64]
Kubernetes APIs:         ["EndpointSliceOrEndpoint", "cilium/v2::CiliumClusterwideNetworkPolicy", "cilium/v2::CiliumEndpoint", "cilium/v2::CiliumNetworkPolicy", "cilium/v2::CiliumNode", "cilium/v2alpha1::CiliumCIDRGroup", "core/v1::Namespace", "core/v1::Pods", "core/v1::Service", "networking.k8s.io/v1::NetworkPolicy"]
KubeProxyReplacement:    True   [eth0     10.39.0.189 2001:470:XXXX 2001:470:XXXX fe80::215:XXXX (Direct Routing), kubespan  fd62:f53d:XXXX]
Host firewall:           Disabled
SRv6:                    Disabled
CNI Chaining:            none
CNI Config file:         successfully wrote CNI configuration file to /host/etc/cni/net.d/05-cilium.conflist
Cilium:                  Ok   1.16.6 (v1.16.6-9e9f0989)
NodeMonitor:             Listening for events on 4 CPUs with 64x4096 of shared memory
Cilium health daemon:    Ok   
IPAM:                    IPv4: 5/254 allocated from 10.244.0.0/24, 
IPv4 BIG TCP:            Disabled
IPv6 BIG TCP:            Disabled
BandwidthManager:        Disabled
Routing:                 Network: Tunnel [vxlan]   Host: Legacy
Attach Mode:             TCX
Device Mode:             veth
Masquerading:            IPTables [IPv4: Enabled, IPv6: Disabled]
Controller Status:       35/35 healthy
Proxy Status:            OK, ip 10.244.0.234, 0 redirects active on ports 10000-20000, Envoy: external
Global Identity Range:   min 256, max 65535
Hubble:                  Ok              Current/Max Flows: 4095/4095 (100.00%), Flows/s: 10.07   Metrics: Disabled
Encryption:              Disabled        
Cluster health:          1/1 reachable   (2025-01-31T21:25:16Z)
Modules Health:          Stopped(0) Degraded(0) OK(47)

How to find what is preventing CoreDNS to connect ?

[smira]

Check your Cilium installation options, it should work out of the box.

In your case 10.96.0.10 not working is clearly a problem with your CNI, it has nothing to do with Talos itself.

[mulder999]

@smira
Thanks for your feedback! I actually experienced the exact same behavior with the default Flannel setup. Out of solutions, I then downgraded to Talos 1.9.0, CoreDNS worked fine there. I then upgraded back to 1.9.3, switched to Cilium, removed Flannel, and CoreDNS continued working. Even after a system reset, CoreDNS starts now in pure Cilium mode.

I have no idea what caused the issue or what ultimately fixed it. Why none of my countless resets did not had the same effect as the downgrade ultimately did...

In case that could help investigate the root cause, I have a second cluster on v1.9.2 with a totally different configuration that still has the same CoreDNS issue...

[smira]

If you remove CNIs on the fly, it's up to you to keep it correct. It's not recommened in general to change CNIs on a running cluster.

Either way, it works and it's fully tested on our side, including KubeSpan, and Cilium with kube-proxy replacement, so your issue is on your side, and it might be a misconfiguration.

[mulder999]

My goal is to stay with Cilium but after having this issue I had to find a solution, hence why I switched back to Flannel, just to check if Cilium was at fault or not...

Looking at my latest uncommited bits that could possibly have solved the problems are:

machine:
  sysctls:
    net.ipv4.conf.all.forwarding: 1
    net.ipv6.conf.all.forwarding: 1

  features:
    kubernetesTalosAPIAccess:
      enabled: true
      allowedRoles:
        - os:reader
      allowedKubernetesNamespaces:
        - kube-system

[smira]

None of that should have anything to do with your issue.

Strart from scratch, have proper machine configuration from the start, install Cilium, it works.

[mulder999]

As I mentioned, the previous issue is resolved, though I’m not sure what exactly fixed it. Thanks for your support!