eBPF with IPv6 single stack in manifest mode fails without nodes having IPv4

[abasitt]

I am install eBPF mode using manifest install but I am facing issue with kube-api server not accessible. I initially thought it may be a kernel issue because I was trying on a slightly older kernel of Rocky8 4.18.0-372.26.1.el8_6.x86_64. I upgraded to latest kernel of ubuntu 24.04 6.8.0-48-generic but still the same behavior.

##cluster install
I am using k3s single node IPv6 single stack cluster and I installed it using below.

NODEV6_ADDR=ipv6address
curl -sfL https://get.k3s.io | K3S_KUBECONFIG_MODE="644" INSTALL_K3S_EXEC="--flannel-backend=none --cluster-cidr=fd00::/48 --service-cidr=fd01::/112 --disable-network-policy --disable=traefik --disable=cni --disable=metrics-server --disable=local-storage --disable=servicelb --disable-kube-proxy --kubelet-arg=address=:: --kubelet-arg=node-ip=:: --kube-apiserver-arg=bind-address=:: --node-ip=$NODEV6_ADDR" sh -

Installed calico eBPF manifest by modifying ipam and enabled IPv6. The installation went through and for a moment everything will come up like below correctly.

k get pods -A
NAMESPACE     NAME                                       READY   STATUS    RESTARTS   AGE
kube-system   calico-kube-controllers-65dcc554ff-g85vs   1/1     Running   0          13s
kube-system   calico-node-9rr7g                          1/1     Running   0          13s
kube-system   coredns-7b98449c4-7t47h                    1/1     Running   0          13s

But shortly the status changes to below with error that kube-api server is not reachable.

k get pods -A
NAMESPACE     NAME                                       READY   STATUS    RESTARTS      AGE
kube-system   calico-kube-controllers-65dcc554ff-g85vs   0/1     Running   4 (37s ago)   4m18s
kube-system   calico-node-9rr7g                          1/1     Running   0             4m18s
kube-system   coredns-7b98449c4-7t47h                    0/1     Running   2 (76s ago)   4m18s

2024-11-05 02:05:23.481 [ERROR][1] client.go 287: Error getting cluster information config ClusterInformation="default" error=Get "https://[fd01::1]:443/apis/crd.projectcalico.org/v1/clusterinformations/default": dial tcp [fd01::1]:443: connect: no route to host

 k get felixconfiguration -oyaml
apiVersion: v1
items:
- apiVersion: crd.projectcalico.org/v1
  kind: FelixConfiguration
  metadata:
    annotations:
      projectcalico.org/metadata: '{"creationTimestamp":"2024-11-02T10:09:22Z"}'
    creationTimestamp: "2024-11-02T10:09:22Z"
    generation: 1
    name: default
    resourceVersion: "694"
    uid: fe2f57d7-31fe-47e4-b1f3-f8778cbfa7cd
  spec:
    bpfConnectTimeLoadBalancing: TCP
    bpfHostNetworkedNATWithoutCTLB: Enabled
    bpfLogLevel: ""
    floatingIPs: Disabled
    logSeverityScreen: Info
    reportingInterval: 0s
kind: List
metadata:
  resourceVersion: ""

Attaching node diags. Let me know if you need more details.
diags-20241105_015927.tar.gz

BTW this works if I install calico operator so wondering what's not working in manifest mode ?

Your Environment

 k version
Client Version: v1.30.6+k3s1
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.30.6+k3s1
k8s@testk3s:~$ k calico version
Client Version:    v3.28.2
Git commit:        9a96ee39f
Cluster Version:   v3.28.2
Cluster Type:      k8s,bgp,kdd

[abasitt]

I was comparing the ebpf maps and seems like the issue is related to IntfIPv4:net.IP(nil). The moment I annotate nodes with ipv4 address projectcalico.org/IPv4Address.. Things starting working. IMO this shouldn't be required for a single stack ipv6 cluster

[tomastigera]

The diags bundle does not contain calico-node logs. It would be very helpful to get those.

Did you tell calico-node where to find the api server? #9441

Have you tried operator? Did it work?

[abasitt]

@tomastigera thank you for looking in to this

Yes I provided the api server address because I am skipping kube-proxy.

Yes the same setup works with operator. The difference is that in manifest mode, I am passing env IP: none and no IPv4 autodetection.

Attaching the calico-node logs. let me know if you need more info.
calico-node.log