duplicate Hubble DNS flows and ingress/egress labeling bug when DNS request lands on another Node #584
Labels
Comments
huntergregory
changed the title
huntergregory
changed the title
huntergregory
changed the title
|
This HTTP experiment shows that for TCP, we similarly have double the number of flows as there are packets, yet we specify different toolbox -> dep-http
toolbox <- dep-http
Additional Setup
apiVersion: apps/v1
kind: Deployment
metadata:
name: dep-http
spec:
replicas: 1
selector:
matchLabels:
pod: http
template:
metadata:
labels:
pod: http
spec:
containers:
- command:
- /agnhost
- serve-hostname
- --http=true
- --port
- "80"
image: k8s.gcr.io/e2e-test-images/agnhost:2.33
imagePullPolicy: IfNotPresent
name: cont-80-tcp
ports:
- containerPort: 80
name: serve-80-tcp
protocol: TCP
nodeSelector:
kubernetes.io/os: linux
vm: "1"
HTTP ExperimentPods, Nodes, and IPsNAME READY STATUS RESTARTS AGE IP NODE
dep-http-84f64997bd-j622q 1/1 Running 0 16m 10.224.1.166 aks-nodepool1-32351195-vmss000001
toolbox-pod 1/1 Running 0 44h 10.224.0.106 aks-nodepool1-32351195-vmss000000ExperimentApply these iptables rules to each node to count TCP packets between the two Pods: iptables-nft -I FORWARD -s 10.224.0.106 -d 10.224.1.166 --protocol tcp --dport 80
iptables-nft -I FORWARD -d 10.224.0.106 -s 10.224.1.166 --protocol tcp --sport 80Curl the http server: k exec toolbox-pod -- curl 10.224.1.166:80Before curling, start observing flows: $ hubble observe flows --protocol tcp -f | grep "toolbox-pod\|dep-http"
Aug 7 19:39:39.821: toolbox-pod:36600 (ID:1679) -> dep-http-84f64997bd-j622q:80 (ID:23273) to-stack FORWARDED (TCP Flags: SYN:true)
Aug 7 19:39:39.821: toolbox-pod:36600 (ID:1679) -> dep-http-84f64997bd-j622q:80 (ID:23273) to-network FORWARDED (TCP Flags: SYN:true)
Aug 7 19:39:39.823: toolbox-pod:36600 (ID:1679) <- dep-http-84f64997bd-j622q:80 (ID:23273) from-network FORWARDED (TCP Flags: SYN:true ACK:true)
Aug 7 19:39:39.823: toolbox-pod:36600 (ID:1679) <- dep-http-84f64997bd-j622q:80 (ID:23273) to-endpoint FORWARDED (TCP Flags: SYN:true ACK:true)
Aug 7 19:39:39.824: dep-http-84f64997bd-j622q:80 (ID:23273) <- toolbox-pod:36600 (ID:1679) to-stack FORWARDED (TCP Flags: ACK:true)
Aug 7 19:39:39.824: dep-http-84f64997bd-j622q:80 (ID:23273) <- toolbox-pod:36600 (ID:1679) to-network FORWARDED (TCP Flags: ACK:true)
Aug 7 19:39:39.831: dep-http-84f64997bd-j622q:80 (ID:23273) <- toolbox-pod:36600 (ID:1679) to-stack FORWARDED (TCP Flags: PSH:true ACK:true)
Aug 7 19:39:39.831: dep-http-84f64997bd-j622q:80 (ID:23273) <- toolbox-pod:36600 (ID:1679) to-network FORWARDED (TCP Flags: PSH:true ACK:true)
Aug 7 19:39:39.832: toolbox-pod:36600 (ID:1679) <- dep-http-84f64997bd-j622q:80 (ID:23273) from-network FORWARDED (TCP Flags: ACK:true)
Aug 7 19:39:39.832: toolbox-pod:36600 (ID:1679) <- dep-http-84f64997bd-j622q:80 (ID:23273) to-endpoint FORWARDED (TCP Flags: ACK:true)
Aug 7 19:39:39.832: toolbox-pod:36600 (ID:1679) <- dep-http-84f64997bd-j622q:80 (ID:23273) from-network FORWARDED (TCP Flags: PSH:true ACK:true)
Aug 7 19:39:39.832: toolbox-pod:36600 (ID:1679) <- dep-http-84f64997bd-j622q:80 (ID:23273) to-endpoint FORWARDED (TCP Flags: PSH:true ACK:true)
Aug 7 19:39:39.832: dep-http-84f64997bd-j622q:80 (ID:23273) <- toolbox-pod:36600 (ID:1679) to-stack FORWARDED (TCP Flags: ACK:true)
Aug 7 19:39:39.832: dep-http-84f64997bd-j622q:80 (ID:23273) <- toolbox-pod:36600 (ID:1679) to-network FORWARDED (TCP Flags: ACK:true)
Aug 7 19:39:39.844: dep-http-84f64997bd-j622q:80 (ID:23273) <- toolbox-pod:36600 (ID:1679) to-stack FORWARDED (TCP Flags: FIN:true ACK:true)
Aug 7 19:39:39.844: dep-http-84f64997bd-j622q:80 (ID:23273) <- toolbox-pod:36600 (ID:1679) to-network FORWARDED (TCP Flags: FIN:true ACK:true)
Aug 7 19:39:39.845: toolbox-pod:36600 (ID:1679) <- dep-http-84f64997bd-j622q:80 (ID:23273) from-network FORWARDED (TCP Flags: FIN:true ACK:true)
Aug 7 19:39:39.845: toolbox-pod:36600 (ID:1679) <- dep-http-84f64997bd-j622q:80 (ID:23273) to-endpoint FORWARDED (TCP Flags: FIN:true ACK:true)
Aug 7 19:39:39.845: dep-http-84f64997bd-j622q:80 (ID:23273) <- toolbox-pod:36600 (ID:1679) to-stack FORWARDED (TCP Flags: ACK:true)
Aug 7 19:39:39.845: dep-http-84f64997bd-j622q:80 (ID:23273) <- toolbox-pod:36600 (ID:1679) to-network FORWARDED (TCP Flags: ACK:true)
Aug 7 19:39:39.825: toolbox-pod:36600 (ID:1679) -> dep-http-84f64997bd-j622q:80 (ID:23273) from-network FORWARDED (TCP Flags: SYN:true)
Aug 7 19:39:39.825: toolbox-pod:36600 (ID:1679) -> dep-http-84f64997bd-j622q:80 (ID:23273) to-endpoint FORWARDED (TCP Flags: SYN:true)
Aug 7 19:39:39.825: toolbox-pod:36600 (ID:1679) <- dep-http-84f64997bd-j622q:80 (ID:23273) to-stack FORWARDED (TCP Flags: SYN:true ACK:true)
Aug 7 19:39:39.825: toolbox-pod:36600 (ID:1679) <- dep-http-84f64997bd-j622q:80 (ID:23273) to-network FORWARDED (TCP Flags: SYN:true ACK:true)
Aug 7 19:39:39.826: dep-http-84f64997bd-j622q:80 (ID:23273) <- toolbox-pod:36600 (ID:1679) from-network FORWARDED (TCP Flags: ACK:true)
Aug 7 19:39:39.826: dep-http-84f64997bd-j622q:80 (ID:23273) <- toolbox-pod:36600 (ID:1679) to-endpoint FORWARDED (TCP Flags: ACK:true)
Aug 7 19:39:39.833: dep-http-84f64997bd-j622q:80 (ID:23273) <- toolbox-pod:36600 (ID:1679) from-network FORWARDED (TCP Flags: PSH:true ACK:true)
Aug 7 19:39:39.834: dep-http-84f64997bd-j622q:80 (ID:23273) <- toolbox-pod:36600 (ID:1679) to-endpoint FORWARDED (TCP Flags: PSH:true ACK:true)
Aug 7 19:39:39.834: toolbox-pod:36600 (ID:1679) <- dep-http-84f64997bd-j622q:80 (ID:23273) to-stack FORWARDED (TCP Flags: ACK:true)
Aug 7 19:39:39.834: toolbox-pod:36600 (ID:1679) <- dep-http-84f64997bd-j622q:80 (ID:23273) to-network FORWARDED (TCP Flags: ACK:true)
Aug 7 19:39:39.834: dep-http-84f64997bd-j622q:80 (ID:23273) <- toolbox-pod:36600 (ID:1679) from-network FORWARDED (TCP Flags: ACK:true)
Aug 7 19:39:39.834: dep-http-84f64997bd-j622q:80 (ID:23273) <- toolbox-pod:36600 (ID:1679) to-endpoint FORWARDED (TCP Flags: ACK:true)
Aug 7 19:39:39.846: dep-http-84f64997bd-j622q:80 (ID:23273) <- toolbox-pod:36600 (ID:1679) from-network FORWARDED (TCP Flags: FIN:true ACK:true)
Aug 7 19:39:39.846: dep-http-84f64997bd-j622q:80 (ID:23273) <- toolbox-pod:36600 (ID:1679) to-endpoint FORWARDED (TCP Flags: FIN:true ACK:true)
Aug 7 19:39:39.846: toolbox-pod:36600 (ID:1679) <- dep-http-84f64997bd-j622q:80 (ID:23273) to-stack FORWARDED (TCP Flags: FIN:true ACK:true)
Aug 7 19:39:39.846: toolbox-pod:36600 (ID:1679) <- dep-http-84f64997bd-j622q:80 (ID:23273) to-network FORWARDED (TCP Flags: FIN:true ACK:true)
Aug 7 19:39:39.847: dep-http-84f64997bd-j622q:80 (ID:23273) <- toolbox-pod:36600 (ID:1679) from-network FORWARDED (TCP Flags: ACK:true)
Aug 7 19:39:39.847: dep-http-84f64997bd-j622q:80 (ID:23273) <- toolbox-pod:36600 (ID:1679) to-endpoint FORWARDED (TCP Flags: ACK:true)
Aug 7 19:39:39.834: toolbox-pod:36600 (ID:1679) <- dep-http-84f64997bd-j622q:80 (ID:23273) to-stack FORWARDED (TCP Flags: PSH:true ACK:true)
Aug 7 19:39:39.834: toolbox-pod:36600 (ID:1679) <- dep-http-84f64997bd-j622q:80 (ID:23273) to-network FORWARDED (TCP Flags: PSH:true ACK:true)The packets sent/received were: Node 0: root@aks-nodepool1-32351195-vmss000000:/# iptables -vnL FORWARD | grep 10.224.1.166
# Warning: iptables-legacy tables present, use iptables-legacy to see them
0 0 tcp -- * * 10.224.1.166 10.224.0.106 tcp spt:80
0 0 tcp -- * * 10.224.0.106 10.224.1.166 tcp dpt:80
root@aks-nodepool1-32351195-vmss000000:/#
root@aks-nodepool1-32351195-vmss000000:/# iptables -vnL FORWARD | grep 10.224.1.166
# Warning: iptables-legacy tables present, use iptables-legacy to see them
4 358 tcp -- * * 10.224.1.166 10.224.0.106 tcp spt:80
6 396 tcp -- * * 10.224.0.106 10.224.1.166 tcp dpt:80Node 1: root@aks-nodepool1-32351195-vmss000001:/# iptables -vnL FORWARD | grep 10.224.1.166
# Warning: iptables-legacy tables present, use iptables-legacy to see them
0 0 tcp -- * * 10.224.1.166 10.224.0.106 tcp spt:80
0 0 tcp -- * * 10.224.0.106 10.224.1.166 tcp dpt:80
root@aks-nodepool1-32351195-vmss000001:/#
root@aks-nodepool1-32351195-vmss000001:/# iptables -vnL FORWARD | grep 10.224.1.166
# Warning: iptables-legacy tables present, use iptables-legacy to see them
4 358 tcp -- * * 10.224.1.166 10.224.0.106 tcp spt:80
6 396 tcp -- * * 10.224.0.106 10.224.1.166 tcp dpt:80 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
For a Retina cluster with Hubble enabled, when a Pod makes a DNS request (to the kube-dns service), if the request lands on a coredns Pod on another Node, then running hubble observe flows will produce double the necessary number of flows. Additionally, there seems to be a small bug in labeling the traffic as ingress versus egress.
In the DNS experiment below, there exist identical copies of DNS flows (looking at JSON format), resulting in 48 DNS flows between
toolbox-podandcoredns-5fc6484dd7-bnzlf, while there are 24 UDP packets when adding packets leaving Node 1 (src=toolbox, dst=coredns), entering Node 2 (src=toolbox, dst=coredns), leaving Node 2 (dst=toolbox, src=coredns), then entering Node 1 (dst=toolbox, src=coredns).Also in this experiment, there seems to be a bug as exemplified below. A single packet travels from
toolboxtocoredns-5fc6484dd7-bnzlf, and a single packet is sent back. Yet there is this discrepancy in egress vs. ingress counts. Presumably, the first and second flows are traced on one Node, and the third and fourth flows are traced on the other Node.toolbox -> bnzlf
toolbox <- bnzlf
Setup
mcr.microsoft.com/containernetworking/retina-agent-enterprise:v0.0.8mcr.microsoft.com/oss/cilium/hubble-relay:v1.15.0toolbox-pod:DNS Experiment
Pods, Nodes, and IPs
We will focus on
coredns-5fc6484dd7-bnzlf, which is on a different Node thantoolbox-pod.Experiment
Apply these iptables rules to each node to count UDP packets between the two Pods:
Curl bing.com
$ k exec toolbox-pod -- curl bing.comBefore curling, start observing flows (focus only on the flows to/from
coredns-5fc6484dd7-bnzlf):The packets sent/received were:
Node 0:
Node 1:
When inspecting JSON output of the flows (
hubble observe -ojson), we see the flows are duplicated and there exists the ingress/egress labeling bug described at the top. Here is an example looking at thetester.default.svc.cluster.localquery:The text was updated successfully, but these errors were encountered: