How container ports are mapped to pod's ip?

[dzhlobo]

I had an interesting case today.

I have a pod that has container with a process that listens on port 80. I also have a service of ClusterIP type that maps service port 80 to pod's port 80:

  ports:
  - name: http
    port: 80
    protocol: TCP
    targetPort: 80

It appears that my container port 80 is mapped to pod ip port 80, kubectl describe service the_service:

Port:              http  80/TCP
TargetPort:        80/TCP
Endpoints:         10.0.18.212:80

Are pod containers ports mapped to pod ip 1-to-1? Is it expected behavior?

I noticed it after my service stopped working after upgrade. The pod landed on different node from a pod that connects to it. My security groups only allowed node to node connections via ports 1025-65535 (which is default if you use node_security_group_enable_recommended_rules of https://github.com/terraform-aws-modules/terraform-aws-eks.

I ended up relaxing security group rules to allow node to node communication via ports 80-65535. This was a bit unexpected, since I expected container port would be mapped to random not privileged port of pod's ip.

[dzhlobo]

I didn't know how networking in cluster works.

My assumption was that all the ip addresses is allocated to node and then traffic from containers in pods were routed correctly using NAT configured on node.

It's not the case. Each pod has its own network namespace (shared between all pod's containers). In the namespace there is a network interface with an ip address configured. In case of VPC CNI this is a real address from a VPC. So when a process in container listens to a port it listens to a port on a network interface in the namespace, no mapping is configured at all.

I don't know exactly how VPC CNI plugin works and how prefix delegation in VPC works but I imagine it is possible to create multiple network interfaces on host system (using macvlan?) and then move them to different network namespaces and that's what happens on nodes.