Lima + KinD + MetalLB

[bcollard]

Hi! I'm using Lima with the docker flavor so that I can spawn KinD clusters. That is already awesome!! 🎉

In Kubernetes, when I create some services type LoadBalancer, I get an external IP, thanks to MetalLB.
I can access these external IPs from the Lima VM. Like:

limactl shell
curl 172.18.1.1

Plus, the connection to the docker daemon from my laptop is working fine: export DOCKER_HOST=unix://$HOME/docker.sock
Super awesome! 🤩

Now, I would like to access these Kubernetes services from the host, my MacBook.

My question: what would be the next steps to have, let's say the subnet 172.18.0.0/16 (the one configured in MetalLB), routed to the guest VM? so that my curl requests work from my machine up to Kubernetes?

I've been quickly looking at https://github.com/lima-vm/lima/blob/master/docs/network.md and https://github.com/lima-vm/vde_vmnet but I'm lost! :-)

[rupesh-papneja]

@bcollard Check this out. this may help lima-vm/vde_vmnet#19 (comment)

[bcollard]

Thanks! I'll take a look!

[bcollard]

Quick intermediate update:
I've followed the instructions here https://github.com/lima-vm/vde_vmnet#install
and then here https://github.com/lima-vm/lima/blob/master/docs/network.md#managed-vmnet-networks-via-vde_vmnet

So, now I have a bridge network between my host and the Lima VM. Now, I can ping machines in both directions 👍 (Host to VM, VM to Host)

On my host, I've added this route sudo route add 172.18.0.0/16 192.168.105.2 but not sure what I'm doing and it doesn't help to reach the k8s svc IP, which is 172.18.1.1

I'll take a look at this comment lima-vm/vde_vmnet#19 (comment) later today

[bcollard]

The last piece I was missing is about IP packet forwarding on the Lima VM:

BRIDGE_NET=$(ip -o link show | awk -F': ' '{print $2}' | grep "br-")
SRC_IP=192.168.105.1
DST_NET=172.18.0.0/16
sudo iptables -t filter -A FORWARD -4 -p tcp -s ${SRC_IP} -d ${DST_NET} -j ACCEPT -i lima1 -o ${BRIDGE_NET}

Result: from my MacBook, I can run docker commands and I can spin up multiple KinD clusters. In these k8s clusters, I can create Services type LoadBalancer and access them from my MacBook.
I'll blog about it.

Big kudos to the contributors for this nice project! ❤️

[bcollard]

ip a

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 52:55:55:08:7d:a3 brd ff:ff:ff:ff:ff:ff
    inet 192.168.5.15/24 brd 192.168.5.255 scope global dynamic eth0
       valid_lft 42571sec preferred_lft 42571sec
    inet6 fec0::5055:55ff:fe08:7da3/64 scope site dynamic mngtmpaddr noprefixroute
       valid_lft 86031sec preferred_lft 14031sec
    inet6 fe80::5055:55ff:fe08:7da3/64 scope link
       valid_lft forever preferred_lft forever
3: lima0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 52:55:55:79:c2:11 brd ff:ff:ff:ff:ff:ff
    inet 192.168.105.3/24 brd 192.168.105.255 scope global dynamic lima0
       valid_lft 41707sec preferred_lft 41707sec
    inet6 fe80::5055:55ff:fe79:c211/64 scope link
       valid_lft forever preferred_lft forever
4: lima1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 52:55:55:b9:d3:90 brd ff:ff:ff:ff:ff:ff
    inet 192.168.105.2/24 brd 192.168.105.255 scope global dynamic lima1
       valid_lft 84476sec preferred_lft 84476sec
    inet6 fe80::5055:55ff:feb9:d390/64 scope link
       valid_lft forever preferred_lft forever
5: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default
    link/ether 02:42:3b:a2:f0:b2 brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
       valid_lft forever preferred_lft forever
6: br-2164ca6b12d6: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default
    link/ether 02:42:62:da:0e:95 brd ff:ff:ff:ff:ff:ff
    inet 172.18.0.1/16 brd 172.18.255.255 scope global br-2164ca6b12d6
       valid_lft forever preferred_lft forever
    inet6 fc00:f853:ccd:e793::1/64 scope global
       valid_lft forever preferred_lft forever
    inet6 fe80::42:62ff:feda:e95/64 scope link
       valid_lft forever preferred_lft forever
    inet6 fe80::1/64 scope link
       valid_lft forever preferred_lft forever

no hurry, no worries! :)

[jandubois]

As expected, you have two vmnet interfaces:

3: lima0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    inet 192.168.105.3/24 brd 192.168.105.255 scope global dynamic lima0
4: lima1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    inet 192.168.105.2/24 brd 192.168.105.255 scope global dynamic lima1

You can see that lima0 and lima1 are both on the same subnet, connecting to the same gateway. There is no reason for a single machine to connect to the same network via 2 interfaces, and I'm not sure if that couldn't even cause some issues. Also it is a bit wasteful, as you will be running 2 instances for each of the vde_switch and vde_vmnet daemons.

[rupesh-papneja]

@jandubois @bcollard - one question here. wont it be simpler to use bridged network as compared to shared in lima config docker.yaml ?

[jandubois]

wont it be simpler to use bridged network as compared to shared in lima config docker.yaml ?

It is trivial to switch, just use:

networks:
- lima: bridged

It all depends on what your goals are. bridged gives you a separate IP address on the local network instead of using NAT on the host.

On the other hand, shared automatically provides you with routing to VPN networks configured on the host (and puts you behind the host firewall, so not all your ports are exposed).

Note that you can have both a shared and a bridged network:

networks:
- lima: bridged
- lima: shared

You will probably want to configure things so that the shared network is used for egress, and the bridged network for ingress.

I'm actually looking right now into modifying the routing metrics to make this happen automatically. Right now the default route on eth0 has the lowest metrics, so will be chosen by e.g. k3s as the external address of the cluster, but that is quite useless because the address is non-routable.

[rupesh-papneja]

thank you for the explanation.
I faced the issue with eth0 and had manually configured to route traffic via bridged interface (lima0) to get my setup working.
I am looking forward for the changes to be released. :)