Cannot create a managed cluster due to "failed to get ip for the load balancer"

[cmoulliard]

Issue

The process to create a managed cluster tanzu mc create --ui -b 65.108.148.216:8080 within a VM fails due to this error reported by the control plane:

docker rm -f $(docker ps -a -q)
docker network rm kind
docker system prune -a --volumes -f
rm -fr ~/.config/tanzu
tanzu mc create -i docker -p dev mgmt-test
...
docker exec -it tkg-kind-c9jtd3rsjmgh2n9m8ri0-control-plane bash
...
kubectl get pods -A
NAMESPACE                           NAME                                                                  READY   STATUS    RESTARTS   AGE
capd-system                         capd-controller-manager-86b587cf4b-x9t5b                              1/1     Running   0          14m
capi-kubeadm-bootstrap-system       capi-kubeadm-bootstrap-controller-manager-7ffb6dc8fc-62vkn            1/1     Running   0          14m
capi-kubeadm-control-plane-system   capi-kubeadm-control-plane-controller-manager-667999fdb8-8jww5        1/1     Running   0          14m
capi-system                         capi-controller-manager-65c5769c4c-lb4pf                              1/1     Running   0          14m
...
kubectl logs -n capd-system capd-controller-manager-86b587cf4b-x9t5b
E0426 12:21:40.408823       1 controller.go:317] controller/dockercluster "msg"="Reconciler error" "error"="failed to get ip for the load balancer: load balancer IP cannot be empty: container tata-lb does not have an associated IP address" "name"="tata" "namespace"="tkg-system" "reconciler group"="infrastructure.cluster.x-k8s.io" "reconciler kind"="DockerCluster"

[cmoulliard]

More information

[snowdrop@h01-121 ~]$ docker ps
CONTAINER ID   IMAGE                                                         COMMAND                  CREATED         STATUS         PORTS                                  NAMES
5e1c9fbbf04a   projects.registry.vmware.com/tkg/kind/node:v1.22.5_vmware.1   "/usr/local/bin/entr…"   7 minutes ago   Up 7 minutes   34011/tcp, 127.0.0.1:34011->6443/tcp   tkg-mgmt-docker-20220426170635-control-plane-j5plk
f4bc2b8e6d87   kindest/haproxy:v20210715-a6da3463                            "haproxy -sf 7 -W -d…"   7 minutes ago   Up 7 minutes   33192/tcp, 0.0.0.0:33192->6443/tcp     tkg-mgmt-docker-20220426170635-lb
f6c466a1f18e   projects.registry.vmware.com/tkg/kind/node:v1.22.5_vmware.1   "/usr/local/bin/entr…"   9 minutes ago   Up 9 minutes   127.0.0.1:41174->6443/tcp              tkg-kind-c9k0kvbsjmgpv83itcd0-control-plane
[snowdrop@h01-121 ~]$ docker logs f4bc2b8e6d87
[WARNING] 115/150941 (1) : config : missing timeouts for frontend 'controlPlane'.
   | While not properly invalid, you will certainly encounter various problems
   | with such a configuration. To fix this, please ensure that all following
   | timeouts are set to a non-zero value: 'client', 'connect', 'server'.
[NOTICE] 115/150941 (1) : New worker #1 (9) forked
[WARNING] 115/150947 (1) : Reexecuting Master process
[NOTICE] 115/150947 (1) : haproxy version is 2.2.9-2~bpo10+1
[NOTICE] 115/150947 (1) : path to executable is /usr/sbin/haproxy
[ALERT] 115/150947 (1) : sendmsg()/writev() failed in logger #1: No such file or directory (errno=2)
[WARNING] 115/150947 (9) : Stopping frontend controlPlane in 0 ms.
[WARNING] 115/150947 (9) : Stopping backend kube-apiservers in 0 ms.
[WARNING] 115/150947 (9) : Stopping frontend GLOBAL in 0 ms.
[WARNING] 115/150947 (9) : Proxy controlPlane stopped (cumulated conns: FE: 11, BE: 0).
[WARNING] 115/150947 (9) : Proxy kube-apiservers stopped (cumulated conns: FE: 0, BE: 11).
[WARNING] 115/150947 (9) : Proxy GLOBAL stopped (cumulated conns: FE: 0, BE: 0).
[NOTICE] 115/150947 (1) : New worker #1 (27) forked
[WARNING] 115/150947 (1) : Former worker #1 (9) exited with code 0 (Exit)
[WARNING] 115/150947 (27) : Server kube-apiservers/tkg-mgmt-docker-20220426170635-control-plane-j5plk is DOWN, reason: Layer4 connection problem, info: "SSL handshake failure", check duration: 0ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
[ALERT] 115/150947 (27) : backend 'kube-apiservers' has no server available!

and docker inspect of the lb container returns

[snowdrop@h01-121 ~]$ docker inspect  f4bc2b8e6d87
[
...
        "NetworkSettings": {
            "Bridge": "",
            "SandboxID": "47efa21bf6b63bd46b8683a15f69b7edeae2f60fe7beae34ada4597d6a55b103",
            "HairpinMode": false,
            "LinkLocalIPv6Address": "",
            "LinkLocalIPv6PrefixLen": 0,
            "Ports": {
                "33192/tcp": null,
                "6443/tcp": [
                    {
                        "HostIp": "0.0.0.0",
                        "HostPort": "33192"
                    }
                ]
            },
            "SandboxKey": "/var/run/docker/netns/47efa21bf6b6",
            "SecondaryIPAddresses": null,
            "SecondaryIPv6Addresses": null,
            "EndpointID": "",
            "Gateway": "",
            "GlobalIPv6Address": "",
            "GlobalIPv6PrefixLen": 0,
            "IPAddress": "",
            "IPPrefixLen": 0,
            "IPv6Gateway": "",
            "MacAddress": "",
            "Networks": {
                "kind": {
                    "IPAMConfig": null,
                    "Links": null,
                    "Aliases": [
                        "f4bc2b8e6d87",
                        "tkg-mgmt-docker-20220426170635-lb"
                    ],
                    "NetworkID": "59ab22a331d14cda52106b44123e76696af55e6a10580d3783204f8d7a77cc8f",
                    "EndpointID": "f6c150517db9a262e477640a4f9ed77078ef7908358f672a92115dda7c5470ca",
                    "Gateway": "172.18.0.1",
                    "IPAddress": "172.18.0.3",
                    "IPPrefixLen": 16,
                    "IPv6Gateway": "fc00:f853:ccd:e793::1",
                    "GlobalIPv6Address": "fc00:f853:ccd:e793::3",
                    "GlobalIPv6PrefixLen": 64,
                    "MacAddress": "02:42:ac:12:00:03",
                    "DriverOpts": null
                }
            }
        }
    }
]

[cmoulliard]

Reported the issue to vmware-tanzu/community-edition#4201