Private network with egress gateway

[robertlemke]

I'd like to set up a cluster with nodes only having private IP addresses.

That basically works, when I enable hcloudNetwork in the respective HetznerClusterTemplate. A load balance can then route incoming requests to the respective pods. However, Nodes need some way to send outgoing requests (egress) as well, for example for pulling images. Since I don't want nodes to have a public IP address, I started implementing a SNAT server which acts as a gateway for egress traffic.

The private network needs to have a route pointing to the SNAT server for the destination 0.0.0.0/0.

Now here's the problem: when i enable hcloudNetwork, the reconciliation service will create the network when the cluster is created and deletes the network when the cluster is destroyed. But I need to set up this network myself (with Terraform) in order to configure the route to the SNAT server in time – otherwise the cluster bootstrap fails, because it cannot pull images. If I disable hcloudNetwork, I can set up the network and route with Terraform – but creation of the machines fail because they are not connected to any network.

Two questions:

1. Is there another approach – without a SNAT server – setting up the cluster with private IPs? Or did I take the right one?
2. Is there any configuration I can set up so that machines become part of the private network which I set up previously?

[janiskemper]

There is no option currently to use an existing network, but this could be implemented in theory.
I won't go into details of the rest of your question, because I'm not the expert there.

[robertlemke]

Thanks! Yeah, I guess that can be implemented easily, alternatively an additional option could be introduced, so that you can define a route for the network managed by the CAPH controller.

By the way, apart from enhanced security, the motivation for using the egress gateway server is that we need to have control over which IP address is used for outgoing requests. While playing with auto-scaling, I realized that new nodes sometimes failed during bootstrap because they could not pull an image from the Google artifact registry – because Google blocked the IP address of the machine in HCloud. And apparently (according to forum posts) this is quite a problem for others as well – there are lots of Hetzner IP addressed blocked for Google services because they were abused in the past. When a new node is being created you might be lucky or not, but you never know … 🥴

[janiskemper]

yes that's a known problem. If you are interested in a Managed Kubernetes solution based on CAPH, you can have a look at https://syself.com. We have a working solution for the blocked IP issue that is based on multiple components (e.g. node images containing all important images, our own mirror, etc.)

I asked the team btw for their input on your original question.

Side note: We stopped using the private networks of HCloud because they were not reliable for us.

[robertlemke]

@janiskemper I am aware of your offering at syself.com and it looks great! However, I have requirements which certainly don't fit into an autopilot product. What would help me a lot though would be discussing a few things from developer to developer and getting (paid) support or small new features developed if needed. If you send me a quick mail to robert at flownative dot com, I can tell you more about it …

[robertlemke]

Some thoughts about private networks:

You wrote that you stopped using the private networks of HCloud because they were not reliable. Can you give some more insights into that so that CAPH users avoid those problems?

I'm currently thinking how a cluster without a private network could be secured properly in HCloud. You can apply firewall rules for all nodes using labels. In those rules you can grant access to nodes from the load balancers. However, what you can't do with those rules is allowing intra-node access – for example from the control plane nodes to worker nodes or traffic between the nodes. You would need to set up a rule with a fixed set of source IP addresses or grant access to a large segment of public IP addresses used by Hetzner (which is basically no protection at all).

So, I would still think that a private network is essential and I dearly hope that I can work around any issues which might exist with private networks.