Firewall, routes and Docker networking leak analysis

[jagaimoworks]

Original post

Maybe this is just me being paranoid, but...the discussion #868 mentions that at the time of the post the firewall goes online within 15ms of starting the Gluetun container. So if the network stack starts before the firewall - To be clear, I don´t know if this is the case - there theoretically should be a tiny window between the start of both the network stack and the firewall where packets could potentially be leaked, as the connected containers/services could already be using Gluetun's briefly unprotected network stack.

One way to prevent this could be forcing any connected containers to always start after Gluetun is finished with its firewall setup.

Docker Compose's depends_on in combination with the healthcheck attributes immediately come to mind. However as this only affects the order the services/containers start with Docker Compose you would still be ¨at risk¨ should just the Gluetun container restart for whatever reason while the VPN client containers keep running.

The only surefire way that I can think of right now would be to enforce additional firewall rules on the docker host to only allow Gluetun to connect to its designated VPN Servers, which pretty much defeats the ease of use Gluetun otherwise provides. Also, this may or may not break things in other ways as well.

If maybe someone more qualified than me would be so kind to put my mind at ease, I would be utmost grateful.

Questions from @the-hotmann

1. is the container shipped with ALL routs removed, so that even in a critical race-condition another container (which may starts/boots faster) can not leak its IP? Or are the routs only removed by executing the entrypoint/entry-script and therefore opening a possible window for such race-condition?
2. is the container malfunctions (just like some days ago) is there a way of it booting up and letting traffic out with the real IP?
3. in cases of OOM/crash etc. are there any potential leaks? Will the health status again go down to "unhealthy", or will the container still signal that it is working properly?

A lot of questions, but I genrally would like to understand how the "start-up" and "shut-down" proceedure works, and how it would behave in a crash/faulty state :)

The malfunction 4 days ago (2024-07-27) was a good example on how things should be: the container did not come up and never entered "healthy" state, so all container depending on it could not reach out (since no VPN connection was build up) which is exactly what I liked to see. Better not working, than leaking! So I would be curious if it would work like this also in other scenariosn, or how it behaves if something went wrong while it once was booted up correctly.

I, just like some friends of mine have expirienced this scenario:

container runs smooth and works properly
after a week or so it had no traffic and I was suspicious
tested and realiced, it is healy but does not have any traffic

I use SurfShark with WireGuard and know it is a silent protocol and that it is harder to debug so I thought this might be a security implementation or just the container not reconnecting once a connection drops.

Anyway I guess this is a topic for a different discussion or at least a different thread and the last time I expirienced this was some months ago. So maybe things changed now, with the newer versions ;)

[qdm12]

You are correct. I doubt other containers would spin up and start their program under 15ms as well though, especially with the network_mode: "service:gluetun" dependency. But theoretically it could happen. Now what could it leak? Pretty much not much 😄 Maybe a single DNS request, because 15ms round trip is rather small. For example UDP traffic with a server in the same LAN/room has a ping of 10ms. I also measured time taken by other code pieces before the firewall is configured, and it takes 1 to 2ms, so not really worth fiddling around and moving it below the firewall configuration part. For context, before there was a lot more processing done before and the firewall would be executed after 75ms, which can allow for a few requests to go out and back in outside the tunnel.

You might want to also subscribe to #1697 which might resolve this, although I'm not too sure of how the implementation will look yet to be fair (it may also just be enabled after 15ms).

[jagaimoworks]

Like you said, client containers will probably always be too slow to ever run into this issue.

My worry was basically, that a running connection could leak while Gluetun restarts. A little bit of testing later, I now know that the network_mode: "service:gluetun" dependency basically forces connected containers to restart. Docker compose seems to do this automatically and even when I tried restarting Gluetun with just "docker restart", the connected containers lost their virtual interfaces without ever regaining them until I manually restarted them. At least this was the case with the client containers I use.

I still think that even a tiny leak of one packet would be inacceptable, but with this it seems virtually impossible to run into this issue other than Gluetun arbitrarily hanging at just the right time.

If anyone is till worried about this ever happening as well as any strong believers of Murphy's law , I suggest to implement something like this in your firewall:

iptables -A DOCKER-USER ! -d your-vpn-server-ip-address/32 -i gluetun-bridge-interface -j DROP

[qdm12]

I suggest to implement something like this in your firewall

I have iptables rules for each of my container, which have a fixed ip address each. A bit of a crazy thing to manage, but it's fun.
For Gluetun, I have

iptables -A DOCKER-USER -s $GLUETUN_IP -p udp --dport 1194 -j ACCEPT
iptables -A DOCKER-USER -s $GLUETUN_IP -p udp --dport 51820 -j ACCEPT
iptables -A DOCKER-USER -s $GLUETUN_IP -j DROP

You could specify the network interface as well as VPN server IP address, but those two rules seem enough to me and gives the flexibility to change VPN server/VPN protocol/network interface.

[qdm12]

@jagaimoworks I hijacked your discussion to put merge in questions related in there!
@the-hotmann

1. Routes are not really removed for now (it will be with Feature request: default gateway configuration #1697), but the firewall takes care of blocking everything. The firewall is configured by the Gluetun entrypoint. See the comment above with its reply for more information 😉
2. If Gluetun malfunctions before the firewall is enabled (there is not that much code before the firewall is enabled), it would exit within 5ms, and connected containers would lose connectivity (or would restart again). 5ms is not enough to do anything network-wise as far as I can tell. If it malfunctions after, the firewall is enabled and blocks everything
3. It it OOMs, Gluetun gets killed by Docker/host-system so connected containers would lose connectivity. If it has a critical crash (even not handled in code, for example a panic), it exits, and, same thing, connected containers would lose connectivity. It won't signal it's working properly, not really possible. And anyway, if it's still running, the firewall is still enabled.

The biggest risk would be gross incompetence from me and the firewall being misconfigured 😄 But I'm usually careful testing things out, especially checking iptables -nvL inside the container to verify everything is as expected in various situations, when doing firewall changes.

I use SurfShark with WireGuard and know it is a silent protocol and that it is harder to debug so I thought this might be a security implementation or just the container not reconnecting once a connection drops.

I'm not sure what you mean? If it's healthy, it means network traffic is working, see https://github.com/qdm12/gluetun-wiki/blob/main/faq/healthcheck.md#internal-healthcheck regarding what the healthcheck does. If it's unhealthy, it means the connection is likely not working anymore, and there could be many causes, but not a security cause 😄