DockerNetworking

Docker Networking

If you use iptables and Docker, you don't want to let the Docker daemon control iptables. It is far too permissive. (See note below about swarm.)

Thanks to Francois Lefevre for getting me started with Docker and IPtables.

--iptables=false

First thing to do is to tell Docker not to touch iptables.

If you are running CentOS 6, you should add this to /etc/sysconfig/docker:

other_args='--iptables=false'

Do not bother with /etc/sysconfig/docker-network, which is not referenced in CentOS 6 even though the file is there.

If you are running CentOS 7 Atomic, modify /etc/sysconfig/docker-network to:

DOCKER_NETWORK_OPTIONS='--iptables=false'

This file is read by /usr/lib/systemd/system/docker.service.

net.ipv4.conf.all.forwarding=1

You can't stop docker from turning on all port forwarding. It seems to hardwire this one:

sysctl -w net.ipv4.conf.all.forwarding=1

It's unclear to me how bad this is, but it isn't good. We turn off ip forwarding by default in our sysctl.conf, and we turn off forwarding via iptables for extra measure.

Why does docker need forwarding?

If your containers connect with other hosts on your network or if they need to make connections to the wider Internet (e.g. for a 3rd party service), you will need to turn on forwarding.

Let's verify that requests fail when forwarding is off. You'll need to leave --iptables=true so you can test this simply:

# sysctl -w net.ipv4.conf.all.forwarding=0
# docker run -i --rm busybox nc -w 2 bivio.biz 80 < /dev/null
nc: timed out
#

However, you don't need to turn on all.forwarding. You can be specific:

# sysctl -w net.ipv4.conf.eth0.forwarding=1
# sysctl -w net.ipv4.conf.docker0.forwarding=1
# docker run -i --rm busybox nc -w 2 bivio.biz 80 < /dev/null
#

iptables masquerading

When you turn off Docker's control of iptables, you have to turn on forwarding in iptables with masquerading. Some people recommend forwarding without masquerading, but the following is insufficient:

-A FORWARD -i docker0 -o eth0 -j ACCEPT
-A FORWARD -i eth0 -o docker0 -j ACCEPT

You have to turn on masquerading, because docker0 is on an unroutable network. Usually the Docker network is 172.17.0.0/16, which the same for all hosts running Docker. This is why it is unroutable. The packets go out, but they have no way of getting back to your host's docker network.

To setup masquerading from docker0 to eth0, you have to:

-t nat -A POSTROUTING -o eth0 -j MASQUERADE
-A FORWARD -i eth0 -o docker0 -m state --state RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -i docker0 -o eth0 -j ACCEPT

Publishing ports

Docker has a feature called a userland-proxy, which is enabled by default for the daemon. When you publish a port, you will see something like this:

# docker run -d --name bb -p 8000:8000 busybox httpd -f -p 8000 -h /
# ps ax | grep docker-proxy
19770 ?        Sl     0:00 docker-proxy -proto tcp -host-ip 0.0.0.0 -host-port 8000 -container-ip 172.17.0.20 -container-port 8000
#

Note the container-ip (172.17.0.20 in this case). You can connect directly to it:

# curl http://172.17.0.20:8000/etc/hosts
172.17.0.20     43b03a4eb4ad
127.0.0.1       localhost
[...snip...]

The proxy is a user process which opens up up the port on all network interfaces (0.0.0.0) unless you specify a specific interface, which might be a good idea, especially if you want to expose an interface only to a back net, e.g.

# docker run -d --name bb -p 192.168.1.1:8000:8000 busybox httpd -f -p 8000 -h /
# ps ax | grep docker-proxy
22750 ?        Sl     0:00 docker-proxy -proto tcp -host-ip 192.168.1.1 -host-port 8000 -container-ip 172.17.0.20 -container-port 8000
#

Some people like using EXPOSE Dockerfile and --publish-all, but this isn't really a good idea, since you'll expose the ports to all your interfaces. It's much better to specify ports at container run time. That way you know what you are exposing.

Opening up ports with iptables

Since we have taken over management of iptables from Docker, we need to open the port manually in iptables. That's fairly easy:

-A INPUT -d 192.168.1.11 -p tcp -m state --state NEW --dport 8000 -j ACCEPT

You don't need to expose anything else, because the userland proxy is running.

Some people recommend using NAT for forwarding ports, but you need to be explicit with the container-ip, e.g.

-t nat -A FORWARD -p tcp --dport 8000 -j DNAT --to-destination 172.17.0.20:8000

That's not easy to set up easily, because you have to use docker inspect and execute the iptables change after the container starts. You could hardwire container IPs, and that gets even worse... The userland proxy is not ideal, but it works ok.

Docker Security

Good article:

• [https://opensource.com/business/15/3/docker-security-tuning](Tuning Docker with the newest security enhancements)

Docker Swarm Mode

Docker Swarm Mode needs Docker networking. It relies on forwarding between subnets via iptables and masquerading. You have to let Docker control iptables. At that point, the machine has to be completely behind a firewall. You can't trust it to do the right thing (it doesn't).