DNS resolution works against skydns IP, but not docker0 interface IP

[troyfolger]

Hey - love skydns/skydock, but I've got a problem.

I am trying to understand if I can get skydns and skydock to work without having to modify the docker daemon configuration (by passing the '--dns' option on the docker run command lines). I've had success, but only by specifying the skydns container's IP for the namserver injected into containers ... it doesn't work as the README suggests (using the virtual interface IP).

Here is a quick example:

docker run -d -p 172.17.42.1:53:53/udp --name skydns crosbymichael/skydns -nameserver 8.8.8.8:53 -domain docker

docker inspect --format '{{ .NetworkSettings.IPAddress }}' skydns
:: 172.17.0.52

docker run -d -v /var/run/docker.sock:/docker.sock --name skydock crosbymichael/skydock -ttl 30 -environment dev -s /docker.sock -domain docker -name skydns

docker run -d --name redis1 crosbymichael/redis
docker run -t -i --name redis-cli --dns=172.17.0.52 crosbymichael/redis-cli -h redis1.redis.dev.docker

:: works! now, try again but specify the docker0 IP for DNS:

docker rm redis-cli
docker run -t -i --name redis-cli --dns=172.17.42.1 crosbymichael/redis-cli -h redis1.redis.dev.docker

:: doesn't work - eventually get 'not connected>' prompt in redis-cli

:: interestingly, dig works for both IPs:

$ dig @172.17.42.1 +short dev.docker
172.17.0.53
172.17.0.52
172.17.0.54

$ dig @172.17.0.52 +short dev.docker
172.17.0.53
172.17.0.52
172.17.0.54

Any ideas on why this is happening? The README suggests that using the docker0 interface as the DNS nameserver should work ...

[jswager]

I've got the exact same problem. The workaround solution of specifying the IP address of the skydns container in the --dns option does work, but it's a bit clunky.

[takeshitakenji]

My solution was to add this to iptables:

-A DOCKER -d 172.17.42.1/32 -p udp -m udp --dport 53 -j DNAT --to-destination 172.17.0.2:53

The cause is the ! -i docker0 in the line equivalent generated by Docker:

-A DOCKER -d 172.17.42.1/32 ! -i docker0 -p udp -m udp --dport 53 -j DNAT --to-destination 172.17.0.2:53

Adding the the new rule removes the restriction.

[jswager]

Interesting solution. But it requires modifying the iptables to reflect the private IP address of the SkyDNS server. A little easier to handle than specifying --dns on each container, but still not effective if all containers are deleted and rebuilt.

The original instructions were the most elegant; it wouldn't matter which private IP the DNS server got, it would always work for all containers.

[takeshitakenji]

It only requires modification of iptables on the host; if regular containers are rebuilt, they don't need any changes. Only if the host is rebuilt or the SkyDNS container is moved around does it need to be redone.

[jswager]

True. In our situation, the SkyDNS container is being created and destroyed along with the rest of them, so we would have to constantly modify the iptables. It's a good workaround - but neither it nor our existing workaround of --dns on each container are as elegant as the originally intended solution of the use the docker bridge.

[toddinpal]

OK, so I'm an iptables neophyte. I tried that iptables entry as follows:

iptables -A DOCKER -d 172.17.42.1/32 -p udp -m udp --dport 53 -j DNAT --to-destination 172.17.0.2:53
iptables: No chain/target/match by that name.

yet an iptables-save has:
-A DOCKER -d 172.17.42.1/32 ! -i docker0 -p udp -m udp --dport 53 -j DNAT --to-destination 172.17.0.2:53

Any help would be appreciated.

[takeshitakenji]

Use iptables-save to save the rules, then copy and paste that line to just below itself, then remove the ! -i docker0 part of the new line. Reload it using iptables-restore and everything should be good.

[toddinpal]

Yeah, I realized that just after I posted. Still doesn't fix the problem of accessing the skydns from another container. :-(

I suspect I need another rule as I now get this in the container:

$ dig @172.17.42.1 host1.tuxedo.dev.docker
;; reply from unexpected source: 172.17.0.2#53, expected 172.17.42.1#53

[takeshitakenji]

Huh. I haven't seen anything like that before. DNAT masked it just fine in both directions when I did it.

[toddinpal]

Any chance you could post your iptables so I can compare them against mine?

[simonklb]

I've bumped into this issue recently. Weirdly enough the issue only seem to come up occasionally and not all the time. Removing the "! -i docker0" flag does fix the problem when the issue occurs however.

Has anyone found a better solution to this since it was reported?