big hammer to get ip-transparent working with pf divert-to rules on OpenBSD

[dgwynne]

I've been tinkering with trying to get logging of DNS requests out of unbound so I can see what terrible things are happening on my network(s). That's been ok, except I noticed some devices ignore the DNS server I provide via DHCP, and I would like to log those requests too.

In this setup I am using an OpenBSD box as a router, and it's running unbound too. Assume i have the following interface:

$ ifconfig re1
re1: flags=8b43<UP,BROADCAST,RUNNING,PROMISC,ALLMULTI,SIMPLEX,MULTICAST> mtu 1500
        lladdr fe:e1:ba:d0:b8:c3
        index 2 priority 0 llprio 3
        groups: internal
        media: Ethernet autoselect (1000baseT full-duplex)
        status: active
        inet 10.0.127.1 netmask 0xffffff00 broadcast 10.0.127.255
        inet6 fe80::20d:b9ff:fe3e:d8c9%re1 prefixlen 64 scopeid 0x2

And this up the top of my pf.conf:

$if_internal="re1"
$svc_dns="$if_internal"

I can then use a rule like this to catch the misbehaving hosts:

pass in quick on $if_internal \
        inet proto { tcp udp } to !$svc_dns port domain \
        rdr-to $svc_dns port domain

This is fine, DNS works, and unbound replies to the redirected DNS requests fine. However, in the logs this
looks like the misbehaving clients are doing the same thing as the well behaved ones. I would like to be able to tell the types of devices apart so I can identify/classify them and maybe fix them. An evolution of this is to have
unbound listen somehwere weird and redirect to that so there's something different in the logs.

So I put this in unbound.conf:

interface: 10.0.127.1@35

... and this in pf.conf:

pass in quick on $if_internal \
        inet proto { tcp udp } to !$svc_dns port domain \
        rdr-to $svc_dns port 35

Now I can look for client requests to port 35 in the logs and I have my answer.

However, the ultimate would be being able to see address of the nameserver the misbehaving clients were trying to connect to in the logs. Reading the unbound.conf man page (and a bit of the code), it sounds like ip-transparent is what i want. It's supposed to enable SO_BINDANY on listening sockets and use getsockname()/IP_RECVDSTADDR to figure out
what the original IP was, and then use IP_SENDSRCADDR for replies.

So with this in unbound.conf:

ip-transparent: yes

... and this in pf.conf:

pass in quick on $0f_internal \
        inet proto { tcp udp } to !$svc_dns port domain \
        divert-to $svc_dns port domain

... it should work. For most things it does.

DNS clients using TCP and UDP to talk to 10.0.127.1 port 53 (like they're supposed to) work. DHS clients trying to talk to another NS like 8.8.8.8 port 53 work too. However, DNS clients using UDP to talk to 8.8.8.8 port 53 do not. They look
like this:

$ dig @8.8.8.8 www.google.com     
;; reply from unexpected source: 10.0.127.1#53, expected 8.8.8.8#53

Turns out that unbound isn't using the cmsg stuff to get the original IP addresses. This means when unbound replies, the addresses on the listening socket are used and they are not the right ones for the client. Turns out that's because unbound only enables "ancillary" data collection in specific situations, and at the moment enabling ip-transparent
isn't one of them.

This diff forces unbound to set all UDP sockets up to do the cmsg stuff. I know this is not the right solution, but I thought I'd ask for direction rather than make a guess and have to do the work again.

[wcawijngaards]

Diverting DNS traffic, like this, is not really good, and I think perhaps only useful as a debug aid, it looks like tampering to me, for the end clients. If the inspection of the traffic is wanted, something like wireshark can do that. It can also be set to filter out particular messages and display them.

The ip transparent option sets the socket option. But in addition you want to enable cmsg stuff, and this particular cmsg stuff is enabled by interface-automatic: yes and perhaps also use interface-automatic-ports: "additionalport1 additionalport2". That makes unbound fetch the IP addresses from the connection and use them in the reply so that the source address of the reply is correct.

If the code change still needs to be made, it would be nice to allow this as an additional option that can be enabled by the user for the specific use case, instead of an always active bypass.

Some devices ignore the DHCP provided DNS server, instead relying on others, and cloud providers are popular. There are encrypted connections that can be set up, using for example the dns over TLS and HTTPS options. Unbound can provide this kind of connectivity, and this is precisely to avoid the firewall here. It is possible to configure the certificate that unbound provides, but also wants to admit, to verify the server identity.

[dgwynne]

Diverting DNS traffic, like this, is not really good, and I think perhaps only useful as a debug aid, it looks like tampering to me, for the end clients. If the inspection of the traffic is wanted, something like wireshark can do that. It can also be set to filter out particular messages and display them.

I probably shouldn't have wasted time explaining all the logging background. I agree that capturing or duplicating packets also lets me log requests, but if I ever enable DoT or DoH on my local networks then the best point to observe DNS requests will be from inside unbound.

If my goal is to intercept or tamper with DNS in the clear, then I can do that already with an IP rewrite. The question is more whether ip-transparent is supposed to work in this situation too.

The ip transparent option sets the socket option. But in addition you want to enable cmsg stuff, and this particular cmsg stuff is enabled by interface-automatic: yes and perhaps also use interface-automatic-ports: "additionalport1 additionalport2". That makes unbound fetch the IP addresses from the connection and use them in the reply so that the source address of the reply is correct.

If the code change still needs to be made, it would be nice to allow this as an additional option that can be enabled by the user for the specific use case, instead of an always active bypass.

Sounds to me like the cmsg bits could also be enabled when ip-transparent: yes is set?

Some devices ignore the DHCP provided DNS server, instead relying on others, and cloud providers are popular. There are encrypted connections that can be set up, using for example the dns over TLS and HTTPS options. Unbound can provide this kind of connectivity, and this is precisely to avoid the firewall here. It is possible to configure the certificate that unbound provides, but also wants to admit, to verify the server identity.

There's an interesting philosophical discussion we could have here about ownership and authority over the equipment we purchase and are responsible for operating, and whether than ends at the network port on an individual device or on port on my router facing the public internet. It's one thing for a malicious network operator to intercept or modify my DNS requests, it's another thing for a device that we've bought and put on my network to think it always knows better than me about where it should do DNS lookups. Am I allowed to intercept DNS in the clear that's trying to leave my network and upgrade it to DoT to a more trusted provider on the public internet? What if I'm doing RPZ?

[wcawijngaards]

Yes that is a complicated space of concerns about device ownership and traffic patterns. But I wanted to make you aware, if you weren't, about DoT and so on.

The ip-transparent option should only enable the socket option that it uses. The additional cmsg stuff, to store and use the interface IP address, should be enabled with a separate option. That could then be used in conjunction with the ip-transparent option. This is a good idea, because the functionality is also useful for others, with eg. weird routing setups, that have source address issues. Not that, in that space, the ip-transparent and ip-freebind socket options are also used.

Perhaps the existing interface-automatic options can be used to enable the cmsg source IP capture that is needed here? If that is not useful, perhaps some option like perhaps ip-cmsg-srcaddr that enables the cmsg stuff?

[dgwynne]

I'll try interface-automatic instead of ip-transparent when I get a chance.

When to cmsg or not to cmsg is an annoyingly complicated decision tree. Technically you can use the IP_RECVSRCADDR and IP_SENDSRCADDR cmsgs if you have a listener on a wildcard address. This allows you to listen to all IPs on the system with a single FD (per address family), and reply to messages from any of those addresses. To me this sounds like any wildcard listener should use the cmsgs, regardless of ip-transparent or interface-automatic. Is interface-automatic actually needed?

Without the cmsgs the traditional approach is iterate over all addresses in the system and bind to them on startup. The more advanced approach is listen on the routing socket for new addresses and bind to them as they appear. This needs privs if you're trying to bind to a low port though, which sucks a bit.

The way ip-transparent is documented is that it allows binding to non-local IPs. To me that implies SO_BINDANY/IP_BINDANY. If it's not a wildcard listener then it doesn't need the cmsgs because the address it's bound to provides the source address, which should be the only way a packet will come in on that socket.

The only real gap is the (ab)use case I have, which is that I'm binding to a non-wildcard address and redirecting/diverting packets to it. To divert packets with pf it needs to be bound with SO_BINDANY, and to reply to them it needs IP_SENDSRCADDR.

So, is that 4 possible configurations?

1. vanilla wildcard listener should use cmsgs when available
2. vanilla non-wildcard listener doesn't need SO_BINDANY or cmsgs
3. ip-transparent should be a non-wildcard listener with SO_BINDANY, but doesn't need cmsgs
4. divert-to in pf needs a bind to a non-wildcard address, and SO_BINDANY and cmsgs so it can reply to them

Am I missing anything?

[wcawijngaards]

It may also be possible to enable both the interface-automatic: yes and ip-transparent: yes option at the same time, and that may be a good possible configuration. The ip-transparent: yes config option enables IP_BINDANY. On BSD, it uses IP_TRANSPARENT.

There is an option in Unbound to list the addresses of an interface, with interface: eth0. That lists the addresses of the interface name, at startup of Unbound, and binds them.

I am not sure what the different configurations are you want; but in terms of the code change I was looking for, I thought of adding code that enables a new config option, that can be toggled together with other options, or on its own. The option enables the cmsg handling, the srcaddr handling, and it could be called ip-cmsg-srcaddr: yes perhaps. This is also the gist of the code change proposed, in the temporary code patch, but then as an option.