Server Firewall/Routing requires more than port 51820? ...or possible time zone problem? ...or other?

[tetricky]

I'm using a hosted server, running debian 12, and with ufw as a firewall (I am insufficiently skilled to manually write (nf)iptables rules), with the intention to run innernet-server. Which I have installed (along with the client) from from the third party repository (using jammy for bookworm).

I actually have tested on two servers. One has a load of legacy rules, which I really don't want to have to keep...as they've come about from all manner of installations, tests, and whatnot. innernet works totally as expected on the 'messy' server. On the 'clean' server I have opened up port 51820 (51820/udp ALLOW IN Anywhere). It doesn't work properly unless I take down the firewall....which is not really an option for an internet facing server. Both the "messy" and "clean" server are in the same datacentre with the same networking, and are identical model machines, with the same debian install. So differences can only be local environment/setup issues.

Is this because innernet needs some other form of iptables rules, that are accidentally in place on the 'messy' server?

Both servers are running podman and a bunch of containers (no overlapping CIDR's with innernet), and have caddy handling requests to ports 80/443 which are also open. Additionally both servers and all peers have mutual To: Anywhere ALLOW IN rules. I can't understand why one server installation works, and one doesn't...the only other active difference I can see is that the two servers are set to different time zones (but both accurately synced to the same ntp server).

I appear to have reached the extents of my (limited) knowledge, and the documentation. Accepting that this is all my fault, I am hoping for some assistance to guide me forward from here, and this is an excellent project with (I believe) the ideal balance of features for my intended use case (A couple of internet facing servers to expose services, a k3s private cluster accessible only by innernet doing the backends, possibly a homelab based cluster, and admin access from various locations. to segmented networks in various locations, to administer from remote locations, but also for access to services). For me innernet appears preferable to something like netbird that has a massive resource overhead, and ongoing maintennance complications. Of course manually configuring wireguard would be possible...but highly complex.

I realise that this is a poor explanation, from an ill-informed standpoint...but I would appreciate any advice, however small, that might point me to a solution.

[bschwind]

Hi @tetricky, thanks for the thoughtful words! Innernet shouldn't require more than a single UDP port to function well. I don't have too much experience with configuring UFW, but if disabling it makes things work, then that's a pretty strong indicator that something is being blocked.

I would double check innernet is listening on the expected port with something like netstat -tulpan, and use innernet set-listen-port <INTERFACE> if you haven't already.

Maybe when UFW is disabled you could run tcpdump while interacting with innernet to see what sort of traffic is coming in.

But I wouldn't expect a timezone issue or ports other than the UDP port to cause issues. At the same time, we have stories like the 500 mile email so I don't want to rule anything out quite yet :)

[tetricky]

Thank you very much for the quick response.

On the not working properly server, I see:

# netstat -tulpan | grep innernet
tcp        0      0 10.12.0.1:51820         0.0.0.0:*               LISTEN      178513/innernet-ser 

# innernet show
network: f140
  listening port: 41273
  peer: innernet-server (3W+AAsC/TF...)
    ip: 10.40.0.1
    endpoint: <serverf140_ip>:51820
    last handshake: 1 minute, 7 seconds ago
    transfer: 244.23 KiB received, 71.96 KiB sent
  peer: f512 (ZAiqXkjZQO...)
    ip: 10.40.1.1

# innernet set-listen-port f140
✔ Listen port · 51820
✔ Set listen port to 51820? · yes

[E] Address already in use (os error 98)

Which is the same on the working server:

# netstat -tulpan | grep innernet
tcp        0      0 10.40.0.1:51820         0.0.0.0:*               LISTEN      245227/innernet-ser

network: f512
  listening port: 41748
  peer: innernet-server (/BIWGvOlmO...)
    ip: 10.12.0.1
    endpoint: <serverf140_ip>:51820
    last handshake: 2 minutes, 12 seconds ago
    transfer: 26.21 KiB received, 79.72 KiB sent
  peer: f140 (qmnYdpi022...)
    ip: 10.12.1.1

# innernet set-listen-port f512
✔ Listen port · 51820
✔ Set listen port to 51820? · yes

[E] Address already in use (os error 98)

Both are servers...both are peers to the other server. (hopefully that isn't a problem..). So it seems that is set correctly to me?

As an example of things that don't seem to work: If I try to:
querying the 'messy' server from its admin peer:

# innernet list-associations f140
[*] Fetching CIDRs
[*] Fetching associations

querying the 'clean' server from its admin peer:

# innernet list-associations f140
[*] Fetching CIDRs
    
[E] http://10.12.0.1:51820/v1/admin/cidrs: Connection Failed: Connect error: connection timed out

Querying from either admin peer (the opposite server to the innernet-server in respective cases) innernet list --tree returns the correct CIDR structure.

I'm not sure what to do in terms of traffic. I'm at an early setup stage, and don't really know how to use it for anything yet...although pinging seems to work.

Can one join the server to itself as a peer? Or will that cause a problem (I may have some legacy config where I tried that...I seem to have that on the 'clean' non-working server)?

I tried using the other cloud server as a peer for the cloud server because I couldn't get a connection from my home network...although maybe that was because I hadn't forwarded the wireguard port to my home computer (behind NAT). Can this punch through, or do I need to port forward? How can I do that for multiple machines on my home network if so? Can something like opnsense be used as a router and route for this if so?

So many scattergun questions...sorry...I'm trying to get my head around it.

[tetricky]

TL;DR

---

I need to find an appropriate firewall rule that allows innernet to function - on both client and server.
I also need to work on how to port forward for multiple clients through the same NAT.

Some (maybe not complete?) progress.

I've re-built a network...just with a single additional CIDR, and only connected from a remote peer (no server for different interface on the peer). This initially returned exactly the same result.

I introduced a firewall route to allow to anywhere from the server CIDR . This made the (previously not connecting) 'clean' server work. This a security risk (allowing bogons seems highly problematic) - ? The messy server previously 'worked' without this rule in place...so presumably had something else happening that allowed it to work. Maybe it didn't really work?

Could I restrict the rule to something exclusively local and retain functionality...like the innernet server ip address....or the localhost ip, or an interface? I don't know what innernet is sending it's control from...what might be required on a server, but also on a client, in terms of most restrictive but functional firewall rules?

The 'clean' server that wasn't working had these set in /etc/sysctl.conf

net.ipv4.ip_forward=1
net.ipv6.conf.all.forwarding=1

Could that have interfered with innernet's routing? It remains an unknown why one server appeared to work...and another didn't. Maybe it didn't really work? Certainly it seems more logical that it works with ufw allow from <server_CIDR> added. Than not.

Anyway, with a firewall off and innernet listening port forwarded through a NAT, and also on an internet facing server, with a firewall, but with the allow CIDR rule (and after an innernet fetch to refresh the clients), all clients can connect and ping each other prroperly.

I need to find an appropriate firewall rule that allows innernet to function - on both client and server.

I also need to work on how to port forward for multiple clients through the same NAT. Does innernet support different listening ports for clients? I assume so. That's what I will try.

---edit---
It seems that the connectivity remains if the server CIDR rule is removed after handshaking or an innernet fetch. So it would seem to be innernet-server control pane related.

[bschwind]

Sorry for the radio silence, I've been on an extended vacation but intend to reply when I'm back.

[tetricky]

Absolutely no problem at all. You are entitled to life stuff, I do not expect full focus on what may not be an issue in scope. I intend to experiment more, but currently addressing other issues.

[bschwind]

I'm afraid I won't be able to address everything in your comments as I don't have much experience debugging the act of setting up innernet servers and such. I have set up a few, but all were on public IPs with port forwarding and things worked from the start.

Do you think you could recreate this scenario with a set of docker containers? I know that's asking a lot, but I'm not sure how else I could help debug the state of a running system.

Does innernet support different listening ports for clients?

Yes, you can run innernet set-listen-port <INTERFACE> to change the port the client will bind to.

You can also run innernet override-endpoint <INTERFACE> to change the endpoint to use for the client, where you can also specify which port others should use to contact you (typically this port is the "external" port on your NAT, not the "internal" port your client has bound to).

I would also like to soon add an option to only specify an external port without having to specify an IP address endpoint as well (#308)

[tetricky]

Apologies for the late reply. I've just had some time to drop back onto this.

"Do you think you could recreate this scenario with a set of docker containers? I know that's asking a lot, but I'm not sure how else I could help debug the state of a running system."

No chance. I don't use docker (podman or k3s). I am sufficiently unsure of the networking within a docker environment, and the use of ufw/firewall within it...it's just going to cause way more problems to set up than it will possibly solve.

I have cloud servers, the actual use case that I am trying to address, that I can provide information on.

I will have a look today at playing with innernet set-listen-port <INTERFACE> and innernet override-endpoint <INTERFACE> to see if forcing traffic to open ports solves the firewall problem.

You say: "all were on public IPs with port forwarding and things worked from the start." - Which ports were forwarded? All? My servers are also with public ip's and port forwarded (but when I put up the firewall something breaks - so I'm not allowing all the traffic that I need to through the firewall).

Thank you for taking the time to reply.