fail2ban behind Traefik

[pdehne]

I followed the Mailserver behind Proxy documentation, to put my DMS behind a Traefik reverse proxy. I did not enable proxy on existing ports, but configured services with separate ports for the PROXY protocol just like it is described in the linked document.

Mail sending / receiving works but it seems fail2ban is not able to ban ips with this configuration. Even if I add ips manually using setup fail2ban ban, I still see many connection attempts from the banned ips.

Can anyone point me into the right direction how to change the fail2ban configuration in a way that it works behind Traefik. Or is it not possible at all?

Many thanks!

[glime-ai[bot]]

Please note that this response was created using generative AI, leveraging previously available information from the repository. This information may be helpful. However, it is our strong recommendation that you carefully consider the information before blindly applying it.

This response is generated only once - you will not be able to interact with the glime-ai.

---

When using Docker Mailserver (DMS) behind a Traefik reverse proxy with the PROXY protocol, Fail2Ban may face challenges in banning IPs because the original client IP is preserved by the PROXY protocol, but the banning mechanism might not be correctly applied.

Here are some steps and considerations to help you configure Fail2Ban to work in this setup:

1. Ensure Fail2Ban is Monitoring the Correct Logs:
   Fail2Ban should be configured to monitor the Postfix and Dovecot logs, which will contain the original client IPs due to the PROXY protocol.

2. Adjust Fail2Ban Configuration:
   You may need to adjust the Fail2Ban configuration to ensure it correctly interprets the logs and applies bans. This might involve customizing the jail configurations to match the log entries.

3. Network Configuration:
   If Traefik and DMS are on the same host, Fail2Ban should be able to modify the host's firewall to ban the IPs. However, if they are on different nodes (e.g., in a Kubernetes setup), Fail2Ban might only see the proxy IP and not the original client IP.

4. Manual Banning:
   When manually banning IPs using the setup.sh fail2ban ban <IP> command, ensure that the IPs are correctly added to the firewall rules. You might need to verify the firewall rules to ensure they are applied as expected.

5. Considerations for Kubernetes or Multi-Node Setups:
   In a Kubernetes or similar multi-node setup, you might need to implement additional network policies or use a centralized firewall management tool to ensure that bans are applied across nodes.

6. Documentation and Community Support:
   Refer to the Mailserver behind Proxy documentation for detailed guidance. Additionally, you can seek support from the community or issue tracker if you encounter specific problems.

For more detailed guidance, you can refer to the following sources:

• Mailserver behind Proxy
• Fail2Ban Configuration

This comment was generated by Glime.

[polarathene]

Mail sending / receiving works but it seems fail2ban is not able to ban ips with this configuration. Even if I add ips manually using setup fail2ban ban, I still see many connection attempts from the banned ips.

• Traefik is directly connected to the client right? No other proxy/load-balancer in-between?
• Traefik and DMS are operating on the same host?
• Without Traefik, if using DMS ports directly (you could temporary map port 587/465 to public interface instead of traefik), does this ban correctly?
• Are you using Docker Compose, not another container runtime or abstraction (Portainer)?

---

I am not familiar with Fail2Ban that much, this is an area of expertise for @casperklein . If you could provide that additional information, it would likely be helpful for them.

From what I understand Fail2Ban should be monitoring the Postfix and Dovecot logs for the client IPs. These should be the proper client IP with PROXY protocol configured correctly.

The banning itself AFAIK is via a syscall to the kernel or something like this to interface with iptables/nftables to apply the ban.

When the connection is made to the Postfix or Dovecot service from Traefik, I think this would always be allowed. So for the IP to be banned effectively, it needs to be the one connecting to the host, which should prevent even Traefik receiving the connection.

PROXY protocol should avoid any issues with IPv6 to IPv4 (docker network gateway IP) mishaps that could happen with Docker. So I don't think it'd be related to this. You also mention you have explicitly tried banning the client IPs and they're still able to connect to the host 🤷‍♂️

---

Until @casperklein is available, my troubleshooting would be:

• Can I get a ban working without Traefik / PROXY protocol involved?
• If yes, can I get a ban to work by manually banning that IP.
• If yes, bring back Traefik and PROXY protocol into the mix. This is where it breaks?
  • An explicit manual ban should still work AFAIK, thus PROXY protocol shouldn't be relevant at this point and could be ignored.
  • Focus needs to be on why Traefik would receive a connection from the banned IP when DMS would not.

Without knowing fail2ban that well, I am not sure how much relevance the containerized service in DMS is. I would focus on replicating the ban manually on the host without it if necessary, as the ban itself needs to work correctly before anything else is relevant 😓

I know that with kubernetes when Traefik is run on a separate host, Fail2Ban is ineffective and another approach is required to have the ingress/gateway manage banned clients.

Once the manual banning is working as expected, you may still have an issue with Fail2Ban monitoring Postfix/Dovecot if ports are relevant context. I'm not quite sure on what monitoring rules are used there.

[pdehne]

• Traefik is directly connected to the client right? No other proxy/load-balancer in-between?
• Traefik and DMS are operating on the same host?

There are no other proxy/Load-balancers in between. This is the configuration:

                       ┌──────────────────────────┐ 
             Port      │          Docker          │ 
  ┌──────┐ Forwarding  │ ┌───────┐       ┌─────┐  │ 
  │Router├────────────►│ │Traefik├──────►│ DMS │  │ 
  └──────┘             │ └───────┘       └─────┘  │ 
 192.168.*.1           │ 172.21.0.4    172.21.0.5 │ 
                       └──────────────────────────┘ 
                              192.168.*.33                                                              

• Without Traefik, if using DMS ports directly (you could temporary map port 587/465 to public interface instead of traefik), does this ban correctly?

I am not planning to expose the DMS ports. Currently only Traefik knows about them. But I'll double check that anyway for testing purposes. Many thanks for the idea!

• Are you using Docker Compose, not another container runtime or abstraction (Portainer)?

Yes, I am using Docker Compose directly.

From what I understand Fail2Ban should be monitoring the Postfix and Dovecot logs for the client IPs. These should be the proper client IP with PROXY protocol configured correctly.

I can see that /var/log/mail.log shows the correct client IPs for e.g. smtp-proxyprotocol/postscreen:

2024-06-11T04:26:36.873278+02:00 mail smtp-proxyprotocol/postscreen[41379]: CONNECT from [185.208.158.235]:64844 to [172.21.0.4]:25
2024-06-11T04:26:36.908343+02:00 mail smtp-proxyprotocol/postscreen[41379]: PREGREET 11 after 0.04 from [185.208.158.235]:64844: EHLO User\r\n
2024-06-11T04:26:36.982454+02:00 mail smtp-proxyprotocol/postscreen[41379]: DISCONNECT [185.208.158.235]:64844
2

The banning itself AFAIK is via a syscall to the kernel or something like this to interface with iptables/nftables to apply the ban.

When the connection is made to the Postfix or Dovecot service from Traefik, I think this would always be allowed. So for the IP to be banned effectively, it needs to be the one connecting to the host, which should prevent even Traefik receiving the connection.

PROXY protocol should avoid any issues with IPv6 to IPv4 (docker network gateway IP) mishaps that could happen with Docker. So I don't think it'd be related to this. You also mention you have explicitly tried banning the client IPs and they're still able to connect to the host 🤷‍♂️

When you say host, you mean Docker itself? I am (currently) not using host networking. So I think neither Traefik nor DMS do know about the Docker host, its ip or firewall. So that wouldn't work in my configuration.

But this gives me an idea. I am wondering how fail2ban knows which ports to actually block. I have created a separate set of proxied services, SMTP on port 25 is proxied to 12525, ESMTP on port 465 to 10465 and Dovecot to 10993 as described in the tutorial about DMS behind proxy. There is a jail named postfix, but will this try to block port 25 or port 1252? In my environment it would need to block port 1252.

Until @casperklein is available, my troubleshooting would be:

• Can I get a ban working without Traefik / PROXY protocol involved?
• If yes, can I get a ban to work by manually banning that IP.
• If yes, bring back Traefik and PROXY protocol into the mix. This is where it breaks?

I will do these tests later this evening. Thanks for the pointers!

• An explicit manual ban should still work AFAIK, thus PROXY protocol shouldn't be relevant at this point and could be ignored.
• Focus needs to be on why Traefik would receive a connection from the banned IP when DMS would not.

Manual bans don't work. This is something I tried already. Currently my only idea ist that it is the wrong jail / port that get's banned.

I know that with kubernetes when Traefik is run on a separate host, Fail2Ban is ineffective and another approach is required to have the ingress/gateway manage banned clients.

I see. Maybe a separate fail2ban container that works with Traefik itself would be better for my environment. I plan to have other services exposed to the internet too. This would be a more general solution that could work for all my exposed services.

[polarathene]

Apologies for the late reply!

The banning itself AFAIK is via a syscall to the kernel or something like this to interface with iptables/nftables to apply the ban.

When the connection is made to the Postfix or Dovecot service from Traefik, I think this would always be allowed. So for the IP to be banned effectively, it needs to be the one connecting to the host, which should prevent even Traefik receiving the connection.

PROXY protocol should avoid any issues with IPv6 to IPv4 (docker network gateway IP) mishaps that could happen with Docker. So I don't think it'd be related to this. You also mention you have explicitly tried banning the client IPs and they're still able to connect to the host 🤷‍♂️

When you say host, you mean Docker itself? I am (currently) not using host networking. So I think neither Traefik nor DMS do know about the Docker host, its ip or firewall. So that wouldn't work in my configuration.

I refer to the host that Docker runs on, aka Docker host. Nothing to do with host mode networking within Docker.

The networking still occurs from the host public interfaces, and then is routed to the docker networks to reach the containers.

AFAIK the way Fail2Ban integration works (which requires you to relax container security by granting additional capability required), it will configure the ban to be effective on the host itself. It is container / docker agnostic.

---

But this gives me an idea. I am wondering how fail2ban knows which ports to actually block.

I am not 100% sure, but assume the ban is port agnostic?

If not that is a good point if you have a port mismatch from what would come from the host port, as that would be the port that is banned I think? @casperklein may have better input on this.

[pdehne]

I created a fail2ban-jail.cf and added the proxied ports.

[dovecot]
port    = pop3,pop3s,imap,imaps,submission,465,sieve,12525,10465,10993

[postfix]
port    = smtp,465,submission

[postfix-sasl]
port     = smtp,465,submission,imap,imaps,pop3,pop3s,12525,10465,10993

[custom]
port = smtp,pop3,pop3s,imap,imaps,submission,submissions,sieve,12525,10465,10993

I'd say fail2ban correctly detects malicious actors, but the ban does not acutally work.

2024-06-11 08:01:21,518 fail2ban.filter         [1043]: INFO    [dovecot] Found 194.169.175.20 - 2024-06-11 08:01:21
2024-06-11 08:01:21,762 fail2ban.actions        [1043]: WARNING [dovecot] 194.169.175.20 already banned
2024-06-11 08:01:51,264 fail2ban.filter         [1043]: INFO    [dovecot] Found 194.169.175.20 - 2024-06-11 08:01:51

According to documentation all traffic from these ips, e.g. 194.169.175.20 should be dropped. So my understanding is that the last entry should not be possible because the ip is already banned. Banning is probably not working because the connection is proxied and the firewall adds the source ip and doesn't know about Traefik. I have currently no idea how to change that.

I'll try your suggestions above just to make sure too, later this evening.

[polarathene]

Banning is probably not working because the connection is proxied and the firewall adds the source ip and doesn't know about Traefik. I have currently no idea how to change that.

How are you verifying the bans btw? If you are using the same host system and not a separate one to make the connection, this could be misleading. Especially with connections to/from Docker containers.

You could look into the /etc/docker/daemon.json setting to disable userland-proxy (see our IPv6 docs for a snippet). This would disable some port mapping routing logic Docker does to try be helpful (IPv6 host to IPv4 container, localhost IPv4/IPv6 loopback routing to containers).

[pdehne]

Just to make sure I removed fail2ban-jail.cf and double checked the fail2banl logs. Detection still works. The problem seems to be that fail2ban can not enforce the ban.

nft list ruleset returns the following which look like the rules for dropping traffic from the banned ips are set correctly for the detected source ip. But the firewall probably doesn't know that these are proxied through Traefik and will let them through because the originating ip is Traefik and not the source ip.

table inet f2b-table {
        set addr-set-postfix {
                type ipv4_addr
                flags interval
                elements = { 194.169.175.10 }
        }

        set addr-set-dovecot {
                type ipv4_addr
                flags interval
                elements = { 194.169.175.10, 194.169.175.20,
                             217.86.191.38 }
        }

        set addr-set-postfix-sasl {
                type ipv4_addr
                flags interval
                elements = { 194.169.175.10, 194.169.175.20 }
        }

        set addr-set-custom {
                type ipv4_addr
                flags interval
                elements = { 194.169.175.10, 194.169.175.20 }
        }
}

[polarathene]

nft list ruleset returns the following which look like the rules for dropping traffic from the banned ips are set correctly for the detected source ip.

At a glance from this article, I assume you have the other rules for the actual ban that do seem to be port oriented?

These rules should apply at the host when doing the connection routing. It's been a while since I was familiar with the syntax but the screenshot I think is showing that the source IP (saddr) connecting to the destination port (dport) will be rejected if there's a match to the banned IP.

---

Not nftables but iptables focused (and nothing to do with f2b), I documented a routing issue in the past: moby/moby#45610

Might be a useful resource for how I approached testing a reproduction there. I lack the time right now to attempt a reproduction of the issue you're facing here, but without traefik our test suite has some F2B coverage that passes. Would be interesting to know if this is a problem Traefik introduces.

Docker itself may interfere with rules based on priorities. I know that it punches through any UFW / Firewalld firewall rules (even if these don't allow a port to be accessed, when you port map a container with Docker, the port could be accessed anyway due to it's rule insertion having higher precedence). That's potentially happening here with the routing? (unlikely since Fail2Ban works properly AFAIK without Traefik in the mix, but perhaps a possibility).

If you have firewalld, you may want to ensure docker zone is used, as my linked issue mentions when comparing to UFW.

Likewise, if your kernel or OS is dated, I know that can introduce problems. This has been the case with Fail2Ban not working previously for some users that deploy DMS on a NAS system. Those had kernels that weren't compatible.

[pdehne]

I tested some more.

Until @casperklein is available, my troubleshooting would be:
Can I get a ban working without Traefik / PROXY protocol involved?

Yes, this works. I just tested without Traefik.

If yes, can I get a ban to work by manually banning that IP.

Automatic ban without Traefik works. I did not test manual ban without Traefik. But I assume this should work too if automatic ban works.

If yes, bring back Traefik and PROXY protocol into the mix. This is where it breaks?

Yes this is where it breaks. The ip's get correctly detected, but the ban is not enforced, I see the ips still trying to connect in fail2ban.log. Without Traefik the ips will no longer show up in fail2ban.log. When I do nft list ruleset in the container, one time with Traefik and one time wihtout Traefik, the two outputed rulesets look exactly the same.

table ip nat {
        chain DOCKER_OUTPUT {
                meta l4proto tcp ip daddr 127.0.0.11 XT match tcp not found
 counter packets 3 bytes 180 XT target DNAT not found

                meta l4proto udp ip daddr 127.0.0.11 XT match udp not found
 counter packets 93 bytes 7346 XT target DNAT not found

        }

        chain OUTPUT {
                type nat hook output priority -100; policy accept;
                ip daddr 127.0.0.11 counter packets 96 bytes 7526 jump DOCKER_OUTPUT
        }

        chain DOCKER_POSTROUTING {
                meta l4proto tcp ip saddr 127.0.0.11 XT match tcp not found
 counter packets 0 bytes 0 XT target SNAT not found

                meta l4proto udp ip saddr 127.0.0.11 XT match udp not found
 counter packets 0 bytes 0 XT target SNAT not found

        }
                                                                                                                                                                                                                                                    chain POSTROUTING {
                type nat hook postrouting priority srcnat; policy accept;
                ip daddr 127.0.0.11 counter packets 96 bytes 7526 jump DOCKER_POSTROUTING
        }
}
table inet f2b-table {
        set addr-set-dovecot {
                type ipv4_addr
                flags interval
                elements = { 194.169.175.10, 194.169.175.20,                                                                                                                                                                                                             194.169.175.66 }
        }

        set addr-set-postfix {                                                                                                                                                                                                                              type ipv4_addr
                flags interval
                elements = { 194.169.175.10, 194.169.175.20,
                             194.169.175.66 }
        }

        set addr-set-postfix-sasl {
                type ipv4_addr
                flags interval
                elements = { 194.169.175.10, 194.169.175.20,
                             194.169.175.66 }                                                                                                                                                                                                       }

        chain f2b-chain {
                type filter hook input priority filter - 1; policy accept;
                meta l4proto tcp ip saddr @addr-set-dovecot drop
                meta l4proto tcp ip saddr @addr-set-postfix drop
                meta l4proto tcp ip saddr @addr-set-postfix-sasl drop
        }
}

At a glance from this article, I assume you have the other rules for the actual ban that do seem to be port oriented?
These rules should apply at the host when doing the connection routing.

Yes, they are in the f2b-chain of the container, not the host. This is the same if Traefik is used or not though.

How are you verifying the bans btw?

There are three IPs that try to do a brute foce attack all the time. So this is a real world scenario :-)

I am out of ideas at the moment what the problem could be or where to look.

[pdehne]

I am not sure what is happening for Traefik to affect the F2B ban like that. I thought the ban would have been applied to clients connecting to the host and containers wouldn't have had relevance as the connection should have been banned before then.

The rules are applied in the container as can be seen in the nft rules output from the container. So as you say, maybe that's the problem after all. When proxying, these rules must be applied on the docker host, not the container. Or, again as you say, it could only work if Traefik is using the HAProxy protocol somehow to inform the firewall in the container of the real source ip addresses, but if this actually happens, or if there is a problem in that area, I don't know. Here my networking knowledge ends :-)

For now I removed Traefik and use direct port forwarding to the docker mailserver. This way fail2ban is working. I am thinking about a more general solution, that will work for my other services that are exposed to the internet as well, anyway. So while I am happy to further help tracking this down, it is not that important for me in the long run. It seems this is a not often used edge case, so it may be perfectly fine to just say fail2ban behind Traefik is not supported.

I may have a question / suggestion to make it easier to get my more general solution going, but I'll do that in another thread.

[polarathene]

The rules are applied in the container as can be seen in the nft rules output from the container. So as you say, maybe that's the problem after all.

Oh? I thought for sure that the rules were applied external of the container. Hence the need for the additional capabilities. Docker manages it's network rules via iptables on the Docker host for each container.

That definitely explains the problem though if F2B isn't blocking connections at the host-level though 😓

[casperklein]

A bit late to the party: I can confirm, that the rules take place in the container, not the host.

[polarathene]

the rules take place in the container, not the host.

I suppose we should document that then? I don't think we mention that using a reverse proxy would break compatibility with the internal F2B?

I am a little surprised that it's restricted to the container given the need for the additional kernel capability 🤔 I assume there isn't a way to adjust the ban to be at the host level?

[pdehne]

the rules take place in the container, not the host.

I suppose we should document that then? I don't think we mention that using a reverse proxy would break compatibility with the internal F2B?

I am a little surprised that it's restricted to the container given the need for the additional kernel capability 🤔 I assume there isn't a way to adjust the ban to be at the host level?

A (iptables) DOCKER-USER chain is mentioned here: https://github.com/linuxserver/fail2ban-confs/blob/master/README.md#docker-user that should be able to drop connections to all docker containers. Maybe this can be utilized with nftables too?