I cannot get bouncer to deny http/https connection with Docker but blocking ping works

[vspaziani]

As the title suggests, I have crowdsec-firewall-bouncer setup and it appears to be working. When I ping the machine it does appear to be blocked but when I navigate to a hosted webpage I am still able to connect. I believe the issue is with the connection to Docker. The full setup I have is MailCow running in Docker. I followed the instructions at Protecting Mailcow with Crowdsec and went a little further than the instructions by modifying the bouncer config to include the forward and docker-user chains. Any help in troubleshooting would be great!

crowdsec-firewall-bouncer.yaml (I had to put the ticks so it was formatted correctly)

mode: iptables
pid_dir: /var/run/
update_frequency: 10s
daemonize: true
log_mode: file
log_dir: /var/log/
log_level: info
log_compression: true
log_max_size: 100
log_max_backups: 3
log_max_age: 30
api_url: http://127.0.0.1:8080/
api_key: API KEY**
insecure_skip_verify: false
disable_ipv6: false
deny_action: DROP
deny_log: false
supported_decisions_types:
' - ban
#to change log prefix
#deny_log_prefix: "crowdsec: "
#to change the blacklists name
blacklists_ipv4: crowdsec-blacklists
blacklists_ipv6: crowdsec6-blacklists
#type of ipset to use
ipset_type: nethash
#if present, insert rule in those chains
iptables_chains:
' - INPUT
' - FORWARD
' - DOCKER-USER

nftables

nftables:
ipv4:
enabled: true
set-only: false
table: crowdsec
chain: crowdsec-chain
priority: -10
ipv6:
enabled: true
set-only: false
table: crowdsec6
chain: crowdsec6-chain
priority: -10

nftables_hooks:
' - input
' - forward

' # packet filter
pf:
' # an empty string disables the anchor
anchor_name: ""

prometheus:
prometheus:
enabled: true
listen_addr: 127.0.0.1
listen_port: 60601

crowdsec-firewall-bouncer.log

time="05-08-2023 16:52:20" level=info msg="backend type : iptables"
time="05-08-2023 16:52:20" level=info msg="config is valid"
time="05-08-2023 16:52:20" level=info msg="backend type : iptables"
time="05-08-2023 16:52:20" level=info msg="iptables for ipv4 initiated"
time="05-08-2023 16:52:20" level=info msg="iptables clean-up : /usr/sbin/iptables -D INPUT -m set --match-set crowdsec-blacklists src -j DROP"
time="05-08-2023 16:52:20" level=error msg="error while removing set entry in iptables : exit status 1 --> iptables: Bad rule (does a matching rule exist in that chain?).\n"
time="05-08-2023 16:52:20" level=info msg="iptables clean-up : /usr/sbin/iptables -D FORWARD -m set --match-set crowdsec-blacklists src -j DROP"
time="05-08-2023 16:52:20" level=error msg="error while removing set entry in iptables : exit status 1 --> iptables: Bad rule (does a matching rule exist in that chain?).\n"
time="05-08-2023 16:52:20" level=info msg="iptables clean-up : /usr/sbin/iptables -D DOCKER-USER -m set --match-set crowdsec-blacklists src -j DROP"
time="05-08-2023 16:52:20" level=error msg="error while removing set entry in iptables : exit status 1 --> iptables: Bad rule (does a matching rule exist in that chain?).\n"
time="05-08-2023 16:52:20" level=info msg="ipset clean-up : /usr/sbin/ipset -exist destroy crowdsec-blacklists"
time="05-08-2023 16:52:20" level=info msg="Checking existing set"
time="05-08-2023 16:52:20" level=info msg="ipset set-up : /usr/sbin/ipset -exist create crowdsec-blacklists nethash timeout 300 maxelem 65536"
time="05-08-2023 16:52:21" level=info msg="Rule doesn't exist (/usr/sbin/iptables -C INPUT -m set --match-set crowdsec-blacklists src -j DROP)"
time="05-08-2023 16:52:21" level=info msg="Rule doesn't exist (/usr/sbin/iptables -C FORWARD -m set --match-set crowdsec-blacklists src -j DROP)"
time="05-08-2023 16:52:21" level=info msg="Rule doesn't exist (/usr/sbin/iptables -C DOCKER-USER -m set --match-set crowdsec-blacklists src -j DROP)"
time="05-08-2023 16:52:21" level=info msg="iptables set-up : /usr/sbin/iptables -I INPUT -m set --match-set crowdsec-blacklists src -j DROP"
time="05-08-2023 16:52:21" level=info msg="iptables set-up : /usr/sbin/iptables -I FORWARD -m set --match-set crowdsec-blacklists src -j DROP"
time="05-08-2023 16:52:21" level=info msg="iptables set-up : /usr/sbin/iptables -I DOCKER-USER -m set --match-set crowdsec-blacklists src -j DROP"
time="05-08-2023 16:52:21" level=info msg="iptables for ipv6 initiated"
time="05-08-2023 16:52:21" level=info msg="iptables clean-up : /usr/sbin/ip6tables -D INPUT -m set --match-set crowdsec6-blacklists src -j DROP"
time="05-08-2023 16:52:21" level=error msg="error while removing set entry in iptables : exit status 1 --> ip6tables: Bad rule (does a matching rule exist in that chain?).\n"
time="05-08-2023 16:52:21" level=info msg="iptables clean-up : /usr/sbin/ip6tables -D FORWARD -m set --match-set crowdsec6-blacklists src -j DROP"
time="05-08-2023 16:52:21" level=error msg="error while removing set entry in iptables : exit status 1 --> ip6tables: Bad rule (does a matching rule exist in that chain?).\n"
time="05-08-2023 16:52:21" level=info msg="iptables clean-up : /usr/sbin/ip6tables -D DOCKER-USER -m set --match-set crowdsec6-blacklists src -j DROP"
time="05-08-2023 16:52:21" level=error msg="error while removing set entry in iptables : exit status 1 --> ip6tables: Bad rule (does a matching rule exist in that chain?).\n"
time="05-08-2023 16:52:21" level=info msg="ipset clean-up : /usr/sbin/ipset -exist destroy crowdsec6-blacklists"
time="05-08-2023 16:52:21" level=info msg="Checking existing set"
time="05-08-2023 16:52:21" level=info msg="ipset set-up : /usr/sbin/ipset -exist create crowdsec6-blacklists nethash timeout 300 family inet6 maxelem 65536"
time="05-08-2023 16:52:22" level=warning msg="iptables check command (/usr/sbin/ip6tables -C INPUT -m set --match-set crowdsec6-blacklists src -j DROP) failed : exit status 1"
time="05-08-2023 16:52:22" level=warning msg="iptables check command (/usr/sbin/ip6tables -C FORWARD -m set --match-set crowdsec6-blacklists src -j DROP) failed : exit status 1"
time="05-08-2023 16:52:22" level=warning msg="iptables check command (/usr/sbin/ip6tables -C DOCKER-USER -m set --match-set crowdsec6-blacklists src -j DROP) failed : exit status 1"
time="05-08-2023 16:52:22" level=info msg="iptables set-up : /usr/sbin/ip6tables -I INPUT -m set --match-set crowdsec6-blacklists src -j DROP"
time="05-08-2023 16:52:22" level=info msg="iptables set-up : /usr/sbin/ip6tables -I FORWARD -m set --match-set crowdsec6-blacklists src -j DROP"
time="05-08-2023 16:52:22" level=info msg="iptables set-up : /usr/sbin/ip6tables -I DOCKER-USER -m set --match-set crowdsec6-blacklists src -j DROP"
time="05-08-2023 16:52:22" level=info msg="Using API key auth"
time="05-08-2023 16:52:22" level=info msg="Serving metrics at 127.0.0.1:60601/metrics"
time="05-08-2023 16:52:22" level=info msg="Processing new and deleted decisions . . ."
time="05-08-2023 16:53:19" level=info msg="15000 decisions added"
time="05-08-2023 16:57:42" level=info msg="1 decision added"

[LaurenceJJones]

Could you provide the output of iptables -L ?

[vspaziani]

Output of Sudo iptables -L

'# Warning: iptables-legacy tables present, use iptables-legacy to see them
Chain INPUT (policy ACCEPT)
target prot opt source destination
DROP all -- anywhere anywhere match-set crowdsec-blacklists src

Chain FORWARD (policy DROP)
target prot opt source destination
DROP all -- anywhere anywhere match-set crowdsec-blacklists src
DOCKER-USER all -- anywhere anywhere
DOCKER-ISOLATION-STAGE-1 all -- anywhere anywhere
ACCEPT all -- anywhere anywhere ctstate RELATED,ESTABLISHED
DOCKER all -- anywhere anywhere
ACCEPT all -- anywhere anywhere
ACCEPT all -- anywhere anywhere
ACCEPT all -- anywhere anywhere ctstate RELATED,ESTABLISHED
DOCKER all -- anywhere anywhere
ACCEPT all -- anywhere anywhere
ACCEPT all -- anywhere anywhere

Chain OUTPUT (policy ACCEPT)
target prot opt source destination

Chain DOCKER (2 references)
target prot opt source destination
ACCEPT tcp -- anywhere 172.22.1.249 tcp dpt:redis
ACCEPT tcp -- anywhere 172.22.1.5 tcp dpt:8983
ACCEPT tcp -- anywhere 172.22.1.7 tcp dpt:mysql
ACCEPT tcp -- anywhere 172.22.1.253 tcp dpt:submission
ACCEPT tcp -- anywhere 172.22.1.253 tcp dpt:submissions
ACCEPT tcp -- anywhere 172.22.1.253 tcp dpt:smtp
ACCEPT tcp -- anywhere 172.22.1.250 tcp dpt:12345
ACCEPT tcp -- anywhere 172.22.1.250 tcp dpt:sieve
ACCEPT tcp -- anywhere 172.22.1.250 tcp dpt:pop3s
ACCEPT tcp -- anywhere 172.22.1.250 tcp dpt:imaps
ACCEPT tcp -- anywhere 172.22.1.250 tcp dpt:imap2
ACCEPT tcp -- anywhere 172.22.1.250 tcp dpt:pop3
ACCEPT tcp -- anywhere 172.22.1.10 tcp dpt:8443
ACCEPT tcp -- anywhere 172.22.1.10 tcp dpt:8085

Chain DOCKER-ISOLATION-STAGE-1 (1 references)
target prot opt source destination
DOCKER-ISOLATION-STAGE-2 all -- anywhere anywhere
DOCKER-ISOLATION-STAGE-2 all -- anywhere anywhere
RETURN all -- anywhere anywhere

Chain DOCKER-ISOLATION-STAGE-2 (2 references)
target prot opt source destination
DROP all -- anywhere anywhere
DROP all -- anywhere anywhere
RETURN all -- anywhere anywhere

Chain DOCKER-USER (1 references)
target prot opt source destination
DROP all -- anywhere anywhere match-set crowdsec-blacklists src
RETURN all -- anywhere anywhere

[LaurenceJJones]

So everything that I expect to see within the chains is there. However, you might want to investigate Warning: iptables-legacy tables present, use iptables-legacy to see them this as there may be some rules that are not shown that are causing issues?

[vspaziani]

I think you are right. While I was waiting for you to reply before, I put in IPTable rules to deny traffic using the same IP list Crowdsec does on every chain but didn't do it on the legacy ones mostly because I didn't make that connection. So with a rule on all the chains I sent you before, I was still able to hit the web page. So it must be using the legacy tables or something. I was thinking that the IP tables are like firewall rules and the first one takes priority, my mistake was assuming that the legacy ones are called from the non-legacy ones. I will update you as soon as I know more. Thanks for all you help so far. Just the point in the IPtable direction has been a huge leap because I didn't know that was the mechanism blocking.

[vspaziani]

Ok so now I have tried a couple things and still cannot get it to work but have more information....

I removed all of the legacy IP tables and disabled their creation so that I could enable the bouncer. With the legacy ones removed, I still cannot get it to work correctly. The webpage is denied (as expected) if I hit the server with the IP:Port. Where I am having trouble is that when I try to load the webpage using the FQDN, through my reverse proxy, the webpage is not denied. I am going to try some additional tests like setting up the bouncer to deny access on port 25 or 465 but that might take me a little to setup. If you have any ideas or think I should try something else, any help is appreciated.

[LaurenceJJones]

To be honest, I got nothing more to add. The bouncers is doing what it suppose to do and injecting the rules to the top of the chains. Remember to constantly check iptables -L as our bouncer only injects the rules once then if they are flushed they wont be reinjected till bouncer restarts

[vspaziani]

OK I got it working. Thanks again to @LaurenceJJones for pointing me in the right direction. Your help allowed me to at least know what services to google to take deeper dives. There were several areas that needed to be modified on my part. I am documenting this all so that if anyone runs into a similar issue, maybe they can use my lessons learned to help.

First, the reason that I was still able to hit the webpage was that the mail server was behind a reverse proxy. I looked into how the data was being sent to the mailcow server by enabling logging on IPTABLES. When I looked at the data, it appears that the proxy server was making the HTTP requests. When an IP was banned, I needed it to be banned on the proxy server AND the mail server. It needs to be banned on both because I have all port 80 and 443 traffic port forwarded in my router to the proxy server but the SMTPS(465),POP(995)... all the ports for mail server stuff are forwarded to the mail server directly. There is a tutorial an setting up crowdsec with multiple servers How to set up a CrowdSec multi-server installation and CrowdSec Multiserver Docker. This issue being that I needed both since my proxy server is Ngnix Proxy Manager running in docker. That configuration did not expose the bouncer port (8080) so I had two modify the docker-compose file. The part :

Now edit /etc/systemd/system/crowdsec.service and add the -no-api

was really important as without it the 'mothership' (or server 2&3) would not receive heartbeats from the mail server.

fail2ban is automatically enabled on Mailcow. I had to modify the configuration so that it would not ban an IP before Crowdsec was able to. Again, the vision being that banned IP addresses be banned everywhere.

lastly, once I get everything working. I ran into an issue where a blocked IP would not be blocked by the individual machine but would be blocked by all the other machines- through the Proxy server. I took a look at the Bouncer logs and found an error "time="19-08-2023 14:56:04" level=info msg="Error while inserting in set (/usr/sbin/ipset -exist add crowdsec-blacklists 116.212.142.42 timeout 583196): exit status 1 --> ipset v7.15: Hash is full, cannot add more elements\n" this is already an identified issue #260. After I made the change, everything completely worked.