Downtime for https://opnxng.com

[extremelyonline]

My instance has been under a swarm of bots/crawlers attack. I got logs like *** uWSGI listen queue of socket "0.0.0.0:8080" (fd: 4) full !!! (101/100) *** in searxng and a lot of ERR logs in caddy. (The attacks are mostly from China.)

The site is very slow and even inaccessible for most of today. I'm trying to troubleshot this. Meanwhile, my instance is unstable and maybe shut down.

[extremelyonline]

I just installed a Caddy Maxmind module. The problem went away after blocking the ips from China...

The engines have banned my ip still, but I think it will resolve itself.

Is geo ip blocking acceptable for public instances? Thank you.

[unixfox]

I'm not sure if it is acceptable or not.
We technically don't have a rule for that.

This could be considered as applying some defenses against bots which better than having a bad experience for the users of the instance.

I think what would be better is to block only the range of IP addresses that attack your instance and not the whole China country.

@return42 what do you think about that?

[extremelyonline]

May be too soon to tell, but it seems the module has improved the engine errors in #321.

Will try isolating the IP range or the city when I have the time.

[extremelyonline]

Tried lifting the ban, the engines have Too Many Requests right away.

The ips came from different provinces and are hard to isolate. I do notice that they are all made with non-standard remote_port. I'll enforce the ban for now and look for better ways to block bots.

[return42]

Is geo ip blocking acceptable for public instances? Thank you.

@return42 what do you think about that?

I think its up to the maintainer for which auditorium he wants to provide a search engine / I myself don't have geo IP blocking, but my configuration is optimized for a German audience.

I also get bots from China .. when I see my ddg or google engine gets blocked, I debug my instance for 5min and see what IPs are making trouble .. then I add this IPs to a blocklist I implemented on top of SearXNG's master: return42/searxng@5ec0956542

---

On the long term I want to get rid of this block list and want to improve the bot detection of the limiter --> searxng/searxng#2357