Limit for searx.work instance

[ice9smart]

Recently, there is a great amount of requests from China attacking my instance https://searx.work. These requests come from different IP and ISP, but they all simulate the same iOS device and share the same User-Agent header (according to my observation). Therefore, I forbid all requests with this User-Agent header: Mozilla/5.0 (iPhone; CPU iPhone OS 15_7 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/15.6,2 Mobile/15E148 Safari/604.1.

Corresponding NGINX config:

if ($http_user_agent = "Mozilla/5.0 (iPhone; CPU iPhone OS 15_7 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/15.6,2 Mobile/15E148 Safari/604.1") {
    return 403;
}

This discussion is created so that if some normal users are influenced, they may find a way to contact me and I will adjust the limit accordingly.

[return42]

Recently, there is a great amount of requests from China attacking my instance .. These requests come from different IP and ISP

Same to my instance .. and I suppose to many others. My first attempt was to identify the IP of the bots, but this is a never ending task since these bots have endless IP lists --> return42/searxng@736c3c2

Currently I'm working on another solution where the bots are recognized by the fact that they don't request a CSS.

• [mod] limiter: reduce request rates for requests without a ping searxng#2357

The first tests on my instance were already quite promising --> searxng/searxng#2357 (comment)

However, more tests need to be done and a subnet (/48) of the IPv6 has to be implemented ..

Hint: there are many more IP addresses available in the IPv6 address space and the limiter we currently have in SearXNG does not consider any IPv6 prefix yet --> Do you have bots routing over IPv6 to your host?

[unixfox]

I don't think it's a good idea to block bots using a generic user agent as it might very well block legitimate traffic.

Can't you block based on the ISP? Are those requests from datacenters?

[ice9smart]

Are those requests from datacenters?

No. They are mostly from China Telecom/Mobile/Unicom, which are the three major ISP in China. Blocking them means blocking nearly all visitors in China.