You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I've been analyzing the top user agents on each of my websites daily, but I'm struggling to know if they are legitimate users or just crawlers. Every day I find a new user agent there, and I don't know where to investigate if they're are good or bad.
Is there a website or database that shares user agents and where are they from?
What do you guys do after reading the user agents that reach your websites?
Are there any crawlers that I shouldn't block?
Until today I've blocked this ones:
80legs
colly
ALittle
IonCrawl
Gregarius
axios
python-requests
Go-http-client
masscan
btc-lib-file
SemrushBot
Gulper
Ruby
FAST-WebCrawler
ELinks
Iframely
The text was updated successfully, but these errors were encountered:
I've been analyzing the top user agents on each of my websites daily, but I'm struggling to know if they are legitimate users or just crawlers. Every day I find a new user agent there, and I don't know where to investigate if they're are good or bad.
Is there a website or database that shares user agents and where are they from?
What do you guys do after reading the user agents that reach your websites?
Are there any crawlers that I shouldn't block?
Until today I've blocked this ones:
The text was updated successfully, but these errors were encountered: