Bot check failing on many bots due to only checking the comment

[msaspence]

Loads of bots are being missed because the bot check is only checking content[:comment]

Some that are coming through for use and not being caught include:

facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)
compatible; Googlebot/2.1; +http://www.google.com/bot.html
compatible; YandexBot/3.0; +http://yandex.com/bots
compatible; PaperLiBot/2.1; http://support.paper.li/entries/20023257-what-is-paper-li
compatible; AhrefsBot/4.0; +http://ahrefs.com/robot/
Twitterbot/1.0
LinkedInBot/1.0 (compatible; Mozilla/5.0; Jakarta Commons-HttpClient/3.1 +http://www.linkedin.com)
http://showyou.com/crawler
compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm
compatible; TweetmemeBot/3.0; +http://tweetmeme.com/
+http://search.msn.com/msnbot.htm

Is there a good reason why only the comment section is checked?
user_agents starting "compatible;" dont seem to be parsed at all

[kevinelliott]

Thanks, I'll look into it.

On Jan 18, 2013, at 4:59 AM, msaspence [email protected] wrote:

Loads of bots are being missed because the bot check is only checking content[:comment]

Some that are coming through for use and not being caught include:

facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)
compatible; Googlebot/2.1; +http://www.google.com/bot.html
compatible; YandexBot/3.0; +http://yandex.com/bots
compatible; PaperLiBot/2.1; http://support.paper.li/entries/20023257-what-is-paper-li
compatible; AhrefsBot/4.0; +http://ahrefs.com/robot/
Twitterbot/1.0
LinkedInBot/1.0 (compatible; Mozilla/5.0; Jakarta Commons-HttpClient/3.1 +http://www.linkedin.com)
http://showyou.com/crawler
compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm
compatible; TweetmemeBot/3.0; +http://tweetmeme.com/
+http://search.msn.com/msnbot.htm

—
Reply to this email directly or view it on GitHub.

[be9]

Implemented named-based check in PR #38