You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
It would be nice if the "robots.txt request timed out" would tell for which uri it actually failed.
This is the method in question:
def self.get_robots_txt(uri, user_agent)
begin
Timeout::timeout(Robots.timeout) do
io = URI.join(uri.to_s, "/robots.txt").open("User-Agent" => user_agent) rescue nil
end
rescue Timeout::Error
STDERR.puts "robots.txt request timed out"
end
end
The text was updated successfully, but these errors were encountered:
It would be nice if the "robots.txt request timed out" would tell for which uri it actually failed.
This is the method in question:
The text was updated successfully, but these errors were encountered: