Open
Description
It would be nice if the "robots.txt request timed out" would tell for which uri it actually failed.
This is the method in question:
def self.get_robots_txt(uri, user_agent)
begin
Timeout::timeout(Robots.timeout) do
io = URI.join(uri.to_s, "/robots.txt").open("User-Agent" => user_agent) rescue nil
end
rescue Timeout::Error
STDERR.puts "robots.txt request timed out"
end
end
Metadata
Metadata
Assignees
Labels
No labels