Network error during robots.txt fetch causes file to be ignored

• Key: NUTCH-105
• Type: Bug
• Status: Closed
• Resolution: Fixed
• Priority: Critical
• Assignee: Unassigned
• Reporter: Rod Taylor
• Component/s: fetcher
• Affects Version/s: 0.8, 0.8.1, 0.9.0
• Fix Version/s: 0.8.1, 0.9.0

Description

Earlier we had a small network glitch which prevented us from retrieving
the robots.txt file for a site we were crawling at the time:

nutch-root-tasktracker-sbider1.sitebuildit.com.log:051005 193021
task_m_h02y5t Couldn't get robots.txt for
http://www.japanesetranslator.co.uk/portfolio/:
org.apache.commons.httpclient.ConnectTimeoutException: The host
did not accept the connection within timeout of 10000 ms
nutch-root-tasktracker-sbider1.sitebuildit.com.log:051005 193031
task_m_h02y5t Couldn't get robots.txt for
http://www.japanesetranslator.co.uk/translation/:
org.apache.commons.httpclient.ConnectTimeoutException: The host
did not accept the connection within timeout of 10000 ms

Nutch then assumed that because we were unable to retrieve the file due
to network issues, that it didn't exist and we could crawl the entire
website. Nutch then successfully grabbed a few pages which were listed
in the robots.txt as being disallowed.

I think Nutch should continue attempting to retrieve the robots.txt file
until, at very least, we are able to establish a connection to the host,
otherwise the host should be ignored until the next round of fetches.

The webmaster of japanesetranslator.co.uk filed a complaint informing us
of the issue.