[NUTCH-578] URL fetched with 403 is generated over and over again

• Type: Bug
• Status: Closed
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 1.0.0
• Fix Version/s: 1.9
• Component/s: generator

Environment

Ubuntu Gutsy Gibbon (7.10) running on VMware server. I have checked out the most recent version of the trunk as of Nov 20, 2007

• Patch Info: Patch Available
• Assignee: Markus Jelsma
• Reporter: Nathaniel Powell
• Created: 20/Nov/07 21:39
• Updated: 13/Mar/24 14:51
• Resolved: 07/Jul/14 12:39

Description

I have not changed the following parameter in the nutch-default.xml:

<property>
<name>db.fetch.retry.max</name>
<value>3</value>
<description>The maximum number of times a url that has encountered
recoverable errors is generated for fetch.</description>
</property>

However, there is a URL which is on the site that I'm crawling, www.teachertube.com, which keeps being generated over and over again for almost every segment (many more times than 3):

fetch of http://www.teachertube.com/images/ failed with: Http code=403, url=http://www.teachertube.com/images/

This is a bug, right?

Thanks.

Issue Links

• is duplicated by: NUTCH-813 Repetitive crawl 403 status page
• relates to: NUTCH-1245 URL gone with 404 after db.fetch.interval.max stays db_unfetched in CrawlDb and is generated over and over again
• relates to: NUTCH-1247 CrawlDatum.retries should be int