[NUTCH-293] support for Crawl-delay in Robots.txt

• Type: Improvement
• Status: Closed
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 0.8
• Fix Version/s: 0.8
• Component/s: fetcher
• Assignee: Unassigned
• Reporter: Stefan Groschupf
• Created: 02/Jun/06 00:22
• Updated: 19/Jul/06 22:05
• Resolved: 19/Jul/06 22:05

Description

Nutch need support for Crawl-delay defined in robots.txt, it is not a standard but a de-facto standard.
See:
http://help.yahoo.com/help/us/ysearch/slurp/slurp-03.html
Webmasters start blocking nutch since we do not support it.