[NUTCH-2754] fetcher.max.crawl.delay ignored if exceeding 5 min. / 300 sec.

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.16
• Fix Version/s: 1.17
• Component/s: fetcher, robots
• Assignee: Unassigned
• Reporter: Sebastian Nagel
• Created: 13/Nov/19 13:51
• Updated: 28/Jan/21 13:16
• Resolved: 23/Dec/19 10:57

Description

Sites specifying a Crawl-Delay of more than 5 minutes (301 seconds or more) are always ignored, even if fetcher.max.crawl.delay is set to a higher value.

We need to pass a higher value of fetcher.max.crawl.delay to crawler-commons' robots.txt parser otherwise it will use the internal default value of 300 sec. and disallow all sites specifying a longer Crawl-Delay in their robots.txt.

Issue Links

• links to: GitHub Pull Request #487