[NUTCH-2630] Fetcher to log skipped records by robots.txt

• Type: Improvement
• Status: Closed
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 1.15
• Fix Version/s: 1.16
• Component/s: fetcher
• Assignee: Unassigned
• Reporter: Markus Jelsma
• Created: 01/Aug/18 11:34
• Updated: 28/Jan/21 13:55
• Resolved: 14/Nov/18 12:05

Description

To analyze problems it would be helpful if fetcher logs URLs which are disallowed in the robots.txt - see discussion on user mailing list.

Issue Links

• links to: GitHub Pull Request #387