[NUTCH-2300] Fetcher to optionally save robots.txt

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.13
• Component/s: fetcher, protocol, segment
• Patch Info: Patch Available
• Flags: Patch
• Assignee: Sebastian Nagel
• Reporter: Sebastian Nagel
• Created: 19/Aug/16 13:42
• Updated: 22/Aug/16 21:52
• Resolved: 22/Aug/16 21:51

Description

For debugging or archival purposes it may be useful to let Fetcher store the robots.txt response (content and HTTP status). Of course, this should be optional and not by default.

Issue Links

• links to: GitHub Pull Request #141