[NUTCH-2398] Fetcher saving redirected robots.txt under redirect target URL

• Type: Bug
• Status: Closed
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 1.13
• Fix Version/s: 1.14
• Component/s: fetcher
• Patch Info: Patch Available
• Assignee: Sebastian Nagel
• Reporter: Sebastian Nagel
• Created: 05/Jul/17 16:11
• Updated: 13/Mar/24 14:51
• Resolved: 17/Jul/17 14:31

Description

NUTCH-2300 lets the Fetcher store optionally the robots.txt response (content and HTTP status). If the '.../robots.txt' is redirected, the redirected content is also stored but with the redirect source URL as key. It should use the redirect target URL instead. Otherwise one of the responses is overwritten in the segments map file.