[NUTCH-2776] Fetcher to temporarily deduplicate followed redirects

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Implemented
• Affects Version/s: 1.16
• Fix Version/s: 1.17
• Component/s: fetcher
• Patch Info: Patch Available
• Assignee: Sebastian Nagel
• Reporter: Sebastian Nagel
• Created: 20/Mar/20 18:52
• Updated: 28/Jan/21 13:16
• Resolved: 30/Apr/20 08:43

Description

If fetcher follows redirect (http.redirect.max > 0), it may happen that many redirects of a site point to the same URL. In this situation, it might be good if fetcher could temporarily (for a configurable time period) deduplicate the redirect targets and skip all redirects except the first one. Typical examples of duplicated redirect targets are:

• instead of responding with HTTP status 404:

  /
  /resource-not-found
  /search/
  /404
  /error/not-found
  /err/notfound.html

• a page to accept/decline cookies

  /cookie_usage.php
  

Issue Links

• duplicates: NUTCH-1150 http.redirect.max can lead to multiple parses of the same url
• links to: GitHub Pull Request #505