[NUTCH-2550] Fetcher fails to follow redirects

• Type: Bug
• Status: Closed
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 1.15
• Fix Version/s: 1.15
• Component/s: fetcher
• Assignee: Unassigned
• Reporter: Hans Brende
• Created: 07/Apr/18 23:16
• Updated: 01/Oct/19 14:29
• Resolved: 10/Apr/18 22:52

Description

As I detailed in this github comment, it appears that PR #221 broke redirects. The fetcher will repeatedly fetch the original url rather than the one it's supposed to be redirecting to until http.redirect.max is exceeded, and then end with STATUS_FETCH_GONE.

I noticed this issue when I was trying to crawl a site with a 301 MOVED PERMANENTLY status code.

Should be pretty easy to fix though: I was able to get redirects working again simply by inserting the code

url = fit.url

here and here.

Issue Links

• is caused by: NUTCH-2375 Upgrade the code base from org.apache.hadoop.mapred to org.apache.hadoop.mapreduce
• links to: GitHub Pull Request #309