[NUTCH-2445] Fetcher following outlinks to keep track of already fetched items

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.13
• Fix Version/s: 1.14
• Patch Info: Patch Available
• Assignee: Markus Jelsma
• Reporter: Markus Jelsma
• Created: 20/Oct/17 12:42
• Updated: 23/Oct/17 14:54
• Resolved: 23/Oct/17 13:59

Description

When fetcher.follow.outlinks.depth is non-zero, fetcher follows outlinks. This patch keeps track of already fetched URL's and thus avoid fetching the same URL twice.

A Set is used to keep track of them, hashcodes to reduce memory usage. This is not used if fetcher doesn't follow outlinks.