[NUTCH-2730] SitemapProcessor to treat sitemap URLs as Set instead of List

• Type: Improvement
• Status: Closed
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 1.15
• Fix Version/s: 1.18
• Component/s: sitemap
• Assignee: Markus Jelsma
• Reporter: Markus Jelsma
• Created: 16/Aug/19 10:42
• Updated: 14/Jan/21 04:12
• Resolved: 14/Jul/20 10:49

Description

https://archive.epa.gov/robots.txt lists 160k sitemap URLs, absurd! Almost 160k of them are duplicates, no friendly words to describe this astonishing fact.

And although our Nutch locally chews through this list in 22s, for some weird reason the big job on Hadoop fails, although it is also working on a lot more.

Maybe this is not a problem, maybe it is. Nevertheless, treating them as Set and not List makes sense.

Issue Links

• depends upon: NUTCH-2796 Upgrade to crawler-commons 1.1
• links to: Github PR #535