[NUTCH-2069] Ignore external links based on domain

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.10
• Fix Version/s: 1.11
• Component/s: fetcher, parser
• Patch Info: Patch Available
• Assignee: Unassigned
• Reporter: Julien Nioche
• Created: 29/Jul/15 11:12
• Updated: 06/Apr/17 09:06
• Resolved: 20/Nov/15 16:23

Description

We currently have `db.ignore.external.links` which is a nice way of restricting the crawl based on the hostname. This adds a new parameter 'db.ignore.external.links.domain' to do the same based on the domain.

Issue Links

• relates to: NUTCH-2365 HTTP Redirects to SubDomains don't get crawled if db.ignore.external.links.mode == byDomain