[NUTCH-2221] Introduce db.ignore.internal.links to FetcherThread

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.11
• Fix Version/s: 1.12
• Component/s: fetcher
• Assignee: Markus Jelsma
• Reporter: Markus Jelsma
• Created: 15/Feb/16 14:29
• Updated: 13/Mar/24 14:50
• Resolved: 23/Feb/16 11:56

Description

FetcherThread has support for db.ignore.external.links. In config you can find db.ignore.internal.links as well, but it only operates on LinkDB, which is confusing. This patch will introduce db.ignore.internal.links to FetcherThread, similar to db.ignore.external.links. With both parameter set to true you can limit the crawl to the injected seed list.

Issue Links

• breaks: NUTCH-2144 Plugin to override db.ignore.external to exempt interesting external domain URLs
• depends upon: NUTCH-2220 Rename db.* options used only by the linkdb to linkdb.*
• is related to: NUTCH-2216 db.ignore.*.links to optionally follow internal redirects