[NUTCH-2184] Enable IndexingJob to function with no crawldb

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.17
• Component/s: indexer
• Patch Info: Patch Available
• Flags: Patch
• Assignee: Lewis John McGibbney
• Reporter: Lewis John McGibbney
• Created: 12/Dec/15 02:36
• Updated: 28/Jan/21 13:16
• Resolved: 09/Jan/20 10:38

Description

Sometimes when working with distributed team(s), we have found that we can 'loose' data structures which are currently considered as critical e.g. crawldb, linkdb and/or segments.
In my current scenario I have a requirement to index segment data with no accompanying crawldb or linkdb.
Absence of the latter is OK as linkdb is optional however currently in IndexerMapReduce crawldb is mandatory.
This ticket should enhance the IndexerMapReduce code to support the use case where you ONLY have segments and want to force an index for every record present.

Issue Links

• incorporates: NUTCH-2456 Allow to index pages/URLs not contained in CrawlDb
• links to: GitHub Pull Request #95
• links to: GitHub Pull Request #95
• links to: GitHub Pull Request #486