[NUTCH-1905] Nutch index tool should be resilient to segments that don't have crawl_* data

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Duplicate
• Fix Version/s: 1.11
• Component/s: indexer
• Assignee: Chris A. Mattmann
• Reporter: Chris A. Mattmann
• Created: 04/Jan/15 20:46
• Updated: 13/Mar/24 14:50
• Resolved: 10/Apr/15 23:33

Description

When running the ./bin/nutch index command with the -dir <path/to/segment/dir> I noticed that if you have a segment directory that doesn't include crawl_* or parse_* data, that the indexer fails (correctly). However, the indexer should be more resilient in those cases - we can add a simple check to see if those dirs are present in the segment, and proceed if they are, otherwise, ignore that segment and print a message (and go to the other segments).

Issue Links

• duplicates: NUTCH-1771 Solrindex fails if a segment is corrupted or incomplete