[NUTCH-1616] SegmentMerger missing proper crawl_fetch datum

• Type: Bug
• Status: Closed
• Priority: Critical
• Resolution: Duplicate
• Affects Version/s: 1.7
• Fix Version/s: 1.8
• Assignee: Markus Jelsma
• Reporter: Markus Jelsma
• Created: 19/Jul/13 14:08
• Updated: 13/Mar/24 14:51
• Resolved: 08/Jan/14 13:12

Description

Merged 26036 vs. unmerged 26038 indexed documents! There are two records on the merged segment that no longer have a crawl_fetch CrawlDatum with a fetch_success status. Instead, the only crawl_fetch CrawlDatum has status linked!

The original segment two crawl_fetch CrawlDatums with linked and the fetch_success status.

Without the fetch_success of not_modified status it is not going to be indexed.

Issue Links

• duplicates: NUTCH-1113 Merging segments causes URLs to vanish from crawldb/index?
• is related to: NUTCH-1617 IndexerMapReduce to consider latest fetchDatum
• relates to: NUTCH-1113 Merging segments causes URLs to vanish from crawldb/index?
• relates to: NUTCH-1520 SegmentMerger looses records