[NUTCH-2565] MergeDB incorrectly handles unfetched CrawlDatums

• Type: Bug
• Status: Closed
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 1.14
• Fix Version/s: 1.15
• Component/s: crawldb
• Patch Info: Patch Available
• Flags: Patch
• Assignee: Unassigned
• Reporter: Jurian Broertjes
• Created: 10/Apr/18 10:59
• Updated: 01/Oct/19 14:28
• Resolved: 21/Jun/18 14:43

Description

I ran into this issue when merging a crawlDB originating from sitemaps into our normal crawlDB. CrawlDatums are merged based on output of AbstractFetchSchedule::calculateLastFetchTime(). When CrawlDatums are unfetched, this can overwrite fetchTime or other stuff.

I assume this is a bug and have a simple fix for it that checks if CrawlDatum has status db_unfetched.

 

Issue Links

• links to: GitHub Pull Request #311