[NUTCH-2750] Improve CrawlDbReader & LinkDbReader reader handling

• Type: Improvement
• Status: Closed
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 1.16
• Fix Version/s: 1.17
• Component/s: crawldb, linkdb
• Patch Info: Patch Available
• Flags: Patch
• Assignee: Unassigned
• Reporter: Jurian Broertjes
• Created: 24/Oct/19 12:02
• Updated: 28/Jan/21 13:16
• Resolved: 12/Nov/19 14:38

Description

The current implementation in the CrawlDbReader re-opens readers for every URL. This is not very efficient. I've implemented a modification time check that only re-opens readers on updated crawlDB.

 PR: https://github.com/apache/nutch/pull/483

Issue Links

• links to: GitHub Pull Request #483