[NUTCH-2286] CrawlDbReader -stats to show fetch time and interval

• Type: Improvement
• Status: Closed
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 1.12
• Fix Version/s: 1.13
• Component/s: crawldb
• Patch Info: Patch Available
• Assignee: Sebastian Nagel
• Reporter: Sebastian Nagel
• Created: 23/Jun/16 14:52
• Updated: 13/Mar/24 14:51
• Resolved: 02/Jul/16 10:18

Description

An overview about fetch times and fetch intervals could be useful to configure a crawl. CrawlDbReader could easily calculate min, max and average and show it as part of the statistics job (command-line option -stats):

% bin/nutch readdb .../crawldb/ -stats
...
TOTAL urls:     544910
shortest fetch interval:       7 days, 00:00:00
avg fetch interval:            7 days, 17:43:58
longest fetch interval:       10 days, 12:00:00
earliest fetch time:   Wed May 25 11:42:00 CEST 2016
avg of fetch times:    Sun Jun 05 18:11:00 CEST 2016
latest fetch time:     Wed Jun 22 10:25:00 CEST 2016
...

Issue Links

• links to: GitHub Pull Request #125