[NUTCH-2164] Inconsistent 'Modified Time' in crawl db

• Type: Improvement
• Status: Closed
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 1.11
• Fix Version/s: 1.13
• Component/s: crawldb, fetcher
• Assignee: Unassigned
• Reporter: Thamme Gowda
• Created: 09/Nov/15 08:56
• Updated: 13/Mar/24 14:51
• Resolved: 23/Aug/16 08:41

Description

The 'Modified time' in crawldb is invalid. It is set to (0-Timezone Difference)

How to verify/reproduce:
Run 'nutch readdb /path/to/crawldb -dump yy' and then inspect content of 'yy'

The following improvements can be done:
1. Set modified time by DefaultFetchSchedule
2. Set ProtocolStatus.lastModified if modified time is available in protocol response headers

This issue is also discussed in dev mailing lists: http://www.mail-archive.com/[e-mail]/msg19803.html#

Issue Links

• is duplicated by: NUTCH-2242 lastModified not always set
• links to: GitHub Pull Request #108