[NUTCH-1475] Index-More Plugin -- A better fall back value for date field

• Type: Bug
• Status: Closed
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 2.1, 1.5.1
• Fix Version/s: 1.7, 2.2.1
• Labels: index-more plugins

Environment

All

• Patch Info: Patch Available
• Assignee: Sebastian Nagel
• Reporter: James Sullivan
• Created: 07/Oct/12 01:49
• Updated: 11/Oct/19 15:36
• Resolved: 20/Jun/13 21:00

Description

Among other fields, the more plugin for Nutch 2.x provides a "last modified" and "date" field for the Solr index. The "last modified" field is the last modified date from the http headers if available, if not available it is left empty. Currently, the "date" field is the same as the "last modified" field unless that field is empty in which case getFetchTime is used as a fall back. I think getFetchTime is not a good fall back as it is the next fetch time and often a month or more in the future which doesn't make sense for the date field. Users do not expect webpages/documents with future dates. A more sensible fallback would be current date at the time it is indexed.

This is possible by simply changing line 97 of https://svn.apache.org/repos/asf/nutch/branches/2.x/src/plugin/index-more/src/java/org/apache/nutch/indexer/more/MoreIndexingFilter.java from

time = page.getFetchTime(); // use fetch time

to

time = new Date().getTime();

Users interested in the getFetchTime value can still get it from the "tstamp" field.

Issue Links

• is related to: NUTCH-1589 Port NUTCH-1475 Index-More Plugin -- A better fall back value for date field to 2.x
• relates to: NUTCH-1457 Nutch2 Refactor the update process so that fetched items are only processed once