[NUTCH-933] Fetcher does not save a pages Last-Modified value in CrawlDatum

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Not A Problem
• Affects Version/s: 1.2
• Component/s: fetcher
• Assignee: Unassigned
• Reporter: Joe Kemp
• Created: 31/Oct/10 03:14
• Updated: 28/Jan/21 14:03
• Resolved: 07/Jul/14 13:03

Description

I added the following code in the output method just after the If (content !=null) statement.

String lastModified = metadata.get("Last-Modified");
if (lastModified !=null && !lastModified.equals("")) {

try

{ Date lastModifiedDate = DateUtil.parseDate(lastModified); datum.setModifiedTime(lastModifiedDate.getTime()); }

catch (DateParseException e) {

}
}

I now get 304 for pages that haven't changed when I recrawl. Need to do further testing. Might also need a configuration parameter to turn off this behavior, allowing pages to be forced to be refreshed.