[NUTCH-2229] Allow Jexl expressions on CrawlDatum's fixed attributes

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.11
• Fix Version/s: 1.12
• Component/s: crawldb
• Patch Info: Patch Available
• Assignee: Markus Jelsma
• Reporter: Markus Jelsma
• Created: 23/Feb/16 15:00
• Updated: 03/Feb/17 11:04
• Resolved: 03/Feb/17 11:03

Description

CrawlDatum allows Jexl expressions on its metadata fields nicely, but it lacks the opportunity to select on attributes like fetchTime and modifiedTime.

This includes a rudimentary date parser only supporting the yyyy-MM-dd'T'HH:mm:ss'Z' format:

Dump everything with a modifiedTime higher than 2016-03-20T00:00:00Z

bin/nutch readdb crawl/crawldb/ -dump out -format csv -expr "(modifiedTime > 2016-03-20T00:00:00Z)"

Dump everything that is an HTML file

bin/nutch readdb crawl/crawldb/ -dump out -format csv -expr "(Content_Type == 'text/html' || Content_Type == 'application/xhtml+xml')"

Keep in mind:

• Jexl doesn't allow a hyphen/minus in field identifier, they are transformed to underscores
• string literals must be in quotes, only surrounding qoute needs to be escaped by backslash