[NUTCH-2231] Jexl support in generator job

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.11
• Fix Version/s: 1.12
• Assignee: Markus Jelsma
• Reporter: Markus Jelsma
• Created: 24/Feb/16 12:55
• Updated: 13/Mar/24 14:50
• Resolved: 11/May/16 08:21

Description

Generator should support Jexl expressions. This would make it much easier to implement focussing crawlers that rely on information stored in the CrawlDB. With the HostDB it is possible to restrict the generator to select only interesting records but it is very cumbersome and involves domainblacklist-urlfiltering.

With Jexl support, it is no hassle!

Crawl only english records:

bin/nutch generate crawl/crawldb/ crawl/segments/ -expr "(lang == 'en'')"

Crawl only HTML records:

bin/nutch generate crawl/crawldb/ crawl/segments/ -expr "(Content_Type == 'text/html' || Content_Type == 'application/xhtml+xml')"

Keep in mind:

• Jexl doesn't allow a hyphen/minus in field identifier, they are transformed to underscores
• string literals must be in quotes, only surrounding qoute needs to be escaped by backslash

Issue Links

• supercedes: NUTCH-1179 Option to restrict generated records by metadata
• supercedes: NUTCH-2215 Generator to restrict crawl to mime type