[PIG-3143] Enable TOKENIZE to use any configurable Lucene Tokenizer, if a config parameter is set and the JARs included

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Won't Fix
• Affects Version/s: 0.11
• Component/s: impl, internal-udfs
• Assignee: Unassigned
• Reporter: Russell Jurney
• Created: 28/Jan/13 21:10
• Updated: 28/Apr/15 23:25
• Resolved: 28/Apr/15 23:25

Description

I'll do this in time for 12. TOKENIZE is literally useless as is. See:

http://thedatachef.blogspot.com/2011/04/lucene-text-tokenization-udf-for-apache.html
https://github.com/Ganglion/varaha/blob/master/src/main/java/varaha/text/TokenizeText.java

Issue Links

• duplicates: DATAFU-14 Add NGram Tokenizer to datafu.pig.text.lucene