[PIG-3190] Add LuceneTokenizer and SnowballTokenizer to Pig - useful text tokenization

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 0.11
• Fix Version/s: 0.18.0
• Component/s: internal-udfs
• Assignee: Russell Jurney
• Reporter: Russell Jurney
• Created: 17/Feb/13 00:44
• Updated: 26/May/17 22:35

Description

TOKENIZE is literally useless. The Lucene Standard/Snowball tokenizers in lucene, as used by, varaha is much more useful for actual tasks: https://github.com/Ganglion/varaha/blob/master/src/main/java/varaha/text/TokenizeText.java