[LUCENE-2414] add icu-based tokenizer for unicode text segmentation

• Type: New Feature
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.1
• Fix Version/s: 3.1, 4.0-ALPHA
• Component/s: modules/other
• Lucene Fields: New, Patch Available
• Assignee: Robert Muir
• Reporter: Robert Muir
• Created: 22/Apr/10 20:52
• Updated: 28/Aug/22 12:25
• Resolved: 06/May/10 12:55

Description

I pulled out the last part of LUCENE-1488, the tokenizer itself and cleaned it up some.

The idea is simple:

• First step is to divide text into writing system boundaries (scripts)
• You supply an ICUTokenizerConfig (or just use the default) which lets you tailor segmentation on a per-writing system basis.
• This tailoring can be any BreakIterator, so rule-based or dictionary-based or your own.

The default implementation (if you do not customize) is just to do UAX#29, but with tailorings for stuff with no clear word division:

• Thai (uses dictionary-based word breaking)
• Khmer, Myanmar, Lao (uses custom rules for syllabification)

Additionally as more of an example i have a tailoring for hebrew that treats the punctuation special. (People have asked before
for ways to make standardanalyzer treat dashes differently, etc)

Issue Links

• is part of: LUCENE-1488 multilingual analyzer based on icu