Uploaded image for project: 'Lucene - Core'
  1. Lucene - Core
  2. LUCENE-2414

add icu-based tokenizer for unicode text segmentation

    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 3.1
    • 3.1, 4.0-ALPHA
    • modules/other
    • None
    • New, Patch Available

    Description

      I pulled out the last part of LUCENE-1488, the tokenizer itself and cleaned it up some.

      The idea is simple:

      • First step is to divide text into writing system boundaries (scripts)
      • You supply an ICUTokenizerConfig (or just use the default) which lets you tailor segmentation on a per-writing system basis.
      • This tailoring can be any BreakIterator, so rule-based or dictionary-based or your own.

      The default implementation (if you do not customize) is just to do UAX#29, but with tailorings for stuff with no clear word division:

      • Thai (uses dictionary-based word breaking)
      • Khmer, Myanmar, Lao (uses custom rules for syllabification)

      Additionally as more of an example i have a tailoring for hebrew that treats the punctuation special. (People have asked before
      for ways to make standardanalyzer treat dashes differently, etc)

      Attachments

        1. LUCENE-2414.patch
          96 kB
          Robert Muir
        2. LUCENE-2414.patch
          94 kB
          Robert Muir
        3. LUCENE-2414.patch
          95 kB
          Uwe Schindler
        4. LUCENE-2414.patch
          95 kB
          Robert Muir
        5. LUCENE-2414.patch
          98 kB
          Robert Muir

        Issue Links

          Activity

            People

              rcmuir Robert Muir
              rcmuir Robert Muir
              Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: