Uploaded image for project: 'Lucene - Core'
  1. Lucene - Core
  2. LUCENE-2847

Support all of unicode in StandardTokenizer

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 3.1, 4.0-ALPHA
    • modules/analysis
    • None
    • New, Patch Available

    Description

      StandardTokenizer currently only supports the BMP.

      If it encounters characters outside of the BMP, it just discards them...
      it should instead implement fully implement UAX#29 across all of unicode.

      Attachments

        1. LUCENE-2847.patch
          329 kB
          Steven Rowe
        2. LUCENE-2847.patch
          327 kB
          Steven Rowe
        3. LUCENE-2847.patch
          45 kB
          Robert Muir

        Issue Links

          is part of

          Improvement - An improvement or enhancement to an existing feature or task. LUCENE-1689 supplementary character handling

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          Activity

            People

              sarowe Steven Rowe
              rcmuir Robert Muir
              Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: