Uploaded image for project: 'Lucene - Core'
  1. Lucene - Core
  2. LUCENE-2847

Support all of unicode in StandardTokenizer

          Details

          • Type: Bug
          • Status: Closed
          • Priority: Major
          • Resolution: Fixed
          • Affects Version/s: None
          • Fix Version/s: 3.1, 4.0-ALPHA
          • Component/s: modules/analysis
          • Labels:
            None
          • Lucene Fields:
            New, Patch Available

            Description

            StandardTokenizer currently only supports the BMP.

            If it encounters characters outside of the BMP, it just discards them...
            it should instead implement fully implement UAX#29 across all of unicode.

              Attachments

              1. LUCENE-2847.patch
                329 kB
                Steve Rowe
              2. LUCENE-2847.patch
                327 kB
                Steve Rowe
              3. LUCENE-2847.patch
                45 kB
                Robert Muir

                Issue Links

                is part of

                Improvement - An improvement or enhancement to an existing feature or task. LUCENE-1689 supplementary character handling

                • Minor - Minor loss of function, or other problem where easy workaround is present.
                • Resolved

                  Activity

                    People

                    • Assignee:
                      steve_rowe Steve Rowe
                      Reporter:
                      rcmuir Robert Muir
                    • Votes:
                      0 Vote for this issue
                      Watchers:
                      0 Start watching this issue

                      Dates

                      • Created:
                        Updated:
                        Resolved: