Uploaded image for project: 'Lucene - Core'
  1. Lucene - Core
  2. LUCENE-8972

CharFilter version of ICUTransformFilter, to better support dictionary-based tokenization

Attach filesAttach ScreenshotAdd voteVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

    Details

    • Type: New Feature
    • Status: Open
    • Priority: Minor
    • Resolution: Unresolved
    • Affects Version/s: main (9.0), 8.2
    • Fix Version/s: None
    • Component/s: modules/analysis
    • Labels:
      None
    • Lucene Fields:
      New

      Description

      The ICU Transliteration API is currently exposed through Lucene only post-tokinzer, via ICUTransformFilter. Some tokenizers (particularly dictionary-based) may assume pre-normalized input (e.g., for Chinese characters, there may be an assumption of traditional-only or simplified-only input characters, at the level of either all input, or per-dictionary-defined-token).

      The potential usefulness of a CharFilter that exposes the ICU Transliteration API was suggested in a thread on the Solr mailing list, and my hope is that this issue can facilitate more detailed discussion of the proposed addition.

      A concrete example of mixed traditional/simplified characters that are currently tokenized differently by the ICUTokenizer are:

      • 红楼梦 (SSS)
      • 紅樓夢 (TTT)
      • 紅楼夢 (TST)

      The first two tokens (simplified-only and traditional-only, respectively) are included in the CJ dictionary that backs ICUTokenizer, but the last (a mixture of traditional and simplified characters) is not, and is not recognized as a token. Even if we assume this to be an intentional omission from the dictionary that results in behavior that could be desirable for some use cases, there are surely some use cases that would benefit from a more permissive dictionary-based tokenization strategy (such as could be supported by pre-tokenizer transliteration).

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              mgibney Michael Gibney

              Dates

              • Created:
                Updated:

                Time Tracking

                Estimated:
                Original Estimate - Not Specified
                Not Specified
                Remaining:
                Remaining Estimate - 0h
                0h
                Logged:
                Time Spent - 14h 50m
                14h 50m

                  Issue deployment