Uploaded image for project: 'Lucene - Core'
  1. Lucene - Core
  2. LUCENE-2906

Filter to process output of ICUTokenizer and create overlapping bigrams for CJK

    Details

    • Type: New Feature
    • Status: Closed
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 3.6, 4.0-ALPHA
    • Component/s: modules/analysis
    • Labels:
      None
    • Lucene Fields:
      New

      Description

      The ICUTokenizer produces unigrams for CJK. We would like to use the ICUTokenizer but have overlapping bigrams created for CJK as in the CJK Analyzer. This filter would take the output of the ICUtokenizer, read the ScriptAttribute and for selected scripts (Han, Kana), would produce overlapping bigrams.

        Attachments

        1. LUCENE-2906.patch
          55 kB
          Robert Muir
        2. LUCENE-2906.patch
          54 kB
          Robert Muir
        3. LUCENE-2906.patch
          49 kB
          Robert Muir
        4. LUCENE-2906.patch
          27 kB
          Robert Muir

          Activity

            People

            • Assignee:
              rcmuir Robert Muir
              Reporter:
              tburtonwest Tom Burton-West
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: