Lucene - Core
  1. Lucene - Core
  2. LUCENE-2906

Filter to process output of ICUTokenizer and create overlapping bigrams for CJK

    Details

    • Type: New Feature New Feature
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 3.6, 4.0-ALPHA
    • Component/s: modules/analysis
    • Labels:
      None
    • Lucene Fields:
      New

      Description

      The ICUTokenizer produces unigrams for CJK. We would like to use the ICUTokenizer but have overlapping bigrams created for CJK as in the CJK Analyzer. This filter would take the output of the ICUtokenizer, read the ScriptAttribute and for selected scripts (Han, Kana), would produce overlapping bigrams.

      1. LUCENE-2906.patch
        55 kB
        Robert Muir
      2. LUCENE-2906.patch
        54 kB
        Robert Muir
      3. LUCENE-2906.patch
        49 kB
        Robert Muir
      4. LUCENE-2906.patch
        27 kB
        Robert Muir
      There are no Sub-Tasks for this issue.

        Activity

        Uwe Schindler made changes -
        Status Resolved [ 5 ] Closed [ 6 ]
        Robert Muir made changes -
        Status Open [ 1 ] Resolved [ 5 ]
        Resolution Fixed [ 1 ]
        Robert Muir made changes -
        Attachment LUCENE-2906.patch [ 12508682 ]
        Robert Muir made changes -
        Attachment LUCENE-2906.patch [ 12508659 ]
        Robert Muir made changes -
        Attachment LUCENE-2906.patch [ 12507766 ]
        Robert Muir made changes -
        Assignee Robert Muir [ rcmuir ]
        Simon Willnauer made changes -
        Fix Version/s 3.6 [ 12319070 ]
        Fix Version/s 3.5 [ 12317877 ]
        Michael McCandless made changes -
        Fix Version/s 3.5 [ 12317877 ]
        Fix Version/s 3.4 [ 12316675 ]
        Robert Muir made changes -
        Fix Version/s 3.4 [ 12316675 ]
        Fix Version/s 4.0 [ 12314025 ]
        Fix Version/s 3.3 [ 12316470 ]
        Robert Muir made changes -
        Fix Version/s 3.3 [ 12316470 ]
        Fix Version/s 4.0 [ 12314025 ]
        Fix Version/s 3.2 [ 12316070 ]
        Mark Thomas made changes -
        Workflow Default workflow, editable Closed status [ 12561865 ] jira [ 12582926 ]
        Mark Thomas made changes -
        Workflow jira [ 12544616 ] Default workflow, editable Closed status [ 12561865 ]
        Robert Muir made changes -
        Fix Version/s 3.2 [ 12316070 ]
        Fix Version/s 4.0 [ 12314025 ]
        Robert Muir made changes -
        Field Original Value New Value
        Attachment LUCENE-2906.patch [ 12470397 ]
        Tom Burton-West created issue -

          People

          • Assignee:
            Robert Muir
            Reporter:
            Tom Burton-West
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development