Uploaded image for project: 'Lucene - Core'
  1. Lucene - Core
  2. LUCENE-4286

Add flag to CJKBigramFilter to allow indexing unigrams as well as bigrams

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 4.0-ALPHA, 3.6.1
    • Fix Version/s: 4.0-BETA, 6.0
    • Component/s: None
    • Labels:
      None
    • Lucene Fields:
      New

      Description

      Add an optional flag to the CJKBigramFilter to tell it to also output unigrams. This would allow indexing of both bigrams and unigrams and at query time the analyzer could analyze queries as bigrams unless the query contained a single Han unigram.

      As an example here is a configuration a Solr fieldType with the analyzer for indexing with the "indexUnigrams" flag set and the analyzer for querying without the flag.

      <fieldType name="CJK" autoGeneratePhraseQueries="false">

      <analyzer type="index">
      <tokenizer class="solr.ICUTokenizerFactory"/>
      <filter class="solr.CJKBigramFilterFactory" indexUnigrams="true" han="true"/>
      </analyzer>

      <analyzer type="query">
      <tokenizer class="solr.ICUTokenizerFactory"/>
      <filter class="solr.CJKBigramFilterFactory" han="true"/>
      </analyzer>
      </fieldType>

      Use case: About 10% of our queries that contain Han characters are single character queries. The CJKBigram filter only outputs single characters when there are no adjacent bigrammable characters in the input. This means we have to create a separate field to index Han unigrams in order to address single character queries and then write application code to search that separate field if we detect a single character Han query. This is rather kludgey. With the optional flag, we could configure Solr as above

      This is somewhat analogous to the flags in LUCENE-1370 for the ShingleFilter used to allow single word queries (although that uses word n-grams rather than character n-grams.)

        Attachments

        1. LUCENE-4286.patch_3.x
          20 kB
          Tom Burton-West
        2. LUCENE-4286.patch
          17 kB
          Robert Muir
        3. LUCENE-4286.patch
          7 kB
          Robert Muir

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              tburtonwest Tom Burton-West
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: