Uploaded image for project: 'Solr'
  1. Solr
  2. SOLR-3056

Introduce Japanese field type in schema.xml

    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 3.6, 4.0-ALPHA
    • 3.6, 4.0-ALPHA
    • Schema and Analysis
    • None

    Description

      Kuromoji (LUCENE-3305) is now on both on trunk and branch_3x (thanks again Robert, Uwe and Simon). It would be very good to get a default field type defined for Japanese in schema.xml so we can good Japanese out-of-the-box support in Solr.

      I've been playing with the below configuration today, which I think is a reasonable starting point for Japanese. There's lot to be said about various considerations necessary when searching Japanese, but perhaps a wiki page is more suitable to cover the wider topic?

      In order to make the below text_ja field type work, Kuromoji itself and its analyzers need to be seen by the Solr classloader. However, these are currently in contrib and I'm wondering if we should consider moving them to core to make them directly available. If there are concerns with additional memory usage, etc. for non-Japanese users, we can make sure resources are loaded lazily and only when needed in factory-land.

      Any thoughts?

      <!-- Text field type is suitable for Japanese text using morphological analysis
      
           NOTE: Please copy files
             contrib/analysis-extras/lucene-libs/lucene-kuromoji-x.y.z.jar
             dist/apache-solr-analysis-extras-x.y.z.jar
           to your Solr lib directory (i.e. example/solr/lib) before before starting Solr.
           (x.y.z refers to a version number)
      
           If you would like to optimize for precision, default operator AND with
             <solrQueryParser defaultOperator="AND"/>
           below (this file).  Use "OR" if you would like to optimize for recall (default).
      -->
      <fieldType name="text_ja" class="solr.TextField" positionIncrementGap="100" autoGeneratePhraseQueries="false">
        <analyzer>
          <!-- Kuromoji Japanese morphological analyzer/tokenizer
      
               Use search-mode to get a noun-decompounding effect useful for search.
      
               Example:
                 関西国際空港 (Kansai International Airpart) becomes 関西 (Kansai) 国際 (International) 空港 (airport)
                 so we get a match for 空港 (airport) as we would expect from a good search engine
      
               Valid values for mode are:
                  normal: default segmentation
                  search: segmentation useful for search (extra compound splitting)
                extended: search mode with unigramming of unknown words (experimental)
      
               NOTE: Search mode improves segmentation for search at the expense of part-of-speech accuracy
          -->
          <tokenizer class="solr.KuromojiTokenizerFactory" mode="search"/>
          <!-- Reduces inflected verbs and adjectives to their base/dectionary forms (辞書形) -->	
          <filter class="solr.KuromojiBaseFormFilterFactory"/>
          <!-- Optionally remove tokens with certain part-of-speeches
          <filter class="solr.KuromojiPartOfSpeechStopFilterFactory" tags="stopTags.txt" enablePositionIncrements="true"/> -->
          <!-- Normalizes full-width romaji to half-with and half-width kana to full-width (Unicode NFKC subset) -->
          <filter class="solr.CJKWidthFilterFactory"/>
          <!-- Lower-case romaji characters -->
          <filter class="solr.LowerCaseFilterFactory"/>
        </analyzer>
      </fieldType>
      

      Attachments

        1. SOLR-3056.patch
          4 kB
          Robert Muir
        2. SOLR-3056.patch
          5 kB
          Christian Moen
        3. SOLR-3056_typo.patch
          1.0 kB
          Robert Muir
        4. SOLR-3056_schema40.patch
          2 kB
          Christian Moen
        5. SOLR-3056_schema40.patch
          2 kB
          Christian Moen
        6. SOLR-3056_schema40.patch
          3 kB
          Christian Moen
        7. SOLR-3056_move.patch
          7 kB
          Robert Muir

        Issue Links

          Activity

            People

              Unassigned Unassigned
              cm Christian Moen
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: