Uploaded image for project: 'Solr'
  1. Solr
  2. SOLR-5469

The Analysis Page on the Solr Admin Page does not work with Custom Analyzers

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Minor
    • Resolution: Cannot Reproduce
    • 4.0
    • None
    • Schema and Analysis

    • Windows, Tomcat, Java 1.7

    Description

      The Analysis Page on the Solr Admin Page does not work with Custom Analyzers. To be specific the Analyzer page does not display all of the tokens output by the custom analyzer if the tokens themselves don't have a KeywordAttribute added. It does not matter that the tokens are not keyword tokens, it is just that the Tokens need to have the KeywordAttribute (even if it evaluates to false).

      I'm attaching the json output of the case of a text_en field (solr.Text fieldType) and the json output of the case of a field (mapped to custom fieldType).

      The json generated for the custom fieldType (using a custom analyzer) while similar in all aspects to the json generated for text_en fieldType does not have KeywordAttribute set for the tokens (which seems valid).

      JSON From the analysis page for the custom fieldType (custom analyzer)

      {
      "responseHeader":

      { "status":0, "QTime":0}

      ,
      "analysis":{
      "field_types":{},
      "field_names":{
      "title":{
      "index":[
      "org.apache.lucene.analysis.miscellaneous.RemoveDuplicatesTokenFilter",[

      { "text":"concoct", "raw_bytes":"[63 6f 6e 63 6f 63 74]", "start":5, "end":15, "type":"word", "position":1, "positionHistory":[1]}

      ,

      { "text":"solut", "raw_bytes":"[73 6f 6c 75 74]", "match":true, "start":22, "end":30, "type":"word", "position":4, "positionHistory":[4]}

      ,

      { "text":"trick", "raw_bytes":"[74 72 69 63 6b]", "match":true, "start":31, "end":37, "type":"word", "position":5, "positionHistory":[5]}

      ]],
      "query":[
      "org.apache.lucene.analysis.miscellaneous.RemoveDuplicatesTokenFilter",[

      { "text":"solut", "raw_bytes":"[73 6f 6c 75 74]", "start":7, "end":15, "type":"word", "position":2, "positionHistory":[2]}

      ,

      { "text":"trick", "raw_bytes":"[74 72 69 63 6b]", "start":16, "end":22, "type":"word", "position":3, "positionHistory":[3]}

      ]]}}}}

      JSON from the standard field (using the solr.TextField) which uses the analyzer from the following built-in tokenizers/tokenfilters in the following order.
      solr.HTMLStripCharFilterFactory
      solr.WhitespaceTokenizerFactory
      solr.StopFilterFactory
      solr.WordDelimiterFilterFactory
      solr.ICUFoldingFilterFactory
      solr.EnglishPossessiveFilterFactory
      solr.PorterStemFilterFactory

      {
      "responseHeader":

      { "status":0, "QTime":47}

      ,
      "analysis":{
      "field_types":{
      "text_en":{
      "index":[
      "org.apache.lucene.analysis.charfilter.HTMLStripCharFilter","Concocting the a Solution Tricks",
      "org.apache.lucene.analysis.core.WhitespaceTokenizer",[

      { "text":"Concocting", "raw_bytes":"[43 6f 6e 63 6f 63 74 69 6e 67]", "start":0, "end":10, "position":1, "positionHistory":[1], "type":"word"}

      ,

      { "text":"the", "raw_bytes":"[74 68 65]", "start":11, "end":14, "position":2, "positionHistory":[2], "type":"word"}

      ,

      { "text":"a", "raw_bytes":"[61]", "start":15, "end":16, "position":3, "positionHistory":[3], "type":"word"}

      ,

      { "text":"Solution", "raw_bytes":"[53 6f 6c 75 74 69 6f 6e]", "start":17, "end":25, "position":4, "positionHistory":[4], "type":"word"}

      ,

      { "text":"Tricks", "raw_bytes":"[54 72 69 63 6b 73]", "start":26, "end":32, "position":5, "positionHistory":[5], "type":"word"}

      ],
      "org.apache.lucene.analysis.core.StopFilter",[

      { "text":"Concocting", "raw_bytes":"[43 6f 6e 63 6f 63 74 69 6e 67]", "position":1, "positionHistory":[1, 1], "start":0, "end":10, "type":"word"}

      ,

      { "text":"Solution", "raw_bytes":"[53 6f 6c 75 74 69 6f 6e]", "position":4, "positionHistory":[4, 4], "start":17, "end":25, "type":"word"}

      ,

      { "text":"Tricks", "raw_bytes":"[54 72 69 63 6b 73]", "position":5, "positionHistory":[5, 5], "start":26, "end":32, "type":"word"}

      ],
      "org.apache.lucene.analysis.miscellaneous.WordDelimiterFilter",[

      { "text":"Concocting", "raw_bytes":"[43 6f 6e 63 6f 63 74 69 6e 67]", "start":0, "end":10, "type":"word", "position":1, "positionHistory":[1, 1, 1]}

      ,

      { "text":"Solution", "raw_bytes":"[53 6f 6c 75 74 69 6f 6e]", "start":17, "end":25, "type":"word", "position":4, "positionHistory":[4, 4, 4]}

      ,

      { "text":"Tricks", "raw_bytes":"[54 72 69 63 6b 73]", "start":26, "end":32, "type":"word", "position":5, "positionHistory":[5, 5, 5]}

      ],
      "org.apache.lucene.analysis.icu.ICUFoldingFilter",[

      { "text":"concocting", "raw_bytes":"[63 6f 6e 63 6f 63 74 69 6e 67]", "position":1, "positionHistory":[1, 1, 1, 1], "start":0, "end":10, "type":"word"}

      ,

      { "text":"solution", "raw_bytes":"[73 6f 6c 75 74 69 6f 6e]", "position":4, "positionHistory":[4, 4, 4, 4], "start":17, "end":25, "type":"word"}

      ,

      { "text":"tricks", "raw_bytes":"[74 72 69 63 6b 73]", "position":5, "positionHistory":[5, 5, 5, 5], "start":26, "end":32, "type":"word"}

      ],
      "org.apache.lucene.analysis.en.EnglishPossessiveFilter",[

      { "text":"concocting", "raw_bytes":"[63 6f 6e 63 6f 63 74 69 6e 67]", "position":1, "positionHistory":[1, 1, 1, 1, 1], "start":0, "end":10, "type":"word"}

      ,

      { "text":"solution", "raw_bytes":"[73 6f 6c 75 74 69 6f 6e]", "position":4, "positionHistory":[4, 4, 4, 4, 4], "start":17, "end":25, "type":"word"}

      ,

      { "text":"tricks", "raw_bytes":"[74 72 69 63 6b 73]", "position":5, "positionHistory":[5, 5, 5, 5, 5], "start":26, "end":32, "type":"word"}

      ],
      "org.apache.lucene.analysis.en.PorterStemFilter",[

      { "text":"concoct", "raw_bytes":"[63 6f 6e 63 6f 63 74]", "org.apache.lucene.analysis.tokenattributes.KeywordAttribute#keyword":false, "position":1, "positionHistory":[1, 1, 1, 1, 1, 1], "start":0, "end":10, "type":"word"}

      ,

      { "text":"solut", "raw_bytes":"[73 6f 6c 75 74]", "match":true, "org.apache.lucene.analysis.tokenattributes.KeywordAttribute#keyword":false, "position":4, "positionHistory":[4, 4, 4, 4, 4, 4], "start":17, "end":25, "type":"word"}

      ,

      { "text":"trick", "raw_bytes":"[74 72 69 63 6b]", "match":true, "org.apache.lucene.analysis.tokenattributes.KeywordAttribute#keyword":false, "position":5, "positionHistory":[5, 5, 5, 5, 5, 5], "start":26, "end":32, "type":"word"}

      ]],
      "query":[
      "org.apache.lucene.analysis.charfilter.HTMLStripCharFilter","a Solution Tricks",
      "org.apache.lucene.analysis.core.WhitespaceTokenizer",[

      { "text":"a", "raw_bytes":"[61]", "start":0, "end":1, "position":1, "positionHistory":[1], "type":"word"}

      ,

      { "text":"Solution", "raw_bytes":"[53 6f 6c 75 74 69 6f 6e]", "start":2, "end":10, "position":2, "positionHistory":[2], "type":"word"}

      ,

      { "text":"Tricks", "raw_bytes":"[54 72 69 63 6b 73]", "start":11, "end":17, "position":3, "positionHistory":[3], "type":"word"}

      ],
      "org.apache.lucene.analysis.core.StopFilter",[

      { "text":"Solution", "raw_bytes":"[53 6f 6c 75 74 69 6f 6e]", "position":2, "positionHistory":[2, 2], "start":2, "end":10, "type":"word"}

      ,

      { "text":"Tricks", "raw_bytes":"[54 72 69 63 6b 73]", "position":3, "positionHistory":[3, 3], "start":11, "end":17, "type":"word"}

      ],
      "org.apache.lucene.analysis.miscellaneous.WordDelimiterFilter",[

      { "text":"Solution", "raw_bytes":"[53 6f 6c 75 74 69 6f 6e]", "start":2, "end":10, "type":"word", "position":2, "positionHistory":[2, 2, 2]}

      ,

      { "text":"Tricks", "raw_bytes":"[54 72 69 63 6b 73]", "start":11, "end":17, "type":"word", "position":3, "positionHistory":[3, 3, 3]}

      ],
      "org.apache.lucene.analysis.icu.ICUFoldingFilter",[

      { "text":"solution", "raw_bytes":"[73 6f 6c 75 74 69 6f 6e]", "position":2, "positionHistory":[2, 2, 2, 2], "start":2, "end":10, "type":"word"}

      ,

      { "text":"tricks", "raw_bytes":"[74 72 69 63 6b 73]", "position":3, "positionHistory":[3, 3, 3, 3], "start":11, "end":17, "type":"word"}

      ],
      "org.apache.lucene.analysis.en.EnglishPossessiveFilter",[

      { "text":"solution", "raw_bytes":"[73 6f 6c 75 74 69 6f 6e]", "position":2, "positionHistory":[2, 2, 2, 2, 2], "start":2, "end":10, "type":"word"}

      ,

      { "text":"tricks", "raw_bytes":"[74 72 69 63 6b 73]", "position":3, "positionHistory":[3, 3, 3, 3, 3], "start":11, "end":17, "type":"word"}

      ],
      "org.apache.lucene.analysis.en.PorterStemFilter",[

      { "text":"solut", "raw_bytes":"[73 6f 6c 75 74]", "org.apache.lucene.analysis.tokenattributes.KeywordAttribute#keyword":false, "position":2, "positionHistory":[2, 2, 2, 2, 2, 2], "start":2, "end":10, "type":"word"}

      ,

      { "text":"trick", "raw_bytes":"[74 72 69 63 6b]", "org.apache.lucene.analysis.tokenattributes.KeywordAttribute#keyword":false, "position":3, "positionHistory":[3, 3, 3, 3, 3, 3], "start":11, "end":17, "type":"word"}

      ]]}},
      "field_names":{}}}

      The latter json does show up correctly in the analysis page whereas the former doesn't show up correctly.
      Especially if the text involved in analysis involves StopWords.

      Attachments

        Activity

          People

            Unassigned Unassigned
            swamipr Swami Rajamohan
            Votes:
            1 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: