Uploaded image for project: 'Solr'
  1. Solr
  2. SOLR-2726

NullPointerException when using spellcheck.q

    Details

      Description

      When I use spellcheck.q in my query to define what will be "spellchecked", I always have this error, for every configuration I try :

      java.lang.NullPointerException
      at org.apache.solr.handler.component.SpellCheckComponent.getTokens(SpellCheckComponent.java:476)
      at org.apache.solr.handler.component.SpellCheckComponent.process(SpellCheckComponent.java:131)
      at org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:202)
      at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:129)
      at org.apache.solr.core.SolrCore.execute(SolrCore.java:1368)
      at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:356)
      at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:252)
      at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
      at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
      at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
      at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
      at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
      at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
      at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
      at org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:114)
      at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
      at org.mortbay.jetty.Server.handle(Server.java:326)
      at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
      at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
      at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
      at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
      at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
      at org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:228)
      at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)

      All my other functions works great, this is the only thing which doesn't work at all, just when i add "&spellcheck.q=my%20sentence" in the query...

      Example of a query :
      http://localhost:8983/solr/db/suggest_full?q=american%20israel&spellcheck.q=american%20israel

      In solrconfig.xml :

      <searchComponent name="suggest_full" class="solr.SpellCheckComponent">
      <str name="queryAnalyzerFieldType">suggestTextFull</str>
      <lst name="spellchecker">
      <str name="name">suggest_full</str>
      <str name="classname">org.apache.solr.spelling.suggest.Suggester</str>
      <str name="lookupImpl">org.apache.solr.spelling.suggest.tst.TSTLookup</str>
      <str name="field">text_suggest_full</str>
      <str name="fieldType">suggestTextFull</str>
      </lst>
      </searchComponent>

      <requestHandler name="/suggest_full" class="org.apache.solr.handler.component.SearchHandler">
      <lst name="defaults">
      <str name="spellcheck">true</str>
      <str name="spellcheck.dictionary">suggest_full</str>
      <str name="spellcheck.count">10</str>
      <str name="spellcheck.onlyMorePopular">true</str>
      </lst>
      <arr name="components">
      <str>suggest_full</str>
      </arr>
      </requestHandler>

      I'm using SolR 3.3, and I tried it too on SolR 4.0

      1. SOLR-2726.patch
        1 kB
        Bernd Fehling
      2. SOLR-2726.patch
        8 kB
        Robert Muir

        Activity

        Hide
        vshamsnejad valentin added a comment -

        I've run some tests, and I found that it makes this error when i add a spellcheck component to a handler and i try to use spellcheck.q

        So spellcheck.q works with this kind of use :

        http://localhost:8983/solr/db/suggest_full?q=american%20israel&spellcheck.q=american%20israel&qt=spellchecker (with the original spellchecker of db)

        But this spellchecker has the class solr.SpellCheckerRequestHandler that doesn't have all the options I want (like collation).

        Show
        vshamsnejad valentin added a comment - I've run some tests, and I found that it makes this error when i add a spellcheck component to a handler and i try to use spellcheck.q So spellcheck.q works with this kind of use : http://localhost:8983/solr/db/suggest_full?q=american%20israel&spellcheck.q=american%20israel&qt=spellchecker (with the original spellchecker of db) But this spellchecker has the class solr.SpellCheckerRequestHandler that doesn't have all the options I want (like collation).
        Hide
        befehl Bernd Fehling added a comment -

        The NPE comes from SpellCheckComponent.process --> SpellCheckComponent.getTokens because there is no analyzer defined, but if using "spellcheck.q" there is an analyzer required.
        This issue is manifold.

        • there is no default analyzer defined (like the WhitespaceAnalyzer if only using "q" paramter for suggest)
        • there cannot be any analyzer defined at all for this, beause:
          • no analyzer or spellcheck.analyzer parameter is read from solrconfig.xml for this
          • the class SolrSpellChecker has only getQueryAnalyzer() but no setQueryAnalzyer() to set one

        How should we fix this?

        • add a default analyzer?
        • add setQueryAnalyzer() to SolrSpellChecker?
        • set analyzer at Suggester.init or SolrSpellChecker.init or SpellCheckComponent.prepare?

        Any opinions?

        Show
        befehl Bernd Fehling added a comment - The NPE comes from SpellCheckComponent.process --> SpellCheckComponent.getTokens because there is no analyzer defined, but if using "spellcheck.q" there is an analyzer required. This issue is manifold. there is no default analyzer defined (like the WhitespaceAnalyzer if only using "q" paramter for suggest) there cannot be any analyzer defined at all for this, beause: no analyzer or spellcheck.analyzer parameter is read from solrconfig.xml for this the class SolrSpellChecker has only getQueryAnalyzer() but no setQueryAnalzyer() to set one How should we fix this? add a default analyzer? add setQueryAnalyzer() to SolrSpellChecker? set analyzer at Suggester.init or SolrSpellChecker.init or SpellCheckComponent.prepare? Any opinions?
        Hide
        befehl Bernd Fehling added a comment - - edited

        According to SOLR-572 the default analyzer should be WhitespaceAnalyzer.
        With this SOLR-2726.patch I added the WhitespaceAnalyzer to "init" of Suggester.java.
        Now spellcheck.q works without NPE.

        Tip:
        To get suggestions with multiple words like "New Y" for "New York" and also for "New Year" you can use queryAnalyzerFieldType with an analyzer having a PatternReplaceFilterFactory for e.g. "_" (underscore).
        If you now lookup up suggestions with New_Y you will get suggestions for "New York", "New Year", ...

        Show
        befehl Bernd Fehling added a comment - - edited According to SOLR-572 the default analyzer should be WhitespaceAnalyzer. With this SOLR-2726 .patch I added the WhitespaceAnalyzer to "init" of Suggester.java. Now spellcheck.q works without NPE. Tip: To get suggestions with multiple words like "New Y" for "New York" and also for "New Year" you can use queryAnalyzerFieldType with an analyzer having a PatternReplaceFilterFactory for e.g. "_" (underscore). If you now lookup up suggestions with New_Y you will get suggestions for "New York", "New Year", ...
        Hide
        rcmuir Robert Muir added a comment -

        In my opinion, since the base class SolrSpellChecker has this 'analyzer' field (that it wants to be non-null),
        it should at least take care of it in its init() method, and we should make sure subclasses call super.init(args) in their init() methods.

        When i had this bug in directspellchecker i copied-pasted the below code from AbstractLuceneSpellChecker to fix it, but i think its dumb
        to put this in every spellchecker subclass, and its trappy for someone trying to implement their own spellchecker:

            if (field != null && core.getSchema().getFieldTypeNoEx(field) != null)  {
              analyzer = core.getSchema().getFieldType(field).getQueryAnalyzer();
            }
            fieldTypeName = (String) config.get(FIELD_TYPE);
            if (core.getSchema().getFieldTypes().containsKey(fieldTypeName))  {
              FieldType fieldType = core.getSchema().getFieldTypes().get(fieldTypeName);
              analyzer = fieldType.getQueryAnalyzer();
            }
            if (analyzer == null)   {
              LOG.info("Using WhitespaceAnalyzer for dictionary: " + name);
              analyzer = new WhitespaceAnalyzer(core.getSolrConfig().luceneMatchVersion);
            }
        
        Show
        rcmuir Robert Muir added a comment - In my opinion, since the base class SolrSpellChecker has this 'analyzer' field (that it wants to be non-null), it should at least take care of it in its init() method, and we should make sure subclasses call super.init(args) in their init() methods. When i had this bug in directspellchecker i copied-pasted the below code from AbstractLuceneSpellChecker to fix it, but i think its dumb to put this in every spellchecker subclass, and its trappy for someone trying to implement their own spellchecker: if (field != null && core.getSchema().getFieldTypeNoEx(field) != null) { analyzer = core.getSchema().getFieldType(field).getQueryAnalyzer(); } fieldTypeName = (String) config.get(FIELD_TYPE); if (core.getSchema().getFieldTypes().containsKey(fieldTypeName)) { FieldType fieldType = core.getSchema().getFieldTypes().get(fieldTypeName); analyzer = fieldType.getQueryAnalyzer(); } if (analyzer == null) { LOG.info("Using WhitespaceAnalyzer for dictionary: " + name); analyzer = new WhitespaceAnalyzer(core.getSolrConfig().luceneMatchVersion); }
        Hide
        rcmuir Robert Muir added a comment -

        patch: moves the analyzer construction to SolrSpellChecker's init (since SolrSpellChecker has this variable).

        also adds a test.

        if we don't want to do it this way, then the alternative is to make getAnalyzer() abstract in SolrSpellChecker and remove the 'analyzer' variable, this way subclasses know they must deal with this.

        Show
        rcmuir Robert Muir added a comment - patch: moves the analyzer construction to SolrSpellChecker's init (since SolrSpellChecker has this variable). also adds a test. if we don't want to do it this way, then the alternative is to make getAnalyzer() abstract in SolrSpellChecker and remove the 'analyzer' variable, this way subclasses know they must deal with this.
        Hide
        rcmuir Robert Muir added a comment -

        Thanks valentin, Bernd!

        Show
        rcmuir Robert Muir added a comment - Thanks valentin, Bernd!
        Hide
        thetaphi Uwe Schindler added a comment -

        Bulk close after 3.5 is released

        Show
        thetaphi Uwe Schindler added a comment - Bulk close after 3.5 is released

          People

          • Assignee:
            rcmuir Robert Muir
            Reporter:
            vshamsnejad valentin
          • Votes:
            1 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development