Solr
  1. Solr
  2. SOLR-2726

NullPointerException when using spellcheck.q

    Details

      Description

      When I use spellcheck.q in my query to define what will be "spellchecked", I always have this error, for every configuration I try :

      java.lang.NullPointerException
      at org.apache.solr.handler.component.SpellCheckComponent.getTokens(SpellCheckComponent.java:476)
      at org.apache.solr.handler.component.SpellCheckComponent.process(SpellCheckComponent.java:131)
      at org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:202)
      at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:129)
      at org.apache.solr.core.SolrCore.execute(SolrCore.java:1368)
      at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:356)
      at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:252)
      at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
      at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
      at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
      at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
      at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
      at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
      at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
      at org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:114)
      at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
      at org.mortbay.jetty.Server.handle(Server.java:326)
      at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
      at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
      at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
      at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
      at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
      at org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:228)
      at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)

      All my other functions works great, this is the only thing which doesn't work at all, just when i add "&spellcheck.q=my%20sentence" in the query...

      Example of a query :
      http://localhost:8983/solr/db/suggest_full?q=american%20israel&spellcheck.q=american%20israel

      In solrconfig.xml :

      <searchComponent name="suggest_full" class="solr.SpellCheckComponent">
      <str name="queryAnalyzerFieldType">suggestTextFull</str>
      <lst name="spellchecker">
      <str name="name">suggest_full</str>
      <str name="classname">org.apache.solr.spelling.suggest.Suggester</str>
      <str name="lookupImpl">org.apache.solr.spelling.suggest.tst.TSTLookup</str>
      <str name="field">text_suggest_full</str>
      <str name="fieldType">suggestTextFull</str>
      </lst>
      </searchComponent>

      <requestHandler name="/suggest_full" class="org.apache.solr.handler.component.SearchHandler">
      <lst name="defaults">
      <str name="spellcheck">true</str>
      <str name="spellcheck.dictionary">suggest_full</str>
      <str name="spellcheck.count">10</str>
      <str name="spellcheck.onlyMorePopular">true</str>
      </lst>
      <arr name="components">
      <str>suggest_full</str>
      </arr>
      </requestHandler>

      I'm using SolR 3.3, and I tried it too on SolR 4.0

      1. SOLR-2726.patch
        1 kB
        Bernd Fehling
      2. SOLR-2726.patch
        8 kB
        Robert Muir

        Activity

        valentin created issue -
        valentin made changes -
        Field Original Value New Value
        Affects Version/s 4.0 [ 12314992 ]
        Description When I use spellcheck.q in my query to define what will be "spellchecked", I always have this error, for every configuration I try :

        java.lang.NullPointerException
        at org.apache.solr.handler.component.SpellCheckComponent.getTokens(SpellCheckComponent.java:476)
        at org.apache.solr.handler.component.SpellCheckComponent.process(SpellCheckComponent.java:131)
        at org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:202)
        at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:129)
        at org.apache.solr.core.SolrCore.execute(SolrCore.java:1368)
        at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:356)
        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:252)
        at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
        at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
        at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
        at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
        at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
        at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
        at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
        at org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:114)
        at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
        at org.mortbay.jetty.Server.handle(Server.java:326)
        at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
        at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
        at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
        at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
        at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
        at org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:228)
        at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)

        All my other functions works great, this is the only thing which doesn't work at all, just when i add "&spellcheck.q=my%20sentence" in the query...

        I'm using SolR 3.3.
        When I use spellcheck.q in my query to define what will be "spellchecked", I always have this error, for every configuration I try :

        java.lang.NullPointerException
        at org.apache.solr.handler.component.SpellCheckComponent.getTokens(SpellCheckComponent.java:476)
        at org.apache.solr.handler.component.SpellCheckComponent.process(SpellCheckComponent.java:131)
        at org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:202)
        at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:129)
        at org.apache.solr.core.SolrCore.execute(SolrCore.java:1368)
        at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:356)
        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:252)
        at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
        at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
        at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
        at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
        at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
        at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
        at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
        at org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:114)
        at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
        at org.mortbay.jetty.Server.handle(Server.java:326)
        at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
        at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
        at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
        at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
        at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
        at org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:228)
        at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)

        All my other functions works great, this is the only thing which doesn't work at all, just when i add "&spellcheck.q=my%20sentence" in the query...

        I'm using SolR 3.3, and I tried it too on SolR 4.0
        valentin made changes -
        Description When I use spellcheck.q in my query to define what will be "spellchecked", I always have this error, for every configuration I try :

        java.lang.NullPointerException
        at org.apache.solr.handler.component.SpellCheckComponent.getTokens(SpellCheckComponent.java:476)
        at org.apache.solr.handler.component.SpellCheckComponent.process(SpellCheckComponent.java:131)
        at org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:202)
        at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:129)
        at org.apache.solr.core.SolrCore.execute(SolrCore.java:1368)
        at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:356)
        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:252)
        at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
        at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
        at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
        at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
        at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
        at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
        at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
        at org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:114)
        at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
        at org.mortbay.jetty.Server.handle(Server.java:326)
        at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
        at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
        at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
        at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
        at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
        at org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:228)
        at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)

        All my other functions works great, this is the only thing which doesn't work at all, just when i add "&spellcheck.q=my%20sentence" in the query...

        I'm using SolR 3.3, and I tried it too on SolR 4.0
        When I use spellcheck.q in my query to define what will be "spellchecked", I always have this error, for every configuration I try :

        java.lang.NullPointerException
        at org.apache.solr.handler.component.SpellCheckComponent.getTokens(SpellCheckComponent.java:476)
        at org.apache.solr.handler.component.SpellCheckComponent.process(SpellCheckComponent.java:131)
        at org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:202)
        at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:129)
        at org.apache.solr.core.SolrCore.execute(SolrCore.java:1368)
        at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:356)
        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:252)
        at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
        at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
        at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
        at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
        at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
        at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
        at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
        at org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:114)
        at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
        at org.mortbay.jetty.Server.handle(Server.java:326)
        at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
        at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
        at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
        at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
        at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
        at org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:228)
        at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)

        All my other functions works great, this is the only thing which doesn't work at all, just when i add "&spellcheck.q=my%20sentence" in the query...

        Example of a query :
        http://localhost:8983/solr/db/suggest_full?q=american%20israel&spellcheck.q=american%20israel

        In solrconfig.xml :

        <searchComponent name="suggest_full" class="solr.SpellCheckComponent">
        <str name="queryAnalyzerFieldType">suggestTextFull</str>
        <lst name="spellchecker">
        <str name="name">suggest_full</str>
        <str name="classname">org.apache.solr.spelling.suggest.Suggester</str>
        <str name="lookupImpl">org.apache.solr.spelling.suggest.tst.TSTLookup</str>
        <str name="field">text_suggest_full</str>
        <str name="fieldType">suggestTextFull</str>
        </lst>
        </searchComponent>

        <requestHandler name="/suggest_full" class="org.apache.solr.handler.component.SearchHandler">
        <lst name="defaults">
        <str name="spellcheck">true</str>
        <str name="spellcheck.dictionary">suggest_full</str>
        <str name="spellcheck.count">10</str>
        <str name="spellcheck.onlyMorePopular">true</str>
        </lst>
        <arr name="components">
        <str>suggest_full</str>
        </arr>
        </requestHandler>

        I'm using SolR 3.3, and I tried it too on SolR 4.0
        Hide
        valentin added a comment -

        I've run some tests, and I found that it makes this error when i add a spellcheck component to a handler and i try to use spellcheck.q

        So spellcheck.q works with this kind of use :

        http://localhost:8983/solr/db/suggest_full?q=american%20israel&spellcheck.q=american%20israel&qt=spellchecker (with the original spellchecker of db)

        But this spellchecker has the class solr.SpellCheckerRequestHandler that doesn't have all the options I want (like collation).

        Show
        valentin added a comment - I've run some tests, and I found that it makes this error when i add a spellcheck component to a handler and i try to use spellcheck.q So spellcheck.q works with this kind of use : http://localhost:8983/solr/db/suggest_full?q=american%20israel&spellcheck.q=american%20israel&qt=spellchecker (with the original spellchecker of db) But this spellchecker has the class solr.SpellCheckerRequestHandler that doesn't have all the options I want (like collation).
        Hide
        Bernd Fehling added a comment -

        The NPE comes from SpellCheckComponent.process --> SpellCheckComponent.getTokens because there is no analyzer defined, but if using "spellcheck.q" there is an analyzer required.
        This issue is manifold.

        • there is no default analyzer defined (like the WhitespaceAnalyzer if only using "q" paramter for suggest)
        • there cannot be any analyzer defined at all for this, beause:
          • no analyzer or spellcheck.analyzer parameter is read from solrconfig.xml for this
          • the class SolrSpellChecker has only getQueryAnalyzer() but no setQueryAnalzyer() to set one

        How should we fix this?

        • add a default analyzer?
        • add setQueryAnalyzer() to SolrSpellChecker?
        • set analyzer at Suggester.init or SolrSpellChecker.init or SpellCheckComponent.prepare?

        Any opinions?

        Show
        Bernd Fehling added a comment - The NPE comes from SpellCheckComponent.process --> SpellCheckComponent.getTokens because there is no analyzer defined, but if using "spellcheck.q" there is an analyzer required. This issue is manifold. there is no default analyzer defined (like the WhitespaceAnalyzer if only using "q" paramter for suggest) there cannot be any analyzer defined at all for this, beause: no analyzer or spellcheck.analyzer parameter is read from solrconfig.xml for this the class SolrSpellChecker has only getQueryAnalyzer() but no setQueryAnalzyer() to set one How should we fix this? add a default analyzer? add setQueryAnalyzer() to SolrSpellChecker? set analyzer at Suggester.init or SolrSpellChecker.init or SpellCheckComponent.prepare? Any opinions?
        Hide
        Bernd Fehling added a comment - - edited

        According to SOLR-572 the default analyzer should be WhitespaceAnalyzer.
        With this SOLR-2726.patch I added the WhitespaceAnalyzer to "init" of Suggester.java.
        Now spellcheck.q works without NPE.

        Tip:
        To get suggestions with multiple words like "New Y" for "New York" and also for "New Year" you can use queryAnalyzerFieldType with an analyzer having a PatternReplaceFilterFactory for e.g. "_" (underscore).
        If you now lookup up suggestions with New_Y you will get suggestions for "New York", "New Year", ...

        Show
        Bernd Fehling added a comment - - edited According to SOLR-572 the default analyzer should be WhitespaceAnalyzer. With this SOLR-2726 .patch I added the WhitespaceAnalyzer to "init" of Suggester.java. Now spellcheck.q works without NPE. Tip: To get suggestions with multiple words like "New Y" for "New York" and also for "New Year" you can use queryAnalyzerFieldType with an analyzer having a PatternReplaceFilterFactory for e.g. "_" (underscore). If you now lookup up suggestions with New_Y you will get suggestions for "New York", "New Year", ...
        Bernd Fehling made changes -
        Attachment SOLR-2726.patch [ 12492460 ]
        Hide
        Robert Muir added a comment -

        In my opinion, since the base class SolrSpellChecker has this 'analyzer' field (that it wants to be non-null),
        it should at least take care of it in its init() method, and we should make sure subclasses call super.init(args) in their init() methods.

        When i had this bug in directspellchecker i copied-pasted the below code from AbstractLuceneSpellChecker to fix it, but i think its dumb
        to put this in every spellchecker subclass, and its trappy for someone trying to implement their own spellchecker:

            if (field != null && core.getSchema().getFieldTypeNoEx(field) != null)  {
              analyzer = core.getSchema().getFieldType(field).getQueryAnalyzer();
            }
            fieldTypeName = (String) config.get(FIELD_TYPE);
            if (core.getSchema().getFieldTypes().containsKey(fieldTypeName))  {
              FieldType fieldType = core.getSchema().getFieldTypes().get(fieldTypeName);
              analyzer = fieldType.getQueryAnalyzer();
            }
            if (analyzer == null)   {
              LOG.info("Using WhitespaceAnalyzer for dictionary: " + name);
              analyzer = new WhitespaceAnalyzer(core.getSolrConfig().luceneMatchVersion);
            }
        
        Show
        Robert Muir added a comment - In my opinion, since the base class SolrSpellChecker has this 'analyzer' field (that it wants to be non-null), it should at least take care of it in its init() method, and we should make sure subclasses call super.init(args) in their init() methods. When i had this bug in directspellchecker i copied-pasted the below code from AbstractLuceneSpellChecker to fix it, but i think its dumb to put this in every spellchecker subclass, and its trappy for someone trying to implement their own spellchecker: if (field != null && core.getSchema().getFieldTypeNoEx(field) != null) { analyzer = core.getSchema().getFieldType(field).getQueryAnalyzer(); } fieldTypeName = (String) config.get(FIELD_TYPE); if (core.getSchema().getFieldTypes().containsKey(fieldTypeName)) { FieldType fieldType = core.getSchema().getFieldTypes().get(fieldTypeName); analyzer = fieldType.getQueryAnalyzer(); } if (analyzer == null) { LOG.info("Using WhitespaceAnalyzer for dictionary: " + name); analyzer = new WhitespaceAnalyzer(core.getSolrConfig().luceneMatchVersion); }
        Robert Muir made changes -
        Assignee Robert Muir [ rcmuir ]
        Hide
        Robert Muir added a comment -

        patch: moves the analyzer construction to SolrSpellChecker's init (since SolrSpellChecker has this variable).

        also adds a test.

        if we don't want to do it this way, then the alternative is to make getAnalyzer() abstract in SolrSpellChecker and remove the 'analyzer' variable, this way subclasses know they must deal with this.

        Show
        Robert Muir added a comment - patch: moves the analyzer construction to SolrSpellChecker's init (since SolrSpellChecker has this variable). also adds a test. if we don't want to do it this way, then the alternative is to make getAnalyzer() abstract in SolrSpellChecker and remove the 'analyzer' variable, this way subclasses know they must deal with this.
        Robert Muir made changes -
        Attachment SOLR-2726.patch [ 12494662 ]
        Hide
        Robert Muir added a comment -

        Thanks valentin, Bernd!

        Show
        Robert Muir added a comment - Thanks valentin, Bernd!
        Robert Muir made changes -
        Status Open [ 1 ] Resolved [ 5 ]
        Fix Version/s 3.5 [ 12317876 ]
        Fix Version/s 4.0 [ 12314992 ]
        Resolution Fixed [ 1 ]
        Hide
        Uwe Schindler added a comment -

        Bulk close after 3.5 is released

        Show
        Uwe Schindler added a comment - Bulk close after 3.5 is released
        Uwe Schindler made changes -
        Status Resolved [ 5 ] Closed [ 6 ]

          People

          • Assignee:
            Robert Muir
            Reporter:
            valentin
          • Votes:
            1 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development