Solr
  1. Solr
  2. SOLR-2504

Combined usage of Synonyms/SpellChecker causes java.lang.NullPointerException, when searching for a word out of synonyms.txt

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 3.1
    • Fix Version/s: None
    • Labels:
      None

      Description

      After migrating from 1.4 to 3.1 we experience the following behaviour:

      When SpellChecking is turned off, everything works fine.
      When Synonyms are not being used, everything works fine.

      When both, SpellChecking and Synonyms, are being used and a search is triggered, that contains at least one of the words out of synonyms.txt the following error is thrown:

      java.lang.NullPointerException
      at org.apache.lucene.util.AttributeSource.cloneAttributes(AttributeSource.java:542)
      at org.apache.solr.analysis.SynonymFilter.incrementToken(SynonymFilter.java:132)
      at org.apache.lucene.analysis.FilteringTokenFilter.incrementToken(FilteringTokenFilter.java:58)
      at org.apache.solr.handler.component.SpellCheckComponent.getTokens(SpellCheckComponent.java:485)
      at org.apache.solr.handler.component.SpellCheckComponent.process(SpellCheckComponent.java:131)
      at org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:194)
      at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:129)
      at org.apache.solr.core.SolrCore.execute(SolrCore.java:1360)
      at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:356)
      at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:252)
      at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215)
      at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
      at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)
      at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:172)
      at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
      at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:117)
      at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:108)
      at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:151)
      at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:874)
      at org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:665)
      at org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:528)
      at org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:81)
      at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:689)
      at java.lang.Thread.run(Thread.java:619)

      The problem has been described already here:
      http://osdir.com/ml/solr-user.lucene.apache.org/2010-09/msg00945.html

      I have a report of a third person, experiencing the same problem.

        Activity

        Hide
        Uwe Schindler added a comment -

        The line numbers in your stack trace do not correspond to the officially released version 3.1 of Lucene 3.1 / Solr 3.1. Maybe you have some older JAR files in your classpath?

        Show
        Uwe Schindler added a comment - The line numbers in your stack trace do not correspond to the officially released version 3.1 of Lucene 3.1 / Solr 3.1. Maybe you have some older JAR files in your classpath?
        Hide
        Jens Bertheau added a comment -

        Updated: In my first posting I pasted the error message with wrong line numbers, now the line numbers should represent the current JAR files.

        Show
        Jens Bertheau added a comment - Updated: In my first posting I pasted the error message with wrong line numbers, now the line numbers should represent the current JAR files.
        Hide
        Uwe Schindler added a comment -

        Thanks for the correct line numbers! My last comment I deleted was wrong.

        In the current stable branch there is an update related to that, maybe take the current stable branch?

        Show
        Uwe Schindler added a comment - Thanks for the correct line numbers! My last comment I deleted was wrong. In the current stable branch there is an update related to that, maybe take the current stable branch?
        Hide
        Stefan Moises added a comment -

        I am the initial reporter of this bug in the mailing list and currently we are using a nightly build of Solr (apache-solr-4.0-2011-03-16_08-16-50.tar) and still get the error, so I don't think it has been fixed yet, unless those changes you've mentioned have been at a later date or in another branch - here are my line numbers from that nighly build:

        SEVERE: java.lang.NullPointerException
        at org.apache.lucene.util.AttributeSource.cloneAttributes(AttributeSource.java:507)
        at org.apache.lucene.analysis.synonym.SynonymFilter.incrementToken(SynonymFilter.java:131)
        at org.apache.lucene.analysis.util.FilteringTokenFilter.incrementToken(FilteringTokenFilter.java:60)
        at org.apache.lucene.analysis.miscellaneous.WordDelimiterFilter.incrementToken(WordDelimiterFilter.java:224)
        at org.apache.lucene.analysis.core.LowerCaseFilter.incrementToken(LowerCaseFilter.java:54)
        at org.apache.lucene.analysis.miscellaneous.KeywordMarkerFilter.incrementToken(KeywordMarkerFilter.java:76)
        at org.apache.lucene.analysis.snowball.SnowballFilter.incrementToken(SnowballFilter.java:77)
        at org.apache.lucene.analysis.miscellaneous.RemoveDuplicatesTokenFilter.incrementToken(RemoveDuplicatesTokenFilter.java:54)
        at org.apache.solr.spelling.SpellingQueryConverter.convert(SpellingQueryConverter.java:115)
        at org.apache.solr.handler.component.SpellCheckComponent.process(SpellCheckComponent.java:137)
        at org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:239)
        at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:129)
        at org.apache.solr.core.SolrCore.execute(SolrCore.java:1298)
        at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:353)
        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:248)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
        at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:465)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
        at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:555)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298)
        at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:852)
        at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:588)
        at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489)
        at java.lang.Thread.run(Thread.java:662)

        Show
        Stefan Moises added a comment - I am the initial reporter of this bug in the mailing list and currently we are using a nightly build of Solr (apache-solr-4.0-2011-03-16_08-16-50.tar) and still get the error, so I don't think it has been fixed yet, unless those changes you've mentioned have been at a later date or in another branch - here are my line numbers from that nighly build: SEVERE: java.lang.NullPointerException at org.apache.lucene.util.AttributeSource.cloneAttributes(AttributeSource.java:507) at org.apache.lucene.analysis.synonym.SynonymFilter.incrementToken(SynonymFilter.java:131) at org.apache.lucene.analysis.util.FilteringTokenFilter.incrementToken(FilteringTokenFilter.java:60) at org.apache.lucene.analysis.miscellaneous.WordDelimiterFilter.incrementToken(WordDelimiterFilter.java:224) at org.apache.lucene.analysis.core.LowerCaseFilter.incrementToken(LowerCaseFilter.java:54) at org.apache.lucene.analysis.miscellaneous.KeywordMarkerFilter.incrementToken(KeywordMarkerFilter.java:76) at org.apache.lucene.analysis.snowball.SnowballFilter.incrementToken(SnowballFilter.java:77) at org.apache.lucene.analysis.miscellaneous.RemoveDuplicatesTokenFilter.incrementToken(RemoveDuplicatesTokenFilter.java:54) at org.apache.solr.spelling.SpellingQueryConverter.convert(SpellingQueryConverter.java:115) at org.apache.solr.handler.component.SpellCheckComponent.process(SpellCheckComponent.java:137) at org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:239) at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:129) at org.apache.solr.core.SolrCore.execute(SolrCore.java:1298) at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:353) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:248) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191) at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:465) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102) at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:555) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298) at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:852) at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:588) at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489) at java.lang.Thread.run(Thread.java:662)
        Hide
        Uwe Schindler added a comment -

        From looking at the code: It's impossible to get an NPE at this place. As this bug only appears in some combinations of components in Solr, this maybe an Hotspot bug in Suns JVM. Can you try to run your server using the -Xbatch switch, to disable background hotspot compilation (may slowdown your server immense)?

        Show
        Uwe Schindler added a comment - From looking at the code: It's impossible to get an NPE at this place. As this bug only appears in some combinations of components in Solr, this maybe an Hotspot bug in Suns JVM. Can you try to run your server using the -Xbatch switch, to disable background hotspot compilation (may slowdown your server immense)?
        Hide
        Uwe Schindler added a comment -

        Stefan Moises: Changes in AttributeSource are dated 2011-04-23 (LUCENE-3042).

        Show
        Uwe Schindler added a comment - Stefan Moises: Changes in AttributeSource are dated 2011-04-23 ( LUCENE-3042 ).
        Hide
        Stefan Moises added a comment -

        Uwe: thanks for the info! Would it make sense to update to the latest nightly? But I'm with you, the code doesn't look like there should ever be a NPE ... So I'll try the JVM switch first I think.

        Show
        Stefan Moises added a comment - Uwe: thanks for the info! Would it make sense to update to the latest nightly? But I'm with you, the code doesn't look like there should ever be a NPE ... So I'll try the JVM switch first I think.
        Hide
        Jens Bertheau added a comment -

        OK, I tried the current snapshot (1100496) and it actually solves the problem!
        @Stefan: Apparently you are using already Solr 4. Not sure, if the problem has been fixed in the 4-branch...

        Show
        Jens Bertheau added a comment - OK, I tried the current snapshot (1100496) and it actually solves the problem! @Stefan: Apparently you are using already Solr 4. Not sure, if the problem has been fixed in the 4-branch...
        Hide
        Uwe Schindler added a comment -

        Its also fixed in trunk since the above date!

        Show
        Uwe Schindler added a comment - Its also fixed in trunk since the above date!
        Hide
        Stefan Moises added a comment -

        Awesome, I'll update ASAP and let you know if everythings ok in Solr 4.x now, too!

        Show
        Stefan Moises added a comment - Awesome, I'll update ASAP and let you know if everythings ok in Solr 4.x now, too!
        Hide
        Jens Bertheau added a comment -

        Fixed in version of 2011-04-23

        Show
        Jens Bertheau added a comment - Fixed in version of 2011-04-23
        Hide
        Stefan Moises added a comment -

        Just for the record - works now for me in trunk, too! Many thanks for the quick help!

        Show
        Stefan Moises added a comment - Just for the record - works now for me in trunk, too! Many thanks for the quick help!
        Hide
        Uwe Schindler added a comment -

        Can we close this issue then? The reason for this problem is still unknown, I suspect an JVM bug, because NPE is impossible at that point. The changes in code simply removed the problem - but I don't understand why.

        Can we confirm that it works with current updated 3.1 branch, future 3.2 (branch_3x) and 4.0 (trunk)?

        Show
        Uwe Schindler added a comment - Can we close this issue then? The reason for this problem is still unknown, I suspect an JVM bug, because NPE is impossible at that point. The changes in code simply removed the problem - but I don't understand why. Can we confirm that it works with current updated 3.1 branch, future 3.2 (branch_3x) and 4.0 (trunk)?

          People

          • Assignee:
            Uwe Schindler
            Reporter:
            Jens Bertheau
          • Votes:
            1 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development