Uploaded image for project: 'Solr'
  1. Solr
  2. SOLR-8807

NPE during spell checking when result collapsing is activated and index got more than one segment.

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 5.4
    • Fix Version/s: 6.6, 7.0
    • Component/s: spellchecker
    • Environment:

      Solr 5.4 with an index cosisting of two segments

      Description

      When using spellchecker with collapse/expand results, I got an NPE. Only happend when the index consists of more than one segment.

      11:30:33,505 WARN  [org.apache.solr.spelling.SpellCheckCollator] (http-/0.0.0.0:8080-2) Exception trying to re-query to check if a spell check possibility would return any hits.: java.lang.NullPointerException
              at org.apache.solr.search.CollapsingQParserPlugin$OrdScoreCollector.finish(CollapsingQParserPlugin.java:631) [solr-core-5.4.0.jar:5.4.0 1718046 - upayavira - 2015-12-04 23:16:46]
              at org.apache.solr.search.CollapsingQParserPlugin$OrdScoreCollector.finish(CollapsingQParserPlugin.java:681) [solr-core-5.4.0.jar:5.4.0 1718046 - upayavira - 2015-12-04 23:16:46]
              at org.apache.solr.search.SolrIndexSearcher.buildAndRunCollectorChain(SolrIndexSearcher.java:213) [solr-core-5.4.0.jar:5.4.0 1718046 - upayavira - 2015-12-04 23:16:46]
              at org.apache.solr.search.SolrIndexSearcher.getDocListNC(SolrIndexSearcher.java:1672) [solr-core-5.4.0.jar:5.4.0 1718046 - upayavira - 2015-12-04 23:16:46]
              at org.apache.solr.search.SolrIndexSearcher.getDocListC(SolrIndexSearcher.java:1491) [solr-core-5.4.0.jar:5.4.0 1718046 - upayavira - 2015-12-04 23:16:46]
              at org.apache.solr.search.SolrIndexSearcher.search(SolrIndexSearcher.java:557) [solr-core-5.4.0.jar:5.4.0 1718046 - upayavira - 2015-12-04 23:16:46]
              at org.apache.solr.handler.component.QueryComponent.process(QueryComponent.java:525) [solr-core-5.4.0.jar:5.4.0 1718046 - upayavira - 2015-12-04 23:16:46]
              at org.apache.solr.spelling.SpellCheckCollator.collate(SpellCheckCollator.java:147) [solr-core-5.4.0.jar:5.4.0 1718046 - upayavira - 2015-12-04 23:16:46]
              at org.apache.solr.handler.component.SpellCheckComponent.addCollationsToResponse(SpellCheckComponent.java:238) [solr-core-5.4.0.jar:5.4.0 1718046 - upayavira - 2015-12-04 23:16:46]
              at org.apache.solr.handler.component.SpellCheckComponent.process(SpellCheckComponent.java:203) [solr-core-5.4.0.jar:5.4.0 1718046 - upayavira - 2015-12-04 23:16:46]
              at org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:281) [solr-core-5.4.0.jar:5.4.0 1718046 - upayavira - 2015-12-04 23:16:46]
              at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:156) [solr-core-5.4.0.jar:5.4.0 1718046 - upayavira - 2015-12-04 23:16:46]
              at org.apache.solr.core.SolrCore.execute(SolrCore.java:2073) [solr-core-5.4.0.jar:5.4.0 1718046 - upayavira - 2015-12-04 23:16:46]
      
        <requestHandler name="/select" class="solr.SearchHandler">
      	<arr name="last-components">
      	  <str>spellchecker</str>
      	</arr>    
          </requestHandler>
      

      The query parameters are:

            "spellcheck.maxCollations": "5",
            "q.op": "AND",
            "fq": "{!collapse field=type}",
            "spellcheck.maxCollationTries": "10",
            "spellcheck.collateMaxCollectDocs": "100000",
            "spellcheck.alternativeTermCount": "10",
            "spellcheck.extendedResults": "true",
            "spellcheck.dictionary": [
              "dest_wordbreak",
              "dest_fuzzy"
            ],
            "q": "kosamui thailand",
            "defType": "edismax",
            "expand": "true",
            "spellcheck.maxResultsForSuggest": "3",
            "qf": "country_name region_name",
            "spellcheck": "true",
            "spellcheck.accuracy": "0.5",
            "spellcheck.count": "20",
            "spellcheck.collate": "true",
      
      1. SOLR-8806-failing-unit-test.patch
        9 kB
        James Dyer
      2. SOLR-8807.patch
        10 kB
        James Dyer
      3. SOLR-8807.patch
        2 kB
        Matthias Krueger

        Issue Links

          Activity

          Hide
          joel.bernstein Joel Bernstein added a comment -

          Are you sure this is related to the spellchecker? Does the collapse query work if the spellchecker is turned off?

          Show
          joel.bernstein Joel Bernstein added a comment - Are you sure this is related to the spellchecker? Does the collapse query work if the spellchecker is turned off?
          Hide
          joel.bernstein Joel Bernstein added a comment -

          Wondering if the spellchecker is enabling the early termination collector. It appears that not all the segments are being visited in the search that is being executed. I'll dig into the spellchecker and see what I can find.

          Show
          joel.bernstein Joel Bernstein added a comment - Wondering if the spellchecker is enabling the early termination collector. It appears that not all the segments are being visited in the search that is being executed. I'll dig into the spellchecker and see what I can find.
          Hide
          joel.bernstein Joel Bernstein added a comment - - edited

          Just found this line in the SpellcheckCollator:

           checkResponse.setFieldFlags(f |= SolrIndexSearcher.TERMINATE_EARLY);            
          

          This is problematic for the CollapsingQParserPlugin because not all segments will get visited during the search. Right now the CollapsingQParserPlugin relies on all segments getting visited or you get the NPE that is happening in this ticket. There probably is a way to deal with this but it's going to take some looking into.

          Show
          joel.bernstein Joel Bernstein added a comment - - edited Just found this line in the SpellcheckCollator: checkResponse.setFieldFlags(f |= SolrIndexSearcher.TERMINATE_EARLY); This is problematic for the CollapsingQParserPlugin because not all segments will get visited during the search. Right now the CollapsingQParserPlugin relies on all segments getting visited or you get the NPE that is happening in this ticket. There probably is a way to deal with this but it's going to take some looking into.
          Hide
          joel.bernstein Joel Bernstein added a comment - - edited

          If you stop setting spellcheck.collateMaxCollectDocs it should turn off the early termination. Not sure if this will kill your performance, but it should make the NPE go away.

          Show
          joel.bernstein Joel Bernstein added a comment - - edited If you stop setting spellcheck.collateMaxCollectDocs it should turn off the early termination. Not sure if this will kill your performance, but it should make the NPE go away.
          Hide
          markus17 Markus Jelsma added a comment -

          FYI, for some reason the NPE disappeared after a collection optimize. I cannot reproduce it like i did a few minutes ago just before the optimize.

          Show
          markus17 Markus Jelsma added a comment - FYI, for some reason the NPE disappeared after a collection optimize. I cannot reproduce it like i did a few minutes ago just before the optimize.
          Hide
          markus17 Markus Jelsma added a comment -

          Ah, to verify, i reindexed documents over the existing collection to force maxDoc > numDoc and the NPE appeared again. After an optimize, the NPE disappeared again.

          Show
          markus17 Markus Jelsma added a comment - Ah, to verify, i reindexed documents over the existing collection to force maxDoc > numDoc and the NPE appeared again. After an optimize, the NPE disappeared again.
          Hide
          mkrio Matthias Krueger added a comment -

          My client is affected by this NPE, too. Some oddities of the SpellCheckCollator seem to misalign with behaviour of the CollapsingQParserPlugin's collectors.

          • The checkResponse ResponseBuilder in SpellCheckCollator#collate is being built using the original params (including the fq={!collapseā€¦} param) and the original filters.
          • The checkResponse ends up having the CollapsingQParserPlugin$CollapsingPostFilter twice in the filters list.
          • They will end up in a DelegatingCollector chain CollapsingQParserPlugin$OrdScoreCollector -> CollapsingQParserPlugin$OrdScoreCollector.
          • Unfortunately, this does not work well when there are documents returned from less than all segments. The two OrdScoreCollectors are instantiated with the number of segments and open an empty array of LeafReaderContexts but their values are only populated for the first OrdScoreCollector in IndexSearcher#search (via Collector#getLeafCollector -> OrdScoreCollector#doSetNextReader) calls.
          • OrdScoreCollector (and the other collectors in CollapsingQParserPlugin) only passes the LeafReaderContexts to the delegate within finish() (not within #doSetNextReader) but stops when it deems all necessary segments consumed. When the delegate itself then performs the iteration it will NPE in
            nextDocBase = currentContext+1 < contexts.length ? contexts[currentContext+1].docBase : maxDoc;
            
          Show
          mkrio Matthias Krueger added a comment - My client is affected by this NPE, too. Some oddities of the SpellCheckCollator seem to misalign with behaviour of the CollapsingQParserPlugin's collectors. The checkResponse ResponseBuilder in SpellCheckCollator#collate is being built using the original params (including the fq={!collapseā€¦} param) and the original filters. The checkResponse ends up having the CollapsingQParserPlugin$CollapsingPostFilter twice in the filters list. They will end up in a DelegatingCollector chain CollapsingQParserPlugin$OrdScoreCollector -> CollapsingQParserPlugin$OrdScoreCollector. Unfortunately, this does not work well when there are documents returned from less than all segments. The two OrdScoreCollectors are instantiated with the number of segments and open an empty array of LeafReaderContexts but their values are only populated for the first OrdScoreCollector in IndexSearcher#search (via Collector#getLeafCollector -> OrdScoreCollector#doSetNextReader) calls. OrdScoreCollector (and the other collectors in CollapsingQParserPlugin) only passes the LeafReaderContexts to the delegate within finish() (not within #doSetNextReader) but stops when it deems all necessary segments consumed. When the delegate itself then performs the iteration it will NPE in nextDocBase = currentContext+1 < contexts.length ? contexts[currentContext+1].docBase : maxDoc;
          Hide
          mkrio Matthias Krueger added a comment - - edited

          This seems to be very much related to SOLR-9104, SOLR-10336, and SOLR-7435 or even the same cause.

          Show
          mkrio Matthias Krueger added a comment - - edited This seems to be very much related to SOLR-9104 , SOLR-10336 , and SOLR-7435 or even the same cause.
          Hide
          mkrio Matthias Krueger added a comment -

          A simple solution would be to have CollapsingQParserPlugin behave like other DelegatingCollectors and call

          super.doSetNextReader(context); 
          

          in doSetNextReader. Not sure if omission was intentional though. This at least avoids the NPE we're seeing.

          Show
          mkrio Matthias Krueger added a comment - A simple solution would be to have CollapsingQParserPlugin behave like other DelegatingCollectors and call super .doSetNextReader(context); in doSetNextReader. Not sure if omission was intentional though. This at least avoids the NPE we're seeing.
          Hide
          jdyer James Dyer added a comment -

          While this won't address the other cases where the Collapsing Post-Filter can throw an NPE, we should probably just omit the post-filter when testing collation candidates. As with Result Grouping, SpellCheckCollator can execute the query without the post-filter and then report the # of un-grouped results the collation returns. (or, in this specific case, an estimated un-grouped hit-count) We also don't want to be doing anything unnecessary as we want to test the collations for hits in a way that does as little work as possible, so omitting the post-filter seems to be the best approach all around.

          Show
          jdyer James Dyer added a comment - While this won't address the other cases where the Collapsing Post-Filter can throw an NPE, we should probably just omit the post-filter when testing collation candidates. As with Result Grouping, SpellCheckCollator can execute the query without the post-filter and then report the # of un-grouped results the collation returns. (or, in this specific case, an estimated un-grouped hit-count) We also don't want to be doing anything unnecessary as we want to test the collations for hits in a way that does as little work as possible, so omitting the post-filter seems to be the best approach all around.
          Hide
          jdyer James Dyer added a comment -

          This patch SOLR-8806-failing-unit-test.patch contains a unit test that fails with most seeds. It fails with...

          java.lang.NullPointerException
          	at org.apache.solr.search.CollapsingQParserPlugin$IntScoreCollector.finish(CollapsingQParserPlugin.java:870)
          

          ...reproducing this issue's problem.

          Show
          jdyer James Dyer added a comment - This patch SOLR-8806-failing-unit-test.patch contains a unit test that fails with most seeds. It fails with... java.lang.NullPointerException at org.apache.solr.search.CollapsingQParserPlugin$IntScoreCollector.finish(CollapsingQParserPlugin.java:870) ...reproducing this issue's problem.
          Hide
          jdyer James Dyer added a comment -

          With this patch, SOLR-8807.patch , the new test passes. When checking candidate queries for hit-counts, SpellCheckCollator removes the collapse post-filter and the "expand" parameter from the query.

          If there are no objections to this approach, I will commit this in a few days.

          Show
          jdyer James Dyer added a comment - With this patch, SOLR-8807.patch , the new test passes. When checking candidate queries for hit-counts, SpellCheckCollator removes the collapse post-filter and the "expand" parameter from the query. If there are no objections to this approach, I will commit this in a few days.
          Hide
          mkrio Matthias Krueger added a comment -

          Thanks, James Dyer, looks good to me. This won't help the linked CollapsingQParser issues (which all seem to have the same root cause) but will solve the SpellCheckCollator's problem.

          Show
          mkrio Matthias Krueger added a comment - Thanks, James Dyer , looks good to me. This won't help the linked CollapsingQParser issues (which all seem to have the same root cause) but will solve the SpellCheckCollator's problem.
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit 55e36615eb4e84abbb674e52eaae0954b47e5b11 in lucene-solr's branch refs/heads/master from jdyer1
          [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=55e3661 ]

          SOLR-8807: disable the CollapseQParser Plugin when testing spellcheck collations for hit-counts

          Show
          jira-bot ASF subversion and git services added a comment - Commit 55e36615eb4e84abbb674e52eaae0954b47e5b11 in lucene-solr's branch refs/heads/master from jdyer1 [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=55e3661 ] SOLR-8807 : disable the CollapseQParser Plugin when testing spellcheck collations for hit-counts
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit d2092f7c55014c5366c3c5a021550c1f91c7e80b in lucene-solr's branch refs/heads/branch_6x from jdyer1
          [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=d2092f7 ]

          SOLR-8807: disable the CollapseQParser Plugin when testing spellcheck collations for hit-counts

          Show
          jira-bot ASF subversion and git services added a comment - Commit d2092f7c55014c5366c3c5a021550c1f91c7e80b in lucene-solr's branch refs/heads/branch_6x from jdyer1 [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=d2092f7 ] SOLR-8807 : disable the CollapseQParser Plugin when testing spellcheck collations for hit-counts
          Hide
          jdyer James Dyer added a comment -

          Thank you Christian for reporting this bug.

          Show
          jdyer James Dyer added a comment - Thank you Christian for reporting this bug.

            People

            • Assignee:
              jdyer James Dyer
              Reporter:
              cdanningerCV Christian Danninger
            • Votes:
              2 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development