Solr
  1. Solr
  2. SOLR-8295

collapse parser can trigger NPE when a segment has empty DocValues for collapse field (ie: no doc in segment has a value)

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 5.4, 6.0
    • Component/s: None
    • Labels:
      None

      Description

      When collapsing on a field that doesn't exist for all documents, the existence of a segment that does not contain any documents that have a value can cause an NPE as the collapse parser assumes the DocValues for a segment will always be non-null.

        Issue Links

          Activity

          Hide
          Hoss Man added a comment -

          Updated summary & description to clarify root cause.
          The underlying bug predates SOLR-6168, (i think it has been arround for as long as the collapse parser has existed?) but was uncovered due to random testing added by that issue.

          Initial report...

          From https://builds.apache.org/job/Lucene-Solr-Tests-trunk-Java8/629/ - reproduces for me:

             [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestRandomCollapseQParserPlugin -Dtests.method=testRandomCollpaseWithSort -Dtests.seed=EB40CE4410753BB -Dtests.multiplier=2 -Dtests.slow=true -Dtests.locale=es_PY -Dtests.timezone=Asia/Oral -Dtests.asserts=true -Dtests.file.encoding=US-ASCII
             [junit4] ERROR   11.7s J2 | TestRandomCollapseQParserPlugin.testRandomCollpaseWithSort <<<
             [junit4]    > Throwable #1: org.apache.solr.client.solrj.SolrServerException: org.apache.solr.client.solrj.SolrServerException: java.lang.NullPointerException
             [junit4]    > 	at __randomizedtesting.SeedInfo.seed([EB40CE4410753BB:B097F2D6AAF6048]:0)
             [junit4]    > 	at org.apache.solr.client.solrj.embedded.EmbeddedSolrServer.request(EmbeddedSolrServer.java:240)
             [junit4]    > 	at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:150)
             [junit4]    > 	at org.apache.solr.client.solrj.SolrClient.query(SolrClient.java:943)
             [junit4]    > 	at org.apache.solr.client.solrj.SolrClient.query(SolrClient.java:958)
             [junit4]    > 	at org.apache.solr.search.TestRandomCollapseQParserPlugin.testRandomCollpaseWithSort(TestRandomCollapseQParserPlugin.java:164)
             [junit4]    > 	at java.lang.Thread.run(Thread.java:745)
             [junit4]    > Caused by: org.apache.solr.client.solrj.SolrServerException: java.lang.NullPointerException
             [junit4]    > 	at org.apache.solr.client.solrj.embedded.EmbeddedSolrServer.checkForExceptions(EmbeddedSolrServer.java:252)
             [junit4]    > 	at org.apache.solr.client.solrj.embedded.EmbeddedSolrServer.request(EmbeddedSolrServer.java:180)
             [junit4]    > 	... 43 more
             [junit4]    > Caused by: java.lang.NullPointerException
             [junit4]    > 	at org.apache.solr.search.CollapsingQParserPlugin$IntFieldValueCollector.collect(CollapsingQParserPlugin.java:1109)
             [junit4]    > 	at org.apache.lucene.search.MatchAllDocsQuery$1$1.score(MatchAllDocsQuery.java:56)
             [junit4]    > 	at org.apache.lucene.search.BulkScorer.score(BulkScorer.java:39)
             [junit4]    > 	at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:670)
             [junit4]    > 	at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:474)
             [junit4]    > 	at org.apache.solr.search.SolrIndexSearcher.buildAndRunCollectorChain(SolrIndexSearcher.java:203)
             [junit4]    > 	at org.apache.solr.search.SolrIndexSearcher.getDocListNC(SolrIndexSearcher.java:1692)
             [junit4]    > 	at org.apache.solr.search.SolrIndexSearcher.getDocListC(SolrIndexSearcher.java:1508)
             [junit4]    > 	at org.apache.solr.search.SolrIndexSearcher.search(SolrIndexSearcher.java:562)
             [junit4]    > 	at org.apache.solr.handler.component.QueryComponent.process(QueryComponent.java:519)
             [junit4]    > 	at org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:276)
             [junit4]    > 	at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:151)
             [junit4]    > 	at org.apache.solr.core.SolrCore.execute(SolrCore.java:2079)
             [junit4]    > 	at org.apache.solr.client.solrj.embedded.EmbeddedSolrServer.request(EmbeddedSolrServer.java:179)
             [junit4]    > 	... 43 more
          
          Show
          Hoss Man added a comment - Updated summary & description to clarify root cause. The underlying bug predates SOLR-6168 , (i think it has been arround for as long as the collapse parser has existed?) but was uncovered due to random testing added by that issue. Initial report... From https://builds.apache.org/job/Lucene-Solr-Tests-trunk-Java8/629/ - reproduces for me: [junit4] 2> NOTE: reproduce with: ant test -Dtestcase=TestRandomCollapseQParserPlugin -Dtests.method=testRandomCollpaseWithSort -Dtests.seed=EB40CE4410753BB -Dtests.multiplier=2 -Dtests.slow=true -Dtests.locale=es_PY -Dtests.timezone=Asia/Oral -Dtests.asserts=true -Dtests.file.encoding=US-ASCII [junit4] ERROR 11.7s J2 | TestRandomCollapseQParserPlugin.testRandomCollpaseWithSort <<< [junit4] > Throwable #1: org.apache.solr.client.solrj.SolrServerException: org.apache.solr.client.solrj.SolrServerException: java.lang.NullPointerException [junit4] > at __randomizedtesting.SeedInfo.seed([EB40CE4410753BB:B097F2D6AAF6048]:0) [junit4] > at org.apache.solr.client.solrj.embedded.EmbeddedSolrServer.request(EmbeddedSolrServer.java:240) [junit4] > at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:150) [junit4] > at org.apache.solr.client.solrj.SolrClient.query(SolrClient.java:943) [junit4] > at org.apache.solr.client.solrj.SolrClient.query(SolrClient.java:958) [junit4] > at org.apache.solr.search.TestRandomCollapseQParserPlugin.testRandomCollpaseWithSort(TestRandomCollapseQParserPlugin.java:164) [junit4] > at java.lang.Thread.run(Thread.java:745) [junit4] > Caused by: org.apache.solr.client.solrj.SolrServerException: java.lang.NullPointerException [junit4] > at org.apache.solr.client.solrj.embedded.EmbeddedSolrServer.checkForExceptions(EmbeddedSolrServer.java:252) [junit4] > at org.apache.solr.client.solrj.embedded.EmbeddedSolrServer.request(EmbeddedSolrServer.java:180) [junit4] > ... 43 more [junit4] > Caused by: java.lang.NullPointerException [junit4] > at org.apache.solr.search.CollapsingQParserPlugin$IntFieldValueCollector.collect(CollapsingQParserPlugin.java:1109) [junit4] > at org.apache.lucene.search.MatchAllDocsQuery$1$1.score(MatchAllDocsQuery.java:56) [junit4] > at org.apache.lucene.search.BulkScorer.score(BulkScorer.java:39) [junit4] > at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:670) [junit4] > at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:474) [junit4] > at org.apache.solr.search.SolrIndexSearcher.buildAndRunCollectorChain(SolrIndexSearcher.java:203) [junit4] > at org.apache.solr.search.SolrIndexSearcher.getDocListNC(SolrIndexSearcher.java:1692) [junit4] > at org.apache.solr.search.SolrIndexSearcher.getDocListC(SolrIndexSearcher.java:1508) [junit4] > at org.apache.solr.search.SolrIndexSearcher.search(SolrIndexSearcher.java:562) [junit4] > at org.apache.solr.handler.component.QueryComponent.process(QueryComponent.java:519) [junit4] > at org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:276) [junit4] > at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:151) [junit4] > at org.apache.solr.core.SolrCore.execute(SolrCore.java:2079) [junit4] > at org.apache.solr.client.solrj.embedded.EmbeddedSolrServer.request(EmbeddedSolrServer.java:179) [junit4] > ... 43 more
          Hide
          Hoss Man added a comment -

          patch that:

          • adds specific tests for this situation that fails
          • improves the error reporting for the randomized test to make it easier to identify what the params were that caused an error
          • fixes all of the above failures by using the helper method DocValues.getNumeric (which returns an empty NumericDocValues object instead of null like IndexReader..getNumericDocValues)
          Show
          Hoss Man added a comment - patch that: adds specific tests for this situation that fails improves the error reporting for the randomized test to make it easier to identify what the params were that caused an error fixes all of the above failures by using the helper method DocValues.getNumeric (which returns an empty NumericDocValues object instead of null like IndexReader..getNumericDocValues)
          Hide
          ASF subversion and git services added a comment -

          Commit 1714701 from hossman@apache.org in branch 'dev/trunk'
          [ https://svn.apache.org/r1714701 ]

          SOLR-8295: Fix NPE in collapse QParser when collapse field is missing from all docs in a segment

          Show
          ASF subversion and git services added a comment - Commit 1714701 from hossman@apache.org in branch 'dev/trunk' [ https://svn.apache.org/r1714701 ] SOLR-8295 : Fix NPE in collapse QParser when collapse field is missing from all docs in a segment
          Hide
          ASF subversion and git services added a comment -

          Commit 1714709 from hossman@apache.org in branch 'dev/branches/branch_5x'
          [ https://svn.apache.org/r1714709 ]

          SOLR-8295: Fix NPE in collapse QParser when collapse field is missing from all docs in a segment (merge r1714701)

          Show
          ASF subversion and git services added a comment - Commit 1714709 from hossman@apache.org in branch 'dev/branches/branch_5x' [ https://svn.apache.org/r1714709 ] SOLR-8295 : Fix NPE in collapse QParser when collapse field is missing from all docs in a segment (merge r1714701)
          Hide
          ASF subversion and git services added a comment -

          Commit 1714843 from hossman@apache.org in branch 'dev/trunk'
          [ https://svn.apache.org/r1714843 ]

          SOLR-8295: fix buggy test assumption

          Show
          ASF subversion and git services added a comment - Commit 1714843 from hossman@apache.org in branch 'dev/trunk' [ https://svn.apache.org/r1714843 ] SOLR-8295 : fix buggy test assumption
          Hide
          ASF subversion and git services added a comment -

          Commit 1714848 from hossman@apache.org in branch 'dev/branches/branch_5x'
          [ https://svn.apache.org/r1714848 ]

          SOLR-8295: fix buggy test assumption (merge r1714843)

          Show
          ASF subversion and git services added a comment - Commit 1714848 from hossman@apache.org in branch 'dev/branches/branch_5x' [ https://svn.apache.org/r1714848 ] SOLR-8295 : fix buggy test assumption (merge r1714843)
          Hide
          Joel Bernstein added a comment -

          Ah, this is already fixed. I was looking into a bug report today on the users list for 5.3.1. This bug was introduced in the 5x branch when the numeric docValues support was added. I suspect it took a long time to surface because many people are still using the 4x line of code. In the 4x line of code everything was based on a top level field cache which couldn't have this problem. I'll report back to the users list about this.

          Show
          Joel Bernstein added a comment - Ah, this is already fixed. I was looking into a bug report today on the users list for 5.3.1. This bug was introduced in the 5x branch when the numeric docValues support was added. I suspect it took a long time to surface because many people are still using the 4x line of code. In the 4x line of code everything was based on a top level field cache which couldn't have this problem. I'll report back to the users list about this.

            People

            • Assignee:
              Hoss Man
              Reporter:
              Steve Rowe
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development