Uploaded image for project: 'Solr'
  1. Solr
  2. SOLR-6744

fl renaming / alias of uniqueKey field generates null pointer exception in SolrCloud configuration

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 4.10.1
    • Fix Version/s: 6.2.1, 6.3, 7.0
    • Component/s: SolrCloud
    • Labels:
      None
    • Environment:

      Multiple replicas on SolrCloud config. This specific example with 4 shard, 3 replica per shard config. This bug does NOT exist when query is handled by single core.

      Description

      If trying to rename the uniqueKey field using 'fl' in a distributed query (ie: SolrCloud config), an NPE is thrown.

      The workarround is to redundently request the uniqueKey field, once with the desired alias, and once with the original name

      Example...

      http://localhost:8983/solr/cloudcollection/select?q=*%3A*&wt=xml&indent=true&fl=key:id

      Work around:

      http://localhost:8983/solr/cloudcollection/select?q=*%3A*&wt=xml&indent=true&fl=key:id&fl=id

      Error w/o work around...

      <response><lst name="responseHeader"><int name="status">500</int><int name="QTime">11</int><lst name="params"><str name="q">*:*</str><str name="indent">true</str><str name="fl">key:id</str><str name="wt">xml</str></lst></lst><lst name="error"><str name="trace">java.lang.NullPointerException
      	at org.apache.solr.handler.component.QueryComponent.returnFields(QueryComponent.java:1257)
      	at org.apache.solr.handler.component.QueryComponent.handleRegularResponses(QueryComponent.java:720)
      	at org.apache.solr.handler.component.QueryComponent.handleResponses(QueryComponent.java:695)
      	at org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:324)
      	at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)
      	at org.apache.solr.core.SolrCore.execute(SolrCore.java:1967)
      	at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:777)
      	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:418)
      	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:207)
      	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1419)
      	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:455)
      	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
      	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:557)
      	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
      	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1075)
      	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:384)
      	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
      	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1009)
      	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
      	at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:255)
      	at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:154)
      	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
      	at org.eclipse.jetty.server.Server.handle(Server.java:368)
      	at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489)
      	at org.eclipse.jetty.server.BlockingHttpConnection.handleRequest(BlockingHttpConnection.java:53)
      	at org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:942)
      	at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:1004)
      	at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:640)
      	at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)
      	at org.eclipse.jetty.server.BlockingHttpConnection.handle(BlockingHttpConnection.java:72)
      	at org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run(SocketConnector.java:264)
      	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
      	at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
      	at java.lang.Thread.run(Thread.java:745)
      </str><int name="code">500</int></lst></response>
      
      1. SOLR-6744.patch
        7 kB
        Mike Drob
      2. SOLR-6744.patch
        7 kB
        Mike Drob
      3. SOLR-6744.patch
        5 kB
        Mike Drob

        Activity

        Hide
        hossman Hoss Man added a comment -

        i edited the summary & description a bit to fix up the formatting and point out the workaround (as noted by Jeon Woosung on the mailing list)

        the root issue is that the the distributed processing code relies on the uniqueKey field to rectify & merge information from each shard across the various distributed process from shards, and the NPE comes from trying to do this lookup on the uniqueKey field name.

        Show
        hossman Hoss Man added a comment - i edited the summary & description a bit to fix up the formatting and point out the workaround (as noted by Jeon Woosung on the mailing list) the root issue is that the the distributed processing code relies on the uniqueKey field to rectify & merge information from each shard across the various distributed process from shards, and the NPE comes from trying to do this lookup on the uniqueKey field name.
        Hide
        aarmcm Aaron McMillin added a comment -

        I have this problem in 4.9.0

        Show
        aarmcm Aaron McMillin added a comment - I have this problem in 4.9.0
        Hide
        laigood12345 laigood added a comment -

        I have this problem in 5.2.1

        Show
        laigood12345 laigood added a comment - I have this problem in 5.2.1
        Hide
        akuchling Andrew Kuchling added a comment -

        The bug is still present in Solr 6.1.

        Show
        akuchling Andrew Kuchling added a comment - The bug is still present in Solr 6.1.
        Hide
        mdrob Mike Drob added a comment -

        Attaching a patch for master that addresses this + includes a test.

        Hoss Man - can you take a look?

        Show
        mdrob Mike Drob added a comment - Attaching a patch for master that addresses this + includes a test. Hoss Man - can you take a look?
        Hide
        tomasflobbe Tomás Fernández Löbbe added a comment -

        I think the fix looks good. Maybe it would be better to check for the rename of the id field outside of the loop instead?
        Also, it would be nice if the test would also validate that the "id" field is not included in the returned fields.

        Show
        tomasflobbe Tomás Fernández Löbbe added a comment - I think the fix looks good. Maybe it would be better to check for the rename of the id field outside of the loop instead? Also, it would be nice if the test would also validate that the "id" field is not included in the returned fields.
        Hide
        mdrob Mike Drob added a comment -

        Tomás Fernández Löbbe - I'll update the test to check for no id field returned.

        Moving the check outside of the loop helped me to catch another bug where we could rename id to something else, and then another field to id and Solr would still be looking in the wrong place.

        There are some comments in SolrReturnFields that indicate multiple renames might be possible, like id->foo->bar. I wasn't able to figure out the syntax for making that happen but that might be something worth investigating in a follow-on issue. Seems esoteric enough to safely skip for now.

        Show
        mdrob Mike Drob added a comment - Tomás Fernández Löbbe - I'll update the test to check for no id field returned. Moving the check outside of the loop helped me to catch another bug where we could rename id to something else, and then another field to id and Solr would still be looking in the wrong place. There are some comments in SolrReturnFields that indicate multiple renames might be possible, like id->foo->bar. I wasn't able to figure out the syntax for making that happen but that might be something worth investigating in a follow-on issue. Seems esoteric enough to safely skip for now.
        Hide
        tomasflobbe Tomás Fernández Löbbe added a comment -

        Many tests fail with the latest patch with NPE. Would you mind taking a look at that?

        Seems esoteric enough to safely skip for now.

        Yes, I agree

        Show
        tomasflobbe Tomás Fernández Löbbe added a comment - Many tests fail with the latest patch with NPE. Would you mind taking a look at that? Seems esoteric enough to safely skip for now. Yes, I agree
        Hide
        mdrob Mike Drob added a comment -

        Had time to get back to this, the null pointer was caused by a missing initialization on the rename map. This should be good now.

        Show
        mdrob Mike Drob added a comment - Had time to get back to this, the null pointer was caused by a missing initialization on the rename map. This should be good now.
        Hide
        jira-bot ASF subversion and git services added a comment -

        Commit b3d12d265bb389f1ec239e8a96f044f7b89c01b1 in lucene-solr's branch refs/heads/master from Tomas Fernandez Lobbe
        [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=b3d12d2 ]

        SOLR-6744: Consider uniqueKey rename when handling shard responses in distributed search

        Show
        jira-bot ASF subversion and git services added a comment - Commit b3d12d265bb389f1ec239e8a96f044f7b89c01b1 in lucene-solr's branch refs/heads/master from Tomas Fernandez Lobbe [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=b3d12d2 ] SOLR-6744 : Consider uniqueKey rename when handling shard responses in distributed search
        Hide
        jira-bot ASF subversion and git services added a comment -

        Commit 43d03430b9d63d77168098a72ff219b746aa3939 in lucene-solr's branch refs/heads/branch_6x from Tomas Fernandez Lobbe
        [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=43d0343 ]

        SOLR-6744: Consider uniqueKey rename when handling shard responses in distributed search

        Show
        jira-bot ASF subversion and git services added a comment - Commit 43d03430b9d63d77168098a72ff219b746aa3939 in lucene-solr's branch refs/heads/branch_6x from Tomas Fernandez Lobbe [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=43d0343 ] SOLR-6744 : Consider uniqueKey rename when handling shard responses in distributed search
        Hide
        tomasflobbe Tomás Fernández Löbbe added a comment -

        Thanks Mike! I did a trivial change (only assign the "renameFields" if there are any renames) and committed.

        Show
        tomasflobbe Tomás Fernández Löbbe added a comment - Thanks Mike! I did a trivial change (only assign the "renameFields" if there are any renames) and committed.
        Hide
        shalinmangar Shalin Shekhar Mangar added a comment -

        Re-opening to backport to 6.2.1

        Show
        shalinmangar Shalin Shekhar Mangar added a comment - Re-opening to backport to 6.2.1
        Hide
        shalinmangar Shalin Shekhar Mangar added a comment -

        Closing after 6.2.1 release

        Show
        shalinmangar Shalin Shekhar Mangar added a comment - Closing after 6.2.1 release
        Hide
        gesias Klas Gesias added a comment -

        I found something that may be related to this. Tested this standalone for 4.10.4 and 4.8.1. Solrcloud 4.10.4.

        I made some slight simplifications on the queries to make it easier to read.

        In single core mode I can do this, have an alias with id and pick the name aswell
        http://localhost:8983/solr/select/?q=*.*&fl=id:django_id&fl=name

        Gives:
        {"responseHeader":{"params":{"q":":","fl":"id:django_id"} ,"response":{"numFound":8101, "docs":[{"id":"2875206", "name": "some_name"},{"id":"2875205", "name": "other_name"} ]}

        When in SolrCloud mode however I get some behavior I do not really grasp, if I specify only id:django_id I get the fields
        Request:
        http://solrcloud:8988/solr/select/?q=*.*&fl=id:django_id
        gives

        Response:
        {"responseHeader":{"params":{"q":"*:*","fl":"id:django_id"}},"response":{"numFound":284,"start":0,"maxScore":1.0,"docs":[{"id":"product.66"},{"id":"product.69"}...]}}
        Where "product.69" is really the "id" not "django_id" which I tried to alias into the "id" field. So it totally ignores my alias request. I could type anything after ":", I could write "id:spamalot" and it would still return the "id" field and id-value.

        But if I add one or more fl fields when trying to alias the id:django_id, I get no documents at all! I do get hits but I am pretty certain they are not the correct amount and no documents are retrieved.
        Request:
        http://solrcloud:8988/solr/select/?q=*.*&fl=id:django_id&fl=name

        Response:
        {"responseHeader":{"params":{"q":"*:*","fl":["id:django_id","name"]}},"response":{"numFound":274,"maxScore":1.0,"docs":[]}}

        Is this a known and expected behaviour? If I do not use aliases this does not happen at all.

        Show
        gesias Klas Gesias added a comment - I found something that may be related to this. Tested this standalone for 4.10.4 and 4.8.1. Solrcloud 4.10.4. I made some slight simplifications on the queries to make it easier to read. In single core mode I can do this, have an alias with id and pick the name aswell http://localhost:8983/solr/select/?q=*.*&fl=id:django_id&fl=name Gives: {"responseHeader":{"params":{"q":" : ","fl":"id:django_id"} ,"response":{"numFound":8101, "docs":[{"id":"2875206", "name": "some_name"},{"id":"2875205", "name": "other_name"} ]} When in SolrCloud mode however I get some behavior I do not really grasp, if I specify only id:django_id I get the fields Request: http://solrcloud:8988/solr/select/?q=*.*&fl=id:django_id gives Response: {"responseHeader":{"params":{"q":"*:*","fl":"id:django_id"}},"response":{"numFound":284,"start":0,"maxScore":1.0,"docs":[{"id":"product.66"},{"id":"product.69"}...]}} Where "product.69" is really the "id" not "django_id" which I tried to alias into the "id" field. So it totally ignores my alias request. I could type anything after ":", I could write "id:spamalot" and it would still return the "id" field and id-value. But if I add one or more fl fields when trying to alias the id:django_id , I get no documents at all! I do get hits but I am pretty certain they are not the correct amount and no documents are retrieved. Request: http://solrcloud:8988/solr/select/?q=*.*&fl=id:django_id&fl=name Response: {"responseHeader":{"params":{"q":"*:*","fl":["id:django_id","name"]}},"response":{"numFound":274,"maxScore":1.0,"docs":[]}} Is this a known and expected behaviour? If I do not use aliases this does not happen at all.
        Hide
        mdrob Mike Drob added a comment -

        Klas Gesias - Yes, I think it's the same issue. There were two parts to this JIRA: field renaming id to something else causes an error, and renaming something else to id returns the wrong results.

        At this point it is a known bug in older versions, and fixed to behave properly in 6.2.1+

        Show
        mdrob Mike Drob added a comment - Klas Gesias - Yes, I think it's the same issue. There were two parts to this JIRA: field renaming id to something else causes an error, and renaming something else to id returns the wrong results. At this point it is a known bug in older versions, and fixed to behave properly in 6.2.1+
        Hide
        gesias Klas Gesias added a comment -

        Mike Drob I understand, thanks for making the time to come back here and verify!

        Show
        gesias Klas Gesias added a comment - Mike Drob I understand, thanks for making the time to come back here and verify!

          People

          • Assignee:
            tomasflobbe Tomás Fernández Löbbe
            Reporter:
            gdgrimm Garth Grimm
          • Votes:
            1 Vote for this issue
            Watchers:
            12 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development