Uploaded image for project: 'Solr'
  1. Solr
  2. SOLR-9448

[subquery] calls another collection fails with "undefined field" or NPE from mergeIds

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Workaround
    • Affects Version/s: 6.1, 6.2
    • Fix Version/s: 6.4, 7.0
    • Component/s: SolrCloud
    • Security Level: Public (Default Security Level. Issues are Public)
    • Labels:
      None

      Description

      If subquery collection has a different unique key field name (let's say foo_id is different to id in primary collection), you've got NullPoniterException from QueryComponent.mergeIds(). To accommodate the difference between uniqueKey field names between collections, add the following parameters foo.fl=id:foo_id&foo.distrib.singlePass=true. The former one renames uniqueKey field, the later switches to single pass search. There is no way to rename field in default processing.
      In a rare case when a collection under subquery has no unqueKey at all, it leads to something like "undefined field", but it's not going to work anyway.

      1. SOLR-9448.patch
        7 kB
        Mikhail Khludnev
      2. SOLR-9448.patch
        4 kB
        Mikhail Khludnev
      3. SOLR-9448.patch
        15 kB
        Mikhail Khludnev
      4. SOLR-9448.patch
        14 kB
        Mikhail Khludnev

        Issue Links

          Activity

          Hide
          mkhludnev Mikhail Khludnev added a comment -

          ok. reproduced it with attached test case. The synopsis is:

          if we request one collection from another /people/select?q=department_text:foo&collection=departments the query is unnecessary parsed in people collection where it can hit a wall if schemas are different.

            @Test
            public void testJustCrossCollectionRequest() throws SolrServerException, IOException { 
              cluster.getSolrClient().request(new QueryRequest(
                  params( "q","departments_text:foo",
                               "collection","departments")
              ), "people");
            }
          

          org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://127.0.0.1:52548/solr/people_shard1_replica1: undefined field departments_text
          at __randomizedtesting.SeedInfo.seed([1E3C35306C9B6800:E9590C1B5403419]:0)
          at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:608)
          at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:261)
          at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:250)
          at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:415)
          at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:367)
          at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1280)
          at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:1050)
          at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:992)
          at org.apache.solr.response.transform.TestSubQueryTransformerDistrib.testJustCrossCollectionRequest(TestSubQueryTransformerDistrib.java:200)

          So, suggested workaround is just to declare fields from the secondary collection or wildcard just to pass unnecessary query parsing.

          Question to developers

          Don't we need to bypass query parsing in QueryComponent.prepare() when it's followed by distributed processing? If we do, I can come up with the patch.

          Show
          mkhludnev Mikhail Khludnev added a comment - ok. reproduced it with attached test case. The synopsis is: if we request one collection from another /people/select?q=department_text:foo&collection=departments the query is unnecessary parsed in people collection where it can hit a wall if schemas are different. @Test public void testJustCrossCollectionRequest() throws SolrServerException, IOException { cluster.getSolrClient().request( new QueryRequest( params( "q" , "departments_text:foo" , "collection" , "departments" ) ), "people" ); } org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://127.0.0.1:52548/solr/people_shard1_replica1: undefined field departments_text at __randomizedtesting.SeedInfo.seed( [1E3C35306C9B6800:E9590C1B5403419] :0) at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:608) at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:261) at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:250) at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:415) at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:367) at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1280) at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:1050) at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:992) at org.apache.solr.response.transform.TestSubQueryTransformerDistrib.testJustCrossCollectionRequest(TestSubQueryTransformerDistrib.java:200) So, suggested workaround is just to declare fields from the secondary collection or wildcard just to pass unnecessary query parsing. Question to developers Don't we need to bypass query parsing in QueryComponent.prepare() when it's followed by distributed processing? If we do, I can come up with the patch.
          Hide
          mkhludnev Mikhail Khludnev added a comment -
            @Test
            public void testJustCrossCollectionRequest() throws SolrServerException, IOException { 
              cluster.getSolrClient().request(new QueryRequest(
                  params( "q","*:*",
                               "collection","departments")
              ), "people");
            }
          

          if we bypassing query parsing stage, we hit NPE in reduce phase because primary collection ID field name is used for reducing shards results.

          [departments_shard1_replica1] webapp=/solr path=/select params=

          Unknown macro: {distrib=false&_stateVer_=people}

          hits=8 status=0 QTime=45
          43234 ERROR (qtp2060379030-30) [n:127.0.0.1:53331_solr c:people s:shard1 r:core_node1 x:people_shard1_replica2] o.a.s.h.RequestHandlerBase java.lang.NullPointerException
          at org.apache.solr.handler.component.QueryComponent.mergeIds(QueryComponent.java:1113)
          at org.apache.solr.handler.component.QueryComponent.handleRegularResponses(QueryComponent.java:757)
          at org.apache.solr.handler.component.QueryComponent.handleResponses(QueryComponent.java:736)
          at org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:422)
          at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:154)
          at org.apache.solr.core.SolrCore.execute(SolrCore.java:2107)
          at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:653)
          at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:460)
          at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:257)

          Shouldn't it be fixed?

          Show
          mkhludnev Mikhail Khludnev added a comment - @Test public void testJustCrossCollectionRequest() throws SolrServerException, IOException { cluster.getSolrClient().request( new QueryRequest( params( "q" , "*:*" , "collection" , "departments" ) ), "people" ); } if we bypassing query parsing stage, we hit NPE in reduce phase because primary collection ID field name is used for reducing shards results. [departments_shard1_replica1] webapp=/solr path=/select params= Unknown macro: {distrib=false&_stateVer_=people} hits=8 status=0 QTime=45 43234 ERROR (qtp2060379030-30) [n:127.0.0.1:53331_solr c:people s:shard1 r:core_node1 x:people_shard1_replica2] o.a.s.h.RequestHandlerBase java.lang.NullPointerException at org.apache.solr.handler.component.QueryComponent.mergeIds(QueryComponent.java:1113) at org.apache.solr.handler.component.QueryComponent.handleRegularResponses(QueryComponent.java:757) at org.apache.solr.handler.component.QueryComponent.handleResponses(QueryComponent.java:736) at org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:422) at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:154) at org.apache.solr.core.SolrCore.execute(SolrCore.java:2107) at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:653) at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:460) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:257) Shouldn't it be fixed?
          Hide
          mkhludnev Mikhail Khludnev added a comment -

          There reason is that we set caller collection uniqKey even it's different in another (calee) collection.
          https://github.com/apache/lucene-solr/blob/master/solr/core/src/java/org/apache/solr/handler/component/QueryComponent.java#L914

          Show
          mkhludnev Mikhail Khludnev added a comment - There reason is that we set caller collection uniqKey even it's different in another (calee) collection. https://github.com/apache/lucene-solr/blob/master/solr/core/src/java/org/apache/solr/handler/component/QueryComponent.java#L914
          Hide
          mkhludnev Mikhail Khludnev added a comment - - edited

          attaching simpler approach to reproduce:

          15551 ERROR (qtp1315527986-39) [n:127.0.0.1:56344_solr c:people s:shard2 r:core_node1 x:people_shard2_replica1] o.a.s.h.RequestHandlerBase java.lang.NullPointerException
          	at org.apache.solr.handler.component.QueryComponent.mergeIds(QueryComponent.java:1118)
          	at org.apache.solr.handler.component.QueryComponent.handleRegularResponses(QueryComponent.java:763)
          	at org.apache.solr.handler.component.QueryComponent.handleResponses(QueryComponent.java:742)
          	at org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:422)
          
          Show
          mkhludnev Mikhail Khludnev added a comment - - edited attaching simpler approach to reproduce: 15551 ERROR (qtp1315527986-39) [n:127.0.0.1:56344_solr c:people s:shard2 r:core_node1 x:people_shard2_replica1] o.a.s.h.RequestHandlerBase java.lang.NullPointerException at org.apache.solr.handler.component.QueryComponent.mergeIds(QueryComponent.java:1118) at org.apache.solr.handler.component.QueryComponent.handleRegularResponses(QueryComponent.java:763) at org.apache.solr.handler.component.QueryComponent.handleResponses(QueryComponent.java:742) at org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:422)
          Hide
          mkhludnev Mikhail Khludnev added a comment -

          SOLR-9448.patch proposes a workaround (when a collection for subquery has a different uniqueKey):

          • add field alias for expected uniqueKey: subq.fl=id:subq_coll_id
          • add subq.distrib.singlePass=true
          Show
          mkhludnev Mikhail Khludnev added a comment - SOLR-9448.patch proposes a workaround (when a collection for subquery has a different uniqueKey): add field alias for expected uniqueKey: subq.fl=id:subq_coll_id add subq.distrib.singlePass=true
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit 54d8574f9662f89598a06fbb47de9a376ef5d2bc in lucene-solr's branch refs/heads/master from Mikhail Khludnev
          [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=54d8574 ]

          SOLR-9448: providing a test for workaround of a differently named uniqueKey field

          Show
          jira-bot ASF subversion and git services added a comment - Commit 54d8574f9662f89598a06fbb47de9a376ef5d2bc in lucene-solr's branch refs/heads/master from Mikhail Khludnev [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=54d8574 ] SOLR-9448 : providing a test for workaround of a differently named uniqueKey field
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit 6cf1f0f51f5f11750b4d9d9ebe157453550b76eb in lucene-solr's branch refs/heads/branch_6x from Mikhail Khludnev
          [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=6cf1f0f ]

          SOLR-9448: providing a test for workaround of a differently named uniqueKey field

          Show
          jira-bot ASF subversion and git services added a comment - Commit 6cf1f0f51f5f11750b4d9d9ebe157453550b76eb in lucene-solr's branch refs/heads/branch_6x from Mikhail Khludnev [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=6cf1f0f ] SOLR-9448 : providing a test for workaround of a differently named uniqueKey field

            People

            • Assignee:
              mkhludnev Mikhail Khludnev
              Reporter:
              mkhludnev Mikhail Khludnev
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development