Solr
  1. Solr
  2. SOLR-5768

Add a distrib.singlePass parameter to make EXECUTE_QUERY phase fetch all fields and skip GET_FIELDS

    Details

    • Type: Improvement Improvement
    • Status: Resolved
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 4.10, 6.0
    • Component/s: None
    • Labels:
      None

      Description

      Suggested by Yonik on solr-user:
      http://www.mail-archive.com/solr-user@lucene.apache.org/msg95045.html

      Although it seems like it should be relatively simple to make it work
      with other fields as well, by passing down the complete "fl" requested
      if some optional parameter is set (distrib.singlePass?)

      1. SOLR-5768.diff
        16 kB
        Shalin Shekhar Mangar
      2. SOLR-5768.diff
        14 kB
        Shalin Shekhar Mangar
      3. SOLR-5768.diff
        14 kB
        Shalin Shekhar Mangar
      4. SOLR-5768.diff
        14 kB
        Shalin Shekhar Mangar
      5. SOLR-5768.diff
        10 kB
        Gregg Donovan
      6. SOLR-5768.diff
        10 kB
        Gregg Donovan

        Issue Links

          Activity

          Hide
          Shikhar Bhushan added a comment -

          seems like the JIRA title has it the other way round

          Show
          Shikhar Bhushan added a comment - seems like the JIRA title has it the other way round
          Hide
          Shalin Shekhar Mangar added a comment -

          see it is already implemented

          thanks, I'll fix.

          Show
          Shalin Shekhar Mangar added a comment - see it is already implemented thanks, I'll fix.
          Hide
          Gregg Donovan added a comment -

          Is this all that's needed for a distrib.singlePass parameter? It seems like SOLR-1880 may have done most of the work.

          Show
          Gregg Donovan added a comment - Is this all that's needed for a distrib.singlePass parameter? It seems like SOLR-1880 may have done most of the work.
          Hide
          Shalin Shekhar Mangar added a comment -

          Thanks Gregg.

          1. We need tests for this.
          2. Any field other than id and score are not being passed to the shard queries so this patch is incomplete.
          Show
          Shalin Shekhar Mangar added a comment - Thanks Gregg. We need tests for this. Any field other than id and score are not being passed to the shard queries so this patch is incomplete.
          Hide
          Gregg Donovan added a comment -

          Thanks, Shalin! You're right – it's not quite so easy. Here's an updated patch with a test that returns multiple fields in a single pass. I'm not sure if it's better to work with the fields as represented by rb.rsp.getReturnFields() or by rb.req.getParams().get(CommonParams.FL).

          Show
          Gregg Donovan added a comment - Thanks, Shalin! You're right – it's not quite so easy. Here's an updated patch with a test that returns multiple fields in a single pass. I'm not sure if it's better to work with the fields as represented by rb.rsp.getReturnFields() or by rb.req.getParams().get(CommonParams.FL).
          Hide
          Gregg Donovan added a comment -

          Updated patch.

          Show
          Gregg Donovan added a comment - Updated patch.
          Hide
          Gregg Donovan added a comment -

          Updated patch with better matching of whether score was requested in request params.

          Show
          Gregg Donovan added a comment - Updated patch with better matching of whether score was requested in request params.
          Hide
          Uwe Schindler added a comment -

          Move issue to Solr 4.9.

          Show
          Uwe Schindler added a comment - Move issue to Solr 4.9.
          Hide
          Shalin Shekhar Mangar added a comment -

          Here's an updated patch which applies to trunk.

          The DistributedQueryComponentOptimizationTest doesn't pass with the patch. This is the error:

          org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: java.lang.NullPointerException
          	at org.apache.solr.handler.component.QueryComponent.regularFinishStage(QueryComponent.java:779)
          	at org.apache.solr.handler.component.QueryComponent.finishStage(QueryComponent.java:733)
          	at org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:333)
          	at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)
          	at org.apache.solr.core.SolrCore.execute(SolrCore.java:1980)
          	at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:780)
          	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:420)
          	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:208)
          	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1419)
          	at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:136)
          	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1419)
          	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:455)
          	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:229)
          	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
          	at org.eclipse.jetty.server.handler.GzipHandler.handle(GzipHandler.java:301)
          	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1077)
          	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:384)
          	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
          	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1009)
          	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
          	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
          	at org.eclipse.jetty.server.Server.handle(Server.java:368)
          	at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489)
          	at org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:942)
          	at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:1004)
          	at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:640)
          	at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)
          	at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)
          	at org.eclipse.jetty.io.nio.SslConnection.handle(SslConnection.java:196)
          	at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:628)
          	at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52)
          	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
          	at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
          	at java.lang.Thread.run(Thread.java:744)
          
          	at __randomizedtesting.SeedInfo.seed([5353CAD02801E6C4:D2B544C85F5E86F8]:0)
          	at org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:554)
          	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:210)
          	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:206)
          	at org.apache.solr.client.solrj.request.QueryRequest.process(QueryRequest.java:91)
          	at org.apache.solr.client.solrj.SolrServer.query(SolrServer.java:301)
          	at org.apache.solr.BaseDistributedSearchTestCase.queryServer(BaseDistributedSearchTestCase.java:508)
          	at org.apache.solr.BaseDistributedSearchTestCase.query(BaseDistributedSearchTestCase.java:556)
          	at org.apache.solr.BaseDistributedSearchTestCase.query(BaseDistributedSearchTestCase.java:538)
          	at org.apache.solr.BaseDistributedSearchTestCase.query(BaseDistributedSearchTestCase.java:517)
          	at org.apache.solr.handler.component.DistributedQueryComponentOptimizationTest.doTest(DistributedQueryComponentOptimizationTest.java:83)
          	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:863)
          	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
          	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
          	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
          

          I'm trying to track down why the handleResponses method was not called and the response not set.

          Show
          Shalin Shekhar Mangar added a comment - Here's an updated patch which applies to trunk. The DistributedQueryComponentOptimizationTest doesn't pass with the patch. This is the error: org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: java.lang.NullPointerException at org.apache.solr.handler.component.QueryComponent.regularFinishStage(QueryComponent.java:779) at org.apache.solr.handler.component.QueryComponent.finishStage(QueryComponent.java:733) at org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:333) at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135) at org.apache.solr.core.SolrCore.execute(SolrCore.java:1980) at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:780) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:420) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:208) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1419) at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:136) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1419) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:455) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:229) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137) at org.eclipse.jetty.server.handler.GzipHandler.handle(GzipHandler.java:301) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1077) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:384) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1009) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116) at org.eclipse.jetty.server.Server.handle(Server.java:368) at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489) at org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:942) at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:1004) at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:640) at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235) at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82) at org.eclipse.jetty.io.nio.SslConnection.handle(SslConnection.java:196) at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:628) at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543) at java.lang. Thread .run( Thread .java:744) at __randomizedtesting.SeedInfo.seed([5353CAD02801E6C4:D2B544C85F5E86F8]:0) at org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:554) at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:210) at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:206) at org.apache.solr.client.solrj.request.QueryRequest.process(QueryRequest.java:91) at org.apache.solr.client.solrj.SolrServer.query(SolrServer.java:301) at org.apache.solr.BaseDistributedSearchTestCase.queryServer(BaseDistributedSearchTestCase.java:508) at org.apache.solr.BaseDistributedSearchTestCase.query(BaseDistributedSearchTestCase.java:556) at org.apache.solr.BaseDistributedSearchTestCase.query(BaseDistributedSearchTestCase.java:538) at org.apache.solr.BaseDistributedSearchTestCase.query(BaseDistributedSearchTestCase.java:517) at org.apache.solr.handler.component.DistributedQueryComponentOptimizationTest.doTest(DistributedQueryComponentOptimizationTest.java:83) at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:863) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618) I'm trying to track down why the handleResponses method was not called and the response not set.
          Hide
          Shalin Shekhar Mangar added a comment -

          The last patch had the test failure because of an incorrect merge. This patch fixes that problem. I am gonna run all tests and do some manual testing before committing this.

          Show
          Shalin Shekhar Mangar added a comment - The last patch had the test failure because of an incorrect merge. This patch fixes that problem. I am gonna run all tests and do some manual testing before committing this.
          Hide
          Shalin Shekhar Mangar added a comment -

          It's probably better to use rb.rsp.getReturnFields() because that is set in the prepare method and it takes care of missing "fl" parameter as well. Otherwise we get a NPE in updateFl method. This patch fixes that problem.

          I also added a test to assert that response returned from distrib and distrib.singlePass method are the same. I think this is ready.

          Show
          Shalin Shekhar Mangar added a comment - It's probably better to use rb.rsp.getReturnFields() because that is set in the prepare method and it takes care of missing "fl" parameter as well. Otherwise we get a NPE in updateFl method. This patch fixes that problem. I also added a test to assert that response returned from distrib and distrib.singlePass method are the same. I think this is ready.
          Hide
          Shalin Shekhar Mangar added a comment - - edited

          The last patch had a bug where if uniqueKey was omitted from the "fl" param then you'd get a NPE in mergeIds.

          This patch adds the uniqueKey field if not requested to the individual shard requests so that we can always merge the shard responses.

          Edit - I have added tests using debug=track which assert that indeed this optimization works and that no shard requests are sent in the GET_FIELDS stage when using this param and the automatic optimization added in SOLR-1880

          Show
          Shalin Shekhar Mangar added a comment - - edited The last patch had a bug where if uniqueKey was omitted from the "fl" param then you'd get a NPE in mergeIds. This patch adds the uniqueKey field if not requested to the individual shard requests so that we can always merge the shard responses. Edit - I have added tests using debug=track which assert that indeed this optimization works and that no shard requests are sent in the GET_FIELDS stage when using this param and the automatic optimization added in SOLR-1880
          Hide
          ASF subversion and git services added a comment -

          Commit 1607360 from shalin@apache.org in branch 'dev/trunk'
          [ https://svn.apache.org/r1607360 ]

          SOLR-5768: Add a distrib.singlePass parameter to make EXECUTE_QUERY phase fetch all fields and skip GET_FIELDS

          Show
          ASF subversion and git services added a comment - Commit 1607360 from shalin@apache.org in branch 'dev/trunk' [ https://svn.apache.org/r1607360 ] SOLR-5768 : Add a distrib.singlePass parameter to make EXECUTE_QUERY phase fetch all fields and skip GET_FIELDS
          Hide
          ASF subversion and git services added a comment -

          Commit 1607361 from shalin@apache.org in branch 'dev/branches/branch_4x'
          [ https://svn.apache.org/r1607361 ]

          SOLR-5768: Add a distrib.singlePass parameter to make EXECUTE_QUERY phase fetch all fields and skip GET_FIELDS

          Show
          ASF subversion and git services added a comment - Commit 1607361 from shalin@apache.org in branch 'dev/branches/branch_4x' [ https://svn.apache.org/r1607361 ] SOLR-5768 : Add a distrib.singlePass parameter to make EXECUTE_QUERY phase fetch all fields and skip GET_FIELDS
          Hide
          Shalin Shekhar Mangar added a comment -

          Thanks Gregg!

          Show
          Shalin Shekhar Mangar added a comment - Thanks Gregg!
          Hide
          Gregg Donovan added a comment -

          Thank you, Shalin!

          Show
          Gregg Donovan added a comment - Thank you, Shalin!

            People

            • Assignee:
              Shalin Shekhar Mangar
              Reporter:
              Shalin Shekhar Mangar
            • Votes:
              3 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development