Solr
  1. Solr
  2. SOLR-3134

Include shard Information in response

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 4.0-ALPHA
    • Labels:
      None

      Description

      For distributed search where each shard represents a logically different index (or physical location), it would be great to know the hit count for each shard.

      In addition, it would be nice to get error info for each shard rather then aborting the whole request when something fails.

      1. SOLR-3134-shard-info-fix.patch
        15 kB
        Russell Black
      2. SOLR-3134-shard-info-3_x-backport.patch
        9 kB
        Russell Black
      3. SOLR-3134-shard-info-3_6-backport.patch
        4 kB
        Igor Galić
      4. SOLR-3134-shard-info-3_6.patch
        9 kB
        Russell Black
      5. SOLR-3134-shard-info-3_5-backport.patch
        9 kB
        Russell Black
      6. SOLR-3134-shard-info.patch
        10 kB
        Ryan McKinley

        Issue Links

          Activity

          Hide
          Ryan McKinley added a comment -

          This patch addes two parameters:

            /** Request detailed match info for each shard (true/false) */
            public static final String SHARDS_INFO = "shards.info";
          
            /** Should things fail if there is an error? (true/false) */
            public static final String SHARDS_TOLERANT = "shards.tolerant";
          

          When 'shards.info=true', it adds adds a node to the response:

          <lst name="shards.info">
            <lst name="localhost:7777/solr">
              <long name="numFound">1333</long>
              <float name="maxScore">1.0</float>
              <long name="time">686</long>
            </lst>
            <lst name="localhost:8888/solr">
              <long name="numFound">342</long>
              <float name="maxScore">1.0</float>
              <long name="time">602</long>
            </lst>
          </lst>
          

          When 'shards.tolerant=true' it does not abort when a request has an error. If 'shards.info=true', it will add error info to the response for that shard, like:

          <lst name="shards.info">
            <lst name="badurl:3456/solr">
              <str name="error">java.net.UnknownHostException: badurl</str>
              <str name="trace">java.net.UnknownHostException: badurl
          	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:176)
          	at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:157)
          	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:391)
          	at java.net.Socket.connect(Socket.java:579)
          	at java.net.Socket.connect(Socket.java:528)
          	at java.net.Socket.<init>(Socket.java:425)
          	at java.net.Socket.<init>(Socket.java:280)
          	at org.apache.commons.httpclient.protocol.DefaultProtocolSocketFactory.createSocket(DefaultProtocolSocketFactory.java:80)
          	at org.apache.commons.httpclient.protocol.DefaultProtocolSocketFactory.createSocket(DefaultProtocolSocketFactory.java:122)
          	at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707)
          	at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361)
          	at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387)
          	at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171)
          	at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397)
          	at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323)
          	at org.apache.solr.client.solrj.impl.CommonsHttpSolrServer.request(CommonsHttpSolrServer.java:425)
          	at org.apache.solr.client.solrj.impl.CommonsHttpSolrServer.request(CommonsHttpSolrServer.java:251)
          	at org.apache.solr.handler.component.HttpShardHandler$1.call(HttpShardHandler.java:153)
          	at org.apache.solr.handler.component.HttpShardHandler$1.call(HttpShardHandler.java:1)
          	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
          	at java.util.concurrent.FutureTask.run(FutureTask.java:166)
          	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
          	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
          	at java.util.concurrent.FutureTask.run(FutureTask.java:166)
          	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
          	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
          	at java.lang.Thread.run(Thread.java:722)</str>
              <long name="time">2622</long>
            </lst>
            <lst name="localhost:8888/solr">
              <long name="numFound">342</long>
              <float name="maxScore">1.0</float>
              <long name="time">602</long>
            </lst>
          </lst>
          
          Show
          Ryan McKinley added a comment - This patch addes two parameters: /** Request detailed match info for each shard ( true / false ) */ public static final String SHARDS_INFO = "shards.info" ; /** Should things fail if there is an error? ( true / false ) */ public static final String SHARDS_TOLERANT = "shards.tolerant" ; When 'shards.info=true', it adds adds a node to the response: <lst name= "shards.info" > <lst name= "localhost:7777/solr" > <long name= "numFound" > 1333 </long> <float name= "maxScore" > 1.0 </float> <long name= "time" > 686 </long> </lst> <lst name= "localhost:8888/solr" > <long name= "numFound" > 342 </long> <float name= "maxScore" > 1.0 </float> <long name= "time" > 602 </long> </lst> </lst> When 'shards.tolerant=true' it does not abort when a request has an error. If 'shards.info=true', it will add error info to the response for that shard, like: <lst name= "shards.info" > <lst name= "badurl:3456/solr" > <str name= "error" > java.net.UnknownHostException: badurl </str> <str name= "trace" > java.net.UnknownHostException: badurl at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:176) at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:157) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:391) at java.net.Socket.connect(Socket.java:579) at java.net.Socket.connect(Socket.java:528) at java.net.Socket. <init> (Socket.java:425) at java.net.Socket. <init> (Socket.java:280) at org.apache.commons.httpclient.protocol.DefaultProtocolSocketFactory.createSocket(DefaultProtocolSocketFactory.java:80) at org.apache.commons.httpclient.protocol.DefaultProtocolSocketFactory.createSocket(DefaultProtocolSocketFactory.java:122) at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) at org.apache.solr.client.solrj.impl.CommonsHttpSolrServer.request(CommonsHttpSolrServer.java:425) at org.apache.solr.client.solrj.impl.CommonsHttpSolrServer.request(CommonsHttpSolrServer.java:251) at org.apache.solr.handler.component.HttpShardHandler$1.call(HttpShardHandler.java:153) at org.apache.solr.handler.component.HttpShardHandler$1.call(HttpShardHandler.java:1) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:722) </str> <long name= "time" > 2622 </long> </lst> <lst name= "localhost:8888/solr" > <long name= "numFound" > 342 </long> <float name= "maxScore" > 1.0 </float> <long name= "time" > 602 </long> </lst> </lst>
          Hide
          Ryan McKinley added a comment -

          Any thoughts on this?

          I'd like to commit this soon, but I assume there would be more opinions on this

          Show
          Ryan McKinley added a comment - Any thoughts on this? I'd like to commit this soon, but I assume there would be more opinions on this
          Hide
          Ryan McKinley added a comment -

          added in revision: 1292717

          Show
          Ryan McKinley added a comment - added in revision: 1292717
          Hide
          Russell Black added a comment -

          Nice work! I may try to backport this to 3.5.

          Show
          Russell Black added a comment - Nice work! I may try to backport this to 3.5.
          Hide
          Russell Black added a comment -

          I backported to 3_5 but I'm not posting the patch because it doesn't operate correctly on about half the requests. I'm testing it with two shards, one up and one down. Half of the time it works as expected: I get back shard info for both shards plus results from the "up" shard. The other half of the time it returns shard info for only the "down" shard and zero results. I'm guessing there is a race condition based on which which server responds (or doesn't) first. I wonder if this bug exists on the 4.0 branch.

          Show
          Russell Black added a comment - I backported to 3_5 but I'm not posting the patch because it doesn't operate correctly on about half the requests. I'm testing it with two shards, one up and one down. Half of the time it works as expected: I get back shard info for both shards plus results from the "up" shard. The other half of the time it returns shard info for only the "down" shard and zero results. I'm guessing there is a race condition based on which which server responds (or doesn't) first. I wonder if this bug exists on the 4.0 branch.
          Hide
          Russell Black added a comment - - edited

          Yup, bug is on trunk as well. Looking into it.

          Show
          Russell Black added a comment - - edited Yup, bug is on trunk as well. Looking into it.
          Hide
          Russell Black added a comment -

          Ryan,

          I have fixed the bug and submitted a patch. This is a patch against the current 4.0 trunk and is intended to be applied on top of the changes represented by your first patch. I've enriched the test case as well. Please take a look and tell me what you think.

          Show
          Russell Black added a comment - Ryan, I have fixed the bug and submitted a patch. This is a patch against the current 4.0 trunk and is intended to be applied on top of the changes represented by your first patch. I've enriched the test case as well. Please take a look and tell me what you think.
          Hide
          Russell Black added a comment -

          Hi Ryan,

          I backported this patch to 3.5 and 3.x. I'd like to get this feature in the next 3.x release. Would it be possible to get this patch committed to the 3x branch?

          Show
          Russell Black added a comment - Hi Ryan, I backported this patch to 3.5 and 3.x. I'd like to get this feature in the next 3.x release. Would it be possible to get this patch committed to the 3x branch?
          Hide
          Ryan McKinley added a comment -

          Thanks Russell – that test is very helpful!

          I added to trunk in revision: 1294995

          I'll take a stab at merging with 3.x

          Show
          Ryan McKinley added a comment - Thanks Russell – that test is very helpful! I added to trunk in revision: 1294995 I'll take a stab at merging with 3.x
          Hide
          Russell Black added a comment -

          Ryan,

          Backporting to 3x should be straightforward, I've attached patch targeted at 3x which you can use if you wish.

          Show
          Russell Black added a comment - Ryan, Backporting to 3x should be straightforward, I've attached patch targeted at 3x which you can use if you wish.
          Hide
          Russell Black added a comment -

          Ryan,
          Are you still planning to commit the 3x patch? We'd love to have this feature in 3.6.

          Show
          Russell Black added a comment - Ryan, Are you still planning to commit the 3x patch? We'd love to have this feature in 3.6.
          Hide
          Russell Black added a comment -

          Hi Ryan,

          Sorry to keep bothering you about this, but can you comment on your intentions with respect to committing the 3x patch?

          Show
          Russell Black added a comment - Hi Ryan, Sorry to keep bothering you about this, but can you comment on your intentions with respect to committing the 3x patch?
          Hide
          Igor Galić added a comment - - edited

          Fix 3_x-backport to work with 3_6

          oh, well.. Doesn't actually compile :-/

          Show
          Igor Galić added a comment - - edited Fix 3_x-backport to work with 3_6 oh, well.. Doesn't actually compile :-/
          Hide
          Russell Black added a comment -

          Igor,

          Your patch was close it was just missing a few things. Here is the 3.6 patch that we have been using in our system for some time. It compiles and test cases pass. I named it SOLR-3134-shard-info-3_6.patch

          Show
          Russell Black added a comment - Igor, Your patch was close it was just missing a few things. Here is the 3.6 patch that we have been using in our system for some time. It compiles and test cases pass. I named it SOLR-3134 -shard-info-3_6.patch

            People

            • Assignee:
              Ryan McKinley
              Reporter:
              Ryan McKinley
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development