Solr
  1. Solr
  2. SOLR-3134

Include shard Information in response

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 4.0-ALPHA
    • Labels:
      None

      Description

      For distributed search where each shard represents a logically different index (or physical location), it would be great to know the hit count for each shard.

      In addition, it would be nice to get error info for each shard rather then aborting the whole request when something fails.

      1. SOLR-3134-shard-info-fix.patch
        15 kB
        Russell Black
      2. SOLR-3134-shard-info-3_x-backport.patch
        9 kB
        Russell Black
      3. SOLR-3134-shard-info-3_6-backport.patch
        4 kB
        Igor Galić
      4. SOLR-3134-shard-info-3_6.patch
        9 kB
        Russell Black
      5. SOLR-3134-shard-info-3_5-backport.patch
        9 kB
        Russell Black
      6. SOLR-3134-shard-info.patch
        10 kB
        Ryan McKinley

        Issue Links

          Activity

          Ryan McKinley created issue -
          Hide
          Ryan McKinley added a comment -

          This patch addes two parameters:

            /** Request detailed match info for each shard (true/false) */
            public static final String SHARDS_INFO = "shards.info";
          
            /** Should things fail if there is an error? (true/false) */
            public static final String SHARDS_TOLERANT = "shards.tolerant";
          

          When 'shards.info=true', it adds adds a node to the response:

          <lst name="shards.info">
            <lst name="localhost:7777/solr">
              <long name="numFound">1333</long>
              <float name="maxScore">1.0</float>
              <long name="time">686</long>
            </lst>
            <lst name="localhost:8888/solr">
              <long name="numFound">342</long>
              <float name="maxScore">1.0</float>
              <long name="time">602</long>
            </lst>
          </lst>
          

          When 'shards.tolerant=true' it does not abort when a request has an error. If 'shards.info=true', it will add error info to the response for that shard, like:

          <lst name="shards.info">
            <lst name="badurl:3456/solr">
              <str name="error">java.net.UnknownHostException: badurl</str>
              <str name="trace">java.net.UnknownHostException: badurl
          	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:176)
          	at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:157)
          	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:391)
          	at java.net.Socket.connect(Socket.java:579)
          	at java.net.Socket.connect(Socket.java:528)
          	at java.net.Socket.<init>(Socket.java:425)
          	at java.net.Socket.<init>(Socket.java:280)
          	at org.apache.commons.httpclient.protocol.DefaultProtocolSocketFactory.createSocket(DefaultProtocolSocketFactory.java:80)
          	at org.apache.commons.httpclient.protocol.DefaultProtocolSocketFactory.createSocket(DefaultProtocolSocketFactory.java:122)
          	at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707)
          	at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361)
          	at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387)
          	at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171)
          	at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397)
          	at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323)
          	at org.apache.solr.client.solrj.impl.CommonsHttpSolrServer.request(CommonsHttpSolrServer.java:425)
          	at org.apache.solr.client.solrj.impl.CommonsHttpSolrServer.request(CommonsHttpSolrServer.java:251)
          	at org.apache.solr.handler.component.HttpShardHandler$1.call(HttpShardHandler.java:153)
          	at org.apache.solr.handler.component.HttpShardHandler$1.call(HttpShardHandler.java:1)
          	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
          	at java.util.concurrent.FutureTask.run(FutureTask.java:166)
          	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
          	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
          	at java.util.concurrent.FutureTask.run(FutureTask.java:166)
          	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
          	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
          	at java.lang.Thread.run(Thread.java:722)</str>
              <long name="time">2622</long>
            </lst>
            <lst name="localhost:8888/solr">
              <long name="numFound">342</long>
              <float name="maxScore">1.0</float>
              <long name="time">602</long>
            </lst>
          </lst>
          
          Show
          Ryan McKinley added a comment - This patch addes two parameters: /** Request detailed match info for each shard ( true / false ) */ public static final String SHARDS_INFO = "shards.info" ; /** Should things fail if there is an error? ( true / false ) */ public static final String SHARDS_TOLERANT = "shards.tolerant" ; When 'shards.info=true', it adds adds a node to the response: <lst name= "shards.info" > <lst name= "localhost:7777/solr" > <long name= "numFound" > 1333 </long> <float name= "maxScore" > 1.0 </float> <long name= "time" > 686 </long> </lst> <lst name= "localhost:8888/solr" > <long name= "numFound" > 342 </long> <float name= "maxScore" > 1.0 </float> <long name= "time" > 602 </long> </lst> </lst> When 'shards.tolerant=true' it does not abort when a request has an error. If 'shards.info=true', it will add error info to the response for that shard, like: <lst name= "shards.info" > <lst name= "badurl:3456/solr" > <str name= "error" > java.net.UnknownHostException: badurl </str> <str name= "trace" > java.net.UnknownHostException: badurl at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:176) at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:157) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:391) at java.net.Socket.connect(Socket.java:579) at java.net.Socket.connect(Socket.java:528) at java.net.Socket. <init> (Socket.java:425) at java.net.Socket. <init> (Socket.java:280) at org.apache.commons.httpclient.protocol.DefaultProtocolSocketFactory.createSocket(DefaultProtocolSocketFactory.java:80) at org.apache.commons.httpclient.protocol.DefaultProtocolSocketFactory.createSocket(DefaultProtocolSocketFactory.java:122) at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) at org.apache.solr.client.solrj.impl.CommonsHttpSolrServer.request(CommonsHttpSolrServer.java:425) at org.apache.solr.client.solrj.impl.CommonsHttpSolrServer.request(CommonsHttpSolrServer.java:251) at org.apache.solr.handler.component.HttpShardHandler$1.call(HttpShardHandler.java:153) at org.apache.solr.handler.component.HttpShardHandler$1.call(HttpShardHandler.java:1) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:722) </str> <long name= "time" > 2622 </long> </lst> <lst name= "localhost:8888/solr" > <long name= "numFound" > 342 </long> <float name= "maxScore" > 1.0 </float> <long name= "time" > 602 </long> </lst> </lst>
          Ryan McKinley made changes -
          Field Original Value New Value
          Attachment SOLR-3134-shard-info.patch [ 12514596 ]
          Hide
          Ryan McKinley added a comment -

          Any thoughts on this?

          I'd like to commit this soon, but I assume there would be more opinions on this

          Show
          Ryan McKinley added a comment - Any thoughts on this? I'd like to commit this soon, but I assume there would be more opinions on this
          Hide
          Ryan McKinley added a comment -

          added in revision: 1292717

          Show
          Ryan McKinley added a comment - added in revision: 1292717
          Ryan McKinley made changes -
          Status Open [ 1 ] Resolved [ 5 ]
          Resolution Fixed [ 1 ]
          Ryan McKinley made changes -
          Link This issue is related to SOLR-1143 [ SOLR-1143 ]
          Hide
          Russell Black added a comment -

          Nice work! I may try to backport this to 3.5.

          Show
          Russell Black added a comment - Nice work! I may try to backport this to 3.5.
          Hide
          Russell Black added a comment -

          I backported to 3_5 but I'm not posting the patch because it doesn't operate correctly on about half the requests. I'm testing it with two shards, one up and one down. Half of the time it works as expected: I get back shard info for both shards plus results from the "up" shard. The other half of the time it returns shard info for only the "down" shard and zero results. I'm guessing there is a race condition based on which which server responds (or doesn't) first. I wonder if this bug exists on the 4.0 branch.

          Show
          Russell Black added a comment - I backported to 3_5 but I'm not posting the patch because it doesn't operate correctly on about half the requests. I'm testing it with two shards, one up and one down. Half of the time it works as expected: I get back shard info for both shards plus results from the "up" shard. The other half of the time it returns shard info for only the "down" shard and zero results. I'm guessing there is a race condition based on which which server responds (or doesn't) first. I wonder if this bug exists on the 4.0 branch.
          Hide
          Russell Black added a comment - - edited

          Yup, bug is on trunk as well. Looking into it.

          Show
          Russell Black added a comment - - edited Yup, bug is on trunk as well. Looking into it.
          Hide
          Russell Black added a comment -

          Ryan,

          I have fixed the bug and submitted a patch. This is a patch against the current 4.0 trunk and is intended to be applied on top of the changes represented by your first patch. I've enriched the test case as well. Please take a look and tell me what you think.

          Show
          Russell Black added a comment - Ryan, I have fixed the bug and submitted a patch. This is a patch against the current 4.0 trunk and is intended to be applied on top of the changes represented by your first patch. I've enriched the test case as well. Please take a look and tell me what you think.
          Russell Black made changes -
          Attachment SOLR-3134-shard-info-fix.patch [ 12516488 ]
          Hide
          Russell Black added a comment -

          Hi Ryan,

          I backported this patch to 3.5 and 3.x. I'd like to get this feature in the next 3.x release. Would it be possible to get this patch committed to the 3x branch?

          Show
          Russell Black added a comment - Hi Ryan, I backported this patch to 3.5 and 3.x. I'd like to get this feature in the next 3.x release. Would it be possible to get this patch committed to the 3x branch?
          Russell Black made changes -
          Attachment SOLR-3134-shard-info-3_5-backport.patch [ 12516520 ]
          Attachment SOLR-3134-shard-info-3_x-backport.patch [ 12516521 ]
          Hide
          Ryan McKinley added a comment -

          Thanks Russell – that test is very helpful!

          I added to trunk in revision: 1294995

          I'll take a stab at merging with 3.x

          Show
          Ryan McKinley added a comment - Thanks Russell – that test is very helpful! I added to trunk in revision: 1294995 I'll take a stab at merging with 3.x
          Hide
          Russell Black added a comment -

          Ryan,

          Backporting to 3x should be straightforward, I've attached patch targeted at 3x which you can use if you wish.

          Show
          Russell Black added a comment - Ryan, Backporting to 3x should be straightforward, I've attached patch targeted at 3x which you can use if you wish.
          Russell Black made changes -
          Attachment SOLR-3134-shard-info-3_x-backport.patch [ 12516611 ]
          Russell Black made changes -
          Attachment SOLR-3134-shard-info-3_x-backport.patch [ 12516521 ]
          Hide
          Russell Black added a comment -

          Ryan,
          Are you still planning to commit the 3x patch? We'd love to have this feature in 3.6.

          Show
          Russell Black added a comment - Ryan, Are you still planning to commit the 3x patch? We'd love to have this feature in 3.6.
          Hide
          Russell Black added a comment -

          Hi Ryan,

          Sorry to keep bothering you about this, but can you comment on your intentions with respect to committing the 3x patch?

          Show
          Russell Black added a comment - Hi Ryan, Sorry to keep bothering you about this, but can you comment on your intentions with respect to committing the 3x patch?
          Hoss Man made changes -
          Link This issue incorporates SOLR-3335 [ SOLR-3335 ]
          Russell Black made changes -
          Link This issue relates to SOLR-3369 [ SOLR-3369 ]
          Hide
          Igor Galić added a comment - - edited

          Fix 3_x-backport to work with 3_6

          oh, well.. Doesn't actually compile :-/

          Show
          Igor Galić added a comment - - edited Fix 3_x-backport to work with 3_6 oh, well.. Doesn't actually compile :-/
          Igor Galić made changes -
          Attachment SOLR-3134-shard-info-3_6-backport.patch [ 12533314 ]
          Hide
          Russell Black added a comment -

          Igor,

          Your patch was close it was just missing a few things. Here is the 3.6 patch that we have been using in our system for some time. It compiles and test cases pass. I named it SOLR-3134-shard-info-3_6.patch

          Show
          Russell Black added a comment - Igor, Your patch was close it was just missing a few things. Here is the 3.6 patch that we have been using in our system for some time. It compiles and test cases pass. I named it SOLR-3134 -shard-info-3_6.patch
          Russell Black made changes -
          Attachment SOLR-3134-shard-info-3_6.patch [ 12533400 ]
          Uwe Schindler made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Shalin Shekhar Mangar made changes -
          Link This issue is duplicated by SOLR-2587 [ SOLR-2587 ]
          Transition Time In Source Status Execution Times Last Executer Last Execution Date
          Open Open Resolved Resolved
          8d 5h 33m 1 Ryan McKinley 23/Feb/12 09:07
          Resolved Resolved Closed Closed
          442d 1h 31m 1 Uwe Schindler 10/May/13 11:39

            People

            • Assignee:
              Ryan McKinley
              Reporter:
              Ryan McKinley
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development