Solr
  1. Solr
  2. SOLR-659

Explicitly set start and rows per shard for more efficient bulk queries across distributed Solr

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: 1.3
    • Fix Version/s: 1.4
    • Component/s: search
    • Labels:
      None

      Description

      The default behavior of setting start and rows on distributed solr (SOLR-303) is to set start at 0 across all shards and set rows to start+rows across each shard. This ensures all results are returned for any arbitrary start and rows setting, but during "bulk queries" (where start is incrementally increased and rows is kept consistent) the client would need finer control of the per-shard start and rows parameter as retrieving many thousands of documents becomes intractable as start grows higher.

      Attaching a patch that creates a &shards.start and &shards.rows parameter. If used, the logic that sets rows to start+rows per shard is overridden and each shard gets the exact start and rows set in shards.start and shards.rows. The client will receive up to shards.rows * nShards results and should set rows accordingly. This makes bulk queries across distributed solr possible.

      1. shards.start_rows.patch
        3 kB
        Brian Whitman
      2. SOLR-659.patch
        3 kB
        Brian Whitman
      There are no Sub-Tasks for this issue.

        Activity

        Brian Whitman created issue -
        Hide
        Brian Whitman added a comment -

        Attaching patch.

        Show
        Brian Whitman added a comment - Attaching patch.
        Brian Whitman made changes -
        Field Original Value New Value
        Attachment shards.start_rows.patch [ 12386884 ]
        Hide
        Brian Whitman added a comment -

        An example of a bulk query using this patch. Without this patch such bulk queries will eventually time out or cause exceptions in the server as too much data is passed back and forth.

        public SolrDocumentList blockQuery(SolrQuery q, int blockSize, int maxResults) {
            SolrDocumentList allResults = new SolrDocumentList();
            if(blockSize > maxResults) { blockSize = maxResults;  }
            for(int i=0; i<maxResults; i=i+blockSize) {
              // Sets rows of this query to the most results that could ever come back - the blockSize * the number of shards
              q.setRows(blockSize * getNumberOfHosts());
              // Don't set a start on the main query
              q.setStart(0);
              // But do set start and rows on the individual shards. 
              q.set("shards.start", String.valueOf(i));
              q.set("shards.rows", String.valueOf(blockSize));
              // Perform the query.
              QueryResponse sub = query(q);
              // For each returned document (up to blockSize*numberOfHosts() of them), append them to the main result
              for(SolrDocument s : sub.getResults()) {
                allResults.add(s);
                // Break if we've reached our requested limit
                if(allResults.size() > maxResults) { break; }
              }
              if(allResults.size() > maxResults) { break; }
            }
            return allResults;
          }
        
        Show
        Brian Whitman added a comment - An example of a bulk query using this patch. Without this patch such bulk queries will eventually time out or cause exceptions in the server as too much data is passed back and forth. public SolrDocumentList blockQuery(SolrQuery q, int blockSize, int maxResults) { SolrDocumentList allResults = new SolrDocumentList(); if (blockSize > maxResults) { blockSize = maxResults; } for ( int i=0; i<maxResults; i=i+blockSize) { // Sets rows of this query to the most results that could ever come back - the blockSize * the number of shards q.setRows(blockSize * getNumberOfHosts()); // Don't set a start on the main query q.setStart(0); // But do set start and rows on the individual shards. q.set( "shards.start" , String .valueOf(i)); q.set( "shards.rows" , String .valueOf(blockSize)); // Perform the query. QueryResponse sub = query(q); // For each returned document (up to blockSize*numberOfHosts() of them), append them to the main result for (SolrDocument s : sub.getResults()) { allResults.add(s); // Break if we've reached our requested limit if (allResults.size() > maxResults) { break ; } } if (allResults.size() > maxResults) { break ; } } return allResults; }
        Hide
        Mike Klaas added a comment -

        IMO it is too late in the release process for new features.

        Show
        Mike Klaas added a comment - IMO it is too late in the release process for new features.
        Mike Klaas made changes -
        Fix Version/s 1.3 [ 12312486 ]
        Hide
        Otis Gospodnetic added a comment -

        This looks simple enough. I haven't tried it. Brian, do you have a unit test you could attach?

        Or would it make more sense to have a custom QueryComponent for something like this? (I don't know yet)

        Show
        Otis Gospodnetic added a comment - This looks simple enough. I haven't tried it. Brian, do you have a unit test you could attach? Or would it make more sense to have a custom QueryComponent for something like this? (I don't know yet)
        Otis Gospodnetic made changes -
        Fix Version/s 1.4 [ 12313351 ]
        Hide
        Brian Whitman added a comment -

        New patch syncs w/ trunk

        Show
        Brian Whitman added a comment - New patch syncs w/ trunk
        Brian Whitman made changes -
        Attachment SOLR-659.patch [ 12399776 ]
        Hide
        Shalin Shekhar Mangar added a comment -

        If I understand this correctly, it makes bulk queries cheaper at the expense of less precise scoring. But if I'm paging through some results and you modify the shard.start and shard.rows then I'll get inconsistent results. Is that correct?

        The client will receive up to shards.rows * nShards results and should set rows accordingly. This makes bulk queries across distributed solr possible.

        I do not understand that. Why will the client get more than rows? Or by client, did you mean the solr server to which the initial request is sent?

        Show
        Shalin Shekhar Mangar added a comment - If I understand this correctly, it makes bulk queries cheaper at the expense of less precise scoring. But if I'm paging through some results and you modify the shard.start and shard.rows then I'll get inconsistent results. Is that correct? The client will receive up to shards.rows * nShards results and should set rows accordingly. This makes bulk queries across distributed solr possible. I do not understand that. Why will the client get more than rows? Or by client, did you mean the solr server to which the initial request is sent?
        Hide
        Yonik Seeley added a comment -

        I agree this makes sense to enable efficient bulk operations, and also fits in with a past idea I had about mapping shards.param=foo to param=foo during a sub-request.

        I'll give it a couple of days and commit if there are no objections.

        Show
        Yonik Seeley added a comment - I agree this makes sense to enable efficient bulk operations, and also fits in with a past idea I had about mapping shards.param=foo to param=foo during a sub-request. I'll give it a couple of days and commit if there are no objections.
        Yonik Seeley made changes -
        Assignee Yonik Seeley [ yseeley@gmail.com ]
        Hide
        Yonik Seeley added a comment -

        Thanks Brian, I just committed this.

        Show
        Yonik Seeley added a comment - Thanks Brian, I just committed this.
        Yonik Seeley made changes -
        Status Open [ 1 ] Resolved [ 5 ]
        Resolution Fixed [ 1 ]
        Hide
        johnson.hong added a comment -

        This is really helpful to bulk queries ,but how to handle the pagination of query results.
        e.g.at the first query,I set shards.start to 0 and set shards.rows to 30,it may return 50 documents,and i get 30 documents to show ,the other 20 documents is discarded ;then how to get the next 30 documents ?

        Show
        johnson.hong added a comment - This is really helpful to bulk queries ,but how to handle the pagination of query results. e.g.at the first query,I set shards.start to 0 and set shards.rows to 30,it may return 50 documents,and i get 30 documents to show ,the other 20 documents is discarded ;then how to get the next 30 documents ?
        Hide
        Grant Ingersoll added a comment -

        Bulk close for Solr 1.4

        Show
        Grant Ingersoll added a comment - Bulk close for Solr 1.4
        Grant Ingersoll made changes -
        Status Resolved [ 5 ] Closed [ 6 ]
        Transition Time In Source Status Execution Times Last Executer Last Execution Date
        Open Open Resolved Resolved
        409d 4h 23m 1 Yonik Seeley 07/Sep/09 19:30
        Resolved Resolved Closed Closed
        63d 21h 21m 1 Grant Ingersoll 10/Nov/09 15:51

          People

          • Assignee:
            Yonik Seeley
            Reporter:
            Brian Whitman
          • Votes:
            1 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development