Uploaded image for project: 'Solr'
  1. Solr
  2. SOLR-6816 Review SolrCloud Indexing Performance.
  3. SOLR-7333

Make the poll queue time configurable and use knowledge that a batch is being processed to poll efficiently

    Details

    • Type: Sub-task
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 5.2, 6.0
    • Component/s: SolrCloud
    • Labels:
      None

      Description

      StreamingSolrClients uses ConcurrentUpdateSolrServer to stream documents from leader to replica, by default it sets the pollQueueTime for CUSS to 0 so that we don't impose an unnecessary wait when processing single document updates or the last doc in a batch. However, the downside is that replicas receive many more update requests than leaders; I've seen up to 40x number of update requests between replica and leader.

      If we're processing a batch of docs, then ideally the poll queue time should be greater than 0 up until the last doc is pulled off the queue. If we're processing a single doc, then the poll queue time should always be 0 as we don't want the thread to wait unnecessarily for another doc that won't come.

      Rather than force indexing applications to provide this optional parameter in an update request, it would be better for server-side code that can detect whether an update request is a single document or batch of documents to override this value internally, i.e. it'll be 0 by default, but since JavaBinUpdateRequestCodec can determine when it's seen the last doc in a batch, it can override the pollQueueTime to something greater than 0.

      This means that current indexing clients will see a boost when doing batch updates without making any changes on their side.

        Attachments

        1. SOLR-7333.patch
          14 kB
          Timothy Potter
        2. SOLR-7333.patch
          9 kB
          Timothy Potter

          Issue Links

            Activity

              People

              • Assignee:
                thelabdude Timothy Potter
                Reporter:
                thelabdude Timothy Potter
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: