Uploaded image for project: 'Solr'
  1. Solr
  2. SOLR-3221

Make Shard handler threadpool configurable

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 3.6, 4.0-ALPHA
    • 3.6, 4.0-ALPHA
    • None

    Description

      From profiling of monitor contention, as well as observations of the
      95th and 99th response times for nodes that perform distributed search
      (or ‟aggregator‟ nodes) it would appear that the HttpShardHandler code
      currently does a suboptimal job of managing outgoing shard level
      requests.

      Presently the code contained within lucene 3.5's SearchHandler and
      Lucene trunk / 3x's ShardHandlerFactory create arbitrary threads in
      order to service distributed search requests. This is done presently to
      limit the size of the threadpool such that it does not consume resources
      in deployment configurations that do not use distributed search.

      This unfortunately has two impacts on the response time if the node
      coordinating the distribution is under high load.

      The usage of the MaxConnectionsPerHost configuration option results in
      aggressive activity on semaphores within HttpCommons, it has been
      observed that the aggregator can have a response time far greater than
      that of the searchers. The above monitor contention would appear to
      suggest that in some cases its possible for liveness issues to occur and
      for simple queries to be starved of resources simply due to a lack of
      attention from the viewpoint of context switching.

      With, as mentioned above the http commons connection being hotly
      contended

      The fair, queue based configuration eliminates this, at the cost of
      throughput.

      This patch aims to make the threadpool largely configurable allowing for
      those using solr to choose the throughput vs latency balance they
      desire.

      Attachments

        1. SOLR-3221-3x_branch.patch
          6 kB
          Greg Bowyer
        2. SOLR-3221-3x_branch.patch
          6 kB
          Greg Bowyer
        3. SOLR-3221-3x_branch.patch
          6 kB
          Greg Bowyer
        4. SOLR-3221-3x_branch.patch
          8 kB
          Greg Bowyer
        5. SOLR-3221-3x_branch.patch
          6 kB
          Greg Bowyer
        6. SOLR-3221-trunk.patch
          7 kB
          Greg Bowyer
        7. SOLR-3221-trunk.patch
          7 kB
          Greg Bowyer
        8. SOLR-3221-trunk.patch
          7 kB
          Greg Bowyer
        9. SOLR-3221-trunk.patch
          9 kB
          Greg Bowyer
        10. SOLR-3221-trunk.patch
          7 kB
          Greg Bowyer

        Activity

          People

            erickerickson Erick Erickson
            gbowyer@fastmail.co.uk Greg Bowyer
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: