Solr
  1. Solr
  2. SOLR-2184

Allow configuration of max connections for shards

    Details

    • Type: Improvement Improvement
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      Shard-ed requests are hard coded

      mgr.getParams().setDefaultMaxConnectionsPerHost(20);
      mgr.getParams().setMaxTotalConnections(10000);
      

      We should allow config parameters such as
      shard-max-conn-per-host, shard-max-total-conn

      To allow you to tune as needed.

      1. shard_request_hosts.patch
        1 kB
        Eric Pugh
      2. SOLR-2184.patch
        3 kB
        patrick o'leary

        Activity

        Hide
        Greg Bowyer added a comment -

        Did this get solved / changed with SOLR-3221 ?

        Show
        Greg Bowyer added a comment - Did this get solved / changed with SOLR-3221 ?
        Hide
        Eric Pugh added a comment -

        Example of the change, and I have a bit of debugging to highlight which shard didn't respond.

        Show
        Eric Pugh added a comment - Example of the change, and I have a bit of debugging to highlight which shard didn't respond.
        Hide
        Eric Pugh added a comment -

        I recently ran into this same problem when I moved from 10 to 60+ shards. With 60+ shards, I would get back frequent connection errors from various shards. Changing the default max connection per host to a larger number, like 2048 solved the problem. When debugging this, I saw that in Solr 4, the default value is much larger. Here is a link to the commit: http://svn.apache.org/viewvc/lucene/dev/trunk/solr/core/src/java/org/apache/solr/handler/component/HttpShardHandlerFactory.java?r1=1291530&r2=1292353&diff_format=h. I don't know if this needs to be configurable, or just bumped up to a larger number, like the 256 value from Solr 4.

        Show
        Eric Pugh added a comment - I recently ran into this same problem when I moved from 10 to 60+ shards. With 60+ shards, I would get back frequent connection errors from various shards. Changing the default max connection per host to a larger number, like 2048 solved the problem. When debugging this, I saw that in Solr 4, the default value is much larger. Here is a link to the commit: http://svn.apache.org/viewvc/lucene/dev/trunk/solr/core/src/java/org/apache/solr/handler/component/HttpShardHandlerFactory.java?r1=1291530&r2=1292353&diff_format=h . I don't know if this needs to be configurable, or just bumped up to a larger number, like the 256 value from Solr 4.
        Hide
        patrick o'leary added a comment -

        Provides the ability to configure the
        MultiThreadedHttpConnectionManager MaxConnectionsPerHost & MaxTotalConnections

        Which are currently hard coded to 20 and 10,000 respectively, these should be configurable to suit your hardware specifications.

        <requestHandler name="/search" class="org.apache.solr.handler.component.SearchHandler">
            <lst name="defaults">
              <str name="echoParams">explicit</str>
            </lst>
            <int name="shard-max-conn-per-host">100</int>
            <int name="shard-max-total-conn">10000</int>
           
          </requestHandler>
        
        Show
        patrick o'leary added a comment - Provides the ability to configure the MultiThreadedHttpConnectionManager MaxConnectionsPerHost & MaxTotalConnections Which are currently hard coded to 20 and 10,000 respectively, these should be configurable to suit your hardware specifications. <requestHandler name= "/search" class= "org.apache.solr.handler.component.SearchHandler" > <lst name= "defaults" > <str name= "echoParams" >explicit</str> </lst> < int name= "shard-max-conn-per-host" >100</ int > < int name= "shard-max-total-conn" >10000</ int > </requestHandler>

          People

          • Assignee:
            Unassigned
            Reporter:
            patrick o'leary
          • Votes:
            3 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:

              Development