Uploaded image for project: 'Solr'
  1. Solr
  2. SOLR-4956

make maxBufferedAddsPerServer configurable



    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 4.3, 6.0
    • None
    • None
    • None


      Anecdotal user's list evidence indicates that in high-throughput situations, the default of 10 docs/batch for inter-shard batching can generate significant CPU load. See the thread titled "Sharding and Replication" on June 19th, but the gist is below.

      I haven't poked around, but it's a little surprising on the surface that Asif is seeing this kind of difference. So I'm wondering if this change indicates some other underlying issue. Regardless, this seems like it would be good to investigate.

      Here's the gist of Asif's experience from the thread:

      Its a completely practical problem - we are exploring Solr to build a real
      time analytics/data solution for a system handling about 1000 qps. We have
      various metrics that are stored as different collections on the cloud,
      which means very high amount of writes. The cloud also needs to support
      about 300-400 qps.

      We initially tested with a single Solr node on a 16 core / 24 GB box for a
      single metric. We saw that writes were not a issue at all - Solr was
      handling it extremely well. We were also able to achieve about 200 qps from
      a single node.

      When we set up the cloud ( a ensemble on 6 boxes), we saw very high CPU
      usage on the replicas. Up to 10 cores were getting used for writes on the
      replicas. Hence my concern with respect to batch updates for the replicas.

      BTW, I altered the maxBufferedAddsPerServer to 1000 - and now CPU usage is
      very similar to single node installation.


        Issue Links



              Unassigned Unassigned
              erickerickson Erick Erickson
              6 Vote for this issue
              11 Start watching this issue