Uploaded image for project: 'Solr'
  1. Solr
  2. SOLR-10524

Better ZkStateWriter batching

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 6.6, 7.0
    • None
    • None

    Description

      There are several JIRAs (I'll link in a second) about trying to be more efficient about processing overseer messages as the overseer can become a bottleneck, especially with very large numbers of replicas in a cluster. One of the approaches mentioned near the end of SOLR-5872 (15-Mar) was to "read large no:of items say 10000. put them into in memory buckets and feed them into overseer....".

      This JIRA is to break out that part of the discussion as it might be an easy win whereas "eliminating the Overseer queue" would be quite an undertaking.

      Attachments

        1. SOLR-10524.patch
          12 kB
          Cao Manh Dat
        2. SOLR-10524.patch
          13 kB
          Noble Paul
        3. SOLR-10524.patch
          16 kB
          Cao Manh Dat
        4. SOLR-10524.patch
          2 kB
          Cao Manh Dat
        5. SOLR-10524-NPE-fix.patch
          0.6 kB
          Christine Poerschke
        6. SOLR-10524-dragonsinth.patch
          4 kB
          Scott Blum

        Issue Links

          is a child of

          Improvement - An improvement or enhancement to an existing feature or task. SOLR-10265 Overseer can become the bottleneck in very large clusters

          • Major - Major loss of function.
          • Open
          is related to

          Bug - A problem which impairs or prevents the functions of the product. SOLR-10277 On 'downnode', lots of wasteful mutations are done to ZK

          • Major - Major loss of function.
          • Closed

          Improvement - An improvement or enhancement to an existing feature or task. SOLR-5872 Eliminate overseer queue

          • Major - Major loss of function.
          • Open

          Improvement - An improvement or enhancement to an existing feature or task. SOLR-10285 Skip LEADER messages when there are leader only shards

          • Major - Major loss of function.
          • Closed
          relates to

          Bug - A problem which impairs or prevents the functions of the product. SOLR-10641 Explore asychronous read/write for state queue/work queue read/delete

          • Major - Major loss of function.
          • Open
          links to

          Web Link GitHub Pull Request #294

          Activity

            People

              caomanhdat Cao Manh Dat
              erickerickson Erick Erickson
              Votes:
              0 Vote for this issue
              Watchers:
              13 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: