Uploaded image for project: 'Solr'
  1. Solr
  2. SOLR-10524

Better ZkStateWriter batching

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 6.6, 7.0
    • None
    • None

    Description

      There are several JIRAs (I'll link in a second) about trying to be more efficient about processing overseer messages as the overseer can become a bottleneck, especially with very large numbers of replicas in a cluster. One of the approaches mentioned near the end of SOLR-5872 (15-Mar) was to "read large no:of items say 10000. put them into in memory buckets and feed them into overseer....".

      This JIRA is to break out that part of the discussion as it might be an easy win whereas "eliminating the Overseer queue" would be quite an undertaking.

      Attachments

        1. SOLR-10524.patch
          12 kB
          Cao Manh Dat
        2. SOLR-10524.patch
          13 kB
          Noble Paul
        3. SOLR-10524.patch
          16 kB
          Cao Manh Dat
        4. SOLR-10524.patch
          2 kB
          Cao Manh Dat
        5. SOLR-10524-NPE-fix.patch
          0.6 kB
          Christine Poerschke
        6. SOLR-10524-dragonsinth.patch
          4 kB
          Scott Blum

        Issue Links

        is a child of

        Improvement - An improvement or enhancement to an existing feature or task. SOLR-10265 Overseer can become the bottleneck in very large clusters

        • Major - Major loss of function.
        • Open
        is related to

        Bug - A problem which impairs or prevents the functions of the product. SOLR-10277 On 'downnode', lots of wasteful mutations are done to ZK

        • Major - Major loss of function.
        • Closed

        Improvement - An improvement or enhancement to an existing feature or task. SOLR-5872 Eliminate overseer queue

        • Major - Major loss of function.
        • Open

        Improvement - An improvement or enhancement to an existing feature or task. SOLR-10285 Skip LEADER messages when there are leader only shards

        • Major - Major loss of function.
        • Closed
        relates to

        Bug - A problem which impairs or prevents the functions of the product. SOLR-10641 Explore asychronous read/write for state queue/work queue read/delete

        • Major - Major loss of function.
        • Open
        links to

        Web Link GitHub Pull Request #294

        Activity
          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            caomanhdat Cao Manh Dat
            erickerickson Erick Erickson
            Votes:
            0 Vote for this issue
            Watchers:
            13 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment