Uploaded image for project: 'Cassandra'
  1. Cassandra
  2. CASSANDRA-5272

Hinted Handoff Throttle based on cluster size

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Fix Version/s: 1.2.6
    • Component/s: None
    • Labels:

      Description

      For a 12-node EC2 m1.xlarge cluster, restarting a node causes it to get completely overloaded with the default 2-thread, 1024KB setting in 1.2.x. This seemed to be a smaller problem when it was 6-nodes, but still required us to abort handoffs. The old defaults in 1.1.x were WAY more conservative. I've dropped this way down to 128KB on our production cluster which is really conservative, but appears to have solved it. The default seems way too high on any cluster that is non-trivial in size.

      After putting some thought to this, it seems that this should really be based on cluster size, making the throttle a "target" for how much write load a single node can swallow. As the cluster grows, the amount of hints that can be delivered by each other node in the cluster goes down, so the throttle should self-adjust to take that into account.

        Attachments

        1. 5272.txt
          2 kB
          Jonathan Ellis

          Activity

            People

            • Assignee:
              jbellis Jonathan Ellis
              Reporter:
              rbranson Rick Branson
              Reviewer:
              Rick Branson
              Tester:
              Daniel Meyer
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: