Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-5319

Add a tool to make cluster replica and leader balance

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Patch Available
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 0.10.2.1
    • Fix Version/s: None
    • Component/s: admin
    • Labels:
    • Flags:
      Patch, Important

      Description

      When a new broker is added to cluster,there is not any topics in the new broker.When we use console command to create a topic without 'replicaAssignmentOpt',Kafka use AdminUtils.assignReplicasToBrokers to get replicaAssignment.Even though it is balance at the creating time if the cluster never change,with more and more brokers added to cluster the replica balanced will become not well. We also can use 'kafka-reassign-partitions.sh' to balance ,but the big amount of topics make it not easy.And at the topic created time , Kafka choose a PreferredReplicaLeader which be put at the first position of the AR to make leader balance.But the balance will be destroyed when cluster changed.Using 'kafka-reassign-partitions.sh' to make partition reassignment may be also destroy the leader balance ,because user can change the AR of the partition . It may be not balanced , but Kafka believe cluster leader balance is well with every leaders is the first on at AR.
      So we create a tool to make the number of replicas and number of leaders on every brokers is balanced.It uses a algorithm to get a balanced replica and leader reassignment,then uses ReassignPartitionsCommand to real balance the cluster.
      It can be used to make balance when cluster added brokers or cluster is not balanced .It does not deal with moving replicas of a dead broker to an alive broker,it only makes replicas and leaders on alive brokers is balanced.

        Attachments

        1. KAFKA-5319.patch
          15 kB
          Ma Tianchi

          Issue Links

            Activity

              People

              • Assignee:
                markTC Ma Tianchi
                Reporter:
                markTC Ma Tianchi
              • Votes:
                1 Vote for this issue
                Watchers:
                9 Start watching this issue

                Dates

                • Created:
                  Updated: