[KAFKA-5319] Add a tool to make cluster replica and leader balance

• Type: Improvement
• Status: Patch Available
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 0.10.2.1
• Component/s: admin
• Labels: balance tool
• Flags: Patch, Important
• Assignee: Ma Tianchi
• Reporter: Ma Tianchi
• Created: 24/May/17 10:37
• Updated: 12/Jul/18 06:10

Description

When a new broker is added to cluster,there is not any topics in the new broker.When we use console command to create a topic without 'replicaAssignmentOpt',Kafka use AdminUtils.assignReplicasToBrokers to get replicaAssignment.Even though it is balance at the creating time if the cluster never change,with more and more brokers added to cluster the replica balanced will become not well. We also can use 'kafka-reassign-partitions.sh' to balance ,but the big amount of topics make it not easy.And at the topic created time , Kafka choose a PreferredReplicaLeader which be put at the first position of the AR to make leader balance.But the balance will be destroyed when cluster changed.Using 'kafka-reassign-partitions.sh' to make partition reassignment may be also destroy the leader balance ,because user can change the AR of the partition . It may be not balanced , but Kafka believe cluster leader balance is well with every leaders is the first on at AR.
So we create a tool to make the number of replicas and number of leaders on every brokers is balanced.It uses a algorithm to get a balanced replica and leader reassignment,then uses ReassignPartitionsCommand to real balance the cluster.
It can be used to make balance when cluster added brokers or cluster is not balanced .It does not deal with moving replicas of a dead broker to an alive broker,it only makes replicas and leaders on alive brokers is balanced.

Issue Links

• links to: GitHub Pull Request #3393
• mentioned in: Page Loading...