[HDFS-8826] Balancer may not move blocks efficiently in some cases

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.8.0, 3.0.0-alpha1
• Component/s: balancer & mover
• Hadoop Flags: Reviewed
• Assignee: Tsz-wo Sze
• Reporter: Tsz-wo Sze
• Created: 28/Jul/15 00:31
• Updated: 14/Oct/16 00:54
• Resolved: 19/Aug/15 02:27

Description

Balancer is inefficient in the following case:

Datanode	Utilization	Rack
D1	95%	A
D2	30%	B
D3, D4, D5	0%	B

The average utilization is 25% so that D2 is within 10% threshold. However, Balancer currently will first move blocks from D2 to D3, D4 and D5 since they are under the same rack. Then, it will move blocks from D1.

Issue Links

• is depended upon by: HDFS-8923 Add -source flag to balancer usage message
• is duplicated by: HDFS-5958 One very large node in a cluster prevents balancer from balancing data
• is related to: HDFS-8921 Add an option to Balancer for specifying the k-most over-utilized DNs or all over-utilized DNs as sources.
• is related to: HDFS-11008 Change unit test for testing parsing "-source" parameter in Balancer CLI