[HDFS-8824] Do not use small blocks for balancing the cluster

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.8.0, 2.7.4, 3.0.0-alpha1
• Component/s: balancer & mover
• Hadoop Flags: Reviewed
• Assignee: Tsz-wo Sze
• Reporter: Tsz-wo Sze
• Created: 27/Jul/15 23:03
• Updated: 11/Apr/18 22:48
• Resolved: 14/Aug/15 21:50

Description

Balancer gets datanode block lists from NN and then move the blocks in order to balance the cluster. It should not use the blocks with small size since moving the small blocks generates a lot of overhead and the small blocks do not help balancing the cluster much.

Issue Links

• duplicates: HDFS-9303 Balancer slowly with too many small file blocks
• relates to: HDFS-9412 getBlocks occupies FSLock and takes too long to complete