[HDFS-14904] Add Option to let Balancer prefer highly utilized nodes in each iteration

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 3.4.0
• Component/s: balancer & mover
• Labels: pull-request-available
• Hadoop Flags: Reviewed
• Assignee: Leon Gao
• Reporter: Leon Gao
• Created: 12/Oct/19 01:48
• Updated: 09/Apr/21 15:06
• Resolved: 02/Dec/20 23:54

Description

Normally the most important purpose for HDFS balancer is to reduce the top used node to prevent datanode usage from being too high.

Currently, balancer almost randomly picks nodes as sources regardless of usage, which makes it slow to bring down the top used datanodes in the cluster, when there are less underutilized nodes in the cluster (consider expansion).

We can add an option to prefer top used nodes first in each iteration, as suggested in HDFS-14894 .

Issue Links

• causes: HDFS-15904 Flaky test TestBalancer#testBalancerWithSortTopNodes()
• links to: GitHub Pull Request #2483