[HDFS-17037] Consider nonDfsUsed when running balancer

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.4.0
• Fix Version/s: 3.4.0
• Component/s: balancer & mover
• Labels: pull-request-available
• Target Version/s: 3.4.0
• Hadoop Flags: Reviewed
• Assignee: Shuyan Zhang
• Reporter: Shuyan Zhang
• Created: 04/Jun/23 11:46
• Updated: 28/Jan/24 02:13
• Resolved: 09/Jun/23 08:21

Description

When we run balancer with `BalancingPolicy.Node` policy, our goal is to make each datanode storage balanced. But in the current implementation, the balancer doesn't account for storage used by non-dfs on the datanodes, which can make the situation worse for datanodes that are already strained on storage.

Issue Links

• links to: GitHub Pull Request #5715