[HDFS-14894] Add balancer parameter to balance top used nodes

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Done
• Component/s: balancer & mover
• Assignee: Leon Gao
• Reporter: Leon Gao
• Created: 05/Oct/19 22:20
• Updated: 04/Jan/24 06:57
• Resolved: 04/Jan/24 06:56

Description

We sometimes see a few of our datanodes reach very high usage (due to various reasons) and we need to reduce their usage in an urgent situation.

We see two ways to achieve it currently,

-Calculate and reset balancing threshold.

-Pick nodes manually according to usage stats and put them in a file and use `-resource` flag.

However, both of them are not very intuitive or too much manual work in an urgent close-to-outage situation. Add a small feature to automatically pick top used hosts will be a straightforward option, for example `-sourceThreshold 95` to only target datanodes with >95% usage. 

Issue Links

• links to: GitHub Pull Request #1636