[HDFS-8825] Enhancements to Balancer

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: balancer & mover
• Assignee: Tsz Wo Nicholas Sze
• Reporter: Tsz Wo Nicholas Sze
• Created: 28/Jul/15 00:13
• Updated: 02/May/17 00:54

Description

This is an umbrella JIRA to enhance Balancer. The goal is to make it runs faster, more efficient and improve its usability.

Issue Links

• relates to: HDFS-11384 Add option for balancer to disperse getBlocks calls to avoid NameNode's rpc.CallQueueLength spike
• relates to: HDFS-11634 Optimize BlockIterator when iterating starts in the middle.

Sub-Tasks

• Allow Balancer to run faster: Resolved
• Do not use small blocks for balancing the cluster: Resolved
• Balancer may not move blocks efficiently in some cases: Resolved
• Separate Balancer specific logic form Dispatcher: Patch Available
• Remove the limitation imposed by dfs.balancer.moverThreads: Open
• HDFS Balancer should consider remaining storage % when checking for under-utilized machines: Resolved
• Optimize the wait time in Balancer for federation scenario: Open
• Add DN metrics for balancer and other block movement scenarios: In Progress
• Make hard coded parameters used by balancer and other tools configurable: Open
• Allow admin to specify which blockpools the balancer should run on: Resolved
• Add an option to Balancer for specifying the k-most over-utilized DNs or all over-utilized DNs as sources.: Open
• Balancer#Parameters class could use a builder pattern: Resolved
• Document -source option for balancer: Resolved
• Incorrect Dispatcher#Source wait/notify leads to early termination: Open
• Balancer should not terminate if only one of the namenodes has error: Patch Available
• Balancer documentation should not be in javadoc: Open