[HDFS-4992] Make balancer's thread count configurable

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.1.0-beta
• Component/s: balancer & mover
• Hadoop Flags: Reviewed
• Assignee: Max Lapan
• Reporter: Max Lapan
• Created: 15/Jul/13 10:52
• Updated: 27/Aug/13 22:07
• Resolved: 16/Jul/13 14:15

Description

By default, balancer has 1000 threads which moves blocks around (mover threads) and 200 threads which decide which block to move (dispatcher threads).

On large clusters, 1000 threads creates significant load on NN, which slows down other HDFS activity. For example, on our cluster, 'hdfs dfs -ls /' command took about 1 minute when balancer is active. When no balancing in progress, the same command finishes in second or two.

This patch makes amount of threads configurable by two new options 'dfs.balancer.moverThreads' and 'dfs.balancer.dispatcherThreads'.