[HDFS-14675] Increase Balancer Defaults Further

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.3.0
• Fix Version/s: 3.3.0
• Component/s: balancer & mover
• Target Version/s: 3.3.0

Release Note

Increase default bandwidth limit for rebalancing per DataNode (dfs.datanode.balance.bandwidthPerSec) from 10MB/s to 100MB/s.

Increase default maximum threads of DataNode balancer (dfs.datanode.balance.max.concurrent.moves) from 50 to 100.

• Assignee: Stephen O'Donnell
• Reporter: Stephen O'Donnell
• Created: 26/Jul/19 11:22
• Updated: 02/Oct/19 17:12
• Resolved: 22/Aug/19 22:21

Description

HDFS-10297 increased the balancer defaults to 50 for dfs.datanode.balance.max.concurrent.moves and to 10MB/s for dfs.datanode.balance.bandwidthPerSec.

We have found that these settings often have to be increased further as users find the balancer operates too slowly with 50 and 10MB/s. We often recommend moving concurrent moves to between 200 and 300 and setting the bandwidth to 100 or even 1000MB/s, and these settings seem to work well in practice.

I would like to suggest we increase the balancer defaults further. I would suggest 100 for concurrent moves and 100MB/s for the bandwidth, but I would like to know what others think on this topic too.