[HBASE-22349] Stochastic Load Balancer skips balancing when node is replaced in cluster

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.0.0-alpha-1, 1.3.0, 1.4.4, 2.0.0
• Fix Version/s: 2.5.0, 3.0.0-alpha-3
• Component/s: Balancer
• Hadoop Flags: Reviewed

Release Note

StochasticLoadBalancer now respects the hbase.regions.slop configuration value as another factor in determining whether to attempt a balancer run. If any regionserver has a region count outside of the target range, the balancer will attempt to balance. Using the default 0.2 value, the target range is 80%-120% of the average (mean) region count per server. Whether the balancer will ultimately move regions will still depend on the weights of StochasticLoadBalancer's cost functions.

• Assignee: David Manning
• Reporter: Suthan Phillips
• Created: 02/May/19 05:34
• Updated: 30/Apr/22 05:10
• Resolved: 28/Apr/22 21:56

Description

HBASE-24139 allows the load balancer to run when one server has 0 regions and another server has more than 1 region. This is a special case of a more generic problem, where one server has far too few or far too many regions. The StochasticLoadBalancer defaults may decide the cluster is "balanced enough" according to hbase.master.balancer.stochastic.minCostNeedBalance, even though one server may have a far higher or lower number of regions compared to the rest of the cluster.

One specific example of this we have seen is when we use RegionMover to move regions back to a restarted RegionServer, if the StochasticLoadBalancer happens to be running. The load balancer sees a newly restarted RegionServer with 0 regions, and after HBASE-24139, it will balance regions to this server. Simultaneously, RegionMover moves back regions. The end result is that the newly restarted RegionServer has twice the load of any other server in the cluster. Future iterations of the load balancer do nothing, as the cluster cost does not exceed minCostNeedBalance.

Another example is if the load balancer makes very slow progress on a cluster, it may not move the average cluster load to a newly restarted regionserver in one iteration. But after the first iteration, the balancer may again not run due to cluster cost not exceeding minCostNeedBalance.

We can propose a solution where we reuse the slop concept in SimpleLoadBalancer and use this to extend the HBASE-24139 logic for deciding to run the balancer as long as there is a "sloppy" server in the cluster.

Previous Description Notes Below, which are relevant, but as stated, were already fixed by HBASE-24139

In EMR cluster, whenever I replace one of the nodes, the regions never get rebalanced.

The default minCostNeedBalance set to 0.05 is too high.

The region count on the servers were: 21, 21, 20, 20, 20, 20, 21, 20, 20, 20 = 203

Once a node(region server) got replaced with a new node (terminated and EMR recreated a node), the region count on the servers became: 23, 0, 23, 22, 22, 22, 22, 23, 23, 23 = 203

From hbase-master-logs, I can see the below WARN which indicates that the default minCostNeedBalance does not hold good for these scenarios.

##

2019-04-29 09:31:37,027 WARN  [ip-172-31-35-122.ec2.internal,16000,1556524892897_ChoreService_1] cleaner.CleanerChore: WALs outstanding under hdfs://ip-172-31-35-122.ec2.internal:8020/user/hbase/oldWALs2019-04-29 09:31:42,920 INFO  [ip-172-31-35-122.ec2.internal,16000,1556524892897_ChoreService_1] balancer.StochasticLoadBalancer: Skipping load balancing because balanced cluster; total cost is 52.041826194833405, sum multiplier is 1102.0 min cost which need balance is 0.05

##

To mitigate this, I had to modify the default minCostNeedBalance to lower value like 0.01f and restart Region Servers and Hbase Master. After modifying this value to 0.01f I could see the regions getting re-balanced.

This has led me to the following questions which I would like to get it answered from the HBase experts.

1)What are the factors that affect the value of total cost and sum multiplier? How could we determine the right minCostNeedBalance value for any cluster?

2)How did Hbase arrive at setting the default value to 0.05f? Is it optimal value? If yes, then what is the recommended way to mitigate this scenario? 

Attached: Steps to reproduce

 

Note: HBase-17565 patch is already applied.

Issue Links

• relates to: HBASE-24139 Balancer should avoid leaving idle region servers
• Testing discovered: HBASE-26988 Balancer should reset to default setting for hbase.master.loadbalance.bytable if dynamically reloading configuration
• Testing discovered: HBASE-26989 TestStochasticLoadBalancer has some slow methods, and inconsistent set, reset, unset of configuration
• links to: GitHub Pull Request #4371