[HBASE-9267] StochasticLoadBalancer goes over its processing time limit

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.95.2
• Fix Version/s: 0.98.0, 0.96.0
• Hadoop Flags: Reviewed
• Assignee: Elliott Neil Clark
• Reporter: Jean-Daniel Cryans
• Created: 19/Aug/13 22:06
• Updated: 20/Nov/15 11:55
• Resolved: 21/Aug/13 23:44

Description

I trying out 0.95.2, I left it running over the weekend (8 RS, average load between 12 and 3 regions) and right now the balancer runs for 12 mins:

2013-08-19 21:54:45,534 DEBUG [jdec2hbase0403-1.vpc.cloudera.com,60000,1376689696384-BalancerChore] org.apache.hadoop.hbase.master.balancer.StochasticLoadBalancer: Could not find a better load balance plan. Tried 0 different configurations in 777309ms, and did not find anything with a computed cost less than 36.32576937689094

It seems it slowly crept up there, yesterday it was doing:

2013-08-18 20:53:17,232 DEBUG [jdec2hbase0403-1.vpc.cloudera.com,60000,1376689696384-BalancerChore] org.apache.hadoop.hbase.master.balancer.StochasticLoadBalancer: Could not find a better load balance plan. Tried 0 different configurations in 257374ms, and did not find anything with a computed cost less than 36.3251082542424

And originally it was doing 1 minute.

In the jstack I see a 1000 of these and jstack doesn't want to show me the whole thing:

at java.util.SubList$1.nextIndex(AbstractList.java:713)