[HBASE-15207] Stuck balancer

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Cannot Reproduce
• Affects Version/s: 1.2.0
• Component/s: Balancer
• Assignee: Unassigned
• Reporter: Michael Stack
• Created: 02/Feb/16 19:36
• Updated: 05/Feb/16 22:38
• Resolved: 05/Feb/16 22:38

Description

Balancer seems to have gotten stuck in 1.2.0RC1 soon after Master joins running cluster (previous Master had been killed by chaos monkey). Investigate. At least fix the crazy logging which made me notice the stuck balancer.

Last night my logs filled with this (10x256MB log files):

....
2016-02-01 11:25:26,958 DEBUG [B.defaultRpcServer.handler=9,queue=0,port=16000] balancer.BaseLoadBalancer: Lowest locality region server with non zero regions is ve0542.halxg.cloudera.com with locality 0.0
2016-02-01 11:25:26,958 DEBUG [B.defaultRpcServer.handler=9,queue=0,port=16000] balancer.BaseLoadBalancer: Lowest locality region index is 0 and its region server contains 1 regions
...

Added by this:

commit 54028140f4f19a6af81c8c8f29dda0c52491a0c9
Author: tedyu <[e-mail]>
Date: Thu Aug 13 09:11:59 2015 -0700

HBASE-13376 Improvements to Stochastic load balancer (Vandana Ayyalasomayajula)

Looks like balancer got stuck. Logging at ten lines a millisecond.

Here is lead up. Nothing in particular jumps out. Rerun doesn't show this.

2016-01-28 05:56:22,572 DEBUG [ve0524.halxg.cloudera.com,16000,1453988766013_ChoreService_1] balancer.BaseLoadBalancer: Server ve0526.halxg.cloudera.com had 0 regions.
2016-01-28 05:56:22,572 DEBUG [ve0524.halxg.cloudera.com,16000,1453988766013_ChoreService_1] balancer.BaseLoadBalancer: Server ve0532.halxg.cloudera.com had 0 regions.
2016-01-28 05:56:22,572 DEBUG [ve0524.halxg.cloudera.com,16000,1453988766013_ChoreService_1] balancer.BaseLoadBalancer: Server ve0538.halxg.cloudera.com had 0 regions.
2016-01-28 05:56:22,572 DEBUG [ve0524.halxg.cloudera.com,16000,1453988766013_ChoreService_1] balancer.BaseLoadBalancer: Lowest locality region server with non zero regions is ve0540.halxg.cloudera.com with locality 0.0
2016-01-28 05:56:22,572 DEBUG [ve0524.halxg.cloudera.com,16000,1453988766013_ChoreService_1] balancer.BaseLoadBalancer:  Lowest locality region index is 0 and its region server contains 1 regions
2016-01-28 05:56:22,573 DEBUG [ve0524.halxg.cloudera.com,16000,1453988766013_ChoreService_1] balancer.BaseLoadBalancer: Server ve0526.halxg.cloudera.com had 0 regions.
2016-01-28 05:56:22,573 DEBUG [ve0524.halxg.cloudera.com,16000,1453988766013_ChoreService_1] balancer.BaseLoadBalancer: Server ve0532.halxg.cloudera.com had 0 regions.
2016-01-28 05:56:22,573 DEBUG [ve0524.halxg.cloudera.com,16000,1453988766013_ChoreService_1] balancer.BaseLoadBalancer: Server ve0538.halxg.cloudera.com had 0 regions.
2016-01-28 05:56:22,573 DEBUG [ve0524.halxg.cloudera.com,16000,1453988766013_ChoreService_1] balancer.BaseLoadBalancer: Lowest locality region server with non zero regions is ve0540.halxg.cloudera.com with locality 0.0
2016-01-28 05:56:22,573 DEBUG [ve0524.halxg.cloudera.com,16000,1453988766013_ChoreService_1] balancer.BaseLoadBalancer:  Lowest locality region index is 0 and its region server contains 1 regions
2016-01-28 05:56:22,573 DEBUG [ve0524.halxg.cloudera.com,16000,1453988766013_ChoreService_1] balancer.BaseLoadBalancer: Server ve0526.halxg.cloudera.com had 0 regions.
2016-01-28 05:56:22,573 DEBUG [ve0524.halxg.cloudera.com,16000,1453988766013_ChoreService_1] balancer.BaseLoadBalancer: Server ve0532.halxg.cloudera.com had 0 regions.
2016-01-28 05:56:22,573 DEBUG [ve0524.halxg.cloudera.com,16000,1453988766013_ChoreService_1] balancer.BaseLoadBalancer: Server ve0538.halxg.cloudera.com had 0 regions.
....

Nothing else is happening on this master

Happens just after a Master joins cluster after being killed by a monkey.

Sub-Tasks

• Undo aggressive load balancer logging at tens of lines per millisecond: Resolved