[HBASE-19937] Ensure createRSGroupTable be called after ProcedureExecutor and LoadBalancer are initialized

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.0.0-beta-2
• Fix Version/s: 2.0.0-beta-2, 1.4.2, 2.0.0
• Component/s: rsgroup
• Hadoop Flags: Reviewed
• Assignee: Xiaolin Ha
• Reporter: Xiaolin Ha
• Created: 05/Feb/18 08:46
• Updated: 01/Feb/19 20:00
• Resolved: 09/Feb/18 06:43

Description

hbase:rsgroup table will be created by calling createRSGroupTable when master load system coprocessors in 

 

844  this.cpHost = new MasterCoprocessorHost(this, this.conf);

when ProcedureExecutor hasn't been initialized before createRSGroupTable, it will encounter Exception as follows,

 

Exception in thread "org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker-localhost,49715,1518088607130" java.lang.IllegalArgumentException
at org.apache.hbase.thirdparty.com.google.common.base.Preconditions.checkArgument(Preconditions.java:120)
at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.submitProcedure(ProcedureExecutor.java:847)
at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.submitProcedure(ProcedureExecutor.java:835)
at org.apache.hadoop.hbase.master.HMaster.createSystemTable(HMaster.java:1795)
at org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker.createRSGroupTable(RSGroupInfoManagerImpl.java:858)
at org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker.waitForGroupTableOnline(RSGroupInfoManagerImpl.java:823)
at org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker.run(RSGroupInfoManagerImpl.java:743)

while ProcedureExecutor  initialized by calling

848  startServiceThreads();

And LoadBalancer is initialized by calling 

868  this.balancer.initialize();

When LoadBalancer hasn't been initialized before createRSGroupTable, it will encounters Exception as follows,

2018-02-02,16:12:45,688 ERROR org.apache.hadoop.hbase.procedure2.ProcedureExecutor: CODE-BUG: Uncaught runtime exception: pid=7, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS; CreateTableProcedure table=hbase:rsgroup
java.lang.NullPointerException
at org.apache.hadoop.hbase.rsgroup.RSGroupBasedLoadBalancer.generateGroupMaps(RSGroupBasedLoadBalancer.java:254)
at org.apache.hadoop.hbase.rsgroup.RSGroupBasedLoadBalancer.roundRobinAssignment(RSGroupBasedLoadBalancer.java:162)
at org.apache.hadoop.hbase.master.assignment.AssignmentManager.createRoundRobinAssignProcedures(AssignmentManager.java:603)
at org.apache.hadoop.hbase.master.procedure.CreateTableProcedure.executeFromState(CreateTableProcedure.java:108)
at org.apache.hadoop.hbase.master.procedure.CreateTableProcedure.executeFromState(CreateTableProcedure.java:51)
at org.apache.hadoop.hbase.procedure2.StateMachineProcedure.execute(StateMachineProcedure.java:182)
at org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:845)
at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1458)
at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1227)
at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$800(ProcedureExecutor.java:78)
at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1738)

 

As a result of CreateTableProcedure.rollbackState, it may then print logs warning TableExistsException as follows,

2018-02-02,16:12:55,503 WARN org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker: Failed to perform check
java.io.IOException: Failed to create group table. org.apache.hadoop.hbase.TableExistsException: hbase:rsgroup
at org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker.createRSGroupTable(RSGroupInfoManagerImpl.java:877)

 

After some auto-retries, it loops running the thread RSGroupStartupWorker, will print logs as follows, 

2018-02-02,16:23:17,626 INFO org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker: RSGroup table=hbase:rsgroup isOnline=true, regionCount=0, assignCount=0, rootMetaFound=true
2018-02-02,16:23:17,730 INFO org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker: RSGroup table=hbase:rsgroup isOnline=true, regionCount=0, assignCount=0, rootMetaFound=true
2018-02-02,16:23:17,834 INFO org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker: RSGroup table=hbase:rsgroup isOnline=true, regionCount=0, assignCount=0, rootMetaFound=true
2018-02-02,16:23:17,937 INFO org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker: RSGroup table=hbase:rsgroup isOnline=true, regionCount=0, assignCount=0, rootMetaFound=true

 

And using shells of rsgroup, it will tips that currently is in "offline mode".

 

The reason of this problem is that the order of createRSGroupTable and initializing of ProcedureExecutor and LoadBalancer is out of control. If the former is excuted earlier, it will encounter Exception mentioned before.

 

Issue Links

• links to: Review Board (branch-2)