[HBASE-20907] Fix Intermittent failure on TestProcedurePriority

• Type: Test
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 3.0.0-alpha-1, 2.2.0, 2.1.1, 2.0.2
• Hadoop Flags: Reviewed
• Assignee: Yu Li
• Reporter: Yu Li
• Created: 18/Jul/18 03:38
• Updated: 04/Mar/19 09:07
• Resolved: 19/Jul/18 04:04

Description

From a local UT check against 2.1.0-RC1, HMaster failed to initialize before time out. Checking the test log we could see below message:

2018-07-17 20:06:37,142 DEBUG [Thread-4003] client.RpcRetryingCallerImpl(131): Call exception, tries=6, retries=6, started=4173 ms ago, cancelled=false, msg=java.io.IOException: Inject error
        at org.apache.hadoop.hbase.master.procedure.TestProcedurePriority$MyCP.preGetOp(TestProcedurePriority.java:92)
        at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$19.call(RegionCoprocessorHost.java:841)
        at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$19.call(RegionCoprocessorHost.java:838)
        at org.apache.hadoop.hbase.coprocessor.CoprocessorHost$ObserverOperationWithoutResult.callObserver(CoprocessorHost.java:540)
        at org.apache.hadoop.hbase.coprocessor.CoprocessorHost.execOperation(CoprocessorHost.java:614)
        at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.preGet(RegionCoprocessorHost.java:838)
        at org.apache.hadoop.hbase.regionserver.RSRpcServices.get(RSRpcServices.java:2520)
        at org.apache.hadoop.hbase.regionserver.RSRpcServices.get(RSRpcServices.java:2460)
        at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:41998)
        at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:409)
        at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
        at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
        at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)
, details=row 'hbase:namespace' on table 'hbase:meta' at region=hbase:meta,,1.1588230740, hostname=hdpdevm1.et2sqa.tbsite.net,59254,1531829189215, seqNum=-1, exception=java.io.IOException: java.io.IOException: Inject error
        at org.apache.hadoop.hbase.master.procedure.TestProcedurePriority$MyCP.preGetOp(TestProcedurePriority.java:92)
        at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$19.call(RegionCoprocessorHost.java:841)
        ...
        at org.apache.hadoop.hbase.client.HTable.get(HTable.java:386)
        at org.apache.hadoop.hbase.client.HTable.get(HTable.java:360)
        at org.apache.hadoop.hbase.MetaTableAccessor.getTableState(MetaTableAccessor.java:1078)
        at org.apache.hadoop.hbase.MetaTableAccessor.tableExists(MetaTableAccessor.java:403)
        at org.apache.hadoop.hbase.master.TableNamespaceManager.start(TableNamespaceManager.java:94)

In current test code we will set FAIL to true w/o checking whether namespace manager is already up, and if not lucky we will run into the above case and get a timeout.

The fix will be quite straight forward.