[HBASE-24178] some tests of org.apache.hadoop.hbase.TestZooKeeper.* failed on aarch64

• Type: Sub-task
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Assignee: Unassigned
• Reporter: huangtianhua
• Created: 13/Apr/20 10:13
• Updated: 13/Apr/20 10:55

Description

I build and test Hbase on aarch64, and tests org.apache.hadoop.hbase.TestZooKeeper.testMasterZKSessionRecoveryFailure and testMasterSessionExpired failed by chance. I found ARM CI in jenkins failed by chance too https://builds.apache.org/view/H-L/view/HBase/job/HBase-Nightly-ARM/73/testReport/junit/org.apache.hadoop.hbase/TestZooKeeper/testMasterZKSessionRecoveryFailure/, see:

Error Message
Unable to instantiate exception received from server:org.apache.hadoop.hbase.master.HMaster$MasterStoppedException.<init>(java.lang.String)
Stacktrace
org.apache.hadoop.hbase.DoNotRetryIOException: Unable to instantiate exception received from server:org.apache.hadoop.hbase.master.HMaster$MasterStoppedException.<init>(java.lang.String)
at org.apache.hadoop.hbase.TestZooKeeper.testSanity(TestZooKeeper.java:153)
at org.apache.hadoop.hbase.TestZooKeeper.testMasterZKSessionRecoveryFailure(TestZooKeeper.java:140)
Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException:
org.apache.hadoop.hbase.master.HMaster$MasterStoppedException
at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2932)
at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:2135)
at org.apache.hadoop.hbase.master.MasterRpcServices.createTable(MasterRpcServices.java:716)
at org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:400)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:133)
at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:338)
at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:318)

It looks like the master is still stopped when we want to create table after stop it. I made thread to sleep 1 second in testSanity function before calling admin.createTable(desc), then I run the tests for more than 10 times and the tests are success:
**************************************************************************

private void testSanity(final String testName) throws Exception {
String tableName = testName + "_" + System.currentTimeMillis();
TableDescriptor desc = TableDescriptorBuilder.newBuilder(TableName.valueOf(tableName))
.setColumnFamily(ColumnFamilyDescriptorBuilder.of("fam")).build();
Thread.sleep(1000);
LOG.info("Creating table " + tableName);
Admin admin = TEST_UTIL.getAdmin();
try

Unknown macro: { admin.createTable(desc); }

finally

Unknown macro: { admin.close(); }

**************************************************************************