[HBASE-24557] HBase upgrade from 2.3 to master fails because master.HMaster fails to become active master

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: master
• Assignee: Unassigned
• Reporter: Zhuqi Jin
• Created: 13/Jun/20 20:59
• Updated: 11/Aug/20 18:12

Description

When we tested upgrading HBase from 2.3 to master(c303f9d329d578d31140e507bdbcbe3aa097042b), the upgraded node failed to start.

The error message is shown below:

2020-06-13 20:23:26,409 ERROR [master/085bbffdc52f:16000:becomeActiveMaster] master.HMaster: Failed to become active master2020-06-13 20:23:26,409 ERROR [master/085bbffdc52f:16000:becomeActiveMaster] master.HMaster: Failed to become active masterjava.lang.IllegalStateException: Expected the service ClusterSchemaServiceImpl [FAILED] to be RUNNING, but the service has FAILED at org.apache.hbase.thirdparty.com.google.common.util.concurrent.AbstractService.checkCurrentState(AbstractService.java:379) at org.apache.hbase.thirdparty.com.google.common.util.concurrent.AbstractService.awaitRunning(AbstractService.java:319) at org.apache.hadoop.hbase.master.HMaster.initClusterSchemaService(HMaster.java:1362) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:1137) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2245) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:626) at java.lang.Thread.run(Thread.java:748)Caused by: org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed 2 actions: RetriesExhaustedException: 2 times, servers with issues:  at org.apache.hadoop.hbase.client.BufferedMutatorOverAsyncBufferedMutator.makeError(BufferedMutatorOverAsyncBufferedMutator.java:107) at org.apache.hadoop.hbase.client.BufferedMutatorOverAsyncBufferedMutator.internalFlush(BufferedMutatorOverAsyncBufferedMutator.java:122) at org.apache.hadoop.hbase.client.BufferedMutatorOverAsyncBufferedMutator.close(BufferedMutatorOverAsyncBufferedMutator.java:166) at org.apache.hadoop.hbase.master.TableNamespaceManager.migrateNamespaceTable(TableNamespaceManager.java:93) at org.apache.hadoop.hbase.master.TableNamespaceManager.start(TableNamespaceManager.java:123) at org.apache.hadoop.hbase.master.ClusterSchemaServiceImpl.doStart(ClusterSchemaServiceImpl.java:61) at org.apache.hbase.thirdparty.com.google.common.util.concurrent.AbstractService.startAsync(AbstractService.java:249) at org.apache.hadoop.hbase.master.HMaster.initClusterSchemaService(HMaster.java:1360) ... 4 more2020-06-13 20:23:26,410 ERROR [master/085bbffdc52f:16000:becomeActiveMaster] master.HMaster: Master server abort: loaded coprocessors are: [org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint]2020-06-13 20:23:26,410 ERROR [master/085bbffdc52f:16000:becomeActiveMaster] master.HMaster: ***** ABORTING master 085bbffdc52f,16000,1592079787433: Unhandled exception. Starting shutdown. *****java.lang.IllegalStateException: Expected the service ClusterSchemaServiceImpl [FAILED] to be RUNNING, but the service has FAILED at org.apache.hbase.thirdparty.com.google.common.util.concurrent.AbstractService.checkCurrentState(AbstractService.java:379) at org.apache.hbase.thirdparty.com.google.common.util.concurrent.AbstractService.awaitRunning(AbstractService.java:319) at org.apache.hadoop.hbase.master.HMaster.initClusterSchemaService(HMaster.java:1362) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:1137) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2245) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:626) at java.lang.Thread.run(Thread.java:748)Caused by: org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed 2 actions: RetriesExhaustedException: 2 times, servers with issues:  at org.apache.hadoop.hbase.client.BufferedMutatorOverAsyncBufferedMutator.makeError(BufferedMutatorOverAsyncBufferedMutator.java:107) at org.apache.hadoop.hbase.client.BufferedMutatorOverAsyncBufferedMutator.internalFlush(BufferedMutatorOverAsyncBufferedMutator.java:122) at org.apache.hadoop.hbase.client.BufferedMutatorOverAsyncBufferedMutator.close(BufferedMutatorOverAsyncBufferedMutator.java:166) at org.apache.hadoop.hbase.master.TableNamespaceManager.migrateNamespaceTable(TableNamespaceManager.java:93) at org.apache.hadoop.hbase.master.TableNamespaceManager.start(TableNamespaceManager.java:123) at org.apache.hadoop.hbase.master.ClusterSchemaServiceImpl.doStart(ClusterSchemaServiceImpl.java:61) at org.apache.hbase.thirdparty.com.google.common.util.concurrent.AbstractService.startAsync(AbstractService.java:249) at org.apache.hadoop.hbase.master.HMaster.initClusterSchemaService(HMaster.java:1360) ... 4 more

It can be reproduced through the following steps:

• Start a single-node HBase 2.3 with the following configuration.

    <property>
      <name>hbase.rootdir</name>
      <value>/var/lib/hbase</value>
    </property>  <property>
      <name>hbase.zookeeper.property.dataDir</name>
      <value>/var/lib/zookeeper</value>
    </property>

• Use HBase performance evaluation tool to generate data.

  /hbase/bin/hbase pe --nomapred --oneCon=true --valueSize=10 --rows=100 sequentialWrite 1

• Upgrade the node to master with the same configuration. The new version node failed to start. And there is error message in /hbase/logs.

When I used the default configuration, the error message is the same as before.

I also attached the entire log file. hbase--master-085bbffdc52f.log