[HBASE-21378] [hbck2] add --skip version check to hbck2 tool (checkHBCKSupport blocks assigning hbase:meta or hbase:namespace when master is not initialized)

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: hbase-operator-tools-1.0.0
• Component/s: hbase-operator-tools, hbck2

Release Note

Adds a general -s,--skip option to hbck2 so you can run the command it first checking it is version compatible.

Should not be needed going forward but in the spirt of our not knowing all the conditions under which we'll be trying to run hbck2, adding it.

• Assignee: Jingyun Tian
• Reporter: Jingyun Tian
• Created: 24/Oct/18 08:22
• Updated: 10/Sep/19 19:54
• Resolved: 29/Oct/18 17:06

Description

When I encounter the scenario that hbase:namespace is not online.

2018-10-24,14:38:16,910 WARN org.apache.hadoop.hbase.master.HMaster: hbase:namespace,,1529933109115.7e0801c8232b2dc15face54532056076. is NOT online; state={7e0801c8232b2dc15face54532056076 state=OPEN, ts=1540363033384, server=c4-hadoop-tst-st30.bj,29100,1540348649479}; ServerCrashProcedures=false. Master startup cannot progress, in holding-pattern until region onlined.

Then I tried to assign it manually, but it throws PleaseHoldException.

Wed Oct 24 15:26:52 CST 2018, RpcRetryingCaller{globalStartTime=1540365754487, pause=200, maxAttempts=16}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing
	at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:3064)
	at org.apache.hadoop.hbase.master.MasterRpcServices.getClusterStatus(MasterRpcServices.java:934)
	at org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java)
	at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:413)
	at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
	at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
	at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)


	at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:144)
	at org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:3133)
	at org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:3125)
	at org.apache.hadoop.hbase.client.HBaseAdmin.getClusterMetrics(HBaseAdmin.java:2161)
	at org.apache.hbase.HBCK2.checkHBCKSupport(HBCK2.java:98)
	at org.apache.hbase.HBCK2.run(HBCK2.java:364)
	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
	at org.apache.hbase.HBCK2.main(HBCK2.java:447)
Caused by: org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing
	at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:3064)
	at org.apache.hadoop.hbase.master.MasterRpcServices.getClusterStatus(MasterRpcServices.java:934)
	at org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java)
	at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:413)
	at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
	at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
	at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)

	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
	at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:100)
	at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:90)
	at org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:361)
	at org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:349)
	at org.apache.hadoop.hbase.client.MasterCallable.call(MasterCallable.java:101)
	at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:107)

Then I check the code and found it is because of checkHBCKSupport(), I assign hbase:namespace successfully by skipping this check. Thus I think the tool need an option to skip this check.