Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-21378

[hbck2] add --skip version check to hbck2 tool (checkHBCKSupport blocks assigning hbase:meta or hbase:namespace when master is not initialized)

VotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Hide
      Adds a general -s,--skip option to hbck2 so you can run the command it first checking it is version compatible.

      Should not be needed going forward but in the spirt of our not knowing all the conditions under which we'll be trying to run hbck2, adding it.
      Show
      Adds a general -s,--skip option to hbck2 so you can run the command it first checking it is version compatible. Should not be needed going forward but in the spirt of our not knowing all the conditions under which we'll be trying to run hbck2, adding it.

    Description

      When I encounter the scenario that hbase:namespace is not online.

      2018-10-24,14:38:16,910 WARN org.apache.hadoop.hbase.master.HMaster: hbase:namespace,,1529933109115.7e0801c8232b2dc15face54532056076. is NOT online; state={7e0801c8232b2dc15face54532056076 state=OPEN, ts=1540363033384, server=c4-hadoop-tst-st30.bj,29100,1540348649479}; ServerCrashProcedures=false. Master startup cannot progress, in holding-pattern until region onlined.
      

      Then I tried to assign it manually, but it throws PleaseHoldException.

      Wed Oct 24 15:26:52 CST 2018, RpcRetryingCaller{globalStartTime=1540365754487, pause=200, maxAttempts=16}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing
      	at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:3064)
      	at org.apache.hadoop.hbase.master.MasterRpcServices.getClusterStatus(MasterRpcServices.java:934)
      	at org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java)
      	at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:413)
      	at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
      	at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
      	at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)
      
      
      	at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:144)
      	at org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:3133)
      	at org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:3125)
      	at org.apache.hadoop.hbase.client.HBaseAdmin.getClusterMetrics(HBaseAdmin.java:2161)
      	at org.apache.hbase.HBCK2.checkHBCKSupport(HBCK2.java:98)
      	at org.apache.hbase.HBCK2.run(HBCK2.java:364)
      	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
      	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
      	at org.apache.hbase.HBCK2.main(HBCK2.java:447)
      Caused by: org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing
      	at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:3064)
      	at org.apache.hadoop.hbase.master.MasterRpcServices.getClusterStatus(MasterRpcServices.java:934)
      	at org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java)
      	at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:413)
      	at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
      	at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
      	at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)
      
      	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
      	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
      	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
      	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
      	at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:100)
      	at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:90)
      	at org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:361)
      	at org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:349)
      	at org.apache.hadoop.hbase.client.MasterCallable.call(MasterCallable.java:101)
      	at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:107)
      

      Then I check the code and found it is because of checkHBCKSupport(), I assign hbase:namespace successfully by skipping this check. Thus I think the tool need an option to skip this check.

      Attachments

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            tianjingyun Jingyun Tian
            tianjingyun Jingyun Tian
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment