Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-13217

Procedure fails due to ZK issue

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 1.0.1, 1.1.0, 0.98.12, 2.0.0
    • 0.98.13, 1.0.2, 1.2.0, 1.1.1, 2.0.0
    • None
    • None

    Description

      When ever I try to flush explicitly in the trunk code the flush procedure fails due to ZK issue

      ERROR: org.apache.hadoop.hbase.errorhandling.ForeignException$ProxyThrowable via stobdtserver3,16040,1426172670959:org.apache.hadoop.hbase.errorhandling.ForeignException$ProxyThrowable: java.io.IOException: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /hbase/flush-table-proc/acquired/TestTable/stobdtserver3,16040,1426172670959
              at org.apache.hadoop.hbase.errorhandling.ForeignExceptionDispatcher.rethrowException(ForeignExceptionDispatcher.java:83)
              at org.apache.hadoop.hbase.procedure.Procedure.isCompleted(Procedure.java:368)
              at org.apache.hadoop.hbase.procedure.flush.MasterFlushTableProcedureManager.isProcedureDone(MasterFlushTableProcedureManager.java:196)
              at org.apache.hadoop.hbase.master.MasterRpcServices.isProcedureDone(MasterRpcServices.java:905)
              at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:47019)
              at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2073)
              at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:107)
              at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130)
              at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107)
              at java.lang.Thread.run(Thread.java:745)
      Caused by: org.apache.hadoop.hbase.errorhandling.ForeignException$ProxyThrowable: java.io.IOException: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /hbase/flush-table-proc/acquired/TestTable/stobdtserver3,16040,1426172670959
              at org.apache.hadoop.hbase.procedure.Subprocedure.cancel(Subprocedure.java:273)
              at org.apache.hadoop.hbase.procedure.ProcedureMember.controllerConnectionFailure(ProcedureMember.java:225)
              at org.apache.hadoop.hbase.procedure.ZKProcedureMemberRpcs.sendMemberAcquired(ZKProcedureMemberRpcs.java:254)
              at org.apache.hadoop.hbase.procedure.Subprocedure.call(Subprocedure.java:166)
              at org.apache.hadoop.hbase.procedure.Subprocedure.call(Subprocedure.java:52)
              at java.util.concurrent.FutureTask.run(FutureTask.java:262)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
              ... 1 more
      
      

      Once this occurs, even on restart of the RS the RS becomes unusable. I have verified that the ZK remains intact and there is no problem with it. a bit older version of trunk ( 3months) does not have this problem.

      Attachments

        1. HBASE-13217-0.98-v2.patch
          7 kB
          Jerry He
        2. HBASE-13217-v2.patch
          8 kB
          Jerry He
        3. HBASE-13217.patch
          7 kB
          Jerry He

        Activity

          People

            jerryhe Jerry He
            ram_krish ramkrishna.s.vasudevan
            Votes:
            0 Vote for this issue
            Watchers:
            12 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: