Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-17572

HMaster: Caught throwable while processing event C_M_MERGE_REGION

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.3.0
    • Fix Version/s: 1.3.1
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Running ITBLL 1B rows against branch-1.3 compiled against Hadoop 2.7.3 with the noKill monkey policy, I see both masters go down with

      master.HMaster: Caught throwable while processing event C_M_MERGE_REGION
      java.lang.reflect.UndeclaredThrowableException

      In ServerManager#sendRegionsMerge we call ProtobufUtil#mergeRegions, which does a doAs, and the code within that block invokes RSRpcServices#mergeRegions, but is not resilient against RegionOpeningException ("region is opening")

      An UndeclaredThrowableException is "thrown by a method invocation on a proxy instance if its invocation handler's invoke method throws a checked exception (a Throwable that is not assignable to RuntimeException or Error) that is not assignable to any of the exception types declared in the throws clause of the method that was invoked on the proxy instance and dispatched to the invocation handler." (http://docs.oracle.com/javase/7/docs/api/java/lang/reflect/UndeclaredThrowableException.html)

      2017-01-31 07:21:17,495 FATAL [MASTER_TABLE_OPERATIONS-node-1:16000-0] master.HMaster: Caught throwable while processing event C_M_MERGE_REGION
      java.lang.reflect.UndeclaredThrowableException
              at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1737)
              at org.apache.hadoop.hbase.protobuf.ProtobufUtil.mergeRegions(ProtobufUtil.java:1990)
              at org.apache.hadoop.hbase.master.ServerManager.sendRegionsMerge(ServerManager.java:925)
              at org.apache.hadoop.hbase.master.handler.DispatchMergingRegionHandler.process(DispatchMergingRegionHandler.java:153)
              at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:129)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)
      Caused by: com.google.protobuf.ServiceException: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.exceptions.RegionOpeningException): org.apache.hadoop.hbase.exceptions.RegionOpeningException: Region IntegrationTestBigLinkedList,|\xFFnk\x1C\x85<[\x1Ef\xFDE\xF9\xAA\xAC\x08,1485846598043.f56ad22121e872777468020c4452a7c7. is opening on node-2.cluster,16020,1485822382322
              at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName(HRegionServer.java:2964)
              at org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegion(RSRpcServices.java:1139)
              at org.apache.hadoop.hbase.regionserver.RSRpcServices.mergeRegions(RSRpcServices.java:1497)
              at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22749)
              at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2355)
              at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123)
              at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188)
              at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168)
      
              at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:244)
              at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:340)
              at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.mergeRegions(AdminProtos.java:23695)
              at org.apache.hadoop.hbase.protobuf.ProtobufUtil$1.run(ProtobufUtil.java:1993)
              at org.apache.hadoop.hbase.protobuf.ProtobufUtil$1.run(ProtobufUtil.java:1990)
              at java.security.AccessController.doPrivileged(Native Method)
              at javax.security.auth.Subject.doAs(Subject.java:422)
              at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1719)
              ... 7 more
      Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.exceptions.RegionOpeningException): org.apache.hadoop.hbase.exceptions.RegionOpeningException: Region IntegrationTestBigLinkedList,|\xFFnk\x1C\x85<[\x1Ef\xFDE\xF9\xAA\xAC\x08,1485846598043.f56ad22121e872777468020c4452a7c7. is opening on node-2.cluster,16020,1485822382322
              at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName(HRegionServer.java:2964)
              at org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegion(RSRpcServices.java:1139)
              at org.apache.hadoop.hbase.regionserver.RSRpcServices.mergeRegions(RSRpcServices.java:1497)
              at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22749)
              at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2355)
              at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123)
              at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188)
              at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168)
      
              at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1286)
              at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:231)
              ... 14 more
      
      

        Attachments

        1. HBASE-17572-branch-1.3.patch
          2 kB
          Andrew Purtell

          Activity

            People

            • Assignee:
              apurtell Andrew Purtell
              Reporter:
              apurtell Andrew Purtell
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: