Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-18141

Regionserver fails to shutdown when abort triggered in RegionScannerImpl during RPC call

    XMLWordPrintableJSON

    Details

    • Hadoop Flags:
      Reviewed

      Description

      When an abort is triggered within the RPC call path by HRegion.RegionScannerImpl, AccessController is incorrectly apply the RPC caller identity in the RegionServerObserver.preStopRegionServer() hook. This leaves the regionserver in a non-responsive state, where its regions are not reassigned and it returns exceptions for all requests.

      When an abort is triggered on the server side, we should not allow a coprocessor to reject the abort at all.

      Here is a sample stack trace:

      17/05/25 06:10:29 FATAL regionserver.HRegionServer: RegionServer abort: loaded coprocessors are: [org.apache.hadoop.hbase.security.access.AccessController, org.apache.hadoop.hbase.security.token.TokenProvider]
      17/05/25 06:10:29 WARN regionserver.HRegionServer: The region server did not stop
      org.apache.hadoop.hbase.security.AccessDeniedException: Insufficient permissions for user 'rpcuser' (global, action=ADMIN)
              at org.apache.hadoop.hbase.security.access.AccessController.requireGlobalPermission(AccessController.java:548)
              at org.apache.hadoop.hbase.security.access.AccessController.requirePermission(AccessController.java:522)
              at org.apache.hadoop.hbase.security.access.AccessController.preStopRegionServer(AccessController.java:2501)
              at org.apache.hadoop.hbase.regionserver.RegionServerCoprocessorHost$1.call(RegionServerCoprocessorHost.java:86)
              at org.apache.hadoop.hbase.regionserver.RegionServerCoprocessorHost.execShutdown(RegionServerCoprocessorHost.java:300)
              at org.apache.hadoop.hbase.regionserver.RegionServerCoprocessorHost.preStop(RegionServerCoprocessorHost.java:82)
              at org.apache.hadoop.hbase.regionserver.HRegionServer.stop(HRegionServer.java:1905)
              at org.apache.hadoop.hbase.regionserver.HRegionServer.abort(HRegionServer.java:2118)
              at org.apache.hadoop.hbase.regionserver.HRegionServer.abort(HRegionServer.java:2125)
              at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.abortRegionServer(HRegion.java:6326)
              at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.handleFileNotFound(HRegion.java:6319)
              at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:5941)
              at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:6084)
              at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:5858)
              at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:2649)
              at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:34950)
              at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2320)
              at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123)
              at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188)
              at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168)
      

      I haven't yet evaluated which other release branches this might apply to.

      I have a patch currently in progress, which I will post as soon as I complete a test case.

        Attachments

        1. HBASE-18141.001.patch
          14 kB
          Gary Helmling
        2. HBASE-18141.branch-1.3.001.patch
          11 kB
          Gary Helmling
        3. HBASE-18141.branch-1.3.002.patch
          12 kB
          Gary Helmling

          Activity

            People

            • Assignee:
              ghelmling Gary Helmling
              Reporter:
              ghelmling Gary Helmling
            • Votes:
              0 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: