[HBASE-18141] Regionserver fails to shutdown when abort triggered in RegionScannerImpl during RPC call

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 1.3.1
• Fix Version/s: 1.4.0, 1.3.2, 2.0.0, 1.2.7
• Component/s: regionserver, security
• Hadoop Flags: Reviewed
• Assignee: Gary Helmling
• Reporter: Gary Helmling
• Created: 01/Jun/17 00:25
• Updated: 01/Aug/18 06:22
• Resolved: 09/Jun/17 01:45

Description

When an abort is triggered within the RPC call path by HRegion.RegionScannerImpl, AccessController is incorrectly apply the RPC caller identity in the RegionServerObserver.preStopRegionServer() hook. This leaves the regionserver in a non-responsive state, where its regions are not reassigned and it returns exceptions for all requests.

When an abort is triggered on the server side, we should not allow a coprocessor to reject the abort at all.

Here is a sample stack trace:

17/05/25 06:10:29 FATAL regionserver.HRegionServer: RegionServer abort: loaded coprocessors are: [org.apache.hadoop.hbase.security.access.AccessController, org.apache.hadoop.hbase.security.token.TokenProvider]
17/05/25 06:10:29 WARN regionserver.HRegionServer: The region server did not stop
org.apache.hadoop.hbase.security.AccessDeniedException: Insufficient permissions for user 'rpcuser' (global, action=ADMIN)
        at org.apache.hadoop.hbase.security.access.AccessController.requireGlobalPermission(AccessController.java:548)
        at org.apache.hadoop.hbase.security.access.AccessController.requirePermission(AccessController.java:522)
        at org.apache.hadoop.hbase.security.access.AccessController.preStopRegionServer(AccessController.java:2501)
        at org.apache.hadoop.hbase.regionserver.RegionServerCoprocessorHost$1.call(RegionServerCoprocessorHost.java:86)
        at org.apache.hadoop.hbase.regionserver.RegionServerCoprocessorHost.execShutdown(RegionServerCoprocessorHost.java:300)
        at org.apache.hadoop.hbase.regionserver.RegionServerCoprocessorHost.preStop(RegionServerCoprocessorHost.java:82)
        at org.apache.hadoop.hbase.regionserver.HRegionServer.stop(HRegionServer.java:1905)
        at org.apache.hadoop.hbase.regionserver.HRegionServer.abort(HRegionServer.java:2118)
        at org.apache.hadoop.hbase.regionserver.HRegionServer.abort(HRegionServer.java:2125)
        at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.abortRegionServer(HRegion.java:6326)
        at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.handleFileNotFound(HRegion.java:6319)
        at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:5941)
        at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:6084)
        at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:5858)
        at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:2649)
        at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:34950)
        at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2320)
        at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123)
        at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188)
        at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168)

I haven't yet evaluated which other release branches this might apply to.

I have a patch currently in progress, which I will post as soon as I complete a test case.