[HBASE-16773] AccessController should access local region if possible

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.4.0, 1.3.1, 2.0.0
• Hadoop Flags: Reviewed
• Assignee: Ted Yu
• Reporter: Ted Yu
• Created: 05/Oct/16 19:30
• Updated: 09/Nov/17 19:10
• Resolved: 09/Oct/16 13:42

Description

We observed the following in the stack trace of region server on a 1.1.2 cluster:

"PriorityRpcServer.handler=19,queue=1,port=60200" #225 daemon prio=5 os_prio=0 tid=0x00007fb562296000 nid=0x81c0 runnable [0x00007fb509a27000]
   java.lang.Thread.State: RUNNABLE
  at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
  at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
  at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
  - locked <0x00000003d4dfd770> (a sun.nio.ch.Util$2)
  - locked <0x00000003d4dfd760> (a java.util.Collections$UnmodifiableSet)
  - locked <0x00000003d4dfd648> (a sun.nio.ch.EPollSelectorImpl)
  at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
  at org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335)
  at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157)
  at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161)
  at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131)
  at java.io.FilterInputStream.read(FilterInputStream.java:133)
  at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
  at java.io.BufferedInputStream.read(BufferedInputStream.java:265)
  - locked <0x00000003d7dae180> (a java.io.BufferedInputStream)
  at java.io.DataInputStream.readInt(DataInputStream.java:387)
  at org.apache.hadoop.hbase.security.HBaseSaslRpcClient.readStatus(HBaseSaslRpcClient.java:151)
  at org.apache.hadoop.hbase.security.HBaseSaslRpcClient.saslConnect(HBaseSaslRpcClient.java:189)
  at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.setupSaslConnection(RpcClientImpl.java:611)
  - locked <0x00000003d5c7edc0> (a org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection)
  at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.access$600(RpcClientImpl.java:156)
  at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection$2.run(RpcClientImpl.java:737)
  at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection$2.run(RpcClientImpl.java:734)
  at java.security.AccessController.doPrivileged(Native Method)
  at javax.security.auth.Subject.doAs(Subject.java:422)
  at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724)
  at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.setupIOstreams(RpcClientImpl.java:734)
  - locked <0x00000003d5c7edc0> (a org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection)
  at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.writeRequest(RpcClientImpl.java:887)
  at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.tracedWriteRequest(RpcClientImpl.java:856)
  at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1199)
  at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:213)
  at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:287)
  at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$BlockingStub.get(ClientProtos.java:32627)
  at org.apache.hadoop.hbase.client.HTable$3.call(HTable.java:854)
  at org.apache.hadoop.hbase.client.HTable$3.call(HTable.java:845)
  at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:126)
  at org.apache.hadoop.hbase.client.HTable.get(HTable.java:862)
  at org.apache.hadoop.hbase.client.HTable.get(HTable.java:828)
  at org.apache.hadoop.hbase.security.access.AccessControlLists.getPermissions(AccessControlLists.java:461)
  at org.apache.hadoop.hbase.security.access.AccessController.updateACL(AccessController.java:260)
  at org.apache.hadoop.hbase.security.access.AccessController.postPut(AccessController.java:1661)
  at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$32.call(RegionCoprocessorHost.java:940)
  at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionOperation.call(RegionCoprocessorHost.java:1673)
  at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1748)
  at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1705)
  at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.postPut(RegionCoprocessorHost.java:936)
  at org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:3287)
  at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2902)
  at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2844)

There were 20 threads stuck in the retrieval of permissions.

AccessController shouldn't use Connection if getPermissions() can be satisfied by accessing local hbase:acl region.