[HBASE-16663] JMX ConnectorServer stopped when unauthorized user try to stop HM/RS/cluster

• Type: Bug
• Status: Closed
• Priority: Critical
• Resolution: Fixed
• Fix Version/s: 1.3.0, 1.2.4, 0.98.24, 1.1.9, 2.0.0
• Component/s: metrics, security
• Assignee: Pankaj Kumar
• Reporter: Pankaj Kumar
• Created: 21/Sep/16 04:19
• Updated: 09/Nov/17 01:41
• Resolved: 22/Dec/16 02:01

Description

After HBASE-16284, unauthorized user will not able allowed to stop HM/RS/cluster, but while executing "cpHost.preStopMaster()", ConnectorServer will be stopped before AccessController validation.

hbase-site.xml,

 <property>
	<name>hbase.coprocessor.master.classes</name>
        <value>org.apache.hadoop.hbase.JMXListener,org.apache.hadoop.hbase.security.access.AccessController</value>
 </property>

  <property>
	<name>hbase.coprocessor.regionserver.classes</name>
        <value>org.apache.hadoop.hbase.JMXListener,org.apache.hadoop.hbase.security.access.AccessController</value>
  </property>

HBaseAdmin.stopMaster(),

2016-09-20 21:12:26,796 INFO  [RpcServer.FifoWFPBQ.priority.handler=19,queue=1,port=16000] hbase.JMXListener: ConnectorServer stopped!
2016-09-20 21:13:55,380 WARN  [RpcServer.FifoWFPBQ.priority.handler=19,queue=1,port=16000] security.ShellBasedUnixGroupsMapping: got exception trying to get groups for user P72981
ExitCodeException exitCode=1: id: P72981: No such user
2016-09-20 21:14:00,495 ERROR [RpcServer.FifoWFPBQ.priority.handler=19,queue=1,port=16000] master.MasterRpcServices: Exception occurred while stopping master
org.apache.hadoop.hbase.security.AccessDeniedException: Insufficient permissions for user 'P72981' (global, action=ADMIN)
	at org.apache.hadoop.hbase.security.access.AccessController.requireGlobalPermission(AccessController.java:546)
	at org.apache.hadoop.hbase.security.access.AccessController.requirePermission(AccessController.java:522)
	at org.apache.hadoop.hbase.security.access.AccessController.preStopMaster(AccessController.java:1297)
	at org.apache.hadoop.hbase.master.MasterCoprocessorHost$68.call(MasterCoprocessorHost.java:821)
	at org.apache.hadoop.hbase.master.MasterCoprocessorHost.execOperation(MasterCoprocessorHost.java:1188)
	at org.apache.hadoop.hbase.master.MasterCoprocessorHost.preStopMaster(MasterCoprocessorHost.java:817)
	at org.apache.hadoop.hbase.master.HMaster.stopMaster(HMaster.java:2352)
	at org.apache.hadoop.hbase.master.MasterRpcServices.stopMaster(MasterRpcServices.java:1364)

HBaseAdmin.stopRegionServer(rs-host-port),

2016-09-20 20:59:01,234 INFO  [RpcServer.FifoWFPBQ.priority.handler=18,queue=0,port=16020] hbase.JMXListener: ConnectorServer stopped!
2016-09-20 20:59:01,250 WARN  [RpcServer.FifoWFPBQ.priority.handler=18,queue=0,port=16020] security.ShellBasedUnixGroupsMapping: got exception trying to get groups for user P72981
ExitCodeException exitCode=1: id: P72981: No such user
2016-09-20 20:59:01,253 WARN  [RpcServer.FifoWFPBQ.priority.handler=18,queue=0,port=16020] regionserver.HRegionServer: The region server did not stop
org.apache.hadoop.hbase.security.AccessDeniedException: Insufficient permissions for user 'P72981' (global, action=ADMIN)
	at org.apache.hadoop.hbase.security.access.AccessController.requireGlobalPermission(AccessController.java:546)
	at org.apache.hadoop.hbase.security.access.AccessController.requirePermission(AccessController.java:522)
	at org.apache.hadoop.hbase.security.access.AccessController.preStopRegionServer(AccessController.java:2501)
	at org.apache.hadoop.hbase.regionserver.RegionServerCoprocessorHost$1.call(RegionServerCoprocessorHost.java:84)
	at org.apache.hadoop.hbase.regionserver.RegionServerCoprocessorHost.execOperation(RegionServerCoprocessorHost.java:256)
	at org.apache.hadoop.hbase.regionserver.RegionServerCoprocessorHost.preStop(RegionServerCoprocessorHost.java:80)
	at org.apache.hadoop.hbase.regionserver.HRegionServer.stop(HRegionServer.java:1905)
	at org.apache.hadoop.hbase.regionserver.RSRpcServices.stopServer(RSRpcServices.java:1961)

HBaseAdmin.shutdown(),

2016-09-21 12:09:08,259 INFO  [RpcServer.FifoWFPBQ.priority.handler=19,queue=1,port=16000] master.MasterRpcServices: Client=P72981//10.18.248.96 shutdown
2016-09-21 12:09:08,261 INFO  [RpcServer.FifoWFPBQ.priority.handler=19,queue=1,port=16000] hbase.JMXListener: ConnectorServer stopped!
2016-09-21 12:09:08,276 WARN  [RpcServer.FifoWFPBQ.priority.handler=19,queue=1,port=16000] security.ShellBasedUnixGroupsMapping: got exception trying to get groups for user P72981
ExitCodeException exitCode=1: id: P72981: No such user
2016-09-21 12:09:08,280 ERROR [RpcServer.FifoWFPBQ.priority.handler=19,queue=1,port=16000] master.MasterRpcServices: Exception occurred in HMaster.shutdown()
org.apache.hadoop.hbase.security.AccessDeniedException: Insufficient permissions for user 'P72981' (global, action=ADMIN)
	at org.apache.hadoop.hbase.security.access.AccessController.requireGlobalPermission(AccessController.java:546)
	at org.apache.hadoop.hbase.security.access.AccessController.requirePermission(AccessController.java:522)
	at org.apache.hadoop.hbase.security.access.AccessController.preShutdown(AccessController.java:1291)
	at org.apache.hadoop.hbase.master.MasterCoprocessorHost$67.call(MasterCoprocessorHost.java:806)
	at org.apache.hadoop.hbase.master.MasterCoprocessorHost.execOperation(MasterCoprocessorHost.java:1188)
	at org.apache.hadoop.hbase.master.MasterCoprocessorHost.preShutdown(MasterCoprocessorHost.java:802)
	at org.apache.hadoop.hbase.master.HMaster.shutdown(HMaster.java:2335)
	at org.apache.hadoop.hbase.master.MasterRpcServices.shutdown(MasterRpcServices.java:1322)
	at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:58551)
	at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270)
	at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123)
	at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188)
	at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168)

Issue Links

• Dependent: HBASE-16723 RMI registry is not destroyed after stopping JMX Connector Server