Caught while testing branch-1.0, shutting down TestMasterMetricsWrapper.
Found one Java-level deadlock:
waiting to lock monitor 0x00007f2a040051c8 (object 0x00000007e36108a8, a org.apache.hadoop.hbase.util.PoolMap),
which is held by "M:0;ip-10-32-130-237:55342"
waiting to lock monitor 0x00007f2a04005118 (object 0x00000007e3610b00, a org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection),
which is held by "MASTER_META_SERVER_OPERATIONS-ip-10-32-130-237:55342-0"
Full stack dump and deadlock debug output attached.
In RpcClientImpl#close(), we obtain lock on connections first:
Then markClosed() tries to obtain lock on connection object:
Another thread, MetaServerShutdownHandler, calls RpcClientImpl$Connection#setupIOstreams() where :
Lock on connection object is obtained first, then lock on connections is attempted, leading to deadlock: