[HBASE-22079] master leaks ZK on shutdown and gets stuck because of netty threads if netty socket is used

• Type: Bug
• Status: Patch Available
• Priority: Major
• Resolution: Unresolved
• Assignee: Sergey Shelukhin
• Reporter: Sergey Shelukhin
• Created: 21/Mar/19 23:21
• Updated: 26/Mar/19 06:56

Description

"master/...:17000:becomeActiveMaster-SendThread(...1)" #311 daemon prio=5 os_prio=0 tid=0x0000000058c61800 nid=0x2dd0 waiting on condition [0x0000000c477fe000]
   java.lang.Thread.State: TIMED_WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x00000000c4a5b3c0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
	at java.util.concurrent.LinkedBlockingDeque.pollFirst(LinkedBlockingDeque.java:522)
	at java.util.concurrent.LinkedBlockingDeque.poll(LinkedBlockingDeque.java:684)
	at org.apache.zookeeper.ClientCnxnSocketNetty.doTransport(ClientCnxnSocketNetty.java:232)
	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1146)

This causes a bunch of netty threads to also leak it looks like, and these are not daemon (by design, apparently)

Issue Links

• is broken by: HBASE-20159 Support using separate ZK quorums for client