[ZOOKEEPER-846] zookeeper client doesn't shut down cleanly on the close call

• Type: Bug
• Status: Closed
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 3.2.2
• Fix Version/s: 3.3.2, 3.4.0
• Component/s: java client
• Hadoop Flags: Reviewed
• Assignee: Patrick D. Hunt
• Reporter: Ted Yu
• Created: 12/Aug/10 17:19
• Updated: 23/Nov/11 19:22
• Resolved: 22/Sep/10 06:39

Description

Using HBase 0.20.6 (with HBASE-2473) we encountered a situation where Regionserver
process was shutting down and seemed to hang.

Here is the bottom of region server log:
http://pastebin.com/YYawJ4jA

zookeeper-3.2.2 is used.

Here is relevant portion from jstack - I attempted to attach jstack twice in my email to [e-mail] but failed:

"DestroyJavaVM" prio=10 tid=0x00002aabb849c800 nid=0x6c60 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE

"regionserver/10.32.42.245:60020" prio=10 tid=0x00002aabb84ce000 nid=0x6c81 in Object.wait() [0x0000000043755000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)

• waiting on <0x00002aaab76633c0> (a org.apache.zookeeper.ClientCnxn$Packet)
  at java.lang.Object.wait(Object.java:485)
  at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1099)
• locked <0x00002aaab76633c0> (a org.apache.zookeeper.ClientCnxn$Packet)
  at org.apache.zookeeper.ClientCnxn.close(ClientCnxn.java:1077)
  at org.apache.zookeeper.ZooKeeper.close(ZooKeeper.java:505)
• locked <0x00002aaabf5e0c30> (a org.apache.zookeeper.ZooKeeper)
  at org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.close(ZooKeeperWrapper.java:681)
  at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:654)
  at java.lang.Thread.run(Thread.java:619)

"main-EventThread" daemon prio=10 tid=0x0000000043474000 nid=0x6c80 waiting on condition [0x00000000413f3000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)

• parking to wait for <0x00002aaabf6e9150> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
  at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
  at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
  at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:414)

Issue Links

• is related to: HBASE-2966 HBase client stuck on org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1278) holding regionLockObject lock
• relates to: ZOOKEEPER-126 zookeeper client close operation may block indefinitely