[HBASE-12185] Deadlock in HConnectionManager$HConnectionImplementation

• Type: Bug
• Status: Closed
• Priority: Critical
• Resolution: Duplicate
• Affects Version/s: 0.94.2
• Component/s: Client

Environment

CDH 4.2.0

• Assignee: Unassigned
• Reporter: Michael Tamm
• Created: 07/Oct/14 09:52
• Updated: 17/Jun/22 17:34
• Resolved: 17/Jun/22 17:34

Description

Here you can see the relevant section of a thread dump:

Found one Java-level deadlock:
=============================
"AsyncSave-700512-Worker-EventThread":
  waiting to lock monitor 0x00007f8d90eecd20 (object 0x00000005c0a8e1d0, a org.apache.hadoop.hbase.zookeeper.RootRegionTracker),
  which is held by "AsyncSave-700546-Worker"
"AsyncSave-700546-Worker":
  waiting to lock monitor 0x00007f8d90149700 (object 0x0000000571404180, a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation),
  which is held by "AsyncSave-700512-Worker-EventThread"

Java stack information for the threads listed above:
===================================================
"AsyncSave-700512-Worker-EventThread":
	at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.stop(ZooKeeperNodeTracker.java:98)
	- waiting to lock <0x00000005c0a8e1d0> (a org.apache.hadoop.hbase.zookeeper.RootRegionTracker)
	at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.resetZooKeeperTrackers(HConnectionManager.java:603)
	- locked <0x0000000571404180> (a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation)
	at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.abort(HConnectionManager.java:1681)
	at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.connectionEvent(ZooKeeperWatcher.java:389)
	at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:286)
	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
"AsyncSave-700546-Worker":
	at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.resetZooKeeperTrackers(HConnectionManager.java:598)
	- waiting to lock <0x0000000571404180> (a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation)
	at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.abort(HConnectionManager.java:1681)
	at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.blockUntilAvailable(ZooKeeperNodeTracker.java:132)
	- locked <0x00000005c0a8e1d0> (a org.apache.hadoop.hbase.zookeeper.RootRegionTracker)
	at org.apache.hadoop.hbase.zookeeper.RootRegionTracker.waitRootRegionLocation(RootRegionTracker.java:83)
	at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:841)
	at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:954)
	at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:852)
	at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:954)
	at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:856)
	at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:813)
	at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.processBatchCallback(HConnectionManager.java:1503)
	at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.processBatch(HConnectionManager.java:1388)
	at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:955)
	at org.apache.hadoop.hbase.client.HTablePool$PooledHTable.flushCommits(HTablePool.java:449)
	at ...