[HBASE-2322] deadlock between put and cacheflusher in 0.20 branch

• Type: Bug
• Status: Closed
• Priority: Blocker
• Resolution: Fixed
• Fix Version/s: 0.20.4, 0.90.0
• Assignee: Michael Stack
• Reporter: Michael Stack
• Created: 14/Mar/10 01:07
• Updated: 12/Oct/12 06:15
• Resolved: 16/Apr/10 01:02

Description

Found one Java-level deadlock:
=============================
"IPC Server handler 59 on 60020":
  waiting for ownable synchronizer 0x00007fec9eb050f8, (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync),
  which is held by "IPC Server handler 54 on 60020"
"IPC Server handler 54 on 60020":
  waiting to lock monitor 0x000000004190e950 (object 0x00007fec64f25258, a java.util.HashSet),
  which is held by "regionserver/10.20.20.186:60020.cacheFlusher"
"regionserver/10.20.20.186:60020.cacheFlusher":
  waiting for ownable synchronizer 0x00007fec651df998, (a java.util.concurrent.locks.ReentrantLock$NonfairSync),
  which is held by "IPC Server handler 19 on 60020"
"IPC Server handler 19 on 60020":
  waiting for ownable synchronizer 0x00007fec9eb050f8, (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync),
  which is held by "IPC Server handler 54 on 60020"

Java stack information for the threads listed above:
===================================================
"IPC Server handler 59 on 60020":
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00007fec9eb050f8> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:747)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:778)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1114)
        at java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.lock(ReentrantReadWriteLock.java:807)
        at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1299)
        at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1281)
        at org.apache.hadoop.hbase.regionserver.HRegionServer.put(HRegionServer.java:1789)
        at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:577)
        at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)
"IPC Server handler 54 on 60020":
        at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.request(MemStoreFlusher.java:172)
        - waiting to lock <0x00007fec64f25258> (a java.util.HashSet)
        at org.apache.hadoop.hbase.regionserver.HRegion.requestFlush(HRegion.java:1549)
        at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1534)
        at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1318)
        at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1281)
        at org.apache.hadoop.hbase.regionserver.HRegionServer.put(HRegionServer.java:1789)
        at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:577)
        at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)
"regionserver/10.20.20.186:60020.cacheFlusher":
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00007fec651df998> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:747)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:778)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1114)
        at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:186)
        at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:262)
        at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:235)
        - locked <0x00007fec64f25258> (a java.util.HashSet)
        at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.run(MemStoreFlusher.java:149)
"IPC Server handler 19 on 60020":
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00007fec9eb050f8> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:747)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:778)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1114)
        at java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.lock(ReentrantReadWriteLock.java:807)
        at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:980)
        at org.apache.hadoop.hbase.regionserver.HRegion.flushcache(HRegion.java:873)
        at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:241)
        at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushSomeRegions(MemStoreFlusher.java:352)
        - locked <0x00007fec64ed96f0> (a org.apache.hadoop.hbase.regionserver.MemStoreFlusher)
        at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.reclaimMemStoreMemory(MemStoreFlusher.java:321)
        - locked <0x00007fec64ed96f0> (a org.apache.hadoop.hbase.regionserver.MemStoreFlusher)
        at org.apache.hadoop.hbase.regionserver.HRegionServer.put(HRegionServer.java:1783)
        at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:577)
        at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)

Found 1 deadlock.

Issue Links

• is related to: HBASE-2248 Provide new non-copy mechanism to assure atomic reads in get and scan