[HBASE-3830] MemStoreFlusher deadlock detected by JVM

• Type: Bug
• Status: Closed
• Priority: Trivial
• Resolution: Duplicate
• Affects Version/s: 0.90.1
• Component/s: regionserver
• Assignee: Unassigned
• Reporter: zhoushuaifeng
• Created: 29/Apr/11 07:40
• Updated: 12/Jun/22 18:19
• Resolved: 21/Mar/12 04:51

Description

Found one Java-level deadlock:
=============================
"IPC Server handler 9 on 60020":
waiting to lock monitor 0x00000000409f3908 (object 0x00007fe7cbacbd48, a org.apache.hadoop.hbase.regionserver.MemStoreFlusher),
which is held by "IPC Server handler 7 on 60020"
"IPC Server handler 7 on 60020":
waiting for ownable synchronizer 0x00007fe7cbb06228, (a java.util.concurrent.locks.ReentrantLock$NonfairSync),
which is held by "regionserver60020.cacheFlusher"
"regionserver60020.cacheFlusher":
waiting to lock monitor 0x00000000409f3908 (object 0x00007fe7cbacbd48, a org.apache.hadoop.hbase.regionserver.MemStoreFlusher),
which is held by "IPC Server handler 7 on 60020"

Java stack information for the threads listed above:
===================================================
"IPC Server handler 9 on 60020":
at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.reclaimMemStoreMemory(MemStoreFlusher.java)

• waiting to lock <0x00007fe7cbacbd48> (a org.apache.hadoop.hbase.regionserver.MemStoreFlusher)
  at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:2558)
  at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  at java.lang.reflect.Method.invoke(Method.java:597)
  at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
  at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1039)
  "IPC Server handler 7 on 60020":
  at sun.misc.Unsafe.$$YJP$$park(Native Method)
• parking to wait for <0x00007fe7cbb06228> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
  at sun.misc.Unsafe.park(Unsafe.java)
  at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
  at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:747)
  at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:778)
  at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1114)
  at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:186)
  at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:262)
  at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.reclaimMemStoreMemory(MemStoreFlusher.java:429)
• locked <0x00007fe7cbacbd48> (a org.apache.hadoop.hbase.regionserver.MemStoreFlusher)
  at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:2558)
  at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  at java.lang.reflect.Method.invoke(Method.java:597)
  at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
  at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1039)
  "regionserver60020.cacheFlusher":
  at java.util.ResourceBundle.endLoading(ResourceBundle.java:1506)
• waiting to lock <0x00007fe7cbacbd48> (a org.apache.hadoop.hbase.regionserver.MemStoreFlusher)
  at java.util.ResourceBundle.findBundle(ResourceBundle.java:1379)
  at java.util.ResourceBundle.findBundle(ResourceBundle.java:1292)
  at java.util.ResourceBundle.getBundleImpl(ResourceBundle.java:1234)
  at java.util.ResourceBundle.getBundle(ResourceBundle.java:832)
  at sun.util.resources.LocaleData$1.run(LocaleData.java:127)
  at java.security.AccessController.$$YJP$$doPrivileged(Native Method)
  at java.security.AccessController.doPrivileged(AccessController.java)
  at sun.util.resources.LocaleData.getBundle(LocaleData.java:125)
  at sun.util.resources.LocaleData.getTimeZoneNames(LocaleData.java:97)
  at sun.util.TimeZoneNameUtility.getBundle(TimeZoneNameUtility.java:115)
  at sun.util.TimeZoneNameUtility.retrieveDisplayNames(TimeZoneNameUtility.java:80)
  at java.util.TimeZone.getDisplayNames(TimeZone.java:399)
  at java.util.TimeZone.getDisplayName(TimeZone.java:350)
  at java.util.Date.toString(Date.java:1025)
  at java.lang.String.valueOf(String.java:2826)
  at java.lang.StringBuilder.append(StringBuilder.java:115)
  at org.apache.hadoop.hbase.regionserver.PriorityCompactionQueue$CompactionRequest.toString(PriorityCompactionQueue.java:114)
  at java.lang.String.valueOf(String.java:2826)
  at java.lang.StringBuilder.append(StringBuilder.java:115)
  at org.apache.hadoop.hbase.regionserver.PriorityCompactionQueue.addToRegionsInQueue(PriorityCompactionQueue.java:145)
• locked <0x00007fe7ccabd258> (a java.util.HashMap)
  at org.apache.hadoop.hbase.regionserver.PriorityCompactionQueue.add(PriorityCompactionQueue.java:188)
  at org.apache.hadoop.hbase.regionserver.CompactSplitThread.requestCompaction(CompactSplitThread.java:140)
• locked <0x00007fe7cbaf08c8> (a org.apache.hadoop.hbase.regionserver.CompactSplitThread)
  at org.apache.hadoop.hbase.regionserver.CompactSplitThread.requestCompaction(CompactSplitThread.java:118)
• locked <0x00007fe7cbaf08c8> (a org.apache.hadoop.hbase.regionserver.CompactSplitThread)
  at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:387)
  at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushOneForGlobalPressure(MemStoreFlusher.java:194)
  at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.run(MemStoreFlusher.java:214)

Found 1 deadlock.