Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-2322

deadlock between put and cacheflusher in 0.20 branch

VotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Blocker
    • Resolution: Fixed
    • None
    • 0.20.4, 0.90.0
    • None
    • None

    Description

      Found one Java-level deadlock:
      =============================
      "IPC Server handler 59 on 60020":
        waiting for ownable synchronizer 0x00007fec9eb050f8, (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync),
        which is held by "IPC Server handler 54 on 60020"
      "IPC Server handler 54 on 60020":
        waiting to lock monitor 0x000000004190e950 (object 0x00007fec64f25258, a java.util.HashSet),
        which is held by "regionserver/10.20.20.186:60020.cacheFlusher"
      "regionserver/10.20.20.186:60020.cacheFlusher":
        waiting for ownable synchronizer 0x00007fec651df998, (a java.util.concurrent.locks.ReentrantLock$NonfairSync),
        which is held by "IPC Server handler 19 on 60020"
      "IPC Server handler 19 on 60020":
        waiting for ownable synchronizer 0x00007fec9eb050f8, (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync),
        which is held by "IPC Server handler 54 on 60020"
      
      Java stack information for the threads listed above:
      ===================================================
      "IPC Server handler 59 on 60020":
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007fec9eb050f8> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
              at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:747)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:778)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1114)
              at java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.lock(ReentrantReadWriteLock.java:807)
              at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1299)
              at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1281)
              at org.apache.hadoop.hbase.regionserver.HRegionServer.put(HRegionServer.java:1789)
              at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
              at java.lang.reflect.Method.invoke(Method.java:597)
              at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:577)
              at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)
      "IPC Server handler 54 on 60020":
              at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.request(MemStoreFlusher.java:172)
              - waiting to lock <0x00007fec64f25258> (a java.util.HashSet)
              at org.apache.hadoop.hbase.regionserver.HRegion.requestFlush(HRegion.java:1549)
              at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1534)
              at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1318)
              at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1281)
              at org.apache.hadoop.hbase.regionserver.HRegionServer.put(HRegionServer.java:1789)
              at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
              at java.lang.reflect.Method.invoke(Method.java:597)
              at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:577)
              at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)
      "regionserver/10.20.20.186:60020.cacheFlusher":
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007fec651df998> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
              at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:747)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:778)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1114)
              at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:186)
              at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:262)
              at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:235)
              - locked <0x00007fec64f25258> (a java.util.HashSet)
              at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.run(MemStoreFlusher.java:149)
      "IPC Server handler 19 on 60020":
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007fec9eb050f8> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
              at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:747)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:778)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1114)
              at java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.lock(ReentrantReadWriteLock.java:807)
              at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:980)
              at org.apache.hadoop.hbase.regionserver.HRegion.flushcache(HRegion.java:873)
              at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:241)
              at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushSomeRegions(MemStoreFlusher.java:352)
              - locked <0x00007fec64ed96f0> (a org.apache.hadoop.hbase.regionserver.MemStoreFlusher)
              at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.reclaimMemStoreMemory(MemStoreFlusher.java:321)
              - locked <0x00007fec64ed96f0> (a org.apache.hadoop.hbase.regionserver.MemStoreFlusher)
              at org.apache.hadoop.hbase.regionserver.HRegionServer.put(HRegionServer.java:1783)
              at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
              at java.lang.reflect.Method.invoke(Method.java:597)
              at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:577)
              at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)
      
      Found 1 deadlock.
      

      Attachments

        1. hbase-2322.png
          63 kB
          Todd Lipcon

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            stack Michael Stack
            stack Michael Stack
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment