Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-818

Deadlock running 'flushSomeRegions'

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Blocker
    • Resolution: Fixed
    • 0.2.0
    • 0.2.1, 0.18.0
    • None
    • None

    Description

      Playing with MR uploading no a regionserver with 60+ regions, I ran into a deadlock:

      Found one Java-level deadlock:
      =============================
      "IPC Server handler 19 on 60020":
        waiting to lock monitor 0x084be38c (object 0xb6f69a70, a org.apache.hadoop.hbase.regionserver.Flusher),
        which is held by "IPC Server handler 16 on 60020"
      "IPC Server handler 16 on 60020":
        waiting to lock monitor 0x080f8dec (object 0xb73610c0, a org.apache.hadoop.hbase.regionserver.HRegion$WriteState),
        which is held by "IPC Server handler 2 on 60020"
      "IPC Server handler 2 on 60020":
        waiting to lock monitor 0x086e8fe8 (object 0xb6f69cf0, a java.util.HashSet),
        which is held by "IPC Server handler 16 on 60020"
      
      Java stack information for the threads listed above:
      ===================================================
      "IPC Server handler 19 on 60020":
              at org.apache.hadoop.hbase.regionserver.Flusher.flushSomeRegions(Flusher.java:261)
              - waiting to lock <0xb6f69a70> (a org.apache.hadoop.hbase.regionserver.Flusher)
              at org.apache.hadoop.hbase.regionserver.Flusher.reclaimMemcacheMemory(Flusher.java:252)
              at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1136)
              at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
              at java.lang.reflect.Method.invoke(Method.java:623)
              at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:473)
              at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
      "IPC Server handler 16 on 60020":
              at org.apache.hadoop.hbase.regionserver.HRegion.flushcache(HRegion.java:948)
              - waiting to lock <0xb73610c0> (a org.apache.hadoop.hbase.regionserver.HRegion$WriteState)
              at org.apache.hadoop.hbase.regionserver.Flusher.flushRegion(Flusher.java:173)
              - locked <0xb6f69cf0> (a java.util.HashSet)
              at org.apache.hadoop.hbase.regionserver.Flusher.flushSomeRegions(Flusher.java:267)
              - locked <0xb6f69a70> (a org.apache.hadoop.hbase.regionserver.Flusher)
              at org.apache.hadoop.hbase.regionserver.Flusher.reclaimMemcacheMemory(Flusher.java:252)
              at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1136)
              at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
              at java.lang.reflect.Method.invoke(Method.java:623)
              at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:473)
              at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
      "IPC Server handler 2 on 60020":
              at org.apache.hadoop.hbase.regionserver.Flusher.addRegion(Flusher.java:237)
              - waiting to lock <0xb6f69cf0> (a java.util.HashSet)
              at org.apache.hadoop.hbase.regionserver.Flusher.request(Flusher.java:114)
              at org.apache.hadoop.hbase.regionserver.HRegion.requestFlush(HRegion.java:1627)
              - locked <0xb73610c0> (a org.apache.hadoop.hbase.regionserver.HRegion$WriteState)
              at org.apache.hadoop.hbase.regionserver.HRegion.update(HRegion.java:1614)
              at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1398)
              at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1137)
              at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
              at java.lang.reflect.Method.invoke(Method.java:623)
              at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:473)
              at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
      
      Found 1 deadlock.
      

      Regionserver is hosed.

      Attachments

        1. 818.patch
          0.5 kB
          Andrew Kyle Purtell

        Activity

          People

            apurtell Andrew Kyle Purtell
            stack Michael Stack
            Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: