[HBASE-864] Deadlock in regionserver

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.2.1, 0.18.0
• Assignee: Unassigned
• Reporter: Michael Stack
• Created: 03/Sep/08 17:30
• Updated: 13/Sep/08 23:22
• Resolved: 03/Sep/08 18:17

Description

Found in 0.2.0 by Michael Bieniosk:

                                                                                                                                                                                                                                                                                                                            Found one Java-level deadlock:============================="IPC Server handler 9 on 60020":  waiting to lock monitor 0x00000000405b6688 (object 0x00007f328f64f260, a org.apache.hadoop.hbase.regionserver.HRegion$WriteState),  which is held by "IPC Server handler 3 on 60020""IPC Server handler 3 on 60020":  waiting to lock monitor 0x00000000403d05a0 (object 0x00007f328ce1e988, a java.util.HashSet),  which is held by "regionserver/0:0:0:0:0:0:0:0:60020.cacheFlusher""regionserver/0:0:0:0:0:0:0:0:60020.cacheFlusher":  waiting to lock monitor 0x00000000405b6688 (object 0x00007f328f64f260, a org.apache.hadoop.hbase.regionserver.HRegion$WriteState),
  which is held by "IPC Server handler 3 on 60020"

Java stack information for the threads listed above:
===================================================
"IPC Server handler 9 on 60020":
        at org.apache.hadoop.hbase.regionserver.HRegion.requestFlush(HRegion.java:1623)
        - waiting to lock <0x00007f328f64f260> (a org.apache.hadoop.hbase.regionserver.HRegion$WriteState)
        at org.apache.hadoop.hbase.regionserver.HRegion.update(HRegion.java:1614)
        at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1398)
        at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1151)
        at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
        at java.lang.reflect.Method.invoke(Unknown Source)
        at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:473)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
"IPC Server handler 3 on 60020":
        at org.apache.hadoop.hbase.regionserver.Flusher.addRegion(Flusher.java:237)
        - waiting to lock <0x00007f328ce1e988> (a java.util.HashSet)
        at org.apache.hadoop.hbase.regionserver.Flusher.request(Flusher.java:114)
        at org.apache.hadoop.hbase.regionserver.HRegion.requestFlush(HRegion.java:1627)
        - locked <0x00007f328f64f260> (a org.apache.hadoop.hbase.regionserver.HRegion$WriteState)
        at org.apache.hadoop.hbase.regionserver.HRegion.update(HRegion.java:1614)
        at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1398)
        at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1151)
        at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
        at java.lang.reflect.Method.invoke(Unknown Source)
        at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:473)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
"regionserver/0:0:0:0:0:0:0:0:60020.cacheFlusher":
        at org.apache.hadoop.hbase.regionserver.HRegion.flushcache(HRegion.java:948)
        - waiting to lock <0x00007f328f64f260> (a org.apache.hadoop.hbase.regionserver.HRegion$WriteState)
        at org.apache.hadoop.hbase.regionserver.Flusher.flushRegion(Flusher.java:173)
        - locked <0x00007f328ce1e988> (a java.util.HashSet)
        at org.apache.hadoop.hbase.regionserver.Flusher.run(Flusher.java:91)

Found 1 deadlock.