HBase
  1. HBase
  2. HBASE-7728

deadlock occurs between hlog roller and hlog syncer

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: 0.94.2
    • Fix Version/s: 0.94.5, 0.95.0
    • Component/s: wal
    • Labels:
      None
    • Environment:

      Linux 2.6.18-164.el5 x86_64 GNU/Linux

    • Hadoop Flags:
      Reviewed

      Description

      the hlog roller thread and hlog syncer thread may occur dead lock with the 'flushLock' and 'updateLock', and then cause all 'IPC Server handler' thread blocked on hlog append. the jstack info is as follow :
      "regionserver60020.logRoller":
      at org.apache.hadoop.hbase.regionserver.wal.HLog.syncer(HLog.java:1305)

      • waiting to lock <0x000000067bf88d58> (a java.lang.Object)
        at org.apache.hadoop.hbase.regionserver.wal.HLog.syncer(HLog.java:1283)
        at org.apache.hadoop.hbase.regionserver.wal.HLog.sync(HLog.java:1456)
        at org.apache.hadoop.hbase.regionserver.wal.HLog.cleanupCurrentWriter(HLog.java:876)
        at org.apache.hadoop.hbase.regionserver.wal.HLog.rollWriter(HLog.java:657)
      • locked <0x000000067d54ace0> (a java.lang.Object)
        at org.apache.hadoop.hbase.regionserver.LogRoller.run(LogRoller.java:94)
        at java.lang.Thread.run(Thread.java:662)
        "regionserver60020.logSyncer":
        at org.apache.hadoop.hbase.regionserver.wal.HLog.syncer(HLog.java:1314)
      • waiting to lock <0x000000067d54ace0> (a java.lang.Object)
      • locked <0x000000067bf88d58> (a java.lang.Object)
        at org.apache.hadoop.hbase.regionserver.wal.HLog.syncer(HLog.java:1283)
        at org.apache.hadoop.hbase.regionserver.wal.HLog.sync(HLog.java:1456)
        at org.apache.hadoop.hbase.regionserver.wal.HLog$LogSyncer.run(HLog.java:1235)
        at java.lang.Thread.run(Thread.java:662)
      1. 7728-0.94.txt
        2 kB
        Ted Yu
      2. 7728-0.94-simplified.txt
        2 kB
        Ted Yu
      3. 7728-0.94-v2.txt
        2 kB
        Ted Yu
      4. 7728-suggest.txt
        2 kB
        Lars Hofhansl
      5. 7728-suggest-0.96.txt
        2 kB
        Lars Hofhansl
      6. 7728-v1.txt
        0.9 kB
        Ted Yu
      7. 7728-v2.txt
        2 kB
        Ted Yu
      8. 7728-v3.txt
        2 kB
        Ted Yu
      9. 7728-v4.txt
        2 kB
        Ted Yu

        Activity

        No work has yet been logged on this issue.

          People

          • Assignee:
            Ted Yu
            Reporter:
            Wang Qiang
          • Votes:
            0 Vote for this issue
            Watchers:
            17 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development