Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-7728

deadlock occurs between hlog roller and hlog syncer

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: 0.94.2
    • Fix Version/s: 0.94.5, 0.95.0
    • Component/s: wal
    • Labels:
      None
    • Environment:

      Linux 2.6.18-164.el5 x86_64 GNU/Linux

    • Hadoop Flags:
      Reviewed

      Description

      the hlog roller thread and hlog syncer thread may occur dead lock with the 'flushLock' and 'updateLock', and then cause all 'IPC Server handler' thread blocked on hlog append. the jstack info is as follow :
      "regionserver60020.logRoller":
      at org.apache.hadoop.hbase.regionserver.wal.HLog.syncer(HLog.java:1305)

      • waiting to lock <0x000000067bf88d58> (a java.lang.Object)
        at org.apache.hadoop.hbase.regionserver.wal.HLog.syncer(HLog.java:1283)
        at org.apache.hadoop.hbase.regionserver.wal.HLog.sync(HLog.java:1456)
        at org.apache.hadoop.hbase.regionserver.wal.HLog.cleanupCurrentWriter(HLog.java:876)
        at org.apache.hadoop.hbase.regionserver.wal.HLog.rollWriter(HLog.java:657)
      • locked <0x000000067d54ace0> (a java.lang.Object)
        at org.apache.hadoop.hbase.regionserver.LogRoller.run(LogRoller.java:94)
        at java.lang.Thread.run(Thread.java:662)
        "regionserver60020.logSyncer":
        at org.apache.hadoop.hbase.regionserver.wal.HLog.syncer(HLog.java:1314)
      • waiting to lock <0x000000067d54ace0> (a java.lang.Object)
      • locked <0x000000067bf88d58> (a java.lang.Object)
        at org.apache.hadoop.hbase.regionserver.wal.HLog.syncer(HLog.java:1283)
        at org.apache.hadoop.hbase.regionserver.wal.HLog.sync(HLog.java:1456)
        at org.apache.hadoop.hbase.regionserver.wal.HLog$LogSyncer.run(HLog.java:1235)
        at java.lang.Thread.run(Thread.java:662)

        Attachments

        1. 7728-v1.txt
          0.9 kB
          Ted Yu
        2. 7728-suggest.txt
          2 kB
          Lars Hofhansl
        3. 7728-suggest-0.96.txt
          2 kB
          Lars Hofhansl
        4. 7728-v2.txt
          2 kB
          Ted Yu
        5. 7728-v3.txt
          2 kB
          Ted Yu
        6. 7728-0.94.txt
          2 kB
          Ted Yu
        7. 7728-v4.txt
          2 kB
          Ted Yu
        8. 7728-0.94-v2.txt
          2 kB
          Ted Yu
        9. 7728-0.94-simplified.txt
          2 kB
          Ted Yu

          Activity

            People

            • Assignee:
              yuzhihong@gmail.com Ted Yu
              Reporter:
              aaronwq Wang Qiang
            • Votes:
              0 Vote for this issue
              Watchers:
              15 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: