Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-23287

WALs not aged off of HDFS because LogCleaner is not added to choreService

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: 1.5.0, 1.6.0, 1.4.11
    • Fix Version/s: 1.6.0, 1.4.12
    • Component/s: wal
    • Labels:
      None
    • Release Note:
      Hide
      <!-- markdown -->

      Prior to this change HBase 1.4.11 and 1.5.0 based clusters would fail to clean up WALs from the oldWALs area automatically. With this change in place clusters should return to the expected behavior of cleaning up WALs that are no longer needed.
      Show
      <!-- markdown --> Prior to this change HBase 1.4.11 and 1.5.0 based clusters would fail to clean up WALs from the oldWALs area automatically. With this change in place clusters should return to the expected behavior of cleaning up WALs that are no longer needed.

      Description

      We found /hbase/oldWALs keeps growing, and we never use replication feature. Then we found OldWALsCleaner thread is always in WAITTING state :

      Thread 176 (OldWALsCleaner-1): State: WAITING Blocked count: 0 Waited count: 1 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@2ef05d36 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.master.cleaner.LogCleaner.deleteFile(LogCleaner.java:185) org.apache.hadoop.hbase.master.cleaner.LogCleaner.access$000(LogCleaner.java:48) org.apache.hadoop.hbase.master.cleaner.LogCleaner$1.run(LogCleaner.java:161) java.lang.Thread.run(Thread.java:745)
      

      Finally, we found that LogCleaner is not added to choreService, so LogCleaner doesn't run periodically.

        Attachments

        1. HBASE-23287.branch-1.0001.patch
          1 kB
          Baiqiang Zhao

          Activity

            People

            • Assignee:
              DeanZ Baiqiang Zhao
              Reporter:
              DeanZ Baiqiang Zhao
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: