Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-23287

WALs not aged off of HDFS because LogCleaner is not added to choreService

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Blocker
    • Resolution: Fixed
    • 1.5.0, 1.6.0, 1.4.11
    • 1.6.0, 1.4.12
    • wal
    • None
    • Hide
      <!-- markdown -->

      Prior to this change HBase 1.4.11 and 1.5.0 based clusters would fail to clean up WALs from the oldWALs area automatically. With this change in place clusters should return to the expected behavior of cleaning up WALs that are no longer needed.
      Show
      <!-- markdown --> Prior to this change HBase 1.4.11 and 1.5.0 based clusters would fail to clean up WALs from the oldWALs area automatically. With this change in place clusters should return to the expected behavior of cleaning up WALs that are no longer needed.

    Description

      We found /hbase/oldWALs keeps growing, and we never use replication feature. Then we found OldWALsCleaner thread is always in WAITTING state :

      Thread 176 (OldWALsCleaner-1): State: WAITING Blocked count: 0 Waited count: 1 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@2ef05d36 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.master.cleaner.LogCleaner.deleteFile(LogCleaner.java:185) org.apache.hadoop.hbase.master.cleaner.LogCleaner.access$000(LogCleaner.java:48) org.apache.hadoop.hbase.master.cleaner.LogCleaner$1.run(LogCleaner.java:161) java.lang.Thread.run(Thread.java:745)
      

      Finally, we found that LogCleaner is not added to choreService, so LogCleaner doesn't run periodically.

      Attachments

        1. HBASE-23287.branch-1.0001.patch
          1 kB
          Baiqiang Zhao

        Activity

          People

            DeanZ Baiqiang Zhao
            DeanZ Baiqiang Zhao
            Votes:
            0 Vote for this issue
            Watchers:
            21 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: