[HBASE-23287] WALs not aged off of HDFS because LogCleaner is not added to choreService

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 1.5.0, 1.6.0, 1.4.11
• Fix Version/s: 1.6.0, 1.4.12
• Component/s: wal

Release Note

<!-- markdown -->

Prior to this change HBase 1.4.11 and 1.5.0 based clusters would fail to clean up WALs from the oldWALs area automatically. With this change in place clusters should return to the expected behavior of cleaning up WALs that are no longer needed.

• Assignee: Baiqiang Zhao
• Reporter: Baiqiang Zhao
• Created: 13/Nov/19 11:30
• Updated: 16/Aug/22 12:40
• Resolved: 15/Nov/19 20:42

Description

We found /hbase/oldWALs keeps growing, and we never use replication feature. Then we found OldWALsCleaner thread is always in WAITTING state :

Thread 176 (OldWALsCleaner-1): State: WAITING Blocked count: 0 Waited count: 1 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@2ef05d36 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.master.cleaner.LogCleaner.deleteFile(LogCleaner.java:185) org.apache.hadoop.hbase.master.cleaner.LogCleaner.access$000(LogCleaner.java:48) org.apache.hadoop.hbase.master.cleaner.LogCleaner$1.run(LogCleaner.java:161) java.lang.Thread.run(Thread.java:745)

Finally, we found that LogCleaner is not added to choreService, so LogCleaner doesn't run periodically.