[HBASE-28680] BackupLogCleaner causes HMaster WALs to pile up indefinitely

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.6.0
• Fix Version/s: 4.0.0-alpha-1, 2.7.0, 3.0.0-beta-2, 2.6.1
• Component/s: backup&restore, master
• Labels: pull-request-available
• Assignee: Ray Mattingly
• Reporter: Ray Mattingly
• Created: 19/Jun/24 15:41
• Updated: 23/Jun/24 06:29
• Resolved: 21/Jun/24 15:14

Description

We have been trying to setup daily incremental backups for hundreds of clusters at my day job. Recently we discovered that old WALs were piling up across many clusters inline with when we began running incremental backups.

This led to the realization that the BackupLogCleaner will always skip archived HMaster WALs. This is a problem because, if a cleaner is skipping a given file, then the CleanerChore will never delete it.

This seems like a misunderstanding of what it means to "skip" a WAL in a BaseLogCleanerDelegate, and, instead, we should always return these HMaster WALs as deletable from the perspective of the BackupLogCleaner. We could subject them to the same scrutiny as RegionServer WALs: are they older than the most recent successful  backup? But, if I understand correctly, HMaster WALs do not contain any data relevant to table backups, so that would be unnecessary.

Issue Links

• links to: GitHub Pull Request #6006
• links to: GitHub Pull Request #6014
• links to: https://github.com/apache/hbase/pull/6006