[HBASE-18108] Procedure WALs are archived but not cleaned; fix - ASF JIRA

Voters

Watch issue

Watchers

Link

Clone

Update Comment Author

Replace String in Comment

Update Comment Visibility

Delete Comments

XML

Word

Printable

JSON

Details

Type: Sub-task
Status: Closed
Priority: Blocker
Resolution: Fixed
Affects Version/s: 2.0.0
Fix Version/s: 2.0.0
Component/s: proc-v2
Labels:
None

Hadoop Flags:

Reviewed
Release Note:

Hide
The archived Procedure WALs are moved to <hbase_root>/oldWALs/masterProcedureWALs
directory. TimeToLiveProcedureWALCleaner class was added which regularly cleans the Procedure WAL files from there.

The TimeToLiveProcedureWALCleaner is added to hbase.master.logcleaner.plugins configuration value.

A new config parameter is added: hbase.master.procedurewalcleaner.ttl, which specifies how long a Procedure WAL should stay in the archive directory.

Show
The archived Procedure WALs are moved to <hbase_root>/oldWALs/masterProcedureWALs directory. TimeToLiveProcedureWALCleaner class was added which regularly cleans the Procedure WAL files from there. The TimeToLiveProcedureWALCleaner is added to hbase.master.logcleaner.plugins configuration value. A new config parameter is added: hbase.master.procedurewalcleaner.ttl, which specifies how long a Procedure WAL should stay in the archive directory.

Description

The Procedure WAL files used to be deleted when done. ~~HBASE-14614~~ keeps them around in case issue but what is missing is a GC for no-longer-needed WAL files. This one is pretty important.

From WALProcedureStore Cleaner TODO in https://docs.google.com/document/d/1eVKa7FHdeoJ1-9o8yZcOTAQbv0u0bblBlCCzVSIn69g/edit#heading=h.r2pc835nb7vi