[HBASE-28919] Soft drop for destructive table actions

• Type: New Feature
• Status: Patch Available
• Priority: Major
• Resolution: Unresolved
• Fix Version/s: 3.0.0, 4.0.0-alpha-1, 2.7.0, 2.5.11, 2.6.2
• Component/s: master, snapshots
• Labels: pull-request-available
• Assignee: Andrew Kyle Purtell
• Reporter: Andrew Kyle Purtell
• Created: 16/Oct/24 18:34
• Updated: 14/Nov/24 18:32

Description

When we administratively drop a table column or entire table, or truncate a table, the process begins rapidly. Procedures are scheduled for immediate execution that then modify or remove descriptors and state in META and on disk, and take unrecoverable actions at the HDFS layer. Although HFiles are copied to the archive in a destructive action, recovery scenarios are not automatic and involve some operator labor to reconstruct the table and re-import the archived data. If the HFileCleaner is not properly configured to facilitate such recovery then some data is not recoverable soon after procedure execution commences and all affected data is not recoverable within minutes. A customer faced with such an accident will be unhappy because the recovery scenarios available to them from this will involve either a restore from backup or from an earlier snapshot, and any changes committed more recently than the time of the last backup or last snapshot will be lost. 

An effective solution is very simple: We can easily prevent the deletion of the HFiles of a deleted table or table column family by taking a snapshot of the table immediately prior to taking any destructive actions. We set a TTL on the snapshot so housekeeping of truly unwanted HFiles remains no touch. Because we take a table snapshot all table structure and metadata is also captured and saved so fast recovery is possible, as either a restore from snapshot, or a clone from snapshot to a new table. For as long as the snapshot is retained it is straightforward to recover the table data by either restoring the table from the snapshot or cloning the snapshot to a new table, at the operator’s discretion.

No manual actions are required to see the table or column family (or families) truly dropped. Once the snapshot TTL expires all the HFiles related to the dropped table become eligible for deletion. When the HFileCleaner chore executes after that time the HDFS level file deletes will commence with associated reduction in storage requirements. 

Design document is attached. 

I have a working implementation of this proposal based on a fork of branch-2.5. 

Issue Links

• links to: GitHub Pull Request #6383