[HBASE-26437] [hboss] Rename does not clean up znodes for src location

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: hbase-filesystem-1.0.0-alpha1
• Fix Version/s: hbase-filesystem-1.0.0-alpha2
• Component/s: hboss
• Hadoop Flags: Reviewed
• Assignee: Josh Elser
• Reporter: Josh Elser
• Created: 09/Nov/21 22:27
• Updated: 18/Nov/21 01:21
• Resolved: 18/Nov/21 01:16

Description

We ran into a fun situation where the partition hosting ZK data was repeatedly filling up while heavy ExportSnapshot+clone_snapshot operations were running (10's of TB). The cluster was previously working just fine.

Upon investigation of the ZK tree, we found a large number of znodes beneath /hboss, specifically many in the corresponding ZK HBOSS path for $hbase.rootdir/.tmp.

Tracing back from the code, we saw that the CloneSnapshotProcedure (like CreateTableProcedure) will create the table filesystem layout in $hbase.rootdir/.tmp and then rename it into $hbase.rootdir/data/<namespace>. However, it appears that, upon rename, HBOSS was not cleaning up the src path's znode. This is a bug as it allows ZK to grow unbounded (which explains why this problem slowly arose and not suddenly).

As a workaround, HBase can be stopped and the corresponding ZK path for $hbase.rootdir/.tmp can be cleaned up to reclaim 1/2 the space taken up by znodes for imported hbase tables (we would still have znodes for $hbase.rootdir/data/...)

Issue Links

• relates to: HBASE-26461 [hboss] Delete self lock without orphaning znode
• Testing discovered: HBASE-26453 [hboss] removeInMemoryLocks can remove still in-use locks
• links to: GitHub Pull Request #29