It is possible for znodes under /hbase/region-in-transition to remain long after a table is deleted. There does not appear to be any cleanup logic for these.
The details are a little fuzzy, but it seems to be fallout from
HBASE-22617. Incidents related to that bug involved regions stuck in transition, and use of hbck to fix clusters. There was a temporary table created and deleted once per day, but somehow it led to receiving FSLimitException$MaxDirectoryItemsExceededException and regions stuck in transition. Even weeks after fixing the bug and upgrading the cluster, the znodes remain under /hbase/region-in-transition. In the most impacted cluster, hbase zkcli ls /hbase/region-in-transition | wc -w returns almost 100,000 entries. This causes very slow region transition times (often 80 seconds), likely due to enumerating all these entries when zk watch on this node is triggered.
Log lines for slow region transitions:
Even during hmaster failover, entries are not cleaned, but the following log lines can be seen:
- Logic to parse the RIT znode during master failover which sees if the table exists. Clean up entries for nonexistent tables.
- New mode for hbck to do cleanup of nonexistent regions under the znode.