[HBASE-23372] ZooKeeper Assignment can result in stale znodes in region-in-transition after table is dropped and hbck run

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 1.3.2
• Component/s: hbck, master, Region Assignment, Zookeeper
• Assignee: Swaroopa Kadam
• Reporter: David Manning
• Created: 05/Dec/19 19:47
• Updated: 12/Dec/19 06:18

Description

It is possible for znodes under /hbase/region-in-transition to remain long after a table is deleted. There does not appear to be any cleanup logic for these.

The details are a little fuzzy, but it seems to be fallout from HBASE-22617. Incidents related to that bug involved regions stuck in transition, and use of hbck to fix clusters. There was a temporary table created and deleted once per day, but somehow it led to receiving FSLimitException$MaxDirectoryItemsExceededException and regions stuck in transition. Even weeks after fixing the bug and upgrading the cluster, the znodes remain under /hbase/region-in-transition. In the most impacted cluster, hbase zkcli ls /hbase/region-in-transition | wc -w returns almost 100,000 entries. This causes very slow region transition times (often 80 seconds), likely due to enumerating all these entries when zk watch on this node is triggered.

Log lines for slow region transitions:

2019-12-05 07:02:14,714 DEBUG [K.Worker-pool3-t7344] master.AssignmentManager - Handling RS_ZK_REGION_CLOSED, server=<<SERVERNAME>>, region=<<REGION_HASH>>, which is more than 15 seconds late, current_state={<<REGION_HASH>> state=PENDING_CLOSE, ts=1575529254635, server=<<SERVERNAME>>}

Even during hmaster failover, entries are not cleaned, but the following log lines can be seen:

2019-11-27 00:26:27,044 WARN  [.activeMasterManager] master.AssignmentManager - Couldn't find the region in recovering region=<<DELETED_TABLE_REGION>>, state=RS_ZK_REGION_FAILED_OPEN, servername=<<SERVERNAME>>, createTime=1565603905404, payload.length=0

Possible solutions:

1. Logic to parse the RIT znode during master failover which sees if the table exists. Clean up entries for nonexistent tables.
2. New mode for hbck to do cleanup of nonexistent regions under the znode.
3. Others?