[HBASE-28271] Infinite waiting on lock acquisition by snapshot can result in unresponsive master

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.0.0-alpha-4, 2.4.17, 2.5.7
• Fix Version/s: 2.6.0, 2.5.8, 3.0.0-beta-2
• Hadoop Flags: Reviewed
• Assignee: Viraj Jasani
• Reporter: Viraj Jasani
• Created: 19/Dec/23 05:39
• Updated: 24/Jan/24 06:22
• Resolved: 23/Jan/24 02:42

Description

When a region is stuck in transition for significant time, any attempt to take snapshot on the table would keep master handler thread in forever waiting state. As part of the creating snapshot on enabled or disabled table, in order to get the table level lock, LockProcedure is executed but if any region of the table is in transition, LockProcedure could not be executed by the snapshot handler, resulting in forever waiting until the region transition is completed, allowing the table level lock to be acquired by the snapshot handler.

In cases where a region stays in RIT for considerable time, if enough attempts are made by the client to create snapshots on the table, it can easily exhaust all handler threads, leading to potentially unresponsive master. Attached a sample thread dump.

Proposal: The snapshot handler should not stay stuck forever if it cannot take table level lock, it should fail-fast.

Issue Links

• links to: GitHub Pull Request #5603