[HBASE-28683] Only allow one TableProcedureInterface for a single table to run at the same time for some special procedure types

• Type: Improvement
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Fix Version/s: 2.7.0, 3.0.0-beta-2, 2.6.1, 2.5.10
• Component/s: master, proc-v2
• Labels: pull-request-available
• Hadoop Flags: Reviewed
• Assignee: Duo Zhang
• Reporter: Duo Zhang
• Created: 20/Jun/24 15:13
• Updated: 10/Sep/24 15:00
• Resolved: 17/Jul/24 01:45

Description

We have a table lock in the MasterProcedureScheduler, which is designed to only allow one procedure to run at the same time when they require exclusive lock.

But there is a problem that for availability, usually we can not always hold the exclusive lock through the whole procedure life time, as if so, we can not execute region assignment for this table too. The solution is to set holdLock to false, which means we will release the table lock after one execution cycle.

In this way, it is possible that different table procedures may execute at the same time, which could mess things up.

Especially that, in HBASE-28522, we find out that it is even impossible for DisableTableProcedure to hold the exclusive lock all the time. If the steps for DisableTableProcedure can be overlapped with other procedures like ModifyTableProcedure or even EnableTableProcedure, things will be definationly messed up...

So we need to find another way to ensure that for a single table, only one of these procedures can be executed at the same time.

Issue Links

• breaks: HBASE-28830 when a procedure on a table executed as a child procedure, further table procedure operations on that table are blocked forever waiting to acquire the table procedure lock
• is related to: HBASE-28522 UNASSIGN proc indefinitely stuck on dead rs
• links to: GitHub Pull Request #6046
• links to: GitHub Pull Request #6084