[HBASE-28522] UNASSIGN proc indefinitely stuck on dead rs

• Type: Improvement
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Fix Version/s: 2.7.0, 3.0.0-beta-2, 2.6.1, 2.5.11
• Component/s: proc-v2, Region Assignment
• Labels: pull-request-available
• Hadoop Flags: Reviewed
• Assignee: Duo Zhang
• Reporter: Prathyusha
• Created: 14/Apr/24 18:50
• Updated: 28/Jul/24 20:32
• Resolved: 27/Jul/24 15:18

Description

One scenario we noticed in production -

we had DisableTableProc and SCP almost triggered at similar time

2024-03-16 17:59:23,014 INFO [PEWorker-11] procedure.DisableTableProcedure -
Set <TABLE_NAME> to state=DISABLING

2024-03-16 17:59:15,243 INFO [PEWorker-26] procedure.ServerCrashProcedure -
Start pid=21592440, state=RUNNABLE:SERVER_CRASH_START, locked=true; ServerCrashProcedure
<regionserver>, splitWal=true, meta=false

DisabeTableProc creates unassign procs, and at this time ASSIGNs of SCP is not completed

2024-03-16 17:59:23,003 DEBUG [PEWorker-40] procedure2.ProcedureExecutor - LOCK_EVENT_WAIT pid=21594220, ppid=21592440, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=<TABLE_NAME>, region=<regionhash>, ASSIGN

UNASSIGN created by DisableTableProc is stuck on the dead regionserver and we had to manually bypass unassign of DisableTableProc and then do ASSIGN.

If we can break the loop for UNASSIGN procedure to not retry if there is scp for that server, we do not need manual intervention?, at least the DisableTableProc can go to a rollback state?

Issue Links

• is related to: HBASE-23636 Disable table may hang when regionserver stop or abort.
• relates to: HBASE-28582 ModifyTableProcedure should not reset TRSP on region node when closing unused region replicas
• relates to: HBASE-28683 Only allow one TableProcedureInterface for a single table to run at the same time for some special procedure types
• links to: GitHub Pull Request #5995
• links to: GitHub Pull Request #6122