[HBASE-24117] Shutdown AssignmentManager before ProcedureExecutor may cause SCP to accidentally skip assigning a region

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Fix Version/s: 3.0.0-alpha-1, 2.3.0, 2.2.6
• Component/s: proc-v2
• Hadoop Flags: Reviewed
• Assignee: Duo Zhang
• Reporter: Michael Stack
• Created: 05/Apr/20 00:31
• Updated: 28/Sep/20 03:23
• Resolved: 29/Jun/20 15:15

Description

I saw this on TestCloseRegionWithRSCrash. The Region 788a516d1f86af98e0a16bcc1afe4fa1 was being moved to RS example.com,62652,1586032098445 just after it was killed. The Move Close fails because the RS has no node in the Master. The Move then tries to 'confirm' the close but it fails because no remote RS. We are then to wait in this state until operator or some other procedure intervenes to 'fix' the state. Normally a ServerCrashProcedure would do the job but in this test the Master is restarted after the RS is killed, a condition we do not accommodate.

Let me attach the test log.

Issue Links

• fixes: HBASE-25023 NPE while shutdown master node
• relates to: HBASE-24118 [Flakey Tests] TestCloseRegionWhileRSCrash
• links to: GitHub Pull Request #1865