[HBASE-19144] [RSgroups] Retry assignments in FAILED_OPEN state when servers (re)join the cluster

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.4.0, 2.0.0
• Component/s: rsgroup
• Hadoop Flags: Reviewed
• Release Note: When regionserver placement groups (RSGroups) is active, as servers join the cluster the Master will attempt to reassign regions in FAILED_OPEN state.
• Assignee: Andrew Kyle Purtell
• Reporter: Andrew Kyle Purtell
• Created: 31/Oct/17 22:26
• Updated: 01/Aug/18 06:22
• Resolved: 03/Nov/17 22:51

Description

After all servers in the RSgroup are down the regions cannot be opened anywhere and transition rapidly into FAILED_OPEN state.

2017-10-31 21:06:25,449 INFO [ProcedureExecutor-13] master.RegionStates: Transition

{c6c8150c9f4b8df25ba32073f25a5143 state=OFFLINE, ts=1509483985448, server=node-5.cluster,16020,1509482700768}

to

{c6c8150c9f4b8df25ba32073f25a5143 state=FAILED_OPEN, ts=1509483985449, server=node-5.cluster,16020,1509482700768}

2017-10-31 21:06:25,449 WARN [ProcedureExecutor-13] master.RegionStates: Failed to open/close d4e2f173e31ffad6aac140f4bd7b02bc on node-5.cluster,16020,1509482700768, set to FAILED_OPEN

Any region in FAILED_OPEN state has to be manually reassigned, or the master can be restarted and this will also cause reattempt of assignment of any regions in FAILED_OPEN state. This is not unexpected but is an operational headache. It would be better if the RSGroupInfoManager could automatically kick reassignments of regions in FAILED_OPEN state when servers rejoin the cluster.