[HBASE-14237] Meta region may be onlined on multi regonservers for bugs of assigning meta

• Type: Bug
• Status: Closed
• Priority: Critical
• Resolution: Incomplete
• Affects Version/s: 0.94.11
• Assignee: Unassigned
• Reporter: Shaohui Liu
• Created: 18/Aug/15 10:16
• Updated: 01/Jul/22 20:56
• Resolved: 01/Jul/22 20:56

Description

When a regionserver failed to open the meta region and crash after setting the RS_ZK_REGION_FAILED_OPEN state of meta region in zookeeper, the master will handle the event of RS_ZK_REGION_FAILED_OPEN and try to assign the meta region again in AssignmentManager#handleRegion. But at the same time, the master will handle the regionserver expired event and start a MetaServerShutdownHandler for the regionserver, because the servername of regionserver is same as the servername of the unassigned node of meta region. In the MetaServerShutdownHandler, the meta region may be assigned for second time.

heliangliang
We have encountered this problem in our production cluster which resulted in inconsistency of region location in meta table. You can see the log from the attachment.

The code of AssignmentManager is so complex and I have not get a solution to fix this problem. Could someone kindly help to give some suggestions? Thanks