[HBASE-24255] GCRegionProcedure doesn't assign region from RegionServer leading to orphans

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 2.2.4
• Component/s: proc-v2, Region Assignment, regionserver

Environment

hbase 2.2.4

hadoop 3.1.3

• Assignee: Unassigned Assign to me
• Reporter: Andrey Elenskiy
• Created: 24/Apr/20 16:12
• Updated: 03/Jun/20 19:31

Description

We've found ourselves in a situation where parents of merged or split regions needed to be opened again on a regionserver due to having to recover from cluster meltdown (HBCK2's fixMeta kicks off GCMultipleMergedRegionsProcedure which requiters all regions to be merged to be open). Then, when a GCProcedure is kicked of to clean a parent region up by GCMultipleMergedRegionsProcedure, it ends up deleting it from hbase:meta, but doesn't unassign it from RegionServer leading for it to show up in "Orphan Regions on RegionServer" in hbck tab of HBase Master. Also, the hbase client doesn't detect that the region is closed either because it's still technically open on a regionserver (it doesn't reread hbase:meta all the time). The only way to recover from this is to restart regionserver which isn't idea as it can lead to other issues in clusters with region inconsistencies.

Issue Links

• relates to: HBASE-24250 CatalogJanitor resubmits GCMultipleMergedRegionsProcedure for the same region