[HBASE-22330] Backport HBASE-20724 (Sometimes some compacted storefiles are still opened after region failover) to branch-1

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.5.0, 1.4.9, 1.3.4
• Fix Version/s: 1.4.10, 1.3.5
• Component/s: Compaction, regionserver
• Hadoop Flags: Reviewed
• Assignee: Abhishek Singh Chouhan
• Reporter: Andrew Kyle Purtell
• Created: 29/Apr/19 19:44
• Updated: 29/Jan/22 01:23
• Resolved: 09/May/19 23:15

Description

There appears to be a race condition between close and split which when combined with a side effect of HBASE-20704, leads to the parent region store files getting archived and cleared while daughter regions still have references to those parent region store files.

Here is the timeline of events observed for an affected region:

1. RS1 faces ZooKeeper connectivity issue for master node and starts shutting itself down. As part of this it starts to close the store and clean up the compacted files (File A)
2. Master starts bulk assigning regions and assign parent region to RS2
3. Region opens on RS2 and ends up opening compacted store file(s) (suspect this is due to HBASE-20724)
4. Now split happens and daughter regions open on RS2 and try to run a compaction as part of post open
5. Split request at this point is complete. However now archiving proceeds on RS1 and ends up archiving the store file that is referenced by the daughter.

Compaction fails due to FileNotFoundException and all subsequent attempts to open the region will fail until manual resolution.

We think having HBASE-20724 would help in such situations since we won't end up loading compacted store files in the first place. 

Issue Links

• is required by: HBASE-26722 Snapshot is corrupted due to interaction between move, warmupRegion, compaction, and HFileArchiver