[HBASE-20933] multiple splits may result into forever uncleaned split region

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Assignee: Vishal Khandelwal
• Reporter: Vishal Khandelwal
• Created: 25/Jul/18 04:11
• Updated: 13/Jun/22 15:19
• Resolved: 13/Jun/22 15:19

Description

Incase of multiple subsequent split and with an open handle on old reference file, it may result into split region which can never be cleaned

 So Here are two issues.

1. Region is getting split even when it has reference to its parent
2. Region is going offline/in archive mode even when there are reference pending in store

Repro Steps

1. Region split (P)
2. Before major compaction starts after split, open a handle on store file on new region (DA & DB)
3. Let compaction completes on DA, (Here compaction will not clear reference store files as it is opened)
4. Split new region (DA) again ( shouldSplit will return true as before compaction even does the cleanup, it removes the compacted files and reference in-memory list)
5. Now CatalogJanitor will not remove this region as it has store references, majorCompaction/CompactedHFilesDischarger will not do the cleanup as it only looks at only online regions
6.  After above steps region-DA which is offline will always be in split regions and never getting cleaned up.

We found that catalog janitor is also not able to clean regions which are offline(split parent) because it has reference of the daughter of it's parent which is not getting cleaned up. This is causing lot of store files not getting cleaned causing more space in local index store and lot of split lingering regions.

Unit test repro the scenario has been attached.

Fix can be in CompactedHFilesDischarger or catalogJanitor to handle such cases. Even if such region exists which are offline and are split region. They should be able to clean t hem selves

Issue Links

• is a parent of: HBASE-20940 HStore.cansplit should not allow split to happen if it has references
• is a parent of: HBASE-20935 HStore.removeCompactedFiles should log in case it is unable to delete a file