[HBASE-22460] Reopen a region if store reader references may have leaked

• Type: Sub-task
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 3.0.0-alpha-1, 1.5.0, 2.3.0
• Fix Version/s: 3.0.0-alpha-1, 2.3.0

Release Note

Leaked store files can not be removed even after it is invalidated via compaction. A reasonable mitigation for a reader reference leak would be a fast reopen of the region on the same server.

Configs:

1. hbase.master.regions.recovery.check.interval :

Regions Recovery Chore interval in milliseconds. This chore keeps running at this interval to find all regions with configurable max store file ref count and reopens them. Defaults to 20 mins

2. hbase.regions.recovery.store.file.ref.count :

This config represents Store files Ref Count threshold value considered for reopening regions. Any region with store files ref count > this value would be eligible for reopening by master. Default value -1 indicates this feature is turned off. Only positive integer value should be provided to enable this feature.

• Assignee: Viraj Jasani
• Reporter: Andrew Kyle Purtell
• Created: 22/May/19 22:27
• Updated: 29/Nov/24 02:19
• Resolved: 25/Oct/19 07:46

Description

• We can leak store reader references if a coprocessor or core function somehow opens a scanner, or wraps one, and then does not take care to call close on the scanner or the wrapped instance. A reasonable mitigation for a reader reference leak would be a fast reopen of the region on the same server (initiated by the RS) This will release all resources, like the refcount, leases, etc. The clients should gracefully ride over this like any other region transition. This reopen would be like what is done during schema change application and ideally would reuse the relevant code. If the refcount is over some ridiculous threshold this mitigation could be triggered along with a fat WARN in the logs.

Issue Links

• breaks: HBASE-25147 Should store the regionNames field in state data for ReopenTableRegionsProcedure
• is a parent of: HBASE-23212 Provide config reload for Auto Region Reopen based on storeFile ref count
• is a parent of: HBASE-23213 Backport HBASE-22460 to branch-1
• is related to: HBASE-23349 Low refCount preventing archival of compacted away files
• relates to: PHOENIX-5296 Ensure store file reader refcount is zero at end of relevant unit tests
• links to: GitHub Pull Request #600
• links to: GitHub Pull Request #750