[HBASE-22457] Harden the HBase HFile reader reference counting

• Type: Brainstorming
• Status: Resolved
• Priority: Major
• Resolution: Implemented
• Assignee: Unassigned
• Reporter: Lars Hofhansl
• Created: 22/May/19 17:17
• Updated: 11/Jun/22 18:54
• Resolved: 11/Jun/22 18:54

Description

The problem that any coprocessor hook that replaces a passed scanner without closing it can cause an incorrect reference count.
This was bad and wrong before of course, but now it has pretty bad consequences, since an incorrect reference could will prevent HFiles from being archived indefinitely.

All hooks that are passed a scanner and return a scanner are suspect, since the returned scanner may or may not close the passed scanner:

• preCompact
• preCompactScannerOpen
• preFlush
• preFlushScannerOpen
• preScannerOpen
• preStoreScannerOpen
• preStoreFileReaderOpen...? (not sure about this one, it could mess with the reader)

I sampled the Phoenix and also Tephra code, and found a few instances where this is happening.
And for those I filed issued: TEPHRA-300, PHOENIX-5291
(We're not using Tephra)

The Phoenix ones should be rare. In our case we are seeing readers with refCount > 1000.
Perhaps there are other issues, a path where not all exceptions are caught and scanner is left open that way perhaps. (Generally I am not a fan of reference counting in complex systems - it's too easy to miss something. But that's a different discussion. ).

Let's brainstorm some way in which we can harden this.

ram_krish, anoop.hbase, apurtell

Issue Links

• is related to: PHOENIX-5291 Ensure that Phoenix coprocessor close all scanners.
• is related to: HBASE-23349 Low refCount preventing archival of compacted away files
• relates to: HBASE-22459 Expose store reader reference count

Sub-Tasks

• Reopen a region if store reader references may have leaked: Resolved
• Backport HBASE-22460 (Reopen a region if store reader references may have leaked) to branch-1: Resolved