[HBASE-22491] Separate the heap HFileBlock and offheap HFileBlock because the heap block won't need refCnt and save into prevBlocks list before shipping - ASF JIRA

XML

Word

Printable

JSON

Details

Type: Sub-task
Status: Closed
Priority: Major
Resolution: Fixed
Affects Version/s: None
Fix Version/s: 3.0.0-alpha-1, 2.3.0
Component/s: None
Labels:
None

Hadoop Flags:

Reviewed

Description

In here [1], anoop.hbase has a comment:

There is a concern here. Even if the block is on an exclusive heap memory area, we will keep ref to that in this list. In a Phoenix Aggregation kind of use case where many blocks might get fetched and not immediately shipped, we are keeping the ref unwantedly here for longer time. This makes the GC not able to reclaim the heap memory area for the blocks. This might be a hidden bomb IMO. Its not good to remove the MemType. Lets create the block with memory type as EXCLUSIVE when the block data is on heap. The block might be coming from LRU cache or by fetching the block data from HDFS into heap memory area. When the block comes from off heap BC or if it is backed by a BB from the pool (While reading from HDFS, read into pooled BB) lets create the block with mem type as SHARED. Every block can have the retain and release method but let the EXCLUSIVE types do a noop here.

We've a discussion about this, say need to address two thing in this jira:
1. separate the HFileBlock into shared or non-shared;
2. Make the retain/release of non-shared block as a noop, say don't do reference count change for heap block.

1. https://github.com/apache/hbase/pull/257/files