Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-21879 Read HFile's block to ByteBuffer directly instead of to byte for reducing young gc purpose
  3. HBASE-22491

Separate the heap HFileBlock and offheap HFileBlock because the heap block won't need refCnt and save into prevBlocks list before shipping



    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 3.0.0-alpha-1, 2.3.0
    • Component/s: None
    • Labels:
    • Hadoop Flags:


      In here [1], Anoop Sam John has a comment:

      There is a concern here. Even if the block is on an exclusive heap memory area, we will keep ref to that in this list. In a Phoenix Aggregation kind of use case where many blocks might get fetched and not immediately shipped, we are keeping the ref unwantedly here for longer time. This makes the GC not able to reclaim the heap memory area for the blocks. This might be a hidden bomb IMO. Its not good to remove the MemType. Lets create the block with memory type as EXCLUSIVE when the block data is on heap. The block might be coming from LRU cache or by fetching the block data from HDFS into heap memory area. When the block comes from off heap BC or if it is backed by a BB from the pool (While reading from HDFS, read into pooled BB) lets create the block with mem type as SHARED. Every block can have the retain and release method but let the EXCLUSIVE types do a noop here.

      We've a discussion about this, say need to address two thing in this jira:
      1. separate the HFileBlock into shared or non-shared;
      2. Make the retain/release of non-shared block as a noop, say don't do reference count change for heap block.

      1. https://github.com/apache/hbase/pull/257/files


        1. HBASE-22491.HBASE-21879.v01.patch
          47 kB
          Zheng Hu
        2. HBASE-22491.HBASE-21879.v02.patch
          62 kB
          Zheng Hu

          Issue Links



              • Assignee:
                openinx Zheng Hu
                openinx Zheng Hu
              • Votes:
                0 Vote for this issue
                5 Start watching this issue


                • Created: