Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-21879 Read HFile's block to ByteBuffer directly instead of to byte for reducing young gc purpose
  3. HBASE-22127

Ensure that the block cached in the LRUBlockCache offheap is allocated from heap

    XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      In here [1], Anoop Sam John pointed out an crtial problem , I pasted here:

      So if we read from HDFS into a pooled BB and then give to LRU cache for caching (ya mostly cache on read might be true) we will cache the block which is backed by this pooled DBB? Unless the block is evicted , this BB wont go back to pool. I think this is some thing we can not livw with !! For LRU cache the sizing itself is based on what % of heap size we can grow. But here in effect we are occupying the off heap space for the cached blocks. All the sizing assumptions and calc going out of control !

      It's indeed an big problem here. so we can only make the block ref to an heap area if we use LRUCache (both LruBlockCache and CombinedBlockCache case). Or we can also make the lru cache offheap ?

      I think we can introduce an switch indicate that whether the lru block cache offheap or not, if heap, then coping those bytes from ByteBuff to heap.

      https://reviews.apache.org/r/70153/diff/6?file=2133545#file2133545line398

        Attachments

        1. HBASE-22127.HBASE-21879.v1.patch
          25 kB
          Zheng Hu
        2. HBASE-22127.HBASE-21879.v2.patch
          41 kB
          Zheng Hu
        3. HBASE-22127.HBASE-21879.v3.patch
          54 kB
          Zheng Hu
        4. HBASE-22127.HBASE-21879.v4.patch
          57 kB
          Zheng Hu
        5. HBASE-22127.HBASE-21879.v5.patch
          60 kB
          Zheng Hu
        6. HBASE-22127.HBASE-21879.v6.patch
          60 kB
          Zheng Hu
        7. HBASE-22127.HBASE-21879.v7.patch
          60 kB
          Zheng Hu
        8. HBASE-22127.HBASE-21879.v8.patch
          62 kB
          Zheng Hu

          Issue Links

            Activity

              People

              • Assignee:
                openinx Zheng Hu
                Reporter:
                openinx Zheng Hu
              • Votes:
                0 Vote for this issue
                Watchers:
                7 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: