[HBASE-21879] Read HFile's block to ByteBuffer directly instead of to byte for reducing young gc purpose

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 3.0.0-alpha-1, 2.3.0
• Hadoop Flags: Reviewed

Release Note

Before this issue, read path was 100% offheap when block is in the BucketCache. But if a cache miss, then the RS needs to read the block via an on-heap API which causes high young-GC pressure.

This issue adds reading the block via offheap even if reading the block from filesystem directly. It requires hadoop version(>=2.9.3) but can also work with older hadoop versions (all works but we continue to read block onheap). It also requires HBASE-21946 which is not yet in place as of this writing/hbase-2.3.0.

We have written a careful doc about the implementation, performance and practice here: https://docs.google.com/document/d/1xSy9axGxafoH-Qc17zbD2Bd--rWjjI00xTWQZ8ZwI_E/edit#heading=h.nch5d72p27ex

• Assignee: Zheng Hu
• Reporter: Zheng Hu
• Created: 12/Feb/19 12:52
• Updated: 01/Jul/22 21:42
• Resolved: 04/Sep/19 08:28

Description

In HFileBlock#readBlockDataInternal, we have the following:

@VisibleForTesting
protected HFileBlock readBlockDataInternal(FSDataInputStream is, long offset,
    long onDiskSizeWithHeaderL, boolean pread, boolean verifyChecksum, boolean updateMetrics)
 throws IOException {
 // .....
  // TODO: Make this ByteBuffer-based. Will make it easier to go to HDFS with BBPool (offheap).
  byte [] onDiskBlock = new byte[onDiskSizeWithHeader + hdrSize];
  int nextBlockOnDiskSize = readAtOffset(is, onDiskBlock, preReadHeaderSize,
      onDiskSizeWithHeader - preReadHeaderSize, true, offset + preReadHeaderSize, pread);
  if (headerBuf != null) {
        // ...
  }
  // ...
 }

In the read path, we still read the block from hfile to on-heap byte[], then copy the on-heap byte[] to offheap bucket cache asynchronously, and in my 100% get performance test, I also observed some frequent young gc, The largest memory footprint in the young gen should be the on-heap block byte[].

In fact, we can read HFile's block to ByteBuffer directly instead of to byte[] for reducing young gc purpose. we did not implement this before, because no ByteBuffer reading interface in the older HDFS client, but 2.7+ has supported this now, so we can fix this now. I think.

Will provide an patch and some perf-comparison for this.

Issue Links

• is related to: HDFS-14535 The default 8KB buffer in requestFileDescriptors#BufferedOutputStream is causing lots of heap allocation in HBase when using short-circut read
• is related to: HDFS-14541 When evictableMmapped or evictable size is zero, do not throw NoSuchElementException
• is related to: HBASE-22582 The Compaction writer may access the lastCell whose memory has been released when appending fileInfo in the final
• is related to: HBASE-22309 Replace Shipper Interface with Netty's ReferenceCounted; add ExtendCell#retain/ExtendCell#release
• is related to: HDFS-14483 Backport HDFS-14111,HDFS-3246 ByteBuffer pread interface to branch-2.9
• is related to: HDFS-14585 Backport HDFS-8901 Use ByteBuffer in DFSInputStream#read to branch2.9
• relates to: HBASE-21946 Use ByteBuffer pread instead of byte[] pread in HFileBlock when applicable
• relates to: HDFS-3246 pRead equivalent for direct read path
• relates to: HDFS-2834 ByteBuffer-based read API for DFSInputStream
• relates to: HBASE-20188 Evaluate and address performance delta between branch-1 and branch-2
• links to: Design Document
• links to: GitHub Pull Request #320

Sub-Tasks

• Abstract an ByteBuffAllocator to allocate/free ByteBuffer in ByteBufferPool: Closed
• Make the HFileBlock#validateChecksum can accept ByteBuff as an input.: Closed
• Notify users if the ByteBufAllocator is always allocating ByteBuffers from heap which means the increacing GC pressure: Closed
• Make the Compression#decompress can accept ByteBuff as input: Closed
• Abstract an ByteBuffOutputStream for building cell block: Closed
• Consider simplifying the logic of BucketCache eviction.: Closed
• Unify refCount of BucketEntry and refCount of hbase.nio.ByteBuff into one: Closed
• Use ByteBuff's refcnt to track the life cycle of data block: Closed
• Rewrite the block reading methods by using hbase.nio.ByteBuff: Closed
• The HFileBlock#CacheableDeserializer should pass ByteBuffAllocator to the newly created HFileBlock: Closed
• Ensure that the block cached in the LRUBlockCache offheap is allocated from heap: Closed
• Change to release mob hfile's block after rpc server shipped response to client: Closed
• ByteBufferIOEngine should support write off-heap ByteBuff to the bufferArray: Closed
• Remove the returnBlock method because we can just call HFileBlock#release directly: Closed
• Evaluate the get/scan performance after reading HFile block into offheap directly: Closed
• Improve the metrics in ByteBuffAllocator: Closed
• Retain an ByteBuff with refCnt=0 when getBlock from LRUCache: Closed
• Add a UT to address the HFileBlock#heapSize() in TestHeapSize: Closed
• Some paths in HFileScannerImpl did not consider block#release which will exhaust the ByteBuffAllocator: Closed
• It's better to use 65KB as the default buffer size in ByteBuffAllocator: Closed
• Separate the heap HFileBlock and offheap HFileBlock because the heap block won't need refCnt and save into prevBlocks list before shipping: Closed
• Optimize the MultiByteBuff#get(ByteBuffer, offset, len): Closed
• The HFileReaderImpl#shouldUseHeap return the incorrect true when disabled BlockCache: Closed
• There's still too much cpu wasting on validating checksum even if buffer.size=65KB: Closed
• Align the config keys and add document for offheap read in HBase Book.: Closed
• Deprecated the hbase.ipc.server.reservoir.initial.buffer.size & hbase.ipc.server.reservoir.initial.max for HBase2.x compatibility: Closed
• Address the final overview reviewing comments of HBASE-21879: Closed
• Backport offheap block reading (HBASE-21879) to branch-2: Closed
• The HeapAllocationRatio in WebUI is not accurate because all of the heap allocation will happen in another separated allocator named HEAP: Closed