[HBASE-21879] Read HFile's block to ByteBuffer directly instead of to byte for reducing young gc purpose

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 3.0.0-alpha-1, 2.3.0
• Hadoop Flags: Reviewed

Release Note

Before this issue, read path was 100% offheap when block is in the BucketCache. But if a cache miss, then the RS needs to read the block via an on-heap API which causes high young-GC pressure.

This issue adds reading the block via offheap even if reading the block from filesystem directly. It requires hadoop version(>=2.9.3) but can also work with older hadoop versions (all works but we continue to read block onheap). It also requires HBASE-21946 which is not yet in place as of this writing/hbase-2.3.0.

We have written a careful doc about the implementation, performance and practice here: https://docs.google.com/document/d/1xSy9axGxafoH-Qc17zbD2Bd--rWjjI00xTWQZ8ZwI_E/edit#heading=h.nch5d72p27ex

• Assignee: Zheng Hu
• Reporter: Zheng Hu
• Created: 12/Feb/19 12:52
• Updated: 01/Jul/22 21:42
• Resolved: 04/Sep/19 08:28

Description

In HFileBlock#readBlockDataInternal, we have the following:

@VisibleForTesting
protected HFileBlock readBlockDataInternal(FSDataInputStream is, long offset,
    long onDiskSizeWithHeaderL, boolean pread, boolean verifyChecksum, boolean updateMetrics)
 throws IOException {
 // .....
  // TODO: Make this ByteBuffer-based. Will make it easier to go to HDFS with BBPool (offheap).
  byte [] onDiskBlock = new byte[onDiskSizeWithHeader + hdrSize];
  int nextBlockOnDiskSize = readAtOffset(is, onDiskBlock, preReadHeaderSize,
      onDiskSizeWithHeader - preReadHeaderSize, true, offset + preReadHeaderSize, pread);
  if (headerBuf != null) {
        // ...
  }
  // ...
 }

In the read path, we still read the block from hfile to on-heap byte[], then copy the on-heap byte[] to offheap bucket cache asynchronously, and in my 100% get performance test, I also observed some frequent young gc, The largest memory footprint in the young gen should be the on-heap block byte[].

In fact, we can read HFile's block to ByteBuffer directly instead of to byte[] for reducing young gc purpose. we did not implement this before, because no ByteBuffer reading interface in the older HDFS client, but 2.7+ has supported this now, so we can fix this now. I think.

Will provide an patch and some perf-comparison for this.

Issue Links

• is related to: HDFS-14535 The default 8KB buffer in requestFileDescriptors#BufferedOutputStream is causing lots of heap allocation in HBase when using short-circut read
• is related to: HDFS-14541 When evictableMmapped or evictable size is zero, do not throw NoSuchElementException
• is related to: HBASE-22582 The Compaction writer may access the lastCell whose memory has been released when appending fileInfo in the final
• is related to: HBASE-22309 Replace Shipper Interface with Netty's ReferenceCounted; add ExtendCell#retain/ExtendCell#release
• is related to: HDFS-14483 Backport HDFS-14111,HDFS-3246 ByteBuffer pread interface to branch-2.9
• is related to: HDFS-14585 Backport HDFS-8901 Use ByteBuffer in DFSInputStream#read to branch2.9
• relates to: HBASE-21946 Use ByteBuffer pread instead of byte[] pread in HFileBlock when applicable
• relates to: HDFS-3246 pRead equivalent for direct read path
• relates to: HDFS-2834 ByteBuffer-based read API for DFSInputStream
• relates to: HBASE-20188 Evaluate and address performance delta between branch-1 and branch-2
• links to: Design Document
• links to: GitHub Pull Request #320