[HBASE-17623] Reuse the bytes array when building the hfile block

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.4.0, 2.0.0
• Fix Version/s: 1.4.0, 2.0.0
• Component/s: HFile
• Hadoop Flags: Reviewed
• Assignee: Chia-Ping Tsai
• Reporter: Chia-Ping Tsai
• Created: 10/Feb/17 05:04
• Updated: 30/Apr/17 22:21
• Resolved: 25/Mar/17 15:54

Description

There are two improvements.

1. The onDiskBlockBytesWithHeader should maintain a bytes array which can be reused when building the hfile.
2. The onDiskBlockBytesWithHeader is copied to an new bytes array only when we need to cache the block.
3. If no block need to be cached, the uncompressedBlockBytesWithHeader will never be created.

HFileBlock.java

    private void finishBlock() throws IOException {
      if (blockType == BlockType.DATA) {
        this.dataBlockEncoder.endBlockEncoding(dataBlockEncodingCtx, userDataStream,
            baosInMemory.getBuffer(), blockType);
        blockType = dataBlockEncodingCtx.getBlockType();
      }
      userDataStream.flush();
      // This does an array copy, so it is safe to cache this byte array when cache-on-write.
      // Header is still the empty, 'dummy' header that is yet to be filled out.
      uncompressedBlockBytesWithHeader = baosInMemory.toByteArray();
      prevOffset = prevOffsetByType[blockType.getId()];

      // We need to set state before we can package the block up for cache-on-write. In a way, the
      // block is ready, but not yet encoded or compressed.
      state = State.BLOCK_READY;
      if (blockType == BlockType.DATA || blockType == BlockType.ENCODED_DATA) {
        onDiskBlockBytesWithHeader = dataBlockEncodingCtx.
            compressAndEncrypt(uncompressedBlockBytesWithHeader);
      } else {
        onDiskBlockBytesWithHeader = defaultBlockEncodingCtx.
            compressAndEncrypt(uncompressedBlockBytesWithHeader);
      }
      // Calculate how many bytes we need for checksum on the tail of the block.
      int numBytes = (int) ChecksumUtil.numBytes(
          onDiskBlockBytesWithHeader.length,
          fileContext.getBytesPerChecksum());

      // Put the header for the on disk bytes; header currently is unfilled-out
      putHeader(onDiskBlockBytesWithHeader, 0,
          onDiskBlockBytesWithHeader.length + numBytes,
          uncompressedBlockBytesWithHeader.length, onDiskBlockBytesWithHeader.length);
      // Set the header for the uncompressed bytes (for cache-on-write) -- IFF different from
      // onDiskBlockBytesWithHeader array.
      if (onDiskBlockBytesWithHeader != uncompressedBlockBytesWithHeader) {
        putHeader(uncompressedBlockBytesWithHeader, 0,
          onDiskBlockBytesWithHeader.length + numBytes,
          uncompressedBlockBytesWithHeader.length, onDiskBlockBytesWithHeader.length);
      }
      if (onDiskChecksum.length != numBytes) {
        onDiskChecksum = new byte[numBytes];
      }
      ChecksumUtil.generateChecksums(
          onDiskBlockBytesWithHeader, 0, onDiskBlockBytesWithHeader.length,
          onDiskChecksum, 0, fileContext.getChecksumType(), fileContext.getBytesPerChecksum());
    }