[HDDS-10411] Support incremental ChunkBuffer checksum calculation

• Type: Sub-task
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Labels: pull-request-available
• Assignee: Siyao Meng
• Reporter: Siyao Meng
• Created: 23/Feb/24 02:32
• Updated: 5 days ago 23:06

Description

Goal

Calculate ChunkBuffer (ByteBuffer) checksum incrementally rather than having to calculating it from scratch every single time in writeChunkToContainer.

Background

Currently ChunkBuffer (ByteBuffer) checksum is always calculated from scratch. As can be seen here in checksum function initialization, which it always calls reset() before feeding any data with update():

newChecksumByteBufferFunction: https://github.com/apache/ozone/blob/5f0925e190f1dbf2d4617daad8ad42401d5079e1/hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/common/Checksum.java#L67-L68

  private static Function<ByteBuffer, ByteString> newChecksumByteBufferFunction(
      Supplier<ChecksumByteBuffer> constructor) {
    final ChecksumByteBuffer algorithm = constructor.get();
    return  data -> {
      algorithm.reset();
      algorithm.update(data);
      return int2ByteString((int)algorithm.getValue());
    };
  }

Each ByteBuffer (4 MB by default) inside a block's ChunkBuffer gets its checksum calculated here:

https://github.com/apache/ozone/blob/5f0925e190f1dbf2d4617daad8ad42401d5079e1/hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/common/Checksum.java#L171-L177

    // Checksum is computed for each bytesPerChecksum number of bytes of data
    // starting at offset 0. The last checksum might be computed for the
    // remaining data with length less than bytesPerChecksum.
    final List<ByteString> checksumList = new ArrayList<>();
    for (ByteBuffer b : data.iterate(bytesPerChecksum)) {
      checksumList.add(computeChecksum(b, function, bytesPerChecksum));
    }

which is called from BlockOutputStream#writeChunkToContainer.

And when the function is applied in the inner computeChecksum, it always calls reset() first. So it calculates the whole ByteBuffer from offset 0.

Motivation

While this may not be a big issue before Ozone hsync() is implemented (or in HDFS, where each chunk is much smaller, at 64 KB by default), it can now contribute to ~10% of hsync latency between client-DN if the client is only appending a few bytes between hsyncs, as can be seen from weichiu's flame graph.

Estimated latency improvement is 0%~20% with this change, depending on the client write/hsync pattern.

Issue Links

• links to: GitHub Pull Request #7189