[HDDS-10288] Checksum to support direct buffers

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.0.0
• Component/s: Ozone Client
• Labels: pull-request-available
• Assignee: Duong
• Reporter: Duong
• Created: 04/Feb/24 01:24
• Updated: 15/Feb/24 17:16
• Resolved: 15/Feb/24 17:16

Description

Ozone client should leverage direct buffers and zero-copy to improve performance while writing and reading data.
However, today the checksum algorithm doesn't support direct buffers yet and instead copies direct buffers to on-heap arrays to calculate checksum. This is because Ozone relies on Java 8 API which doesn't support Checksum.update(ByteBuffer) yet.

Ref: https://github.com/apache/ozone/blob/master/hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/common/ChecksumByteBufferImpl.java#L69-L69

  @Override
  // TODO - when we eventually move to a minimum Java version >= 9 this method
  //        should be refactored to simply call checksum.update(buffer), as the
  //        Checksum interface has been enhanced to allow this since Java 9.
  public void update(ByteBuffer buffer) {
    ....
    if (buffer.hasArray()) {
      checksum.update(buffer.array(), buffer.position() + buffer.arrayOffset(),
          buffer.remaining());
    } else {
      byte[] b = new byte[buffer.remaining()];
      buffer.get(b);
      checksum.update(b, 0, b.length);
    }
  }

While we accept Java 8 as the minimum supported JVM, most Ozone production clusters are running on a newer platform like JDK11.
We can leverage Java 9+ interface using reflection to checksum direct buffers directly in native memory.

Refer to this example in Kafka. https://github.com/a0x8o/kafka/blob/master/clients/src/main/java/org/apache/kafka/common/utils/Checksums.java

Issue Links

• blocks: HDDS-10283 Apply zero copy to Ozone client reads chunks
• blocks: HDDS-9843 Ozone client high memory (heap) utilization
• links to: GitHub Pull Request #6162