[RATIS-2027] Ratis Streaming: Remote Stream copies data to heap

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 3.1.0
• Component/s: Streaming
• Labels: performance
• Assignee: Tsz-wo Sze
• Reporter: Duong
• Created: 14/Feb/24 19:57
• Updated: 29/Feb/24 18:31
• Resolved: 22/Feb/24 23:43

Description

In ratis streaming, the write to RemoteStream uses ByteBuf#nioBuffer() which copies the ByteBuf content to a HeapByteBuffer.

CompletableFuture<DataStreamReply> write(DataStreamRequestByteBuf request, Executor executor) {
  final Timekeeper.Context context = metrics.start();
  return composeAsync(sendFuture, executor,
      n -> out.writeAsync(request.slice().nioBuffer(), addFlush(request.getWriteOptionList()))
          .whenComplete((l, e) -> metrics.stop(context, e == null)));
} 

And this action implies 2 points of inefficiency:

1. The created and discarded HeapBuffer creates the O(N) cost to GC, with N as the data amount flowing through the ratis server.
2. When the HeapBuffer is written to the network, it needs to be copied again to a DriectBuffer by NIO. So, in total, 2 copies. 

See:  dn_write_streaming_allocation.htm and dn_write_streaming.htm

Issue Links

• links to: GitHub Pull Request #1044