[RATIS-1176] Benchmark various ways to stream data

• Type: Sub-task
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: client, Streaming
• Assignee: Unassigned
• Reporter: Tsz-wo Sze
• Created: 24/Nov/20 14:01
• Updated: 04/Dec/23 21:05

Description

In RATIS-1175, we provided a WritableByteChannel view of DataStreamOutput in order to support FileChannel.transferTo. However, runzhiwang pointed out that sun.nio.ch.FileChannelImpl.transferTo has three submethods

• transferToDirectly (fastest)
• transferToTrustedChannel
• transferToArbitraryChannel (slowest, requires buffer copying)

Unfortunately, our current implementation only able to use transferToArbitraryChannel.

There are several ideas below to improve the performance. We should benchmark them.

1. Improve the current implementation of WritableByteChannel so that it may be able to use a faster transferTo method.
2. Use FileChannel.map(..) and pass MappedByteBuffer to our DataStreamOutput.writeAsync method.
3. Add a new API

   //DataStreamOutput
    CompletableFuture<DataStreamReply> writeAsync(File);
   

   Internally, use Netty DefaultFileRegion for zero-copy file transfer:
   https://github.com/netty/netty/blob/4.1/example/src/main/java/io/netty/example/file/FileServerHandler.java#L53

The data flow of client -> primary -> peer as follows
1. If stream file and do not calculate checksum, we use transferTo. In client, there are 1 DMA copy and 1 DMA gather copy, no CPU copy. In primary, there are
3 DMA copy and 3 CPU copy. In peer, there are 2 DMA copy and 2 CPU copy.

2. If stream file and calculate checksum, we use MapByteBuffer. In client, there are 2 DMA copy and 1 CPU copy. In primary, there are
3 DMA copy and 3 CPU copy. In peer, there are 2 DMA copy and 2 CPU copy.

3. If stream data not in file and calculate checksum, we use DirectByteBuffer. In client, there are 2 DMA copy and 2 CPU copy. In primary, there are
3 DMA copy and 3 CPU copy. In peer, there are 2 DMA copy and 2 CPU copy.

4. we should avoid reading data into heap such as HeapByteBuffer. In client, there are 2 DMA copy and 4 CPU copy. In primary, there are
3 DMA copy and 3 CPU copy. In peer, there are 2 DMA copy and 2 CPU copy.

5. The following is flow before ratis streaming and use ProtoBuf to send data. In client there are 2 DMA copy and 4 CPU copy. In leader, there are 3 DMA copy and 7 CPU copy. In follower, there are 2 DMA copy and 5 CPU copy.

Issue Links

• is related to: RATIS-1179 Add a new API to DataStreamOutput for sending a File