[RATIS-1717] Perf: Use global serialize buf to avoid temp buf

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 3.0.0
• Component/s: performance, server
• Assignee: Song Ziyang
• Reporter: Song Ziyang
• Created: 04/Oct/22 15:04
• Updated: 15/Nov/22 19:04
• Resolved: 13/Oct/22 10:27

Description

In current implementation, before a log entry is serialized to RaftLog, it first needs to be serialized to a byte array[1], and then be written to FileChannel using directByteBuffer[2]. It requires two data copies, one from LogEntryProto to byte array, one from byte array to direct ByteBuffer.

We can reduce the data copy by serializing LogEntry to directByteBuffer, removing the byte array:

1. We need a directByteBuffer large enough to hold a single log (maxBufferSize).
2. When we need to serialize a log entry to RaftLog, first we check whether there's enough space remained in directByteBuffer. If not, we flush the directByteBuffer to FileChannel. Otherwise we serialize the data directly into directByteBuffer.
3. When data buffered in directByteBuffer reaches the threshold (say 64KB), we flush the directByteBuffer, the same as current implementation.

[1] https://github.com/apache/ratis/blob/289db3ae64e8cf620eba882468dda661af0439bc/ratis-server/src/main/java/org/apache/ratis/server/raftlog/segmented/SegmentedRaftLogOutputStream.java#L96

[2] https://github.com/apache/ratis/blob/289db3ae64e8cf620eba882468dda661af0439bc/ratis-server/src/main/java/org/apache/ratis/server/raftlog/segmented/BufferedWriteChannel.java#L65

 

Issue Links

• is related to: RATIS-1743 Memory leak in SegmentedRaftLogWorker due to metrics
• is related to: RATIS-1746 Change the shared byte array in SegmentedRaftLogWorker to be configurable.
• links to: GitHub Pull Request #762