[RATIS-1924] Increase the default of raft.server.log.segment.size.max

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 3.0.0
• Component/s: server
• Assignee: Tsz-wo Sze
• Reporter: Tsz-wo Sze
• Created: 31/Oct/23 04:01
• Updated: 20/Nov/23 18:38
• Resolved: 31/Oct/23 16:02

Description

Currently, the default is 8MB which is very small. The servers will roll the log frequently and create a lot of small files.

According to Song Ziyang's comment, 32MB seems a good choice.

It's only that I did experiments earlier and found 32MB is the optimal for IOTDB default workload. The experiments show:

• (end to end throughput) improves substantially as segmented size grow from 8MB to 32MB, slowly from 32MB to 128MB.
• The memory usage in JVM young increases as the segmented size grow, even if we restricted the max-cached-size and max-cached-segments. It turned out to be the open segment.

Issue Links

• is related to: HDDS-8040 Increase ratis log segment default size and default preallocated size in datanode
• links to: GitHub Pull Request #957