[HDDS-10890] [hsync] Increase default value for hdds.container.ratis.log.appender.queue.num-elements

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Done
• Fix Version/s: 2.0.0
• Labels: pull-request-available
• Assignee: Wei-Chiu Chuang
• Reporter: Wei-Chiu Chuang
• Created: 21/May/24 19:36
• Updated: 22/May/24 10:30
• Resolved: 22/May/24 10:30

Description

hdds.container.ratis.log.appender.queue.num-elements default value is 1. This is the bottleneck at DN Ratis.

With the default value, I got about 2500 requests per second using freon dn echo tool.
Increasing it to 32, I managed to 5500 requests per second,

This is still lower than OM echo on the same machine (~16k requests per second), so I'll keep looking for other efficiencies.

NOTE: this Ozone configuration is associated with Ratis property raft.server.log.appender.buffer.element-limit. The default value for Ozone Manager Ratis is 1024 (defined by ozone.om.ratis.log.appender.queue.num-elements)

  <property>
    <name>hdds.container.ratis.log.appender.queue.num-elements
</name>
    <value>1</value>
    <tag>OZONE, DEBUG, CONTAINER, RATIS</tag>
    <description>Limit for number of append entries in ratis leader's
      log appender queue.
    </description>
  </property>

Issue Links

• is a child of: HDDS-10755 [hsync] Investigate why DataNode Echo throughput is so low
• is related to: HDDS-912 Update ozone to latest ratis snapshot build (0.4.0-3b0be02-SNAPSHOT)
• links to: GitHub Pull Request #6711