[HDDS-9784] Datanode Write performance degradation

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Fix Version/s: 1.4.0
• Component/s: Ozone Datanode
• Labels: pull-request-available
• Assignee: ChenXi
• Reporter: ChenXi
• Created: 28/Nov/23 07:08
• Updated: 03/Dec/23 13:15
• Resolved: 03/Dec/23 13:15

Description

Summary

Default value of hdds.ratis.raft.server.log.appender.wait-time.min 1ms causes write new energy to degrade.

 

Testing commands

Datanode Write performance degradation:

command

ozone freon ommg --operation CREATE_KEY -n 10000000 --volume=s3v --bucket=bucket-object -t 50 --size 4k

Bad Performance 

    <property>
      <name>hdds.ratis.raft.server.log.appender.wait-time.min</name>
      <value>1ms</value>
    </property>

 

Normal Performance

    <property>
      <name>hdds.ratis.raft.server.log.appender.wait-time.min</name>
      <value>0</value>
    </property>

Other testing data can be referred to.

0 size performance, about 8000 QPS, this represents the highest write performance of any OM.

the `hdds.ratis.raft.server.log.appender.wait-time.min` of om had been set to 0ms by default.

 

This represents the performance limit of the DN, about 6000 QPS, which can indicate that the cluster itself has normal read/write performance.

 

Cluster info

• Number of Datanodes 6
• 12disks per Datanode
• HA mode for both OM and SCM
• Ozone version: ozone.git -r 60bb0604217fbd44dbc6b115b59fc26a297fcaba

Issue Links

• relates to: HDDS-9232 Write performance degradation
• relates to: RATIS-1886 AppendLog sleep fixed time cause significant drop in write throughput
• relates to: RATIS-1945 INCONSISTENCY replies should be counted as errors.
• links to: GitHub Pull Request #5711