[HDDS-9232] Write performance degradation

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 1.3.0
• Fix Version/s: 1.4.0
• Labels: ozone-performance pull-request-available
• Target Version/s: 1.4.0
• Flags: Important
• Assignee: Tsz-wo Sze
• Reporter: Mladjan Gadzic
• Created: 31/Aug/23 18:18
• Updated: 28/Nov/23 07:09
• Resolved: 06/Oct/23 17:52

Description

It is observed on real cluster using unsecure Ozone HA (3 OMs, 3 SCMs, 3DNs) that write performance degraded significantly after Ratis upgrade to 2.5.1 (https://issues.apache.org/jira/browse/HDDS-8382). At the time of writing this ticket, it takes master branch ~25mins for 100k keys to be written, while it took 1.3.0 release ~2mins for the same number of keys.

Test was done using single freon client on OM node. Command 

ozone freon rk --bufferSize 4096 --factor THREE --keySize 10240 --numOfVolumes 1 --numOfBuckets 1 --numOfKeys 100000 --numOfThreads 10 --replication-type RATIS 

was used for that purpose.

Commit before HDDS-8382, HDDS-8550 (sha 290e5f2ba1826a2084c1745e458872dc76c7ae91) did not exhibit degradation in write performance.

Issue Links

• causes: HDDS-9416 Intermittent fork timeout in TestOMRatisSnapshots
• is related to: HDDS-9784 Datanode Write performance degradation
• is related to: HDDS-8382 Upgrade Ratis to 2.5.1
• relates to: RATIS-1886 AppendLog sleep fixed time cause significant drop in write throughput
• links to: GitHub Pull Request #5338