[HDDS-7038] Significant performance degradation on 1.3.0-SNAPSHOT

• Type: Improvement
• Status: Resolved
• Priority: Critical
• Resolution: Not A Bug
• Affects Version/s: 1.3.0
• Component/s: Ozone Datanode, S3

Environment

Our dist tarball is built with the following commit hash:

commit 843fac2fb646eecfc33103fdb16eaf77a66ca062
Author: Neil Joshi <[e-mail]>
Date:   Wed Jul 13 22:29:59 2022 -0600
 
    HDDS-6433. Refactor OMFailoverProxyProvider to provide a base for OM and GrpcOM FailoverProxyProviders (#3389)
 

• SCM (non-HA)
• OM (non-HA)
• Datanode x 36
• S3 Gateway

• Assignee: Unassigned
• Reporter: Kiyoshi Mizumaru
• Created: 22/Jul/22 17:42
• Updated: 09/Aug/22 16:49
• Resolved: 09/Aug/22 11:03

Description

Found a significant performance degradation on our new cluster testing with 1.3.0-SNAPSHOT. For example, writing 16 files, size 1MiB each, using the aws s3 cp command takes more than 2 minutes against the new cluster.
 

$ time aws s3 cp --endpoint https://<REDACTED> hats-00 s3://maru-test/hats-00 --recursive
upload: hats-00/00000000/00000008 to s3://maru-test/hats-00/00000000/00000008
upload: hats-00/00000000/00000002 to s3://maru-test/hats-00/00000000/00000002
upload: hats-00/00000000/00000009 to s3://maru-test/hats-00/00000000/00000009
upload: hats-00/00000000/00000003 to s3://maru-test/hats-00/00000000/00000003
upload: hats-00/00000000/00000006 to s3://maru-test/hats-00/00000000/00000006
upload: hats-00/00000000/00000011 to s3://maru-test/hats-00/00000000/00000011
upload: hats-00/00000000/00000010 to s3://maru-test/hats-00/00000000/00000010
upload: hats-00/00000000/00000012 to s3://maru-test/hats-00/00000000/00000012
upload: hats-00/00000000/00000007 to s3://maru-test/hats-00/00000000/00000007
upload: hats-00/00000000/00000000 to s3://maru-test/hats-00/00000000/00000000
upload: hats-00/00000000/00000013 to s3://maru-test/hats-00/00000000/00000013
upload: hats-00/00000000/00000001 to s3://maru-test/hats-00/00000000/00000001
upload: hats-00/00000000/00000015 to s3://maru-test/hats-00/00000000/00000015
upload: hats-00/00000000/00000004 to s3://maru-test/hats-00/00000000/00000004
upload: hats-00/00000000/00000014 to s3://maru-test/hats-00/00000000/00000014
upload: hats-00/00000000/00000005 to s3://maru-test/hats-00/00000000/00000005

real	2m9.484s
user	0m0.773s
sys	0m0.198s

It takes less than 2 seconds against our existing production cluster, which runs with the 1.2.1 release.

$ time aws s3 cp --endpoint https://<REDACTED> hats-00 s3://kmizumar/hats-00 --recursive
upload: hats-00/00000000/00000007 to s3://kmizumar/hats-00/00000000/00000007
upload: hats-00/00000000/00000004 to s3://kmizumar/hats-00/00000000/00000004
upload: hats-00/00000000/00000003 to s3://kmizumar/hats-00/00000000/00000003
upload: hats-00/00000000/00000000 to s3://kmizumar/hats-00/00000000/00000000
upload: hats-00/00000000/00000001 to s3://kmizumar/hats-00/00000000/00000001
upload: hats-00/00000000/00000006 to s3://kmizumar/hats-00/00000000/00000006
upload: hats-00/00000000/00000002 to s3://kmizumar/hats-00/00000000/00000002
upload: hats-00/00000000/00000010 to s3://kmizumar/hats-00/00000000/00000010
upload: hats-00/00000000/00000005 to s3://kmizumar/hats-00/00000000/00000005
upload: hats-00/00000000/00000011 to s3://kmizumar/hats-00/00000000/00000011
upload: hats-00/00000000/00000009 to s3://kmizumar/hats-00/00000000/00000009
upload: hats-00/00000000/00000014 to s3://kmizumar/hats-00/00000000/00000014
upload: hats-00/00000000/00000013 to s3://kmizumar/hats-00/00000000/00000013
upload: hats-00/00000000/00000012 to s3://kmizumar/hats-00/00000000/00000012
upload: hats-00/00000000/00000008 to s3://kmizumar/hats-00/00000000/00000008
upload: hats-00/00000000/00000015 to s3://kmizumar/hats-00/00000000/00000015

real	0m1.347s
user	0m0.830s
sys	0m0.183s

I will upload the log files once I get the HDDS- number. Thank you.