Details
-
Improvement
-
Status: Resolved
-
Critical
-
Resolution: Not A Bug
-
1.3.0
-
None
-
None
-
Our dist tarball is built with the following commit hash:
commit 843fac2fb646eecfc33103fdb16eaf77a66ca062 Author: Neil Joshi <neil2021.gres@gmail.com> Date: Wed Jul 13 22:29:59 2022 -0600 HDDS-6433. Refactor OMFailoverProxyProvider to provide a base for OM and GrpcOM FailoverProxyProviders (#3389)
- SCM (non-HA)
- OM (non-HA)
- Datanode x 36
- S3 Gateway
Our dist tarball is built with the following commit hash: commit 843fac2fb646eecfc33103fdb16eaf77a66ca062 Author: Neil Joshi <neil2021.gres@gmail.com> Date: Wed Jul 13 22:29:59 2022 -0600 HDDS-6433. Refactor OMFailoverProxyProvider to provide a base for OM and GrpcOM FailoverProxyProviders (#3389) SCM (non-HA) OM (non-HA) Datanode x 36 S3 Gateway
Description
Found a significant performance degradation on our new cluster testing with 1.3.0-SNAPSHOT. For example, writing 16 files, size 1MiB each, using the aws s3 cp command takes more than 2 minutes against the new cluster.
$ time aws s3 cp --endpoint https://<REDACTED> hats-00 s3://maru-test/hats-00 --recursive upload: hats-00/00000000/00000008 to s3://maru-test/hats-00/00000000/00000008 upload: hats-00/00000000/00000002 to s3://maru-test/hats-00/00000000/00000002 upload: hats-00/00000000/00000009 to s3://maru-test/hats-00/00000000/00000009 upload: hats-00/00000000/00000003 to s3://maru-test/hats-00/00000000/00000003 upload: hats-00/00000000/00000006 to s3://maru-test/hats-00/00000000/00000006 upload: hats-00/00000000/00000011 to s3://maru-test/hats-00/00000000/00000011 upload: hats-00/00000000/00000010 to s3://maru-test/hats-00/00000000/00000010 upload: hats-00/00000000/00000012 to s3://maru-test/hats-00/00000000/00000012 upload: hats-00/00000000/00000007 to s3://maru-test/hats-00/00000000/00000007 upload: hats-00/00000000/00000000 to s3://maru-test/hats-00/00000000/00000000 upload: hats-00/00000000/00000013 to s3://maru-test/hats-00/00000000/00000013 upload: hats-00/00000000/00000001 to s3://maru-test/hats-00/00000000/00000001 upload: hats-00/00000000/00000015 to s3://maru-test/hats-00/00000000/00000015 upload: hats-00/00000000/00000004 to s3://maru-test/hats-00/00000000/00000004 upload: hats-00/00000000/00000014 to s3://maru-test/hats-00/00000000/00000014 upload: hats-00/00000000/00000005 to s3://maru-test/hats-00/00000000/00000005 real 2m9.484s user 0m0.773s sys 0m0.198s
It takes less than 2 seconds against our existing production cluster, which runs with the 1.2.1 release.
$ time aws s3 cp --endpoint https://<REDACTED> hats-00 s3://kmizumar/hats-00 --recursive upload: hats-00/00000000/00000007 to s3://kmizumar/hats-00/00000000/00000007 upload: hats-00/00000000/00000004 to s3://kmizumar/hats-00/00000000/00000004 upload: hats-00/00000000/00000003 to s3://kmizumar/hats-00/00000000/00000003 upload: hats-00/00000000/00000000 to s3://kmizumar/hats-00/00000000/00000000 upload: hats-00/00000000/00000001 to s3://kmizumar/hats-00/00000000/00000001 upload: hats-00/00000000/00000006 to s3://kmizumar/hats-00/00000000/00000006 upload: hats-00/00000000/00000002 to s3://kmizumar/hats-00/00000000/00000002 upload: hats-00/00000000/00000010 to s3://kmizumar/hats-00/00000000/00000010 upload: hats-00/00000000/00000005 to s3://kmizumar/hats-00/00000000/00000005 upload: hats-00/00000000/00000011 to s3://kmizumar/hats-00/00000000/00000011 upload: hats-00/00000000/00000009 to s3://kmizumar/hats-00/00000000/00000009 upload: hats-00/00000000/00000014 to s3://kmizumar/hats-00/00000000/00000014 upload: hats-00/00000000/00000013 to s3://kmizumar/hats-00/00000000/00000013 upload: hats-00/00000000/00000012 to s3://kmizumar/hats-00/00000000/00000012 upload: hats-00/00000000/00000008 to s3://kmizumar/hats-00/00000000/00000008 upload: hats-00/00000000/00000015 to s3://kmizumar/hats-00/00000000/00000015 real 0m1.347s user 0m0.830s sys 0m0.183s
I will upload the log files once I get the HDDS- number. Thank you.