Uploaded image for project: 'Apache Ozone'
  1. Apache Ozone
  2. HDDS-7038

Significant performance degradation on 1.3.0-SNAPSHOT

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Critical
    • Resolution: Not A Bug
    • 1.3.0
    • None
    • Ozone Datanode, S3
    • None

    Description

      Found a significant performance degradation on our new cluster testing with 1.3.0-SNAPSHOT. For example, writing 16 files, size 1MiB each, using the aws s3 cp command takes more than 2 minutes against the new cluster.
       

      $ time aws s3 cp --endpoint https://<REDACTED> hats-00 s3://maru-test/hats-00 --recursive
      upload: hats-00/00000000/00000008 to s3://maru-test/hats-00/00000000/00000008
      upload: hats-00/00000000/00000002 to s3://maru-test/hats-00/00000000/00000002
      upload: hats-00/00000000/00000009 to s3://maru-test/hats-00/00000000/00000009
      upload: hats-00/00000000/00000003 to s3://maru-test/hats-00/00000000/00000003
      upload: hats-00/00000000/00000006 to s3://maru-test/hats-00/00000000/00000006
      upload: hats-00/00000000/00000011 to s3://maru-test/hats-00/00000000/00000011
      upload: hats-00/00000000/00000010 to s3://maru-test/hats-00/00000000/00000010
      upload: hats-00/00000000/00000012 to s3://maru-test/hats-00/00000000/00000012
      upload: hats-00/00000000/00000007 to s3://maru-test/hats-00/00000000/00000007
      upload: hats-00/00000000/00000000 to s3://maru-test/hats-00/00000000/00000000
      upload: hats-00/00000000/00000013 to s3://maru-test/hats-00/00000000/00000013
      upload: hats-00/00000000/00000001 to s3://maru-test/hats-00/00000000/00000001
      upload: hats-00/00000000/00000015 to s3://maru-test/hats-00/00000000/00000015
      upload: hats-00/00000000/00000004 to s3://maru-test/hats-00/00000000/00000004
      upload: hats-00/00000000/00000014 to s3://maru-test/hats-00/00000000/00000014
      upload: hats-00/00000000/00000005 to s3://maru-test/hats-00/00000000/00000005
      
      real	2m9.484s
      user	0m0.773s
      sys	0m0.198s
      

      It takes less than 2 seconds against our existing production cluster, which runs with the 1.2.1 release.

      $ time aws s3 cp --endpoint https://<REDACTED> hats-00 s3://kmizumar/hats-00 --recursive
      upload: hats-00/00000000/00000007 to s3://kmizumar/hats-00/00000000/00000007
      upload: hats-00/00000000/00000004 to s3://kmizumar/hats-00/00000000/00000004
      upload: hats-00/00000000/00000003 to s3://kmizumar/hats-00/00000000/00000003
      upload: hats-00/00000000/00000000 to s3://kmizumar/hats-00/00000000/00000000
      upload: hats-00/00000000/00000001 to s3://kmizumar/hats-00/00000000/00000001
      upload: hats-00/00000000/00000006 to s3://kmizumar/hats-00/00000000/00000006
      upload: hats-00/00000000/00000002 to s3://kmizumar/hats-00/00000000/00000002
      upload: hats-00/00000000/00000010 to s3://kmizumar/hats-00/00000000/00000010
      upload: hats-00/00000000/00000005 to s3://kmizumar/hats-00/00000000/00000005
      upload: hats-00/00000000/00000011 to s3://kmizumar/hats-00/00000000/00000011
      upload: hats-00/00000000/00000009 to s3://kmizumar/hats-00/00000000/00000009
      upload: hats-00/00000000/00000014 to s3://kmizumar/hats-00/00000000/00000014
      upload: hats-00/00000000/00000013 to s3://kmizumar/hats-00/00000000/00000013
      upload: hats-00/00000000/00000012 to s3://kmizumar/hats-00/00000000/00000012
      upload: hats-00/00000000/00000008 to s3://kmizumar/hats-00/00000000/00000008
      upload: hats-00/00000000/00000015 to s3://kmizumar/hats-00/00000000/00000015
      
      real	0m1.347s
      user	0m0.830s
      sys	0m0.183s
      

      I will upload the log files once I get the HDDS- number. Thank you.

      Attachments

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            Unassigned Unassigned
            kmizumar Kiyoshi Mizumaru
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment