Attach filesAttach ScreenshotVotersWatch issueWatchersLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Closed
    • Major
    • Resolution: Won't Do
    • None
    • None
    • None

    Description

      Ratis streaming supports writeAsync(File, position, count) API, which is backed by Netty FileRegion and FileChannel.transferTo.

      Let's introduce a new interface FileRegionStreamOutput to implement zero copy in OzoneDataStreamOutput, KeyDataStreamOutput, BlockDataStreamOutputEntry and BlockDataStreamOutput.

      Our tests shows Streaming(FileRegion) API have better performance over Async API.

      And it's a good performance reference for the Streaming(ByteBuffer) API.

      https://issues.apache.org/jira/browse/HDDS-5685?focusedCommentId=17416557&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17416557

      Test A (single 10GB key)

      API Min Max Avg
      Async 64s 72s 68s
      Streaming (MappedByteBuffer) 55s 68s 62s
      Streaming (FileRegion) 55s 64s 61s

      Test B (24x 1GB keys from 1 client)

      API Min Max Avg
      Async 48s 54s 51s
      Streaming (MappedByteBuffer) 32s >630s 33s
      Streaming (FileRegion) 36s 39s 38s

       

      Attachments

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            ckj Kaijie Chen
            ckj Kaijie Chen
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment