[HDDS-5752] Add FileRegion API for Ozone Streaming

• Type: Sub-task
• Status: Closed
• Priority: Major
• Resolution: Won't Do
• Labels: pull-request-available
• Assignee: Kaijie Chen
• Reporter: Kaijie Chen
• Created: 16/Sep/21 11:10
• Updated: 12/Dec/22 18:38
• Resolved: 29/Sep/21 07:48

Description

Ratis streaming supports writeAsync(File, position, count) API, which is backed by Netty FileRegion and FileChannel.transferTo.

Let's introduce a new interface FileRegionStreamOutput to implement zero copy in OzoneDataStreamOutput, KeyDataStreamOutput, BlockDataStreamOutputEntry and BlockDataStreamOutput.

Our tests shows Streaming(FileRegion) API have better performance over Async API.

And it's a good performance reference for the Streaming(ByteBuffer) API.

https://issues.apache.org/jira/browse/HDDS-5685?focusedCommentId=17416557&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17416557

Test A (single 10GB key)

API	Min	Max	Avg
Async	64s	72s	68s
Streaming (MappedByteBuffer)	55s	68s	62s
Streaming (FileRegion)	55s	64s	61s

Test B (24x 1GB keys from 1 client)

API	Min	Max	Avg
Async	48s	54s	51s
Streaming (MappedByteBuffer)	32s	>630s	33s
Streaming (FileRegion)	36s	39s	38s

 

Issue Links

• links to: GitHub Pull Request #2652
• links to: Github Pull Request #2652