[FLINK-28373] Read a full buffer of data per file IO read request for sort-shuffle

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.16.0
• Component/s: Runtime / Network
• Labels: pull-request-available
• Assignee: Yuxin Tan
• Reporter: Yingjie Cao
• Created: 04/Jul/22 08:13
• Updated: 12/Aug/22 04:31
• Resolved: 09/Aug/22 06:50

Description

Currently, for sort blocking shuffle, the corresponding data readers read shuffle data in buffer granularity. Before compression, each buffer is 32K by default, after compression the size will become smaller (may less than 10K). For file IO, this is pretty smaller. To achieve better performance and reduce IOPS, we can read more data per IO read request and parse buffer header and data in memory.

Issue Links

• causes: FLINK-28942 Deadlock may occurs when releasing readers for SortMergeResultPartition
• causes: FLINK-28789 TPC-DS tests failed due to release input gate for task failure
• links to: GitHub Pull Request #20335
• links to: GitHub Pull Request #20457