[FLINK-32709] Fix the bug of low memory utilization for Hybrid Shuffle

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.18.0
• Fix Version/s: 1.18.0
• Component/s: Runtime / Network
• Labels: pull-request-available
• Assignee: Yuxin Tan
• Reporter: Yuxin Tan
• Created: 28/Jul/23 02:22
• Updated: 11/Aug/23 02:40
• Resolved: 11/Aug/23 02:40

Description

Currently, each subpartition in Disk/Remote has a segment size of 8M. When writing segments to the Disk tier with a parallelism of 1000, only shuffle data exceeding 1000 * 8M can be written to the Memory tier again. However, for most shuffles, the data volume size falls below this limit, significantly impacting Memory tier utilization.
For better performance, it is necessary to address this issue to improve the memory tier utilization.

Issue Links

• links to: GitHub Pull Request #23095