[FLINK-25781] Adjust the maximum number of buffers can be used per result partition for data read of sort-shuffle

• Type: Sub-task
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.15.0
• Component/s: Runtime / Network
• Labels: pull-request-available
• Assignee: Yingjie Cao
• Reporter: Yingjie Cao
• Created: 24/Jan/22 11:43
• Updated: 27/Jan/22 08:05
• Resolved: 27/Jan/22 08:05

Description

In the current sort-shuffle implementation, the maximum number of buffers can be used per result partition for shuffle data read is 32M. However, for large parallelism jobs, 32M is not enough and for small parallelism jobs, 32M may waste buffers. This ticket aims to adjust the maximum number of buffers can be used per result partition to let it adjust according to parallelism and the selected value is an empirical one based on the TPC-DS test results.

Issue Links

• links to: GitHub Pull Request #18473