[FLINK-25774] Restrict the maximum number of buffers can be used per result partition for sort-shuffle

• Type: Sub-task
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.15.0
• Component/s: Runtime / Network
• Labels: pull-request-available
• Assignee: Yingjie Cao
• Reporter: Yingjie Cao
• Created: 24/Jan/22 10:21
• Updated: 27/Jan/22 08:03
• Resolved: 27/Jan/22 08:03

Description

Currently, for sort-shuffle, the maximum number of buffers can be used per result partition is Integer.MAX_VALUE. However, if too many buffers are taken by one result partition, other result partitions and input gates may spend too much time waiting for buffers which can influence performance. This ticket aims to restrict the maximum number of buffers can be used per result partition and the selected value is an empirical one based on the TPC-DS test results.

Issue Links

• links to: GitHub Pull Request #18470