[FLINK-25780] Reduce the maximum size of data output buffers per result partition for sort-shuffle

• Type: Sub-task
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.15.0
• Component/s: Runtime / Network
• Labels: pull-request-available
• Assignee: Yingjie Cao
• Reporter: Yingjie Cao
• Created: 24/Jan/22 11:27
• Updated: 27/Jan/22 08:04
• Resolved: 27/Jan/22 08:04

Description

The data output buffer of sort-shuffle is for better disk IO performance and currently, the total data output buffer size is 16M which is pretty big. However, blocking request too many buffers may influence performance. This ticket aims to reduce the maximum number of data output buffers to reduce the buffer request time. The selected value is an empirical one based on the TPC-DS test results.

Issue Links

• links to: GitHub Pull Request #18471