[FLINK-28377] Decrease the memory size per request for sort-shuffle data read from 8M to 4M

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.16.0
• Component/s: Runtime / Network
• Labels: pull-request-available
• Assignee: Yuxin Tan
• Reporter: Yingjie Cao
• Created: 04/Jul/22 08:54
• Updated: 22/Jul/22 03:35
• Resolved: 22/Jul/22 03:35

Description

Currently, for sort blocking shuffle, the corresponding data reader always allocate a fixed size of 8M buffers for shuffle data reading. FLINK-28373 can increase buffer utilization, after which, we can reduce the buffer size per request to reduce the time waiting for buffers. (This change is guarded by TPC-DS test that there is no performance regression)

Issue Links

• links to: GitHub Pull Request #20325