[FLINK-35166] Improve the performance of Hybrid Shuffle when enable memory decoupling

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Fix Version/s: 2.0.0
• Component/s: Runtime / Network
• Labels: pull-request-available
• Assignee: Jiang Xin
• Reporter: Jiang Xin
• Created: 19/Apr/24 02:03
• Updated: 10/Jul/24 06:57

Description

Currently, the tiered result partition creates the SortBufferAccumulator with the number of expected buffers as min(numSubpartitions+1, 512), thus the SortBufferAccumulator may obtain very few buffers when the parallelism is small. We can easily make the number of expected buffers 512 by default to have a better performance when the buffers are sufficient.

Issue Links

• links to: GitHub Pull Request #24683