[FLINK-22153] Manually test the sort-merge blocking shuffle

• Type: Improvement
• Status: Resolved
• Priority: Blocker
• Resolution: Resolved
• Affects Version/s: 1.13.0
• Fix Version/s: 1.13.0
• Component/s: Runtime / Network
• Assignee: Till Rohrmann
• Reporter: Yingjie Cao
• Created: 08/Apr/21 10:13
• Updated: 27/Apr/21 18:03
• Resolved: 16/Apr/21 08:05

Description

In 1.12, we introduced sort-merge blocking shuffle to Flink and in 1.13, the feature was optimized which improves the usability (fix direct memory OOM issue) and performance (introduce IO scheduling and broadcast optimization).

The sort-merge blocking shuffle can be tested following the bellow process:

1. Write a simple batch job using either sql/table or DataStream API; (Word count should be enough)
2. Enable sort-merge blocking shuffle by setting taskmanager.network.sort-shuffle.min-parallelism to 1 in the Flink configuration file;
3. Submit and run the batch job with different parallelism and data volume;
4. Tune the relevant config options (taskmanager.network.blocking-shuffle.compression.enabled, taskmanager.network.sort-shuffle.min-buffers, taskmanager.memory.framework.off-heap.batch-shuffle.size) and see the influence. 

 

Issue Links

• is related to: FLINK-21967 Add documentation on the operation of blocking result partition
• is related to: FLINK-19614 Further optimization of sort-merge based blocking shuffle

Sub-Tasks

• Document sort-merge blocking shuffle: Closed
• Improve log messages of sort-merge blocking shuffle: Closed