[FLINK-27862] FLIP-235: Hybrid Shuffle Mode

• Type: New Feature
• Status: Closed
• Priority: Major
• Resolution: Done
• Fix Version/s: 1.16.0
• Component/s: Runtime / Network
• Labels: Umbrella

Release Note

We have introduced a new Hybrid Shuffle Mode for batch executions. It combines the advantages of blocking shuffle and pipelined shuffle (in streaming mode).
- Like blocking shuffle, it does not require upstream and downstream tasks to run simultaneously, which allows executing a job with little resources.
- Like pipelined shuffle, it does not require downstream tasks to be executed after upstream tasks finish, which reduces the overall execution time of the job when given sufficient resources.
- It adapts to custom preferences between persisting less data and restarting less tasks on failures, by providing different spilling strategies.

For more details: https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/ops/batch/batch_shuffle

• Assignee: Weijie Guo
• Reporter: Weijie Guo
• Created: 01/Jun/22 06:56
• Updated: 02/Nov/22 13:42
• Resolved: 05/Aug/22 02:56

Description

Introduce a new shuffle mode can overcome some of the problems of Pipelined Shuffle and Blocking Shuffle in batch scenarios, it can make best use of available resources and minimize disk IO load.

More details see FLIP-235

 

Sub-Tasks

• Refactor ResultPartitionType to decouple scheduling and partition release logic: Closed
• Introduce and support HYBRID resultPartitionType: Closed
• Introduce HsMemoryDataManager: Closed
• Introduce HsSpillingStrategy and Implement SelectiveSpillingStrategy、FullSpillingStrategy: Closed
• Introduce HsDataIndex: Closed
• Implement disk read and write logic for hybrid shuffle: Closed
• Introduce HsResultPartition and HsSubpartitionView: Closed
• Add document and tests for hybrid shuffle mode: Closed