[FLINK-31634] FLIP-301: Hybrid Shuffle supports Remote Storage

• Type: New Feature
• Status: Closed
• Priority: Major
• Resolution: Done
• Affects Version/s: 1.18.0
• Component/s: Runtime / Network
• Labels: Umbrella

Release Note

Hybrid Shuffle supports storing the shuffle data in remote storage. The remote storage path can be configured with the option `taskmanager.network.hybrid-shuffle.remote.path`. Hybrid Shuffle uses less network memory than before by decoupling the memory usage from the number of parallelisms, improving the stability and ease of use. For more detailed information, please refer to https://nightlies.apache.org/flink/flink-docs-master/docs/ops/batch/batch_shuffle/#hybrid-shuffle

• Assignee: Yuxin Tan
• Reporter: Yuxin Tan
• Created: 28/Mar/23 04:36
• Updated: 01/Aug/23 10:08
• Resolved: 24/Jul/23 08:27

Description

This is an umbrella ticket for FLIP-301.

Issue Links

• relates to: FLINK-32675 Add doc for the tiered storage of hybrid shuffle

Sub-Tasks

• Support writing records to the new tiered storage architecture: Closed
• Upstream supports reading buffers from tiered storage: Closed
• Implement HashBufferAccumulator: Closed
• Downstream supports reading buffers from tiered storage: Closed
• Introduce tiered storage memory manager: Closed
• Write the accumulated buffers to the right storage tier: Closed
• Introduce memory tier: Closed
• Introduce memory tier reader: Closed
• Implement SortBufferAccumulator: Closed
• Introduce disk tier: Closed
• Introduce disk tier reader: Closed
• Introduce remote storage tier: Closed
• Introduce remote storage tier reader: Closed
• Introduce the configurations for tiered storage: Closed