[FLINK-29975] Let hybrid full spilling strategy supports partition reuse

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Done
• Affects Version/s: 1.17.0
• Fix Version/s: 1.17.0
• Component/s: Runtime / Network
• Labels: pull-request-available
• Assignee: Weijie Guo
• Reporter: Weijie Guo
• Created: 10/Nov/22 08:55
• Updated: 17/Nov/22 10:19
• Resolved: 17/Nov/22 10:19

Description

Partition reuse is a very useful optimization in some topologies. In essence, multiple downstream tasks consume the same subpartition's data. Therefore, hybrid shuffle should also enjoy the benefits it brings. After FLINK-28889, we are finally able to achieve repeated consumption at the subpartition level for hybrid full spilling strategy, so let's make it better.

Issue Links

• links to: GitHub Pull Request #21284