[FLINK-28380] Produce one intermediate dataset for multiple consumers consuming the same data

• Type: Sub-task
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.16.0
• Component/s: Client / Job Submission, Runtime / Coordination, Runtime / Network
• Labels: pull-request-available
• Assignee: Yingjie Cao
• Reporter: Yingjie Cao
• Created: 04/Jul/22 12:42
• Updated: 09/Aug/22 01:59
• Resolved: 09/Aug/22 01:59

Description

Currently, if one output of an upstream job vertex is consumed by multiple downstream job vertices, the upstream vertex will produce multiple dataset. For blocking shuffle, it means serialize and persist the same data multiple times. This ticket aims to optimize this behavior and make the upstream job vertex produce one dataset which will be read by multiple downstream vertex.

Issue Links

• links to: GitHub Pull Request #20351