[FLINK-1833] Refactor partition availability notification in ExecutionGraph

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Won't Do
• Affects Version/s: 0.10.0
• Component/s: Runtime / Coordination
• Assignee: Unassigned
• Reporter: Ufuk Celebi
• Created: 07/Apr/15 09:10
• Updated: 01/Mar/19 07:37
• Resolved: 01/Mar/19 07:37

Description

The mechanism to notify the JobManager about available result partitions is hard to understand. The are two parts to this:

1) JobManager

• The deployment of receivers happens in the Execution class although it is by now totally unrelated to the state of a specific execution. I propose to move this to the respective IntermediateResultPartition.
• The deployment information for a receiver is spread across different components: when creating the TaskDeploymentDescriptor and the "caching" of partition infos at the consuming vertex. This is very hard to follow and results in unnecessary messages being sent (which are discarded at the TM).

2) TaskManager

• Pipelined results notify where you would expect it in the ResultPartition, but blocking results don't have an extra message and are implicitly piggy-backed to the final state transition, after which the job manager deploys receivers if all blocking partitions of a result have been produced.

Issue Links

• is related to: FLINK-10653 Introduce Pluggable Shuffle Service Architecture
• is superceded by: FLINK-10653 Introduce Pluggable Shuffle Service Architecture