Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-1833

Refactor partition availability notification in ExecutionGraph

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Won't Do
    • Affects Version/s: 0.10.0
    • Fix Version/s: None
    • Component/s: Runtime / Coordination
    • Labels:
      None

      Description

      The mechanism to notify the JobManager about available result partitions is hard to understand. The are two parts to this:

      1) JobManager

      • The deployment of receivers happens in the Execution class although it is by now totally unrelated to the state of a specific execution. I propose to move this to the respective IntermediateResultPartition.
      • The deployment information for a receiver is spread across different components: when creating the TaskDeploymentDescriptor and the "caching" of partition infos at the consuming vertex. This is very hard to follow and results in unnecessary messages being sent (which are discarded at the TM).

      2) TaskManager

      • Pipelined results notify where you would expect it in the ResultPartition, but blocking results don't have an extra message and are implicitly piggy-backed to the final state transition, after which the job manager deploys receivers if all blocking partitions of a result have been produced.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                uce Ufuk Celebi
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: