Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-1672

Refactor task registration/unregistration

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 0.9
    • Runtime / Coordination
    • None

    Description

      Current control flow for task registrations

      1. JM submits a TaskDeploymentDescriptor to a TM
        1. TM registers the required JAR files with the LibraryCacheManager and returns the user code class loader
        2. TM creates a Task instance and registers the task in the runningTasks map
        3. TM creates a TaskInputSplitProvider
        4. TM creates a RuntimeEnvironment and sets it as the environment for the task
        5. TM registers the task with the network environment
        6. TM sends async msg to profiler to monitor tasks
        7. TM creates temporary files in file cache
        8. TM tries to start the task

      If any operation >= 1.2 fails:

      • TM calls task.failExternally()
      • TM removes temporary files from file cache
      • TM unregisters the task from the network environment
      • TM sends async msg to profiler to unmonitor tasks
      • TM calls unregisterMemoryManager on task

      If 1.1 fails, only unregister from LibraryCacheManager.

      RuntimeEnvironment, Task, TaskManager separation

      The RuntimeEnvironment has references to certain components of the task manager like memory manager, which are accecssed from the task. Furthermore it implements Runnable, and creates the executing task Thread. The Task instance essentially wraps the RuntimeEnvironment and allows asynchronous state management of the task (RUNNING, FINISHED, etc.).

      The way that the state updates affect the task is not that obvious: state changes trigger messages to the TM, which for final states further trigger a msg to unregister the task. The way that tasks are unregistered again depends on the state of the task.


      I would propose to refactor this to make the way the state handling/registration/unregistration is handled is more transparent.

      Attachments

        Activity

          People

            sewen Stephan Ewen
            uce Ufuk Celebi
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: