[FLINK-3539] Sync running Execution and Task instances via heartbeats

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Duplicate
• Component/s: Runtime / Coordination
• Assignee: Unassigned
• Reporter: Ufuk Celebi
• Created: 29/Feb/16 14:44
• Updated: 30/Aug/19 09:51
• Resolved: 30/Aug/19 09:51

Description

StephanEwen pointed out that it is possible for the job manager and task manager state to get out of sync. If for example a cancel message from the job manager to the task manager is not delivered, the Execution will be failed at the job manager, but the task will keep on running at the task manager.

A simple way to prevent such situations is the following:

• The task manager and job manager heartbeats add information about currently running tasks/executions
• If a task manager reports a task, which is not a running Execution, that task is cancelled
• If a job manager reports a running execution, which is not a running task, the execution is failed

Issue Links

• duplicates: FLINK-11059 JobMaster may continue using an invalid slot if releasing idle slot meet a timeout