[FLINK-17075] Add task status reconciliation between TM and JM

• Type: Improvement
• Status: Closed
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 1.10.0, 1.11.0
• Fix Version/s: 1.12.0
• Component/s: Runtime / Coordination
• Labels: pull-request-available
• Assignee: Chesnay Schepler
• Reporter: Till Rohrmann
• Created: 09/Apr/20 16:51
• Updated: 14/Jan/21 09:26
• Resolved: 19/Oct/20 09:44

Description

In order to harden the TM and JM communication I suggest to let the TaskExecutor send the task statuses back to the JobMaster as part of the heartbeat payload (similar to FLINK-11059). This would allow to reconcile the states of both components in case that a status update message was lost as described by a user on the ML.

https://lists.apache.org/thread.html/ra9ed70866381f0ef0f4779633346722ccab3dc0d6dbacce04080b74e%40%3Cuser.flink.apache.org%3E

Issue Links

• causes: FLINK-18533 Race condition between task acknowledgement and first heartbeat
• is related to: FLINK-19954 Move execution deployment tracking logic from legacy EG code to SchedulerNG
• links to: GitHub Pull Request #12815
• mentioned in: Page Loading...