[MESOS-6026] Tasks mistakenly marked as FAILED due to race b/w sendExecutorTerminatedStatusUpdate() and _statusUpdate().

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.0.2, 1.1.0
• Component/s: agent
• Labels: mesosphere
• Target Version/s: 1.0.2
• Sprint: Mesosphere Sprint 41
• Assignee: Benjamin Mahler
• Reporter: Kapil Arya
• Shepherd: Vinod Kone
• Created: 10/Aug/16 22:57
• Updated: 28/Oct/16 09:27
• Resolved: 25/Aug/16 03:02

Description

Due to a race between ⁠sendExecutorTerminatedStatusUpdate()⁠ and ⁠_statusUpdate()⁠ that happens when the task has just finished and the executor is exiting.

Here is an example of slave log messages:

Aug 10 21:32:53 ip-10-10-0-205 mesos-slave[20413]: I0810 21:32:53.959374 20418 slave.cpp:3211] Handling status update TASK_FINISHED (UUID: fd79d0bd-4ece-41dc-bced-b93491f6bb2e) for task 291 of framework 340dfe26-a09f-4857-85b8-faba5f8d95df-0008 from executor(1)@10.10.0.205:53504
Aug 10 21:32:53 ip-10-10-0-205 mesos-slave[20413]: I0810 21:32:53.959604 20418 slave.cpp:3732] executor(1)@10.10.0.205:53504 exited
Aug 10 21:32:53 ip-10-10-0-205 mesos-slave[20413]: I0810 21:32:53.959643 20418 slave.cpp:4089] Executor '291' of framework 340dfe26-a09f-4857-85b8-faba5f8d95df-0008 exited with status 0
Aug 10 21:32:53 ip-10-10-0-205 mesos-slave[20413]: I0810 21:32:53.959744 20418 slave.cpp:3211] Handling status update TASK_FAILED (UUID: b94722fb-1658-4936-b604-6d642ffe20a0) for task 291 of framework 340dfe26-a09f-4857-85b8-faba5f8d95df-0008 from @0.0.0.0:0

As can be noticed, the task is marked as TASK_FAILED after the executor has exited.

Issue Links

• duplicates: MESOS-2346 Docker tasks exiting normally, but returning TASK_FAILED
• relates to: MESOS-5276 HTTPCommandExecutor should terminate after it receives an ACK from the agent