[MESOS-7865] Agent may process a kill task and still launch the task.

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Fix Version/s: 1.1.3, 1.2.3, 1.3.2, 1.4.0
• Component/s: agent
• Target Version/s: 1.1.3, 1.2.3, 1.3.2, 1.4.0
• Sprint: Mesosphere Sprint 61, Mesosphere Sprint 62
• Story Points: 3
• Assignee: Benjamin Mahler
• Reporter: Benjamin Mahler
• Shepherd: Vinod Kone
• Created: 07/Aug/17 23:44
• Updated: 28/Aug/17 14:18
• Resolved: 23/Aug/17 22:31

Description

Based on the investigation of MESOS-7744, the agent has a race in which "queued" tasks can still be launched after the agent has processed a kill task for them. This race was introduced when Slave::statusUpdate was made asynchronous:

(1) Slave::__run completes, task is now within Executor::queuedTasks
(2) Slave::killTask locates the executor based on the task ID residing in queuedTasks, calls Slave::statusUpdate() with TASK_KILLED
(3) Slave::___run assumes that killed tasks have been removed from Executor::queuedTasks, but this now occurs asynchronously in Slave::_statusUpdate. So, the executor still sees the queued task and delivers it and adds the task to Executor::launchedTasks.
(3) Slave::_statusUpdate runs, removes the task from Executor::launchedTasks and adds it to Executor::terminatedTasks.

Issue Links

• relates to: MESOS-7744 Mesos Agent Sends TASK_KILL status update to Master, and still launches task