[MESOS-9501] Mesos executor fails to terminate and gets stuck after agent host reboot.

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 1.5.1, 1.6.1, 1.7.0
• Fix Version/s: 1.4.3, 1.5.2, 1.6.2, 1.7.1, 1.8.0
• Component/s: containerization
• Sprint: Containerization R9 Sprint 37
• Story Points: 5
• Assignee: Qian Zhang
• Reporter: Meng Zhu
• Shepherd: Gilbert Song
• Created: 27/Dec/18 01:51
• Updated: 02/Apr/19 07:39
• Resolved: 11/Jan/19 21:38

Description

When an agent host reboots, all of its containers are gone but the agent will still try to recover from its checkpointed state after reboot.

The agent will soon discover that all the cgroup hierarchies are gone and assume (correctly) that the containers are destroyed.

However, when trying to terminate the executor, the agent will first try to wait for the exit status of its container:
https://github.com/apache/mesos/blob/master/src/slave/containerizer/mesos/containerizer.cpp#L2631

Agent dose so by `waitpid` on the checkpointed child process pid. If, after the agent host reboot, a new process with the same pid gets spawned, then the parent will wait for the wrong child process. This could get stuck until the wrongly waited-for process is somehow exited, see `ReaperProcess::wait()`: https://github.com/apache/mesos/blob/master/3rdparty/libprocess/src/reap.cpp#L88-L114

This will block the executor termination as well as future task status update (e.g. master might still think the task is running).

Issue Links

• is related to: MESOS-8125 Agent should properly handle recovering an executor when its pid is reused
• is related to: MESOS-9672 Docker containerizer should ignore pids of executors that do not pass the connection check.