[MESOS-9306] Mesos containerizer can get stuck during cgroup cleanup

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Won't Fix
• Affects Version/s: 1.4.0, 1.5.0, 1.6.0, 1.7.0, 1.8.0
• Component/s: agent, containerization
• Labels: containerizer mesosphere
• Sprint: Containerization: RI13 Sp 45, Containerization: RI14 Sp 46, Containerization: RI14 Sp 47
• Story Points: 5
• Assignee: Andrei Budnik
• Reporter: Greg Mann
• Created: 10/Oct/18 20:19
• Updated: 28/May/19 15:34
• Resolved: 28/May/19 15:34

Description

I observed a task group's executor container which failed to be completely destroyed after its associated tasks were killed. The following is an excerpt from the agent log which is filtered to include only lines with the container ID, d463b9fe-970d-4077-bab9-558464889a9e:

2018-10-10 14:20:50: I1010 14:20:50.204756  6799 containerizer.cpp:2963] Container d463b9fe-970d-4077-bab9-558464889a9e has exited
2018-10-10 14:20:50: I1010 14:20:50.204839  6799 containerizer.cpp:2457] Destroying container d463b9fe-970d-4077-bab9-558464889a9e in RUNNING state
2018-10-10 14:20:50: I1010 14:20:50.204859  6799 containerizer.cpp:3124] Transitioning the state of container d463b9fe-970d-4077-bab9-558464889a9e from RUNNING to DESTROYING
2018-10-10 14:20:50: I1010 14:20:50.204960  6799 linux_launcher.cpp:580] Asked to destroy container d463b9fe-970d-4077-bab9-558464889a9e
2018-10-10 14:20:50: I1010 14:20:50.204993  6799 linux_launcher.cpp:622] Destroying cgroup '/sys/fs/cgroup/freezer/mesos/d463b9fe-970d-4077-bab9-558464889a9e'
2018-10-10 14:20:50: I1010 14:20:50.205417  6806 cgroups.cpp:2838] Freezing cgroup /sys/fs/cgroup/freezer/mesos/d463b9fe-970d-4077-bab9-558464889a9e/mesos
2018-10-10 14:20:50: I1010 14:20:50.205477  6810 cgroups.cpp:2838] Freezing cgroup /sys/fs/cgroup/freezer/mesos/d463b9fe-970d-4077-bab9-558464889a9e
2018-10-10 14:20:50: I1010 14:20:50.205708  6808 cgroups.cpp:1229] Successfully froze cgroup /sys/fs/cgroup/freezer/mesos/d463b9fe-970d-4077-bab9-558464889a9e/mesos after 203008ns
2018-10-10 14:20:50: I1010 14:20:50.205878  6800 cgroups.cpp:1229] Successfully froze cgroup /sys/fs/cgroup/freezer/mesos/d463b9fe-970d-4077-bab9-558464889a9e after 339200ns
2018-10-10 14:20:50: I1010 14:20:50.206185  6799 cgroups.cpp:2856] Thawing cgroup /sys/fs/cgroup/freezer/mesos/d463b9fe-970d-4077-bab9-558464889a9e/mesos
2018-10-10 14:20:50: I1010 14:20:50.206226  6808 cgroups.cpp:2856] Thawing cgroup /sys/fs/cgroup/freezer/mesos/d463b9fe-970d-4077-bab9-558464889a9e
2018-10-10 14:20:50: I1010 14:20:50.206455  6808 cgroups.cpp:1258] Successfully thawed cgroup /sys/fs/cgroup/freezer/mesos/d463b9fe-970d-4077-bab9-558464889a9e after 83968ns
2018-10-10 14:20:50: I1010 14:20:50.306803  6810 cgroups.cpp:1258] Successfully thawed cgroup /sys/fs/cgroup/freezer/mesos/d463b9fe-970d-4077-bab9-558464889a9e/mesos after 100.50816ms
2018-10-10 14:20:50: I1010 14:20:50.307531  6805 linux_launcher.cpp:654] Destroying cgroup '/sys/fs/cgroup/systemd/mesos/d463b9fe-970d-4077-bab9-558464889a9e'
2018-10-10 14:21:40: W1010 14:21:40.032855  6809 containerizer.cpp:2401] Skipping status for container d463b9fe-970d-4077-bab9-558464889a9e because: Container does not exist
2018-10-10 14:22:40: W1010 14:22:40.031224  6800 containerizer.cpp:2401] Skipping status for container d463b9fe-970d-4077-bab9-558464889a9e because: Container does not exist
2018-10-10 14:23:40: W1010 14:23:40.031946  6799 containerizer.cpp:2401] Skipping status for container d463b9fe-970d-4077-bab9-558464889a9e because: Container does not exist
2018-10-10 14:24:40: W1010 14:24:40.032979  6804 containerizer.cpp:2401] Skipping status for container d463b9fe-970d-4077-bab9-558464889a9e because: Container does not exist
2018-10-10 14:25:40: W1010 14:25:40.030784  6808 containerizer.cpp:2401] Skipping status for container d463b9fe-970d-4077-bab9-558464889a9e because: Container does not exist
2018-10-10 14:26:40: W1010 14:26:40.032526  6810 containerizer.cpp:2401] Skipping status for container d463b9fe-970d-4077-bab9-558464889a9e because: Container does not exist
2018-10-10 14:27:40: W1010 14:27:40.029932  6801 containerizer.cpp:2401] Skipping status for container d463b9fe-970d-4077-bab9-558464889a9e because: Container does not exist

The last log line from the containerizer's destroy path is:

14:20:50.307531  6805 linux_launcher.cpp:654] Destroying cgroup '/sys/fs/cgroup/systemd/mesos/d463b9fe-970d-4077-bab9-558464889a9e'

(that is the second such log line, from LinuxLauncherProcess::_destroy)
Then we just see

containerizer.cpp:2401] Skipping status for container d463b9fe-970d-4077-bab9-558464889a9e because: Container does not exist

repeatedly, which occurs because the agent's GET_CONTAINERS call is being polled once per minute. This seems to indicate that the container in question is still in the agent's containers_ map.

So, it seems that the containerizer is stuck either in the Linux launcher's destroy() code path, or the containerizer's destroy() code path.