[MESOS-1915] Docker containers that fail to launch are not killed

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: 0.20.1
• Fix Version/s: 0.21.0
• Component/s: agent

Environment

Mesos 0.20.1 using the docker executor with a private docker repository. Images often take up to 5 minutes to launch.

/etc/mesos-slave/executor_registration_timeout is set to '10mins'

• Target Version/s: 0.21.0
• Sprint: Mesosphere Q4 Sprint 1 10/31
• Assignee: Timothy Chen
• Reporter: Daniel Hall
• Created: 14/Oct/14 02:42
• Updated: 05/Nov/14 18:59
• Resolved: 23/Oct/14 17:54

Description

When we launch docker containers on our Mesos cluster using marathon we have noticed that we end up with several docker containers running, with only one of them actually being tracked my Mesos. When inspected the containers both have the same start time.

This seems to be because Mesos gives up on trying to start the container after 1min, but fails to clean up the docker container because it is is not yet running. Eventually the container starts alongside all the other attempts mesos has made and we end up with several containers running with only one being tracked by Mesos.

I've pasted some logs from the slave below filter for that particular task, but it is pretty easy to replicate in our environment so I'm happy to provide further logs, details and analysis as required. This is becoming a bit problem for us so we are happy to help as much as possible.

Oct 13 04:47:42 mesosslave-1 mesos-slave[16647]: I1013 04:47:42.776945 16661 docker.cpp:743] Starting container 'dd113461-4d18-4170-8e3f-9527e6d7f598' for task 'docker-test.11588a48-5294-11e4-adea-42010af0f51e' (and executor 'docker-test.11588a48-5294-11e4-adea-42010af0f51e') of framework '20140918-022627-519434250-5050-6171-0000'
Oct 13 04:48:42 mesosslave-1 mesos-slave[16647]: E1013 04:48:42.819563 16664 slave.cpp:2205] Failed to update resources for container dd113461-4d18-4170-8e3f-9527e6d7f598 of executor docker-test.11588a48-5294-11e4-adea-42010af0f51e running task docker-test.11588a48-5294-11e4-adea-42010af0f51e on status update for terminal task, destroying container: No container found
Oct 13 04:49:29 mesosslave-1 mesos-slave[16647]: I1013 04:49:29.916460 16665 slave.cpp:2538] Monitoring executor 'docker-test.11588a48-5294-11e4-adea-42010af0f51e' of framework '20140918-022627-519434250-5050-6171-0000' in container 'dd113461-4d18-4170-8e3f-9527e6d7f598'
Oct 13 04:49:31 mesosslave-1 mesos-slave[16647]: I1013 04:49:31.103175 16663 docker.cpp:1286] Updated 'cpu.shares' to 102 at /cgroup/cpu/docker/6a581f5c2174dc76bcfb2e5b89fd9a4310732c384d93901a8b37da8aeb700468 for container dd113461-4d18-4170-8e3f-9527e6d7f598
Oct 13 04:49:31 mesosslave-1 mesos-slave[16647]: I1013 04:49:31.105036 16663 docker.cpp:1321] Updated 'memory.soft_limit_in_bytes' to 32MB for container dd113461-4d18-4170-8e3f-9527e6d7f598

Issue Links

• contains: MESOS-1884 Composing Containerizer is not sending calls to still launching containers