[MESOS-9191] Docker command executor may stuck at infinite unkillable loop.

• Type: Bug
• Status: Accepted
• Priority: Major
• Resolution: Unresolved
• Component/s: containerization, docker
• Labels: containerizer
• Target Version/s: 1.5.4, 1.6.3, 1.8.1, 1.9.0, 1.7.4
• Epic Link: Docker Improvements
• Sprint: Mesosphere Sprint 2018-29, Mesosphere RI-6 Sprint 2018-30, Mesosphere RI-6 Sprint 2018-31
• Story Points: 3
• Assignee: Andrei Budnik
• Reporter: Gilbert Song
• Shepherd: Qian Zhang
• Created: 29/Aug/18 07:05
• Updated: 29/Apr/20 05:01

Description

Due to the change from https://issues.apache.org/jira/browse/MESOS-8574, the behavior of docker command executor to discard the future of docker stop was changed. If there is a new killTask() invoked and there is an existing docker stop in pending state, the old one would call discard and then execute the new one. This is ok for most of cases.

However, docker stop could take long (depends on grace period and whether the application could handle SIGTERM). If the framework retry killTask more frequently than grace period (depends on killpolicy API, env var, or agent flags), then the executor may be stuck forever with unkillable tasks. Because everytime before the docker stop finishes, the future of docker stop is discarded by the new incoming killTask.

We should consider re-use grace period before calling discard() to a pending docker stop future.

Issue Links

• relates to: MESOS-8574 Docker executor makes no progress when 'docker inspect' hangs