Uploaded image for project: 'Mesos'
  1. Mesos
  2. MESOS-8573

Container stuck in PULLING when Docker daemon hangs

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.5.0
    • Fix Version/s: 1.4.2, 1.5.1, 1.6.0
    • Component/s: None
    • Labels:

      Description

      When the force argument is not set to true, Docker::pull will always perform a docker inspect call before it does a docker pull. If either of these two Docker CLI calls hangs indefinitely, the Docker container will be stuck in the PULLING state. This means that we make no further progress in the launch() call path, so the executor binary is never executed, the Future associated with the launch() call is never failed or satisfied, and wait() is never called on the container. The agent chains the executor cleanup onto that wait() call which is never made. So, when the executor registration timeout elapses, containerizer->destroy() is called on the executor container, but the rest of the executor cleanup is never performed, and no terminal task status update is sent.

      This leaves the task destined for that Docker executor stuck in TASK_STAGING from the framework's perspective, and attempts to kill the task will fail.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                gilbert Gilbert Song
                Reporter:
                greggomann Greg Mann
                Shepherd:
                Greg Mann
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: