Uploaded image for project: 'Mesos'
  1. Mesos
  2. MESOS-2601

Tasks are not removed after recovery from slave and mesos containerizer

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.22.1
    • Fix Version/s: 0.22.1, 0.23.0
    • Component/s: agent, containerization
    • Labels:
      None

      Description

      We've seen in our test cluster that tasks that were launched with the mesos containerizer are recovered after slave restart, but actual command process is not running anymore and the checkpointed executor is not marked as completed.

      The Mesos containerizer recovers and all the isolators couldn't recover the task, but the containerizer itself is somehow never removed and the monitor kept calling usage on the containerizer.

      Relevant log lines from the beginning of slave recovery:

      I0408 18:06:33.261379 32504 slave.cpp:577] Successfully attached file '/hdd/mesos/slave/slaves/20150401-160104-251662508-5050-2197-S1/frameworks/20141222-194154-218108076-5050-4125-0004/executors/ct:1427921848104:0:EM DataDog Uploader:/runs/990741ed-909e-49cc-83f8-be63298872da'
      ...
      I0408 18:06:36.583277 32511 containerizer.cpp:350] Recovering container '990741ed-909e-49cc-83f8-be63298872da' for executor 'ct:1427921848104:0:EM DataDog Uploader:' of framework 20141222-194154-218108076-5050-4125-0004
      ....
      I0408 18:06:37.017122 32511 linux_launcher.cpp:162] Couldn't find freezer cgroup for container 990741ed-909e-49cc-83f8-be63298872da, assuming already destroyed
      W0408 18:06:37.074916 32496 cpushare.cpp:199] Couldn't find cgroup for container 990741ed-909e-49cc-83f8-be63298872da
      I0408 18:06:37.075173 32486 mem.cpp:158] Couldn't find cgroup for container 990741ed-909e-49cc-83f8-be63298872da
      E0408 18:06:37.092279 32496 containerizer.cpp:1136] Error in a resource limitation for container 990741ed-909e-49cc-83f8-be63298872da: Unknown container
      I0408 18:06:37.092643 32496 containerizer.cpp:906] Destroying container '990741ed-909e-49cc-83f8-be63298872da'
      W0408 18:06:37.229626 32501 containerizer.cpp:807] Ignoring update for currently being destroyed container: 990741ed-909e-49cc-83f8-be63298872da
      W0408 18:06:38.129873 32484 containerizer.cpp:844] Skipping resource statistic for container 990741ed-909e-49cc-83f8-be63298872da because: Unknown container
      W0408 18:06:38.129909 32484 containerizer.cpp:844] Skipping resource statistic for container 990741ed-909e-49cc-83f8-be63298872da because: Unknown container

        Issue Links

          Activity

          Hide
          tnachen Timothy Chen added a comment -

          Yes this is going to be in 0.22.1, going to mark this resolved.

          Show
          tnachen Timothy Chen added a comment - Yes this is going to be in 0.22.1, going to mark this resolved.
          Hide
          adam-mesos Adam B added a comment -

          commit 1a69b55652e867fee80ffb224ee3096b9e6c5390
          Author: Timothy Chen <tnachen@apache.org>
          Date: Wed Apr 15 15:18:35 2015 -0700

          Fixed recover tasks only by the intiated containerizer.

          Review: https://reviews.apache.org/r/33257

          Show
          adam-mesos Adam B added a comment - commit 1a69b55652e867fee80ffb224ee3096b9e6c5390 Author: Timothy Chen <tnachen@apache.org> Date: Wed Apr 15 15:18:35 2015 -0700 Fixed recover tasks only by the intiated containerizer. Review: https://reviews.apache.org/r/33257
          Hide
          jieyu Jie Yu added a comment -

          Timothy Chen Can this ticket be resolved? Do you plan to include this fix into 0.22.1?

          Show
          jieyu Jie Yu added a comment - Timothy Chen Can this ticket be resolved? Do you plan to include this fix into 0.22.1?
          Hide
          adam-mesos Adam B added a comment -
          Show
          adam-mesos Adam B added a comment - Tim's fix: https://reviews.apache.org/r/33257/
          Hide
          benjaminhindman Benjamin Hindman added a comment -

          Proposed fix: https://reviews.apache.org/r/33459

          Looks like Timothy Chen has something in the works too, can you paste it here please Tim?

          Show
          benjaminhindman Benjamin Hindman added a comment - Proposed fix: https://reviews.apache.org/r/33459 Looks like Timothy Chen has something in the works too, can you paste it here please Tim?
          Hide
          tnachen Timothy Chen added a comment -

          Yes I think the problem exists in both docker and mesos containerizer, where we simply try to recover every task.
          We should checkpoint the containerizer as jie mentioned and fix this soon.

          Show
          tnachen Timothy Chen added a comment - Yes I think the problem exists in both docker and mesos containerizer, where we simply try to recover every task. We should checkpoint the containerizer as jie mentioned and fix this soon.
          Hide
          jieyu Jie Yu added a comment -

          Checked the docker containerizer code, looks like docker containerizer will try to recover containers created by Mesos as well. We really need to checkpoint what containerizer creates the container.

          Show
          jieyu Jie Yu added a comment - Checked the docker containerizer code, looks like docker containerizer will try to recover containers created by Mesos as well. We really need to checkpoint what containerizer creates the container.
          Hide
          idownes Ian Downes added a comment -

          IIUC: if you have "--containerizer=mesos,docker" then a container created by the DockerContainerizer (after the MesosContainerizer declined to launch it) will subsequently first attempt to recover with the Mesos containerizer.

          I think it's better to have "--containerizer=docker,mesos"....

          Show
          idownes Ian Downes added a comment - IIUC: if you have "--containerizer=mesos,docker" then a container created by the DockerContainerizer (after the MesosContainerizer declined to launch it) will subsequently first attempt to recover with the Mesos containerizer. I think it's better to have "--containerizer=docker,mesos"....
          Hide
          cmaloney Cody Maloney added a comment -

          Jie Yu The --containerizer flag has never been changed on the host. Isolator flags also haven't changed at runtime ever on the host (only with a full workdir wipeout / reboot / kill all tasks / new slave id).

          Show
          cmaloney Cody Maloney added a comment - Jie Yu The --containerizer flag has never been changed on the host. Isolator flags also haven't changed at runtime ever on the host (only with a full workdir wipeout / reboot / kill all tasks / new slave id).
          Hide
          idownes Ian Downes added a comment -

          It's probably best to recover with the same containerizers and in the same order, at least until we start checkpointing which containerizer created a container.

          Show
          idownes Ian Downes added a comment - It's probably best to recover with the same containerizers and in the same order, at least until we start checkpointing which containerizer created a container.
          Hide
          jieyu Jie Yu added a comment -

          So you changed the --containerizer flag? If yes, then that's a known issue because we don't checkpoint that information.

          Show
          jieyu Jie Yu added a comment - So you changed the --containerizer flag? If yes, then that's a known issue because we don't checkpoint that information.
          Hide
          tnachen Timothy Chen added a comment -

          Actually I just realized that the task was launched as a docker task, so the docker containerizer should be the one only to recover it. But since the executor is still running and stuck because of the issue earlier of having a tasks stuck in staging, the mesos containerizer was able to find it and wait for it to finish.

          I think the gist of the issue is that the mesos containerizer shouldn't recover tasks started by docker containerizer as it could run into problems like thinking it launched the executor and so on.

          Show
          tnachen Timothy Chen added a comment - Actually I just realized that the task was launched as a docker task, so the docker containerizer should be the one only to recover it. But since the executor is still running and stuck because of the issue earlier of having a tasks stuck in staging, the mesos containerizer was able to find it and wait for it to finish. I think the gist of the issue is that the mesos containerizer shouldn't recover tasks started by docker containerizer as it could run into problems like thinking it launched the executor and so on.
          Hide
          jieyu Jie Yu added a comment -

          What's the slave isolation flag used before and after the restart?

          Show
          jieyu Jie Yu added a comment - What's the slave isolation flag used before and after the restart?
          Hide
          idownes Ian Downes added a comment -

          Can you please explain further:

          "The Mesos containerizer recovers and all the isolators couldn't recover the task"
          Are you saying that recovery is not successful? If any isolator fails to recover then the containerizer should fail to recover and the slave shouldn't start...

          Recovery should be successful even if the executor has exited, it'll just be a slight delay until the reaper polls on the executor's pid and notices it has exited . This is not perfect in that it's possible that while a slave is restarting an executor could exit and another process starts with the same pid. The reaper, as implemented now, will not detect that and thus the containerizer won't realize the executor has terminated.

          Show
          idownes Ian Downes added a comment - Can you please explain further: "The Mesos containerizer recovers and all the isolators couldn't recover the task" Are you saying that recovery is not successful? If any isolator fails to recover then the containerizer should fail to recover and the slave shouldn't start... Recovery should be successful even if the executor has exited, it'll just be a slight delay until the reaper polls on the executor's pid and notices it has exited . This is not perfect in that it's possible that while a slave is restarting an executor could exit and another process starts with the same pid. The reaper, as implemented now, will not detect that and thus the containerizer won't realize the executor has terminated.
          Hide
          jieyu Jie Yu added a comment -

          Can you paste the log?

          Show
          jieyu Jie Yu added a comment - Can you paste the log?

            People

            • Assignee:
              tnachen Timothy Chen
              Reporter:
              tnachen Timothy Chen
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development