Uploaded image for project: 'Mesos'
  1. Mesos
  2. MESOS-9334

Container stuck at ISOLATING state due to libevent poll never returns.

    XMLWordPrintableJSON

Details

    • Mesosphere RI-6 Sprint 2018-31, Containerization R7 Sprint 32
    • 5

    Description

      We found UCR container may be stuck at `ISOLATING` state:

      2018-10-03 09:13:23: I1003 09:13:23.274561 2355 containerizer.cpp:3122] Transitioning the state of container 1e5b8fc3-5c9e-4159-a0b9-3d46595a5b54 from PREPARING to ISOLATING
      2018-10-03 09:13:23: I1003 09:13:23.279223 2354 cni.cpp:962] Bind mounted '/proc/5244/ns/net' to '/run/mesos/isolators/network/cni/1e5b8fc3-5c9e-4159-a0b9-3d46595a5b54/ns' for container 1e5b8fc3-5c9e-4159-a0b9-3d46595a5b54
      2018-10-03 09:23:22: I1003 09:23:22.879868 2354 containerizer.cpp:2459] Destroying container 1e5b8fc3-5c9e-4159-a0b9-3d46595a5b54 in ISOLATING state
      

       In the above logs, the state of container `1e5b8fc3-5c9e-4159-a0b9-3d46595a5b54` was transitioned to `ISOLATING` at 09:13:23, but did not transitioned to any other states until it was destroyed due to the executor registration timeout (10 mins). And the destroy can never complete since it needs to wait for the container to finish isolating.

      Attachments

        Issue Links

          Activity

            People

              qianzhang Qian Zhang
              qianzhang Qian Zhang
              Gilbert Song Gilbert Song
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: