Uploaded image for project: 'Mesos'
  1. Mesos
  2. MESOS-9502

IOswitchboard cleanup could get stuck due to FD leak from a race.

    XMLWordPrintableJSON

    Details

      Description

      Our check container got stuck during destroy which in turned stucks the parent container. It is blocked by the I/O switchboard cleanup:

      1223 18:04:41.000000 16269 switchboard.cpp:814] Sending SIGTERM to I/O switchboard server (pid: 62854) since container 4d4074fa-bc87-471b-8659-08e519b68e13.16d02532-675a-4acb-964d-57459ecf6b67.check-e91521a3-bf72-4ac4-8ead-3950e31cf09e is being destroyed
      ....
      1227 04:45:38.000000 5189 switchboard.cpp:916] I/O switchboard server process for container 4d4074fa-bc87-471b-8659-08e519b68e13.16d02532-675a-4acb-964d-57459ecf6b67.check-e91521a3-bf72-4ac4-8ead-3950e31cf09e has terminated (status=N/A)

      Note the timestamp.

      Root Cause:
      Fundamentally, this is caused by a race between .discard() triggered by Check Container TIMEOUT and IOSB extracting ContainerIO object. This race could be exposed by overloaded/slow agent process. Please see how this race be triggered below:

      1. Right after IOSB server process is running, Check container Timed out and the checker process returns a failure, which would close the HTTP connection with agent.
      2. From the agent side, if the connection breaks, the handler will trigger a discard on the returned future and that will result in containerizer->launch()'s future transitioned to DISCARDED state.
      3. In containerizer, the DISCARDED state will be propagated back to IOSB prepare(), which stop its continuation on extracting the containerIO (it implies the object being cleaned up and FDs(one end of pipes created in IOSB) being closed in its destructor).
      4. Agent starts to destroy the container due to its discarded launch result, and asks IOSB to cleanup the container.
      5. IOSB server is still running, so agent sends a SIGTERM.
      6. SIGTERM handler unblocks the IOSB from redirecting (to redirect stdout/stderr from container to logger before exiting).
      7. io::redirect() calls io::splice() and reads the other end of those pipes forever.

      This issue is not easy to reproduce unless on a busy agent, because the timeout has to happen exactly AFTER IOSB server is running and BEFORE IOSB extracts containerIO.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                abudnik Andrei Budnik
                Reporter:
                mzhu Meng Zhu
                Shepherd:
                Gilbert Song
              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: