Uploaded image for project: 'Mesos'
  1. Mesos
  2. MESOS-5723

SSL-enabled libprocess will leak incoming links to forks

    XMLWordPrintableJSON

Details

    • Mesosphere Sprint 38
    • 2

    Description

      Encountered two different buggy behaviors that can be tracked down to the same underlying problem.

      Repro #1 (non-crashy):
      (1) Start a master. Doesn't matter if SSL is enabled or not.
      (2) Start an agent, with SSL enabled. Downgrade support has the same problem. The master/agent link to one another.
      (3) Run a sleep task. Keep this alive. If you inspect FDs at this point, you'll notice the task has inherited the link FD (master -> agent).
      (4) Restart the agent. Due to (3), the master's link stays open.
      (5) Check master's logs for the agent's re-registration message.
      (6) Check the agent's logs for re-registration. The message will not appear. The master is actually using the old link which is not connected to the agent.


      Repro #2 (crashy):
      (1) Start a master. Doesn't matter if SSL is enabled or not.
      (2) Start an agent, with SSL enabled. Downgrade support has the same problem.
      (3) Run ~100 sleep task one after the other, keep them all alive. Each task links back to the agent. Due to an FD leak, each task will inherit the incoming links from all other actors...
      (4) At some point, the agent will run out of FDs and kernel panic.


      It appears that the SSL socket accept call is missing os::nonblock and os::cloexec calls:
      https://github.com/apache/mesos/blob/4b91d936f50885b6a66277e26ea3c32fe942cf1a/3rdparty/libprocess/src/libevent_ssl_socket.cpp#L794-L806

      For reference, here's poll socket's accept:
      https://github.com/apache/mesos/blob/4b91d936f50885b6a66277e26ea3c32fe942cf1a/3rdparty/libprocess/src/poll_socket.cpp#L53-L75

      Attachments

        Activity

          People

            kaysoky Joseph Wu
            kaysoky Joseph Wu
            Joris Van Remoortere Joris Van Remoortere
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: