[MESOS-3363] custom executor's child process intermittently leaks to be a child of slave

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 0.23.0
• Labels: mesosphere

Environment

vagrant@node-1:~$ uname -a
Linux node-1 3.13.0-29-generic #53-Ubuntu SMP Wed Jun 4 21:00:20 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
vagrant@node-1:~$ dpkg -l | grep -e mesos
ii  mesos                               0.23.0-1.0.ubuntu1404            amd64        Cluster resource manager with efficient resource isolation

• Assignee: Unassigned
• Reporter: James DeFelice
• Created: 03/Sep/15 05:41
• Updated: 14/Jul/17 17:07

Description

I was testing a custom executor implementation that manages the life cycle of multiple child processes. When the executor is SIGTERM'd it sends a SIGTERM to each child process and then self-terminates.

In some cases, the child processes do not die, even through the parent process (the custom executor) does. Instead the child procs are re-parented to the slave process where they continue to live on indefinitely.

My custom executor is written in Go, and I've found a useful Go/Linux-specific setting that allows me to configure a signal to be sent to child procs upon the death of the calling thread in the parent. (see https://golang.org/src/syscall/exec_linux.go?s=6285:6843#1 for details). I've since configured the custom executor to specify that a SIGKILL be sent to all child procs upon termination of the executor (parent) process: child procs are still sent a SIGTERM upon receipt of such by the executor, but the SIGKILL upon executor death now acts as a fallback.

Since implementing the above work-around I have not been able to reproduce the problem as previously described. This particular syscall is implemented in very few OS's (the Golang hack only supports Linux) so I'm not sure how I'd go about something similar on Windows, OS X, BSD, etc.

It seems like mesos should take on the responsibility to ensure that when an executor is killed, all of it's child procs are also eventually killed. Given that it's an intermittent and hard to reproduce problem, I'm assuming that mesos does attempt to ensure executor child proc death, but the that the implementation is racy/leaky.

Issue Links

• is related to: MESOS-5893 mesos-executor should adopt and reap orphan child processes
• relates to: MESOS-6933 Executor does not respect grace period