Details
-
Improvement
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
None
-
None
-
None
Description
This has come up during idownes changes to add containerization.
We would like to be able to run commands asynchronously like:
curl -O http://foo.com/bigfile.zip
Currently, there is not an easy way to do this while having:
1. A Future handle on the exit status of the subprocess.
2. The means to 'discard' the future and consequently kill the subprocess (e.g. stalled hadoop command).
3. Handles to stdin, stdout, stderr of the subprocess.
The first issue is that we need to re-work the Reaper to not reap all subprocesses. Rather, we need to allow other components to reap their own forked subprocesses without the slave's Reaper "stealing" the exit status information. I've proposed that we move the Reaper into libprocess initially with the only change being to reap the desired pids. (We can optimize this later using a per-pid blocking thread or SIGCHLD).
One concern is that if we 'leak' child processes by accidentally not reaping, we may fill the process table with zombie processes. However, we have tight control over where our code performs forks, and can enforce proper reaping.