Major Description
Currently, there is no garbage collection of completed executors/frameworks on slaves, leading to local directories periodically filling up.
Design proposal:
- A libprocess process that gets messages when an executor has gotten cleaned up and sets some timeout when it should clean up the directory. Abstracting it this way is nice because we could add other capabilities like copying the directory output to HDFS if requested by the user.
- Making sure directories continue to get cleaned up after a slave restarts
- Make sure frameworks use their executor's working directory for storage rather than a hard-configured path (e.g. /tmp)
