Details
-
New Feature
-
Status: Open
-
Minor
-
Resolution: Unresolved
-
3.1.0
-
None
-
None
Description
Currently, each Spark application run on Mesos leaves behind at least 500MB of data in sandbox directories, coming from Spark binaries and copied URIs. These can build up as a disk leak, causing major issues on Mesos clusters unless their grace period for sandbox directories is very short.
Spark should have a feature to delete these (from both driver and executor sandboxes) on teardown.