Details
-
Bug
-
Status: In Progress
-
Major
-
Resolution: Unresolved
-
3.0.0
-
None
-
None
Description
Currently we only clean up the local directories on application removed. However, when executors die and restart repeatedly, many temp files are left untouched in the local directories, which is undesired behavior and could cause disk space used up gradually.
SPARK-24340 had fixed this problem in standalone mode. But in YARN mode, this issue still exists. Especially, in long running service like Spark thrift-server with dynamic resource allocation disabled, it's very easy causes local disk full.