[SPARK-5836] Highlight in Spark documentation that by default Spark does not delete its temporary files

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Not A Problem
• Fix Version/s: 1.3.1, 1.4.0
• Component/s: Documentation
• Assignee: Ilya Ganelin
• Reporter: Tomasz Dudziak
• Created: 16/Feb/15 13:38
• Updated: 20/Mar/20 18:54
• Resolved: 21/Aug/15 20:50

Description

We recently learnt the hard way (in a prod system) that Spark by default does not delete its temporary files until it is stopped. WIthin a relatively short time span of heavy Spark use the disk of our prod machine filled up completely because of multiple shuffle files written to it. We think there should be better documentation around the fact that after a job is finished it leaves a lot of rubbish behind so that this does not come as a surprise.

Probably a good place to highlight that fact would be the documentation of spark.local.dir property, which controls where Spark temporary files are written.

Issue Links

• is duplicated by: SPARK-3563 Shuffle data not always be cleaned
• is duplicated by: SPARK-4796 Spark does not remove temp files
• is duplicated by: SPARK-6011 Out of disk space due to Spark not deleting shuffle files of lost executors
• is related to: SPARK-31208 Expose the ability for user to cleanup shuffle files
• links to: [Github] Pull Request #5074 (ilganeli)
• links to: [Github] Pull Request #6901 (srowen)