[SPARK-6011] Out of disk space due to Spark not deleting shuffle files of lost executors

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: 1.2.1
• Component/s: YARN

Environment

Running Spark in Yarn-Client mode

• Assignee: Unassigned
• Reporter: pankaj arora
• Created: 25/Feb/15 18:29
• Updated: 25/Feb/15 23:30
• Resolved: 25/Feb/15 23:30

Description

If Executors gets lost abruptly spark does not delete its shuffle files till application ends.
Ours is long running application which is serving requests received through REST APIs and if any of the executor gets lost shuffle files are not deleted and that leads to local disk going out of space.

Issue Links

• duplicates: SPARK-5836 Highlight in Spark documentation that by default Spark does not delete its temporary files
• links to: [Github] Pull Request #4770 (pankajarora12)