[SPARK-24340] Clean up non-shuffle disk block manager files following executor death

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.4.0
• Component/s: Spark Core
• Assignee: Unassigned
• Reporter: Xingbo Jiang
• Created: 22/May/18 07:11
• Updated: 21/Jul/18 14:22
• Resolved: 21/Jul/18 14:22

Description

Currently we only clean up local folders on application removed, and we don't clean up non-shuffle files, such as temp. shuffle blocks, cached RDD/broadcast blocks, spill files, etc. and this can cause disk space leaks when executors periodically die and are replaced.

To avoid this source of disk space leak, we can clean up executor disk store files except for shuffle index and data files on executor finished.

 

Issue Links

• links to: [Github] Pull Request #21390 (jiangxb1987)