[SPARK-36500] The temp_shuffle files are not cleaned up when a task is interrupted

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.4.8, 3.0.3, 3.1.2
• Fix Version/s: 3.2.0, 3.1.3, 3.0.4
• Component/s: Spark Core
• Assignee: Xingbo Jiang
• Reporter: Xingbo Jiang
• Created: 12/Aug/21 18:22
• Updated: 12/Dec/22 18:11
• Resolved: 13/Aug/21 10:26

Description

We discovered a potential file leak on the long running cluster. As the cluster running for a few hours, there are many "temp_shuffle_*" files left in the folder. It seems the files are not cleaned up as expected when the task finishes.

Issue Links

• links to: [Github] Pull Request #33731 (jiangxb1987)