[SPARK-38005] Support cleaning up merged shuffle files and state from external shuffle service

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.2.0
• Fix Version/s: 3.4.0
• Component/s: Shuffle, Spark Core
• Assignee: Unassigned
• Reporter: Chandni Singh
• Created: 24/Jan/22 23:48
• Updated: 08/Feb/23 08:58
• Resolved: 08/Feb/23 08:55

Description

Currently merged shuffle files and state is not cleaned up until an application ends. SPARK-37618 handles the cleanup of regular shuffle files. This jira will address cleaning up of merged shuffle files/state.

Issue Links

• is duplicated by: SPARK-40480 Remove push-based shuffle data after query finished
• is related to: SPARK-37618 Support cleaning up shuffle blocks from external shuffle service
• links to: [Github] Pull Request #38560 (yabola)