[SPARK-44704] Cleanup shuffle files from host node after migration due to graceful decommissioning

• Type: Improvement
• Status: Open
• Priority: Minor
• Resolution: Unresolved
• Affects Version/s: 3.4.1
• Component/s: Block Manager
• Labels: pull-request-available
• Target Version/s: 3.4.2
• Assignee: Unassigned
• Reporter: Deependra Patel
• Created: 07/Aug/23 12:24
• Updated: 18/Nov/23 00:18

Description

Although these files will be deleted at the end of the application by the external shuffle service, doing this early can free up resources and can help in long running applications running out of disk space.

Issue Links

• links to: GitHub Pull Request #42391