[SPARK-35570] Shuffle file leak with external shuffle service enable

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Invalid
• Affects Version/s: 3.1.2
• Component/s: Block Manager, Shuffle
• Assignee: Unassigned
• Reporter: ZiyueGuan
• Created: 31/May/21 03:04
• Updated: 12/Dec/22 18:10
• Resolved: 03/Jun/21 05:43

Description

Unlike rdd block, external shuffle service doesn't offer a cleaning up of shuffle file. The cleaning up of shuffle file mainly rely on alive executors to response the request from context cleaner. As long as the executor exit, the shuffle file left will not be cleaned until application exits. For streaming application or long running application, disk may run out. 

I'm confused that shuffle file was left like above while the lifecycle of rdd block was properly handled. Is there any difference between them?