[SPARK-34142] Support Fallback Storage Cleanup during stopping SparkContext

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.2.0
• Fix Version/s: 3.2.0
• Component/s: Spark Core
• Assignee: Dongjoon Hyun
• Reporter: Dongjoon Hyun
• Created: 17/Jan/21 05:17
• Updated: 18/Jan/21 00:54
• Resolved: 18/Jan/21 00:54

Description

SPARK-33545 added `Support Fallback Storage during worker decommission` for the managed cloud-storage with TTL support. This issue aims to add additional clean-up feature during stopping SparkContext to save some money before TTL or the other HDFS-compatible storage which doesn't have TTL support.

Issue Links

• links to: [Github] Pull Request #31215 (dongjoon-hyun)
• links to: [Github] Pull Request #31215 (dongjoon-hyun)