[SPARK-41694] Add new config to clean up `spark.ui.store.path` directory when SparkContext.stop()

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.4.0
• Fix Version/s: 3.4.0
• Component/s: Spark Core
• Assignee: Yang Jie
• Reporter: Yang Jie
• Created: 23/Dec/22 08:34
• Updated: 05/Jan/23 04:32
• Resolved: 05/Jan/23 04:32

Description

spark.ui.store.path directory not clean up when SparkContext.stop() now:

1. {}The disk space occupied by the spark.ui.store.path directory will continue to grow.
2. When submitting new App and reusing the spark.ui.store.path directory, we will see the content related to the previous App, which is a bit strange

Issue Links

• links to: [Github] Pull Request #39226 (LuciferYang)
• links to: [Github] Pull Request #39226 (LuciferYang)