[SPARK-31438] Support JobCleaned Status in SparkListener

• Type: Improvement
• Status: In Progress
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.1.0
• Component/s: Spark Core
• Assignee: Unassigned
• Reporter: Jackey Lee
• Created: 13/Apr/20 15:30
• Updated: 12/Dec/22 18:10

Description

In Spark, we need do some hook after job cleaned, such as cleaning hive external temporary paths. This has already discussed in SPARK-31346 and GitHub Pull Request #28129.
The JobEnd Status is not suitable for this. As JobEnd is responsible for Job finished, once all result has generated, it should be finished. After finish, Scheduler will leave the still running tasks to be zombie tasks and delete abnormal tasks asynchronously.
Thus, we add JobCleaned Status to enable user to do some hook after all tasks cleaned in Job. The JobCleaned Status can get from TaskSetManagers, which is related to a stage, and once all stages of the job has been cleaned, then the job is cleaned.

Issue Links

• Blocked: SPARK-31346 Add new configuration to make sure temporary directory cleaned
• links to: [Github] Pull Request #28280 (stczwd)
• links to: [Github] Pull Request #28280 (stczwd)