[SPARK-30388] mark all running map stages of finished job as finished

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.4.0
• Fix Version/s: 3.0.0, 3.1.0
• Component/s: Spark Core, Web UI
• Assignee: Xuesen Liang
• Reporter: Xuesen Liang
• Created: 30/Dec/19 12:14
• Updated: 05/Mar/20 13:19
• Resolved: 03/Mar/20 15:33

Description

When a job finished, its running (re-submitted) map stages should be marked as finished if not used by other jobs.

And the ListenerBus should be notified too, otherwise, these map stage items will stay on the "Active Stages" page of web UI and never gone.

Issue Links

• causes: SPARK-31052 Fix flaky test "DAGSchedulerSuite.shuffle fetch failed on speculative task, but original task succeed"
• links to: GitHub Pull Request #27050