[SPARK-18553] Executor loss may cause TaskSetManager to be leaked

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 1.6.0, 2.0.0, 2.1.0
• Fix Version/s: 1.6.4, 2.0.3, 2.1.0, 2.2.0
• Component/s: Scheduler, Spark Core
• Target Version/s: 1.6.4, 2.0.3, 2.1.0
• Assignee: Josh Rosen
• Reporter: Josh Rosen
• Created: 23/Nov/16 03:08
• Updated: 17/May/20 17:46
• Resolved: 01/Dec/16 19:28

Description

Due to a bug in TaskSchedulerImpl, the complete sudden loss of an executor may cause a TaskSetManager to be leaked, causing ShuffleDependencies and other data structures to be kept alive indefinitely, leading to various types of resource leaks (including shuffle file leaks).

In a nutshell, the problem is that TaskSchedulerImpl did not maintain its own mapping from executorId to running task ids, leaving it unable to clean up taskId to taskSetManager maps when an executor is totally lost.

Issue Links

• links to: [Github] Pull Request #15986 (JoshRosen)
• links to: [Github] Pull Request #16045 (JoshRosen)
• links to: [Github] Pull Request #16070 (JoshRosen)