[SPARK-12486] Executors are not always terminated successfully by the worker.

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.6.1, 2.0.0
• Component/s: Spark Core
• Assignee: Nong Li
• Reporter: Nong Li
• Created: 22/Dec/15 21:01
• Updated: 25/Feb/16 19:49
• Resolved: 25/Feb/16 19:49

Description

There are cases when the executor is not killed successfully by the worker.
One way this can happen is if the executor is in a bad state, fails to heartbeat and the master tells the worker to kill the executor. The executor is in such a bad state that the kill request is ignored. This seems to be able to happen if the executor is in heavy GC.

The cause of this is that the Process.destroy() API is not forceful enough. In Java8, a new API, destroyForcibly() was added. We should use that if available.

Issue Links

• links to: [Github] Pull Request #10438 (nongli)