[SPARK-11801] Notify driver when OOM is thrown before executor JVM is killed

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Incomplete
• Affects Version/s: 1.5.1
• Component/s: Spark Core
• Labels: bulk-closed
• Assignee: Unassigned
• Reporter: Srinivas
• Created: 18/Nov/15 02:36
• Updated: 21/May/19 04:35
• Resolved: 21/May/19 04:35

Description

Here is some background for the issue.
Customer got OOM exception in one of the task and executor got killed with kill %p. It is unclear in driver logs/Spark UI why the task is lost or executor is lost. Customer has to look into the executor logs to see OOM is the cause for the task/executor lost.

It would be helpful if driver logs/spark UI shows the reason for task failures by making sure that task updates the driver with OOM.

Issue Links

• is related to: SPARK-2491 When an OOM is thrown,the executor does not stop properly.
• links to: [Github] Pull Request #9866 (vundela)