[SPARK-22827] Avoid throwing OutOfMemoryError in case of exception in spill

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.2.0
• Fix Version/s: 2.3.0
• Component/s: Spark Core
• Assignee: Sital Kedia
• Reporter: Sital Kedia
• Created: 19/Dec/17 00:24
• Updated: 08/Nov/18 05:25
• Resolved: 20/Dec/17 04:21

Description

Currently, the task memory manager throws an OutofMemory error when there is an IO exception happens in spill() - https://github.com/apache/spark/blob/master/core/src/main/java/org/apache/spark/memory/TaskMemoryManager.java#L194. Similarly there any many other places in code when if a task is not able to acquire memory due to an exception we throw an OutofMemory error which kills the entire executor and hence failing all the tasks that are running on that executor instead of just failing one single task.

Issue Links

• links to: [Github] Pull Request #20014 (sitalkedia)
• links to: [Github] Pull Request #22969 (ueshin)
• links to: [Github] Pull Request #22969 (ueshin)