[HIVE-18869] Improve SparkTask OOM Error Parsing Logic

• Type: Sub-task
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: Spark
• Target Version/s: 3.0.0
• Assignee: Unassigned
• Reporter: Sahil Takiar
• Created: 06/Mar/18 04:50
• Updated: 08/Mar/18 01:06

Description

The method SparkTask#isOOMError parses a stack-trace to check if it is due to an OOM error. A few improvements could be made:

• Differentiate between driver OOM and task OOM
• The string Container killed by YARN for exceeding memory limits is printed if a container exceeds its memory limits, but Spark tasks can OOM for other reasons, such as GC overhead limit exceeded