[SPARK-7736] Exception not failing Python applications (in yarn cluster mode)

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.5.1, 1.6.0
• Component/s: YARN

Environment

Spark 1.3.1, Yarn 2.7.0, Ubuntu 14.04

• Target Version/s: 1.5.1, 1.6.0
• Assignee: Marcelo Masiero Vanzin
• Reporter: Shay Rojansky
• Created: 19/May/15 16:12
• Updated: 08/Dec/17 00:01
• Resolved: 09/Sep/15 22:50

Description

It seems that exceptions thrown in Python spark apps after the SparkContext is instantiated don't cause the application to fail, at least in Yarn: the application is marked as SUCCEEDED.

Note that any exception right before the SparkContext correctly places the application in FAILED state.

Issue Links

• is duplicated by: SPARK-9416 Yarn logs say that Spark Python job has succeeded even though job has failed in Yarn cluster mode
• is duplicated by: SPARK-8612 Yarn application status is misreported for failed PySpark apps.
• links to: [Github] Pull Request #7751 (vanzin)
• links to: [Github] Pull Request #8258 (vanzin)