[SPARK-8612] Yarn application status is misreported for failed PySpark apps.

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Duplicate
• Affects Version/s: 1.3.0, 1.3.1, 1.4.0
• Component/s: YARN

Environment

PySpark job run in yarn-client mode on CDH 5.4.2

• Assignee: Unassigned
• Reporter: Juliet Hougland
• Created: 25/Jun/15 01:18
• Updated: 02/Aug/15 07:28
• Resolved: 02/Aug/15 07:28

Description

When a PySpark job fails, the YARN records and reports its status as successful. Hari Shreedharan pointed out to me that the ApplicationMaster records app success when system.exit is called. PySpark always exits by calling os._exit. Because of this, every PySpark application run on yarn is marked as completed successfully.

Issue Links

• duplicates: SPARK-7736 Exception not failing Python applications (in yarn cluster mode)