-
Type:
Bug
-
Status: Resolved
-
Priority:
Minor
-
Resolution: Duplicate
-
Affects Version/s: 1.3.0, 1.3.1, 1.4.0
-
Fix Version/s: None
-
Component/s: YARN
-
Labels:None
-
Environment:
PySpark job run in yarn-client mode on CDH 5.4.2
When a PySpark job fails, the YARN records and reports its status as successful. Hari Shreedharan pointed out to me that the ApplicationMaster records app success when system.exit is called. PySpark always exits by calling os._exit. Because of this, every PySpark application run on yarn is marked as completed successfully.
- duplicates
-
SPARK-7736 Exception not failing Python applications (in yarn cluster mode)
-
- Resolved
-