[SPARK-6058] Log the error for the EXIT_EXCEPTION_USER_CLASS exit code

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 1.2.1
• Fix Version/s: 1.3.0
• Component/s: YARN
• Assignee: Shixiong Zhu
• Reporter: Shixiong Zhu
• Created: 27/Feb/15 09:14
• Updated: 27/Feb/15 13:33
• Resolved: 27/Feb/15 13:32

Description

When running in the yarn-cluster mode, the user cannot find the stack trace of the driver error in the log.

In https://github.com/apache/spark/blob/8cd1692c9092150107bed27777951633cbf945f6/yarn/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala#L488 , it will rethrow `cause`. However, because ApplicationMaster doesn't set SparkUncaughtExceptionHandler. The error won't be logged.

To view this error in the log, we should call `logError` to log it.

Issue Links

• relates to: SPARK-6018 NoSuchMethodError in Spark app is swallowed by YARN AM
• links to: [Github] Pull Request #4813 (zsxwing)