[SPARK-6018] NoSuchMethodError in Spark app is swallowed by YARN AM

• Type: Bug
• Status: Closed
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 1.2.0
• Fix Version/s: 1.2.2, 1.3.0
• Component/s: YARN
• Labels: yarn
• Target Version/s: 1.2.2, 1.3.0
• Assignee: Cheolsoo Park
• Reporter: Cheolsoo Park
• Created: 25/Feb/15 22:58
• Updated: 24/Mar/15 10:58
• Resolved: 26/Feb/15 21:54

Description

I discovered this bug while testing 1.3 RC with old 1.2 Spark job that I had. Due to changes in DF and SchemaRDD, my app failed with java.lang.NoSuchMethodError. However, AM was marked as succeeded, and the error was silently swallowed.

The problem is that pattern matching in Spark AM fails to catch NoSuchMethodError-

15/02/25 20:13:27 INFO cluster.YarnClusterScheduler: YarnClusterScheduler.postStartHook done
Exception in thread "Driver" scala.MatchError: java.lang.NoSuchMethodError: org.apache.spark.sql.hive.HiveContext.table(Ljava/lang/String;)Lorg/apache/spark/sql/SchemaRDD; (of class java.lang.NoSuchMethodError)
	at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:485)

Issue Links

• is duplicated by: SPARK-6449 Driver OOM results in reported application result SUCCESS
• is related to: SPARK-6058 Log the error for the EXIT_EXCEPTION_USER_CLASS exit code
• links to: [Github] Pull Request #4773 (piaozhexiu)