I discovered this bug while testing 1.3 RC with old 1.2 Spark job that I had. Due to changes in DF and SchemaRDD, my app failed with java.lang.NoSuchMethodError. However, AM was marked as succeeded, and the error was silently swallowed.
The problem is that pattern matching in Spark AM fails to catch NoSuchMethodError-
15/02/25 20:13:27 INFO cluster.YarnClusterScheduler: YarnClusterScheduler.postStartHook done Exception in thread "Driver" scala.MatchError: java.lang.NoSuchMethodError: org.apache.spark.sql.hive.HiveContext.table(Ljava/lang/String;)Lorg/apache/spark/sql/SchemaRDD; (of class java.lang.NoSuchMethodError) at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:485)
- is duplicated by
SPARK-6449 Driver OOM results in reported application result SUCCESS
- is related to
SPARK-6058 Log the error for the EXIT_EXCEPTION_USER_CLASS exit code
- links to