[SPARK-8754] YarnClientSchedulerBackend doesn't stop gracefully in failure conditions

• Type: Bug
• Status: Closed
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 1.4.0
• Fix Version/s: 1.4.1, 1.5.0
• Component/s: YARN
• Target Version/s: 1.4.1, 1.5.0
• Assignee: Devaraj Kavali
• Reporter: Devaraj Kavali
• Created: 01/Jul/15 09:35
• Updated: 25/Nov/15 10:01
• Resolved: 02/Jul/15 05:59

Description

java.lang.NullPointerException
        at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.stop(YarnClientSchedulerBackend.scala:151)
        at org.apache.spark.scheduler.TaskSchedulerImpl.stop(TaskSchedulerImpl.scala:421)
        at org.apache.spark.scheduler.DAGScheduler.stop(DAGScheduler.scala:1447)
        at org.apache.spark.SparkContext.stop(SparkContext.scala:1651)
        at org.apache.spark.SparkContext.<init>(SparkContext.scala:572)
        at org.apache.spark.examples.SparkPi$.main(SparkPi.scala:28)
        at org.apache.spark.examples.SparkPi.main(SparkPi.scala)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:621)
        at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:170)
        at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:193)
        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:112)
        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

If the application has FINISHED/FAILED/KILLED or failed to launch application master, monitorThread is not getting initialized but monitorThread.interrupt() is getting invoked as part of stop() without any check and It is causing to throw NPE and also it is preventing to stop the client.

Issue Links

• duplicates: SPARK-8276 NPE in YarnClientSchedulerBackend.stop
• links to: [Github] Pull Request #7153 (devaraj-kavali)