Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-8754

YarnClientSchedulerBackend doesn't stop gracefully in failure conditions

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 1.4.0
    • Fix Version/s: 1.4.1, 1.5.0
    • Component/s: YARN
    • Labels:
      None

      Description

      java.lang.NullPointerException
              at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.stop(YarnClientSchedulerBackend.scala:151)
              at org.apache.spark.scheduler.TaskSchedulerImpl.stop(TaskSchedulerImpl.scala:421)
              at org.apache.spark.scheduler.DAGScheduler.stop(DAGScheduler.scala:1447)
              at org.apache.spark.SparkContext.stop(SparkContext.scala:1651)
              at org.apache.spark.SparkContext.<init>(SparkContext.scala:572)
              at org.apache.spark.examples.SparkPi$.main(SparkPi.scala:28)
              at org.apache.spark.examples.SparkPi.main(SparkPi.scala)
              at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
              at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
              at java.lang.reflect.Method.invoke(Method.java:606)
              at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:621)
              at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:170)
              at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:193)
              at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:112)
              at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
      

      If the application has FINISHED/FAILED/KILLED or failed to launch application master, monitorThread is not getting initialized but monitorThread.interrupt() is getting invoked as part of stop() without any check and It is causing to throw NPE and also it is preventing to stop the client.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                devaraj.k Devaraj K
                Reporter:
                devaraj.k Devaraj K
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: