Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-12339

NullPointerException on stage kill from web UI

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 2.0.0
    • 2.0.0
    • Web UI
    • None

    Description

      The following message is in the logs after killing a stage:

      scala> INFO Executor: Executor killed task 1.0 in stage 7.0 (TID 33)
      INFO Executor: Executor killed task 0.0 in stage 7.0 (TID 32)
      WARN TaskSetManager: Lost task 1.0 in stage 7.0 (TID 33, localhost): TaskKilled (killed intentionally)
      WARN TaskSetManager: Lost task 0.0 in stage 7.0 (TID 32, localhost): TaskKilled (killed intentionally)
      INFO TaskSchedulerImpl: Removed TaskSet 7.0, whose tasks have all completed, from pool
      ERROR LiveListenerBus: Listener SQLListener threw an exception
      java.lang.NullPointerException
      	at org.apache.spark.sql.execution.ui.SQLListener.onTaskEnd(SQLListener.scala:167)
      	at org.apache.spark.scheduler.SparkListenerBus$class.onPostEvent(SparkListenerBus.scala:42)
      	at org.apache.spark.scheduler.LiveListenerBus.onPostEvent(LiveListenerBus.scala:31)
      	at org.apache.spark.scheduler.LiveListenerBus.onPostEvent(LiveListenerBus.scala:31)
      	at org.apache.spark.util.ListenerBus$class.postToAll(ListenerBus.scala:55)
      	at org.apache.spark.util.AsynchronousListenerBus.postToAll(AsynchronousListenerBus.scala:37)
      	at org.apache.spark.util.AsynchronousListenerBus$$anon$1$$anonfun$run$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(AsynchronousListenerBus.scala:80)
      	at org.apache.spark.util.AsynchronousListenerBus$$anon$1$$anonfun$run$1$$anonfun$apply$mcV$sp$1.apply(AsynchronousListenerBus.scala:65)
      	at org.apache.spark.util.AsynchronousListenerBus$$anon$1$$anonfun$run$1$$anonfun$apply$mcV$sp$1.apply(AsynchronousListenerBus.scala:65)
      	at scala.util.DynamicVariable.withValue(DynamicVariable.scala:58)
      	at org.apache.spark.util.AsynchronousListenerBus$$anon$1$$anonfun$run$1.apply$mcV$sp(AsynchronousListenerBus.scala:64)
      	at org.apache.spark.util.Utils$.tryOrStopSparkContext(Utils.scala:1169)
      	at org.apache.spark.util.AsynchronousListenerBus$$anon$1.run(AsynchronousListenerBus.scala:63)
      ERROR LiveListenerBus: Listener SQLListener threw an exception
      java.lang.NullPointerException
      	at org.apache.spark.sql.execution.ui.SQLListener.onTaskEnd(SQLListener.scala:167)
      	at org.apache.spark.scheduler.SparkListenerBus$class.onPostEvent(SparkListenerBus.scala:42)
      	at org.apache.spark.scheduler.LiveListenerBus.onPostEvent(LiveListenerBus.scala:31)
      	at org.apache.spark.scheduler.LiveListenerBus.onPostEvent(LiveListenerBus.scala:31)
      	at org.apache.spark.util.ListenerBus$class.postToAll(ListenerBus.scala:55)
      	at org.apache.spark.util.AsynchronousListenerBus.postToAll(AsynchronousListenerBus.scala:37)
      	at org.apache.spark.util.AsynchronousListenerBus$$anon$1$$anonfun$run$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(AsynchronousListenerBus.scala:80)
      	at org.apache.spark.util.AsynchronousListenerBus$$anon$1$$anonfun$run$1$$anonfun$apply$mcV$sp$1.apply(AsynchronousListenerBus.scala:65)
      	at org.apache.spark.util.AsynchronousListenerBus$$anon$1$$anonfun$run$1$$anonfun$apply$mcV$sp$1.apply(AsynchronousListenerBus.scala:65)
      	at scala.util.DynamicVariable.withValue(DynamicVariable.scala:58)
      	at org.apache.spark.util.AsynchronousListenerBus$$anon$1$$anonfun$run$1.apply$mcV$sp(AsynchronousListenerBus.scala:64)
      	at org.apache.spark.util.Utils$.tryOrStopSparkContext(Utils.scala:1169)
      	at org.apache.spark.util.AsynchronousListenerBus$$anon$1.run(AsynchronousListenerBus.scala:63)
      

      To reproduce, start a job and kill the stage from web UI, e.g.:

      val rdd = sc.parallelize(0 to 9, 2)
      rdd.mapPartitionsWithIndex { case (n, it) => Thread.sleep(10 * 1000); it }.count
      

      Go to web UI and in Stages tab click "kill" for the stage.

      Attachments

        Activity

          People

            ajbozarth Alex Bozarth
            jlaskowski Jacek Laskowski
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: