Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-12339

NullPointerException on stage kill from web UI

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.0.0
    • Fix Version/s: 2.0.0
    • Component/s: Web UI
    • Labels:
      None
    • Target Version/s:

      Description

      The following message is in the logs after killing a stage:

      scala> INFO Executor: Executor killed task 1.0 in stage 7.0 (TID 33)
      INFO Executor: Executor killed task 0.0 in stage 7.0 (TID 32)
      WARN TaskSetManager: Lost task 1.0 in stage 7.0 (TID 33, localhost): TaskKilled (killed intentionally)
      WARN TaskSetManager: Lost task 0.0 in stage 7.0 (TID 32, localhost): TaskKilled (killed intentionally)
      INFO TaskSchedulerImpl: Removed TaskSet 7.0, whose tasks have all completed, from pool
      ERROR LiveListenerBus: Listener SQLListener threw an exception
      java.lang.NullPointerException
      	at org.apache.spark.sql.execution.ui.SQLListener.onTaskEnd(SQLListener.scala:167)
      	at org.apache.spark.scheduler.SparkListenerBus$class.onPostEvent(SparkListenerBus.scala:42)
      	at org.apache.spark.scheduler.LiveListenerBus.onPostEvent(LiveListenerBus.scala:31)
      	at org.apache.spark.scheduler.LiveListenerBus.onPostEvent(LiveListenerBus.scala:31)
      	at org.apache.spark.util.ListenerBus$class.postToAll(ListenerBus.scala:55)
      	at org.apache.spark.util.AsynchronousListenerBus.postToAll(AsynchronousListenerBus.scala:37)
      	at org.apache.spark.util.AsynchronousListenerBus$$anon$1$$anonfun$run$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(AsynchronousListenerBus.scala:80)
      	at org.apache.spark.util.AsynchronousListenerBus$$anon$1$$anonfun$run$1$$anonfun$apply$mcV$sp$1.apply(AsynchronousListenerBus.scala:65)
      	at org.apache.spark.util.AsynchronousListenerBus$$anon$1$$anonfun$run$1$$anonfun$apply$mcV$sp$1.apply(AsynchronousListenerBus.scala:65)
      	at scala.util.DynamicVariable.withValue(DynamicVariable.scala:58)
      	at org.apache.spark.util.AsynchronousListenerBus$$anon$1$$anonfun$run$1.apply$mcV$sp(AsynchronousListenerBus.scala:64)
      	at org.apache.spark.util.Utils$.tryOrStopSparkContext(Utils.scala:1169)
      	at org.apache.spark.util.AsynchronousListenerBus$$anon$1.run(AsynchronousListenerBus.scala:63)
      ERROR LiveListenerBus: Listener SQLListener threw an exception
      java.lang.NullPointerException
      	at org.apache.spark.sql.execution.ui.SQLListener.onTaskEnd(SQLListener.scala:167)
      	at org.apache.spark.scheduler.SparkListenerBus$class.onPostEvent(SparkListenerBus.scala:42)
      	at org.apache.spark.scheduler.LiveListenerBus.onPostEvent(LiveListenerBus.scala:31)
      	at org.apache.spark.scheduler.LiveListenerBus.onPostEvent(LiveListenerBus.scala:31)
      	at org.apache.spark.util.ListenerBus$class.postToAll(ListenerBus.scala:55)
      	at org.apache.spark.util.AsynchronousListenerBus.postToAll(AsynchronousListenerBus.scala:37)
      	at org.apache.spark.util.AsynchronousListenerBus$$anon$1$$anonfun$run$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(AsynchronousListenerBus.scala:80)
      	at org.apache.spark.util.AsynchronousListenerBus$$anon$1$$anonfun$run$1$$anonfun$apply$mcV$sp$1.apply(AsynchronousListenerBus.scala:65)
      	at org.apache.spark.util.AsynchronousListenerBus$$anon$1$$anonfun$run$1$$anonfun$apply$mcV$sp$1.apply(AsynchronousListenerBus.scala:65)
      	at scala.util.DynamicVariable.withValue(DynamicVariable.scala:58)
      	at org.apache.spark.util.AsynchronousListenerBus$$anon$1$$anonfun$run$1.apply$mcV$sp(AsynchronousListenerBus.scala:64)
      	at org.apache.spark.util.Utils$.tryOrStopSparkContext(Utils.scala:1169)
      	at org.apache.spark.util.AsynchronousListenerBus$$anon$1.run(AsynchronousListenerBus.scala:63)
      

      To reproduce, start a job and kill the stage from web UI, e.g.:

      val rdd = sc.parallelize(0 to 9, 2)
      rdd.mapPartitionsWithIndex { case (n, it) => Thread.sleep(10 * 1000); it }.count
      

      Go to web UI and in Stages tab click "kill" for the stage.

        Attachments

          Activity

            People

            • Assignee:
              ajbozarth Alex Bozarth
              Reporter:
              jlaskowski Jacek Laskowski
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: