Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-23860

SQLAppStatusListener should handle the cases that an accumulator may be GCed

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Duplicate
    • Affects Version/s: 2.3.0
    • Fix Version/s: None
    • Component/s: SQL
    • Labels:
      None

      Description

      "SQLAppStatusListener.onTaskEnd" is running in the event thread. When it's called, the Spark job may already finished, and accumulators may be GCed. "SQLAppStatusListener.onTaskEnd" should handle this case.

      Here is an example of this failure (SQLAppStatusListener was called SQLListener in 2.2):

      18/03/30 06:49:58 ERROR LiveListenerBus: Listener SQLListener threw an exception
      java.lang.IllegalStateException: Attempted to access garbage collected accumulator 78705157
      	at org.apache.spark.util.AccumulatorContext$$anonfun$get$1.apply(AccumulatorV2.scala:268)
      	at org.apache.spark.util.AccumulatorContext$$anonfun$get$1.apply(AccumulatorV2.scala:264)
      	at scala.Option.map(Option.scala:146)
      	at org.apache.spark.util.AccumulatorContext$.get(AccumulatorV2.scala:264)
      	at org.apache.spark.util.AccumulatorV2$$anonfun$name$1.apply(AccumulatorV2.scala:90)
      	at org.apache.spark.util.AccumulatorV2$$anonfun$name$1.apply(AccumulatorV2.scala:90)
      	at scala.Option.orElse(Option.scala:289)
      	at org.apache.spark.util.AccumulatorV2.name(AccumulatorV2.scala:90)
      	at org.apache.spark.util.AccumulatorV2.toInfo(AccumulatorV2.scala:111)
      	at org.apache.spark.sql.execution.ui.SQLListener$$anonfun$onTaskEnd$1.apply(SQLListener.scala:227)
      	at org.apache.spark.sql.execution.ui.SQLListener$$anonfun$onTaskEnd$1.apply(SQLListener.scala:227)
      	at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
      	at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
      	at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
      	at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
      	at scala.collection.TraversableLike$class.map(TraversableLike.scala:234)
      	at scala.collection.AbstractTraversable.map(Traversable.scala:104)
      	at org.apache.spark.sql.execution.ui.SQLListener.onTaskEnd(SQLListener.scala:227)
      	at org.apache.spark.scheduler.SparkListenerBus$class.doPostEvent(SparkListenerBus.scala:45)
      	at org.apache.spark.scheduler.LiveListenerBus.doPostEvent(LiveListenerBus.scala:42)
      	at org.apache.spark.scheduler.LiveListenerBus.doPostEvent(LiveListenerBus.scala:42)
      	at org.apache.spark.util.ListenerBus$class.postToAll(ListenerBus.scala:84)
      	at org.apache.spark.scheduler.LiveListenerBus.postToAll(LiveListenerBus.scala:42)
      	at org.apache.spark.scheduler.LiveListenerBus$$anon$1$$anonfun$run$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(LiveListenerBus.scala:100)
      	at org.apache.spark.scheduler.LiveListenerBus$$anon$1$$anonfun$run$1$$anonfun$apply$mcV$sp$1.apply(LiveListenerBus.scala:81)
      	at org.apache.spark.scheduler.LiveListenerBus$$anon$1$$anonfun$run$1$$anonfun$apply$mcV$sp$1.apply(LiveListenerBus.scala:81)
      	at scala.util.DynamicVariable.withValue(DynamicVariable.scala:58)
      	at org.apache.spark.scheduler.LiveListenerBus$$anon$1$$anonfun$run$1.apply$mcV$sp(LiveListenerBus.scala:81)
      	at org.apache.spark.util.Utils$.tryOrStopSparkContext(Utils.scala:1304)
      	at org.apache.spark.scheduler.LiveListenerBus$$anon$1.run(LiveListenerBus.scala:80)
      

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                zsxwing Shixiong Zhu
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: