Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-22371

dag-scheduler-event-loop thread stopped with error Attempted to access garbage collected accumulator 5605982

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.1.0
    • Fix Version/s: 2.3.1, 2.4.0
    • Component/s: Spark Core
    • Labels:
      None

      Description

      Our Spark Jobs are getting stuck on DagScheduler.runJob as dagscheduler thread is stopped because of Attempted to access garbage collected accumulator 5605982.

      from our investigation it look like accumulator is cleaned by GC first and same accumulator is used for merging the results from executor on task completion event.

      As the error java.lang.IllegalAccessError is LinkageError which is treated as FatalError so dag-scheduler loop is finished with below exception.

      ---ERROR stack trace –
      Exception in thread "dag-scheduler-event-loop" java.lang.IllegalAccessError: Attempted to access garbage collected accumulator 5605982
      at org.apache.spark.util.AccumulatorContext$$anonfun$get$1.apply(AccumulatorV2.scala:253)
      at org.apache.spark.util.AccumulatorContext$$anonfun$get$1.apply(AccumulatorV2.scala:249)
      at scala.Option.map(Option.scala:146)
      at org.apache.spark.util.AccumulatorContext$.get(AccumulatorV2.scala:249)
      at org.apache.spark.scheduler.DAGScheduler$$anonfun$updateAccumulators$1.apply(DAGScheduler.scala:1083)
      at org.apache.spark.scheduler.DAGScheduler$$anonfun$updateAccumulators$1.apply(DAGScheduler.scala:1080)
      at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
      at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
      at org.apache.spark.scheduler.DAGScheduler.updateAccumulators(DAGScheduler.scala:1080)
      at org.apache.spark.scheduler.DAGScheduler.handleTaskCompletion(DAGScheduler.scala:1183)
      at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:1647)
      at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1605)
      at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1594)
      at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)

      I am attaching the thread dump of driver as well

        Attachments

        1. driver-thread-dump-spark2.1.txt
          309 kB
          Mayank Agarwal
        2. Helper.scala
          0.8 kB
          Mayank Agarwal
        3. sampledata
          11.08 MB
          Mayank Agarwal
        4. ShuffleIssue.java
          5 kB
          Mayank Agarwal

          Activity

            People

            • Assignee:
              Rudoy Artem Rudoy
              Reporter:
              mayank.agarwal2305 Mayank Agarwal
            • Votes:
              7 Vote for this issue
              Watchers:
              17 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: