Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-24677

TaskSetManager not updating successfulTaskDurations for old stage attempts

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 2.3.1
    • Fix Version/s: 2.2.3, 2.3.3, 2.4.0
    • Component/s: Spark Core
    • Labels:
      None

      Description

      When introducing SPARK-23433 , maybe cause stop sparkcontext.

      ERROR Utils: uncaught error in thread task-scheduler-speculation, stopping SparkContext
      java.util.NoSuchElementException: MedianHeap is empty.
      at org.apache.spark.util.collection.MedianHeap.median(MedianHeap.scala:83)
      at org.apache.spark.scheduler.TaskSetManager.checkSpeculatableTasks(TaskSetManager.scala:968)
      at org.apache.spark.scheduler.Pool$$anonfun$checkSpeculatableTasks$1.apply(Pool.scala:94)
      at org.apache.spark.scheduler.Pool$$anonfun$checkSpeculatableTasks$1.apply(Pool.scala:93)
      at scala.collection.Iterator$class.foreach(Iterator.scala:742)
      at scala.collection.AbstractIterator.foreach(Iterator.scala:1194)
      at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
      at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
      at org.apache.spark.scheduler.Pool.checkSpeculatableTasks(Pool.scala:93)
      at org.apache.spark.scheduler.Pool$$anonfun$checkSpeculatableTasks$1.apply(Pool.scala:94)
      at org.apache.spark.scheduler.Pool$$anonfun$checkSpeculatableTasks$1.apply(Pool.scala:93)
      

        Attachments

          Activity

            People

            • Assignee:
              dzcxzl dzcxzl
              Reporter:
              dzcxzl dzcxzl
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: