[SPARK-24677] TaskSetManager not updating successfulTaskDurations for old stage attempts

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 2.3.1
• Fix Version/s: 2.2.3, 2.3.3, 2.4.0
• Component/s: Spark Core
• Assignee: dzcxzl
• Reporter: dzcxzl
• Created: 28/Jun/18 07:46
• Updated: 20/Sep/18 17:19
• Resolved: 18/Jul/18 18:29

Description

When introducing SPARK-23433 , maybe cause stop sparkcontext.

ERROR Utils: uncaught error in thread task-scheduler-speculation, stopping SparkContext
java.util.NoSuchElementException: MedianHeap is empty.
at org.apache.spark.util.collection.MedianHeap.median(MedianHeap.scala:83)
at org.apache.spark.scheduler.TaskSetManager.checkSpeculatableTasks(TaskSetManager.scala:968)
at org.apache.spark.scheduler.Pool$$anonfun$checkSpeculatableTasks$1.apply(Pool.scala:94)
at org.apache.spark.scheduler.Pool$$anonfun$checkSpeculatableTasks$1.apply(Pool.scala:93)
at scala.collection.Iterator$class.foreach(Iterator.scala:742)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1194)
at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
at org.apache.spark.scheduler.Pool.checkSpeculatableTasks(Pool.scala:93)
at org.apache.spark.scheduler.Pool$$anonfun$checkSpeculatableTasks$1.apply(Pool.scala:94)
at org.apache.spark.scheduler.Pool$$anonfun$checkSpeculatableTasks$1.apply(Pool.scala:93)

Issue Links

• links to: [Github] Pull Request #21656 (cxzl25)