[SPARK-23433] java.lang.IllegalStateException: more than one active taskSet for stage

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.2.1
• Fix Version/s: 2.2.2, 2.3.1, 2.4.0
• Component/s: Spark Core
• Assignee: Imran Rashid
• Reporter: Shixiong Zhu
• Created: 15/Feb/18 01:14
• Updated: 15/Apr/19 12:35
• Resolved: 03/May/18 16:01

Description

This following error thrown by DAGScheduler stopped the cluster:

18/02/11 13:22:27 ERROR DAGSchedulerEventProcessLoop: DAGSchedulerEventProcessLoop failed; shutting down SparkContext
java.lang.IllegalStateException: more than one active taskSet for stage 7580621: 7580621.2,7580621.1
	at org.apache.spark.scheduler.TaskSchedulerImpl.submitTasks(TaskSchedulerImpl.scala:229)
	at org.apache.spark.scheduler.DAGScheduler.submitMissingTasks(DAGScheduler.scala:1193)
	at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$submitStage(DAGScheduler.scala:1059)
	at org.apache.spark.scheduler.DAGScheduler$$anonfun$submitWaitingChildStages$6.apply(DAGScheduler.scala:900)
	at org.apache.spark.scheduler.DAGScheduler$$anonfun$submitWaitingChildStages$6.apply(DAGScheduler.scala:899)
	at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
	at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:186)
	at org.apache.spark.scheduler.DAGScheduler.submitWaitingChildStages(DAGScheduler.scala:899)
	at org.apache.spark.scheduler.DAGScheduler.handleTaskCompletion(DAGScheduler.scala:1427)
	at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:1929)
	at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1880)
	at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1868)
	at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)

Issue Links

• is related to: SPARK-25250 Race condition with tasks running when new attempt for same stage is created leads to other task in the next attempt running on the same partition id retry multiple times
• relates to: SPARK-19868 conflict TasksetManager lead to spark stopped
• relates to: SPARK-27065 avoid more than one active task set managers for a stage
• links to: [Github] Pull Request #21131 (squito)
• links to: GitHub Pull Request #23871
• links to: GitHub Pull Request #23927
• links to: GitHub Pull Request #24006
• links to: GitHub Pull Request #24007
• links to: GitHub Pull Request #24359