[SPARK-31485] Barrier stage can hang if only partial tasks launched

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.4.0
• Fix Version/s: 2.4.6
• Component/s: Spark Core
• Target Version/s: 2.4.6, 3.0.0
• Assignee: wuyi
• Reporter: wuyi
• Shepherd: Holden Karau
• Created: 19/Apr/20 13:40
• Updated: 07/May/20 13:39
• Resolved: 27/Apr/20 06:10

Description

The issue can be reproduced by following test:

 

initLocalClusterSparkContext(2)
val rdd0 = sc.parallelize(Seq(0, 1, 2, 3), 2)
val dep = new OneToOneDependency[Int](rdd0)
val rdd = new MyRDD(sc, 2, List(dep), Seq(Seq("executor_h_0"),Seq("executor_h_0")))
rdd.barrier().mapPartitions { iter =>
  BarrierTaskContext.get().barrier()
  iter
}.collect()

 

Issue Links

• links to: [Github] Pull Request #28476 (Ngone51)
• links to: GitHub Pull Request #28257