[SPARK-25283] A deadlock in UnionRDD

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.4.0
• Fix Version/s: 2.4.0
• Component/s: Spark Core
• Assignee: Max Gekk
• Reporter: Max Gekk
• Created: 30/Aug/18 12:26
• Updated: 08/Oct/18 15:59
• Resolved: 31/Aug/18 21:38

Description

The PR https://github.com/apache/spark/pull/21913 replaced Scala parallel collections in UnionRDD by new parmap function. This changes cause a deadlock in the partitions method. The code demonstrates the problem:

    val wide = 20
    def unionRDD(num: Int): UnionRDD[Int] = {
      val rdds = (0 until num).map(_ => sc.parallelize(1 to 10, 1))
      new UnionRDD(sc, rdds)
    }
    val level0 = (0 until wide).map { _ =>
      val level1 = (0 until wide).map(_ => unionRDD(wide))
      new UnionRDD(sc, level1)
    }
    val rdd = new UnionRDD(sc, level0)

    rdd.partitions.length

Issue Links

• relates to: SPARK-25286 Remove dangerous parmap
• links to: [Github] Pull Request #22283 (MaxGekk)