[SPARK-35767] CoalesceExec can execute child plan twice

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 2.3.0, 2.3.4, 2.4.0, 2.4.8, 3.0.2, 3.1.2, 3.2.0
• Fix Version/s: 3.2.0, 3.1.3, 3.0.4
• Component/s: SQL
• Assignee: Andy Grove
• Reporter: Andy Grove
• Created: 15/Jun/21 16:13
• Updated: 19/Jun/21 00:52
• Resolved: 15/Jun/21 19:01

Description

CoalesceExec calls `child.execute()` in the if condition and throws away the results, then calls `child.execute()` again in the else condition. This could cause a section of the plan to be executed twice.

protected override def doExecute(): RDD[InternalRow] = {
  if (numPartitions == 1 && child.execute().getNumPartitions < 1) {
    // Make sure we don't output an RDD with 0 partitions, when claiming that we have a
    // `SinglePartition`.
    new CoalesceExec.EmptyRDDWithPartitions(sparkContext, numPartitions)
  } else {
    child.execute().coalesce(numPartitions, shuffle = false)
  }
} 

Issue Links

• links to: [Github] Pull Request #32920 (andygrove)