[SPARK-35767] CoalesceExec can execute child plan twice - ASF JIRA

Log work

Agile Board

Rank to Top

Rank to Bottom

Attach files

Attach Screenshot

Bulk Copy Attachments

Bulk Move Attachments

Voters

Watch issue

Watchers

Create sub-task

Convert to sub-task

Move

Link

Clone

Labels

Update Comment Author

Replace String in Comment

Update Comment Visibility

Delete Comments

Delete

XML

Word

Printable

JSON

Details

Type: Bug
Status: Resolved
Priority: Minor
Resolution: Fixed
Affects Version/s: 2.3.0, 2.3.4, 2.4.0, 2.4.8, 3.0.2, 3.1.2, 3.2.0
Fix Version/s: 3.2.0, 3.1.3, 3.0.4
Component/s: SQL
Labels:
None

Description

CoalesceExec calls `child.execute()` in the if condition and throws away the results, then calls `child.execute()` again in the else condition. This could cause a section of the plan to be executed twice.

protected override def doExecute(): RDD[InternalRow] = {
  if (numPartitions == 1 && child.execute().getNumPartitions < 1) {
    // Make sure we don't output an RDD with 0 partitions, when claiming that we have a
    // `SinglePartition`.
    new CoalesceExec.EmptyRDDWithPartitions(sparkContext, numPartitions)
  } else {
    child.execute().coalesce(numPartitions, shuffle = false)
  }
}