[SPARK-34980] Support coalesce partition through union

• Type: Sub-task
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 3.2.0
• Fix Version/s: 3.3.0
• Component/s: SQL
• Assignee: XiDuo You
• Reporter: XiDuo You
• Created: 07/Apr/21 09:53
• Updated: 27/Jan/22 20:03
• Resolved: 29/Sep/21 08:32

Description

The rule `CoalesceShufflePartitions` can only coalesce paritition if

• leaf node is ShuffleQueryStage
• all shuffle have same partition number

With `Union`, it might break the assumption. Let's say we have such plan

Union
   HashAggregate
      ShuffleQueryStage
   FileScan

`CoalesceShufflePartitions` can not optimize it and the result partition would be `shuffle partition + FileScan partition` which can be quite lagre.

It's better to support partial optimize with `Union`.

Issue Links

• links to: [Github] Pull Request #32084 (ulysses-you)