[SPARK-37652] Support optimize skewed join through union

• Type: Sub-task
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 3.2.0
• Fix Version/s: 3.3.0
• Component/s: SQL
• Assignee: Unassigned
• Reporter: mcdull_zhang
• Created: 15/Dec/21 12:20
• Updated: 09/Feb/22 13:54
• Resolved: 09/Feb/22 13:54

Description

`OptimizeSkewedJoin` rule will take effect only when the plan has two ShuffleQueryStageExec。

With `Union`, it might break the assumption. For example, the following plans
scenes 1

Union
    SMJ
        ShuffleQueryStage
        ShuffleQueryStage
    SMJ
        ShuffleQueryStage
        ShuffleQueryStage

scenes 2

Union
    SMJ
        ShuffleQueryStage
        ShuffleQueryStage
    HashAggregate

when one or more of the SMJ data in the above plan is skewed, it cannot be processed at present.

It's better to support partial optimize with Union.

Issue Links

• links to: [Github] Pull Request #34908 (mcdull-zhang)
• links to: [Github] Pull Request #34908 (mcdull-zhang)