[SPARK-33832] Add an option in AQE to mitigate skew even if it causes an new shuffle

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.0.0
• Fix Version/s: 3.3.0
• Component/s: SQL
• Labels: pull-request-available
• Assignee: XiDuo You
• Reporter: Eugene Koifman
• Created: 17/Dec/20 22:08
• Updated: 06/Apr/24 18:16
• Resolved: 13/Sep/21 09:22

Description

Currently OptimizeSkewedJoin will not apply if skew mitigation causes a new shuffle.
There are situations where it's better to mitigate skew even if it means a new shuffle is added, for example if the join outputs small amount of data.

As a first step I propose adding a SQLConf option to enable this.

I'll open a PR shortly to get feedback on the approach.

Issue Links

• causes: SPARK-37328 SPARK-33832 brings the bug that OptimizeSkewedJoin may not work since it was applied on whole plan innstead of new stage plan
• links to: [Github] Pull Request #30829 (ekoifman)
• links to: [Github] Pull Request #30829 (ekoifman)
• links to: [Github] Pull Request #31653 (ekoifman)
• links to: [Github] Pull Request #32816 (ulysses-you)
• links to: [Github] Pull Request #34080 (cloud-fan)
• links to: [Github] Pull Request #34080 (cloud-fan)
• links to: GitHub Pull Request #6260