[SPARK-46590] Coalesce partiton assert error after skew join optimization

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.3.1, 3.3.3, 3.3.2, 3.4.0, 3.4.1, 3.5.0, 4.0.0, 3.5.1, 3.3.4
• Fix Version/s: 4.0.0, 3.5.1
• Component/s: SQL
• Labels: pull-request-available
• Assignee: Jackey Lee
• Reporter: Jackey Lee
• Created: 04/Jan/24 08:36
• Updated: 24/Jan/24 07:44
• Resolved: 23/Jan/24 08:12

Description

Recently when we were testing TPCDS Q71, we found that if `spark.sql.shuffle.partitions` and `spark.sql.adaptive.coalescePartitions.initialPartitionNum` are both set to the number of executor cores, an AssertError may be reported in coalescePartition due to the partitionSpecs of joins after skew are different.

Issue Links

• links to: GitHub Pull Request #44661
• links to: GitHub Pull Request #44854