[SPARK-48012] SPJ: Support Transfrom Expressions for One Side Shuffle

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.4.3
• Fix Version/s: 4.0.0
• Component/s: SQL
• Labels: pull-request-available
• Target Version/s: 4.0.0
• Assignee: Szehon Ho
• Reporter: Szehon Ho
• Created: 26/Apr/24 19:09
• Updated: 09/Jun/24 14:22
• Resolved: 09/Jun/24 14:22

Description

SPARK-41471 allowed Spark to shuffle just one side and still conduct SPJ, if the other side is KeyGroupedPartitioning.  However, the support was just for a KeyGroupedPartition without any partition transform (day, year, bucket).  It will be useful to add support for partition transform as well, as there are many tables partitioned by those transforms.

Issue Links

• links to: GitHub Pull Request #46255