[SPARK-48613] Umbrella: Storage Partition Join Improvements

• Type: New Feature
• Status: Reopened
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 4.0.0
• Fix Version/s: 4.0.0
• Component/s: SQL
• Labels: pull-request-available
• Assignee: Szehon Ho
• Reporter: Szehon Ho
• Created: 13/Jun/24 07:41
• Updated: 19/Jul/24 21:48

Description

As SPARK-37375 is resolved by release of Spark 4.0, this is for tracking further SPJ enhancements post Spark 4.0.

Issue Links

• is a child of: SPARK-37375 Umbrella: Storage Partitioned Join (SPJ)

Sub-Tasks

• SPJ: Add tests for shuffle skipping for aggregate queries: Resolved
• SPJ: Support auto-shuffle one side + less join keys than partition keys: Resolved
• SPJ: Add documentation: Resolved
• SPJ: Runtime Partition Filtering: Resolved