[SPARK-48065] SPJ: allowJoinKeysSubsetOfPartitionKeys is too strict

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.4.3
• Fix Version/s: 4.0.0
• Component/s: SQL
• Labels: pull-request-available
• Assignee: Szehon Ho
• Reporter: Szehon Ho
• Created: 30/Apr/24 22:27
• Updated: 03/May/24 01:50
• Resolved: 02/May/24 23:14

Description

If spark.sql.v2.bucketing.allowJoinKeysSubsetOfPartitionKeys.enabled is true, then SPJ no longer triggers if there are more join keys than partition keys.  It is triggered only if join keys is equal to , or less than, partition keys.

 

We can relax this constraint, as this case was supported if the flag is not enabled.

Issue Links

• links to: GitHub Pull Request #46325