[SPARK-32806] SortMergeJoin with partial hash distribution can be optimized to remove shuffle

• Type: Improvement
• Status: In Progress
• Priority: Minor
• Resolution: Unresolved
• Affects Version/s: 3.1.0
• Component/s: SQL
• Assignee: Unassigned
• Reporter: Terry Kim
• Created: 07/Sep/20 02:02
• Updated: 11/Sep/20 13:42

Description

SortMergeJoin with partial hash distribution can be optimized to remove shuffle if the hash partitioning expressions are a subset of join keys for both sides.

Issue Links

• relates to: SPARK-18067 SortMergeJoin adds shuffle if join predicates have non partitioned columns
• links to: [Github] Pull Request #29655 (imback82)
• links to: [Github] Pull Request #29655 (imback82)