[SPARK-36794] Ignore duplicated join keys when building relation for SEMI/ANTI shuffle hash join

• Type: Sub-task
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 3.3.0
• Fix Version/s: 3.3.0
• Component/s: SQL
• Assignee: Cheng Su
• Reporter: Cheng Su
• Created: 17/Sep/21 18:59
• Updated: 13/Oct/21 03:38
• Resolved: 13/Oct/21 03:38

Description

For LEFT SEMI and LEFT ANTI hash equi-join without extra join condition, we only need to keep one row per unique join key(s) inside hash table (`HashedRelation`) when building the hash table. This can help reduce the size of hash table of join.

Issue Links

• links to: [Github] Pull Request #34034 (c21)
• links to: [Github] Pull Request #34247 (c21)