[SPARK-32399] Support full outer join in shuffled hash join

• Type: Sub-task
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 3.1.0
• Fix Version/s: 3.1.0
• Component/s: SQL
• Assignee: Cheng Su
• Reporter: Cheng Su
• Created: 22/Jul/20 21:22
• Updated: 10/Mar/21 06:46
• Resolved: 16/Aug/20 23:08

Description

Currently for SQL full outer join, spark always does a sort merge join no matter of how large the join children size are. Inspired by recent discussion in https://github.com/apache/spark/pull/29130#discussion_r456502678 and https://github.com/apache/spark/pull/29181, I think we can support full outer join in shuffled hash join in a way that - when looking up stream side keys from build side HashedRelation. Mark this info inside build side HashedRelation, and after reading all rows from stream side, output all non-matching rows from build side based on modified HashedRelation.

Issue Links

• causes: SPARK-34681 Full outer shuffled hash join when building left side produces wrong result
• links to: [Github] Pull Request #29342 (c21)
• links to: [Github] Pull Request #29342 (c21)