[SPARK-34681] Full outer shuffled hash join when building left side produces wrong result

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 3.1.0, 3.1.1, 3.2.0
• Fix Version/s: 3.1.2, 3.2.0
• Component/s: SQL
• Labels: correctness
• Assignee: Cheng Su
• Reporter: Cheng Su
• Created: 10/Mar/21 00:07
• Updated: 10/Mar/21 06:55
• Resolved: 10/Mar/21 06:55

Description

For full outer shuffled hash join with building hash map on left side, and having non-equal condition, the join can produce wrong result.

The root cause is `boundCondition` in `HashJoin.scala` always assumes the left side row is `streamedPlan` and right side row is `buildPlan` (streamedPlan.output ++ buildPlan.output). This is valid assumption, except for full outer + build left case.

The fix is to correct `boundCondition` in `HashJoin.scala` to handle full outer + build left case properly. See reproduce in https://issues.apache.org/jira/browse/SPARK-32399?focusedCommentId=17298414&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17298414 .

Issue Links

• is caused by: SPARK-32399 Support full outer join in shuffled hash join
• links to: [Github] Pull Request #31792 (c21)
• links to: [Github] Pull Request #31792 (c21)