[SPARK-32460] how spark collects non-match results after performing broadcast left outer join

• Type: Question
• Status: Resolved
• Priority: Major
• Resolution: Invalid
• Affects Version/s: 2.4.0
• Component/s: Spark Core, SQL
• Assignee: Unassigned
• Reporter: farshad delavarpour
• Created: 27/Jul/20 19:02
• Updated: 12/Dec/22 18:11
• Resolved: 28/Jul/20 03:14

Description

Does anybody know how spark collects non-match results after performing broadcast hash left outer join?

Suppose we have 4 nodes. 1 driver and 3 executors. We broadcast the left table. After left outer join is performed in each executor, how does spark recognize which records have not been matched, and how does it collect them to the final result?