[SPARK-26864] Query may return incorrect result when python udf is used as a join condition and the udf uses attributes from both legs of left semi join.

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.4.0
• Fix Version/s: 2.4.1, 3.0.0
• Component/s: SQL
• Labels: correctness
• Assignee: Dilip Biswal
• Reporter: Dilip Biswal
• Created: 13/Feb/19 01:12
• Updated: 17/Jul/19 00:56
• Resolved: 16/Feb/19 09:06

Description

In SPARK-25314, we supported the scenario of having a python UDF that refers to attributes from both legs of a join condition by rewriting
the plan to convert an inner join or left semi join to a filter over a cross join. In case of left semi join, this transformation may
cause incorrect results when the right leg of join condition produces duplicate rows based on the join condition.

Issue Links

• links to: GitHub Pull Request #23769
• links to: GitHub Pull Request #23775
• links to: GitHub Pull Request #23780