Details
-
Bug
-
Status: Resolved
-
Minor
-
Resolution: Fixed
-
None
Description
The optimizer pushes down filters for left anti joins. This unfortunately has the opposite effect. For example:
sql("create or replace temporary view tbl_a as values (1, 5), (2, 1), (3, 6) as t(c1, c2)") sql("create or replace temporary view tbl_b as values 1 as t(c1)") sql(""" select * from tbl_a left anti join tbl_b on ((tbl_a.c1 = tbl_a.c2) is null or tbl_a.c1 = tbl_a.c2) """)
Should return rows [1, 5], [2, 1] & [3, 6], but returns no rows.
The upside is that this will only happen when you use a really weird anti-join (only referencing the table on the left hand side).
Attachments
Issue Links
- relates to
-
SPARK-18614 Incorrect predicate pushdown from ExistenceJoin
- Resolved
- links to