Details
-
Bug
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
2.2.0, 2.3.0, 2.4.0, 3.0.0
Description
The current PullupCorrelatedPredicates implementation can accidentally remove predicates for multiple runs.
For example, for the following logical plan, one more optimizer run can remove the predicate in the SubqueryExpresssion.
# Optimized
Project [a#0]
+- Filter a#0 IN (list#4 [(b#1 < d#3)])
: +- Project [c#2, d#3]
: +- LocalRelation <empty>, [c#2, d#3]
+- LocalRelation <empty>, [a#0, b#1]
# Double optimized
Project [a#0]
+- Filter a#0 IN (list#4 [])
: +- Project [c#2, d#3]
: +- LocalRelation <empty>, [c#2, d#3]
+- LocalRelation <empty>, [a#0, b#1]
Attachments
Issue Links
- is duplicated by
-
SPARK-28529 Fix PullupCorrelatedPredicates optimizer rule to enforce idempotence
- Resolved
- links to