The current PullupCorrelatedPredicates implementation can accidentally remove predicates for multiple runs.
For example, for the following logical plan, one more optimizer run can remove the predicate in the SubqueryExpresssion.
# Optimized
Project [a#0]
+- Filter a#0 IN (list#4 [(b#1 < d#3)])
: +- Project [c#2, d#3]
: +- LocalRelation <empty>, [c#2, d#3]
+- LocalRelation <empty>, [a#0, b#1]
# Double optimized
Project [a#0]
+- Filter a#0 IN (list#4 [])
: +- Project [c#2, d#3]
: +- LocalRelation <empty>, [c#2, d#3]
+- LocalRelation <empty>, [a#0, b#1]
- is duplicated by
-
SPARK-28529 Fix PullupCorrelatedPredicates optimizer rule to enforce idempotence
-
- Resolved
-
- links to