[SPARK-28375] Enforce idempotence on the PullupCorrelatedPredicates optimizer rule

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.2.0, 2.3.0, 2.4.0, 3.0.0
• Fix Version/s: 3.0.0
• Component/s: SQL
• Labels: correctness
• Assignee: Dilip Biswal
• Reporter: Yesheng Ma
• Created: 12/Jul/19 22:50
• Updated: 02/Mar/20 21:31
• Resolved: 30/Jul/19 23:31

Description

The current PullupCorrelatedPredicates implementation can accidentally remove predicates for multiple runs.

For example, for the following logical plan, one more optimizer run can remove the predicate in the SubqueryExpresssion.

# Optimized
Project [a#0]
+- Filter a#0 IN (list#4 [(b#1 < d#3)])
   :  +- Project [c#2, d#3]
   :     +- LocalRelation <empty>, [c#2, d#3]
   +- LocalRelation <empty>, [a#0, b#1]

# Double optimized
Project [a#0]
+- Filter a#0 IN (list#4 [])
   :  +- Project [c#2, d#3]
   :     +- LocalRelation <empty>, [c#2, d#3]
   +- LocalRelation <empty>, [a#0, b#1]

 

 

Issue Links

• is duplicated by: SPARK-28529 Fix PullupCorrelatedPredicates optimizer rule to enforce idempotence
• links to: GitHub Pull Request #25145
• links to: GitHub Pull Request #25164
• links to: GitHub Pull Request #25268