[SPARK-17357] Simplified predicates can't be pushed down through operators because of the rule order in Optimizer

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Incomplete
• Component/s: SQL
• Labels: bulk-closed
• Assignee: Unassigned
• Reporter: L. C. Hsieh
• Created: 01/Sep/16 06:37
• Updated: 01/Jun/20 08:33
• Resolved: 21/May/19 04:37

Description

In Optimizer, Filter operator will go through the rules PushDownPredicate, CombineFilters and BooleanSimplification.

Under this rule order, it is possibly that some predicates that should be able to push down, can't be pushed down through operators.

We should fix it.

Issue Links

• relates to: SPARK-31705 Push more possible predicates through Join via CNF
• links to: [Github] Pull Request #14912 (viirya)
• links to: [Github] Pull Request #15558 (viirya)