[SPARK-16343] Improve the PushDownPredicate rule to pushdown predicates currectly in non-deterministic condition

• Type: Improvement
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 1.6.2, 2.0.0
• Fix Version/s: 2.0.2, 2.1.0
• Component/s: SQL
• Labels: correctness
• Assignee: Xingbo Jiang
• Reporter: Xingbo Jiang
• Created: 01/Jul/16 09:08
• Updated: 29/Sep/16 19:04
• Resolved: 14/Jul/16 06:14

Description

Currently our Optimizer may reorder the predicates to run them more efficient, but in non-deterministic condition, change the order between deterministic parts and non-deterministic parts may change the number of input rows. For example:

SELECT a FROM t WHERE rand() < 0.1 AND a = 1

And

SELECT a FROM t WHERE a = 1 AND rand() < 0.1

may call rand() for different times and therefore the output rows differ.

Issue Links

• links to: [Github] Pull Request #14012 (jiangxb1987)