[SPARK-8599] Improve non-deterministic expression handling

• Type: Improvement
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 1.4.0
• Fix Version/s: 1.5.0
• Component/s: SQL
• Target Version/s: 1.5.0
• Assignee: Wenchen Fan
• Reporter: Yin Huai
• Created: 24/Jun/15 20:24
• Updated: 29/Jul/15 04:39
• Resolved: 29/Jul/15 04:39

Description

Right now, we are using expressions for Random distribution generating expressions. But, we have to track them in lots of places in the optimizer to handle them carefully. Otherwise, these expressions will be treated as stateless expressions and have unexpected behaviors (e.g. SPARK-8023).

Issue Links

• blocks: SPARK-7157 Add approximate stratified sampling to DataFrame

Sub-Tasks

• After initializing a DataFrame with random columns and a seed, df.show should return same value: Resolved
• After initializing a DataFrame with random columns and a seed, ordering by that random column should return same sorted order: Resolved
• Filter using non-deterministic expressions should not be pushed down: Resolved
• If order by clause has non-deterministic expressions, we should add a project to materialize results of these expressions: Resolved
• Improve project collapse with nondeterministic expressions: Resolved
• Support mutable state in code gen expressions: Resolved
• make deterministic describing the tree rather than the expression: Resolved
• Initialize nondeterministic expressions in code gen fallback mode: Resolved
• add initialization phase for nondeterministic expression: Resolved