[SPARK-10316] respect non-deterministic expressions in PhysicalOperation

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.6.0
• Component/s: SQL
• Assignee: Wenchen Fan
• Reporter: Wenchen Fan
• Created: 27/Aug/15 14:40
• Updated: 06/Jun/19 23:07
• Resolved: 08/Sep/15 19:06

Description

We did a lot of special handling for non-deterministic expressions in Optimizer. However, PhysicalOperation just collects all Projects and Filters and messed it up. We should respect the operators order caused by non-deterministic expressions in PhysicalOperation.

Issue Links

• causes: SPARK-27969 Non-deterministic expressions in filters or projects can unnecessarily prevent all scan-time column pruning, harming performance
• links to: [Github] Pull Request #8486 (cloud-fan)
• links to: GitHub Pull Request #8486