[SPARK-12957] Derive and propagate data constrains in logical plan

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.0.0
• Component/s: SQL
• Assignee: Sameer Agarwal
• Reporter: Yin Huai
• Created: 21/Jan/16 18:48
• Updated: 26/Jul/17 16:05
• Resolved: 26/Jul/17 16:05

Description

Based on the semantic of a query plan, we can derive data constrains (e.g. if a filter defines a > 10, we know that the output data of this filter satisfy the constrain of a > 10 and a is not null). We should build a framework to derive and propagate constrains in the logical plan, which can help us to build more advanced optimizations.

Issue Links

• is related to: SPARK-19609 Broadcast joins should pushdown join constraints as Filter to the larger relation
• links to: [Github] Pull Request #10844 (sameeragarwal)

Sub-Tasks

• Add initial support for constraint propagation in SparkSQL: Resolved
• Rewrite/Propagate constraints for Aliases: Resolved
• Track constraints in ExpressionSet: Resolved
• Add Null Filters in the query plan for Filters/Joins based on their data constraints: Resolved
• Add IsNotNull constraints for expressions with an inequality: Resolved
• Use ExpressionSets in ConstraintPropagationSuite: Resolved
• Infer additional constraints from attribute equality: Resolved
• Add support for inferring filters from data constraints: Resolved
• Infer IsNotNull constraints from non-nullable attributes: Resolved