[SPARK-20104] Don't estimate IsNull or IsNotNull predicates for non-leaf node

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.2.0
• Fix Version/s: 2.2.0
• Component/s: SQL
• Assignee: Zhenhua Wang
• Reporter: Zhenhua Wang
• Created: 27/Mar/17 03:26
• Updated: 30/Mar/17 01:47
• Resolved: 27/Mar/17 15:42

Description

In current stage, we don't have advanced statistics such as sketches or histograms. As a result, some operator can't estimate `nullCount` accurately. E.g. left outer join estimation does not accurately update `nullCount` currently. So for IsNull and IsNotNull predicates, we only estimate them when the child is a leaf node, whose `nullCount` is accurate.

Issue Links

• links to: [Github] Pull Request #17438 (wzhfy)