[SPARK-36079] Null-based filter estimates should always be non-negative

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.2.0
• Fix Version/s: 3.2.0, 3.1.3, 3.0.4
• Component/s: SQL
• Assignee: Karen Feng
• Reporter: Karen Feng
• Created: 09/Jul/21 22:57
• Updated: 20/Jul/21 13:51
• Resolved: 20/Jul/21 13:51

Description

It's possible for a column's statistics to have a higher `nullCount` than the table's `rowCount`. In this case, the filter estimates come back outside of the reasonable range (between 0 and 1).

Issue Links

• links to: [Github] Pull Request #33286 (karenfeng)
• links to: [Github] Pull Request #33286 (karenfeng)