[HIVE-22239] Scale data size using column value ranges

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 4.0.0-alpha-1
• Component/s: Physical Optimizer
• Labels: pull-request-available
• Assignee: Jesús Camacho Rodríguez
• Reporter: Jesús Camacho Rodríguez
• Created: 24/Sep/19 21:45
• Updated: 27/Feb/24 22:23
• Resolved: 15/Oct/19 21:38

Description

Currently, min/max values for columns are only used to determine whether a certain range filter falls out of range and thus filters all rows or none at all. If it does not, we just use a heuristic that the condition will filter 1/3 of the input rows. Instead of using that heuristic, we can use another one that assumes that data will be uniformly distributed across that range, and calculate the selectivity for the condition accordingly.

Issue Links

• is blocked by: HIVE-22248 Min value for column in stats is not set correctly for some data types
• is related to: HIVE-22311 Propagate min/max column values from statistics to the optimizer for timestamp type
• links to: GitHub Pull Request #787