[IMPALA-10463] Implement ds_theta_sketch() and ds_theta_estimate() functions

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: Impala 4.0.0
• Component/s: Backend, Frontend
• Epic Link: datasketches-theta
• Epic Color: ghx-label-6
• Assignee: Fucun Chu
• Reporter: Fucun Chu
• Created: 29/Jan/21 13:15
• Updated: 19/Feb/21 14:05
• Resolved: 19/Feb/21 14:05

Description

These should be built-in functions that use DataSketches functionality that was integrated by IMPALA-10440.

ds_theta_sketch() should receive a primitive expression and return a sketch.ds_theta_estimate() should receive a sketch and return a primitive that is the cardinality estimate for that set of data provided to the sketch.

Usage:

select ds_theta_estimate(ds_theta_sketch(col_name)) from table_name;Returns a cardinality estimate for that particular column.

Hive change that introduced the same: https://issues.apache.org/jira/browse/HIVE-22940

https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/exec/DataSketchesFunctions.java