[HIVE-22939] Datasketches support

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Assignee: Unassigned
• Reporter: Zoltan Haindrich
• Created: 27/Feb/20 09:39
• Updated: 27/Feb/20 09:42

Description

We could probably integrate with the Datasketches more closely; it has very usefull alogrithms which could utilized various ways:

• provide an optional way to transparently rewrite count(distinct) to use some distinct counting sketch
• fequent items could be gathered during statistics collection; knowing the most frequent elements could extremely helpfull in giving more accurate estimates for our plans
• and...it also has a way to estimate a CDF function; which might be usefull in giving better estimates for range queries

https://datasketches.apache.org/

Sub-Tasks

• Make the datasketches functions available as predefined functions: Closed
• Enable sketch union-s to be rolled up: Closed
• Add option to enable transparent rewrite of count(distinct) into sketch functions: Closed
• MV rebuild should produce the same view as the one configured at creation time: Closed
• Add option to rewrite PERCENTILE_DISC to sketch functions: Closed
• Add option to rewrite CUME_DIST to sketch functions: Closed
• Add option to rewrite NTILE and RANK to sketch functions: Closed
• Upgrade to datasketches 1.1.0: Closed
• SqlFunctionConverter uses both a passed name for the function and the name from UDF's annotation: Open
• Use the SqlSplittableAggFunction in HiveMergeablAggregate functions: Open
• Reorganize sketch functions registration logic: Open
• Introduce ReturnTypeInference for sketch functions: Open
• Map more windowing aggregate functions to calcite: Open
• CUME_DIST should not return NULL when partitioning column is NULL: Open
• Enable users to provide the construction parameters for sketches: Open