[IMPALA-2658] Extend the NDV function to accept a precision

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: Impala 2.2.4
• Fix Version/s: Impala 4.0.0
• Component/s: Backend
• Labels: ramp-up
• Target Version: Product Backlog
• Assignee: Qifan Chen
• Reporter: Peter Ebert
• Created: 10/Nov/15 05:19
• Updated: 27/Apr/21 13:55
• Resolved: 26/Jun/20 19:56

Description

Hyperloglog algorithm used by NDV defaults to a precision of 10. Being able to set this precision would have two benefits:

1. Lower precision sizes can speed up the performance, as a precision of 9 has 1/2 the number of registers as 10 (exponential) and may be just as accurate depending on expected cardinality.
2. Higher precision can help with very large cardinalities (100 million to billion range) and will typically provide more accurate data. Those who are presenting estimates to end users will likely be willing to trade some performance cost for more accuracy, while still out performing the naive approach by a large margin.

Propose adding the overloaded function NDV(expression, int precision)
with accepted range between 18 and 4 inclusive.

Issue Links

• is depended upon by: IMPALA-5449 Implement LinearCounting Functionality of HyperLogLog++ for small cardinalities
• is related to: IMPALA-10538 Document the newly added scale argument of ndv function