[HIVE-23530] Use SQL functions instead of compute_stats UDAF to compute column statistics

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 4.0.0-alpha-1
• Component/s: Statistics
• Labels: pull-request-available
• Assignee: Jesús Camacho Rodríguez
• Reporter: Jesús Camacho Rodríguez
• Created: 22/May/20 03:48
• Updated: 27/Feb/24 22:23
• Resolved: 29/May/20 00:04

Description

Currently we compute column statistics by relying on the compute_stats UDAF. For instance, for a given table tbl, the query to compute statistics for columns is translated internally into:

SELECT compute_stats(c1),
       compute_stats(c2),
       ...
FROM tbl;

compute_stats produces data for the stats available for each column type, e.g., struct<"max":long,"min":long,"countnulls":long,...>.

This issue is to produce a query that relies purely on SQL functions instead:

SELECT max(c1), min(c1), count(case when c1 is null then 1 else null end),
       ...
FROM tbl;

This will allow us to deprecate the compute_stats UDAF since it mostly duplicates functionality found in those other functions. Additionally, many of those functions already provide a vectorized implementation so the approach can potentially improve the performance of column stats collection.

Issue Links

• is related to: HIVE-27712 GenericUDAFNumericStatsEvaluator throws NPE
• relates to: HIVE-26066 Remove deprecated GenericUDAFComputeStats
• links to: GitHub Pull Request #1034
• links to: PR link