[HIVE-7654] A method to extrapolate columnStats for partitions of a table

• Type: New Feature
• Status: Closed
• Priority: Minor
• Resolution: Fixed
• Fix Version/s: 0.14.0
• Component/s: Statistics
• Labels: TODOC14
• Assignee: Pengcheng Xiong
• Reporter: Pengcheng Xiong
• Created: 07/Aug/14 21:52
• Updated: 13/Nov/14 19:42
• Resolved: 23/Aug/14 00:54

Description

In a PARTITIONED table, there are many partitions. For example,

create table if not exists loc_orc (
state string,
locid int,
zip bigint
) partitioned by(year string) stored as orc;

We assume there are 4 partitions, partition(year='2000'), partition(year='2001'), partition(year='2002') and partition(year='2003').

We can use the following command to compute statistics for columns state,locid of partition(year='2001')

analyze table loc_orc partition(year='2001') compute statistics for columns state,locid;

We need to know the “aggregated” column status for the whole table loc_orc. However, we may not have the column status for some partitions, e.g., partition(year='2002') and also we may not have the column status for some columns, e.g., zip bigint for partition(year='2001')

We propose a method to extrapolate the missing column status for the partitions.