[HIVE-1361] table/partition level statistics

• Type: Sub-task
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.7.0
• Component/s: Query Processor, Statistics
• Assignee: Unassigned
• Reporter: Ning Zhang
• Created: 21/May/10 23:16
• Updated: 02/May/13 02:29
• Resolved: 27/Sep/10 18:03

Description

At the first step, we gather table-level stats for non-partitioned table and partition-level stats for partitioned table. Future work could extend the table level stats to partitioned table as well.

There are 3 major milestones in this subtask:
1) extend the insert statement to gather table/partition level stats on-the-fly.
2) extend metastore API to support storing and retrieving stats for a particular table/partition.
3) add an ANALYZE TABLE [PARTITION] statement in Hive QL to gather stats for existing tables/partitions.

The proposed stats are:

Partition-level stats:

• number of rows
• total size in bytes
• number of files
• max, min, average row sizes
• max, min, average file sizes

Table-level stats in addition to partition level stats:

• number of partitions

Issue Links

• is depended upon by: HIVE-1652 Delete temporary stats data after some time
• is depended upon by: HIVE-1653 Ability to enforce correct stats
• is part of: HIVE-33 [Hive]: Add optimizer statistics in Hive
• relates to: HIVE-1652 Delete temporary stats data after some time
• relates to: HIVE-1653 Ability to enforce correct stats
• relates to: HIVE-3917 Support noscan operation for analyze command