[HIVE-3516] Fast incremental statistics computation on columns in Hive tables

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: Statistics
• Assignee: Shreepadma Venugopalan
• Reporter: Shreepadma Venugopalan
• Created: 29/Sep/12 19:28
• Updated: 02/May/13 02:30

Description

Statistics computed on Hive columns in partition can be rolled up to avoid scanning the table again to compute column statistics at the table(global) level. While its straightforward to roll up some statistics such as max, min, avgcollen, maxcollen etc, rolling up other statistics such as ndv requires maintaining intermediate state. This ticket covers the task of a) maintaining the necessary intermediate state needed to roll up partition level statistics b) detecting that the partition level statistics can be rolled up and actually computing table level statistics from partition level statistics.

Issue Links

• is depended upon by: HIVE-3602 Provide ability to roll up column stats from specific partitions
• is related to: HIVE-1362 Optimizer statistics on columns in tables and partitions