[HIVE-3917] Support noscan operation for analyze command

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.11.0
• Fix Version/s: 0.11.0
• Component/s: Statistics
• Hadoop Flags: Reviewed
• Assignee: Gang Tim Liu
• Reporter: Gang Tim Liu
• Created: 19/Jan/13 02:11
• Updated: 15/Jan/16 22:25
• Resolved: 03/Feb/13 17:46

Description

hive supports analyze command to gather statistics from existing tables/partition https://cwiki.apache.org/confluence/display/Hive/StatsDev#StatsDev-ExistingTables

It collects:
1. Number of Rows
2. Number of files
3. Size in Bytes

If table/partition is big, the operation would take time since it will open all files and scan all data.

It would be nice to support fast operation to gather statistics which doesn't require to open all files:
1. Number of files
2. Size in Bytes

Potential syntax is
ANALYZE TABLE tablename [PARTITION(partcol1[=val1], partcol2[=val2], ...)] COMPUTE STATISTICS [noscan];

In the future, all statistics without scan can be retrieved via this optional parameter.

Issue Links

• is depended upon by: HIVE-3958 support partial scan for analyze command - RCFile
• is related to: HIVE-1361 table/partition level statistics
• is related to: HIVE-33 [Hive]: Add optimizer statistics in Hive
• relates to: HIVE-12661 StatsSetupConst.COLUMN_STATS_ACCURATE is not used correctly
• relates to: HIVE-3954 flag indecating statistics is stale