[HIVE-3958] support partial scan for analyze command - RCFile

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.11.0
• Labels: TODOC11
• Hadoop Flags: Reviewed
• Assignee: Gang Tim Liu
• Reporter: Gang Tim Liu
• Created: 29/Jan/13 19:25
• Updated: 02/Jan/18 02:24
• Resolved: 27/Mar/13 14:07

Description

analyze commands allows us to collect statistics on existing tables/partitions. It works great but might be slow since it scans all files.

There are 2 ways to speed it up:
1. collect stats without file scan. It may not collect all stats but good and fast enough for use case. HIVE-3917 addresses it
2. collect stats via partial file scan. It doesn't scan all content of files but part of it to get file metadata. some examples are https://cwiki.apache.org/Hive/rcfilecat.html for RCFile, ORC ( HIVE-3874 ) and HFile of Hbase (Edit: That link should be https://cwiki.apache.org/confluence/display/Hive/RCFileCat.)

This jira is targeted to address the #2. More specifically RCFile format.

Issue Links

• depends upon: HIVE-3917 Support noscan operation for analyze command
• is related to: HIVE-17652 retire ANALYZE TABLE ... PARTIALSCAN
• relates to: HIVE-9491 [Parquet] Support Analyze Table with partial scan
• relates to: HIVE-4177 support partial scan for analyze command - ORC
• relates to: HIVE-11020 support partial scan for analyze command - Avro