[HIVE-461] Optimize RCFile reading by using column pruning results

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.4.0
• Fix Version/s: 0.4.0
• Component/s: Serializers/Deserializers
• Hadoop Flags: Reviewed
• Release Note: HIVE-461. Optimize RCFile reading by using column pruning results. (Yongqiang He via zshao)
• Assignee: He Yongqiang
• Reporter: Zheng Shao
• Created: 30/Apr/09 01:49
• Updated: 17/Dec/11 00:07
• Resolved: 06/Jul/09 01:05

Description

RCFile is a column-based file format introduced in HIVE-352. Column-based storage has shown better compression ratio. On our internal data set (30 columns, most of them are short integer strings), we are seeing gzip-compressed RCFile to be 20%+ smaller than gzip-compressed SequenceFile.

RCFIle also has the potential to improve the reading efficiency a lot since it compresses each column separately.

We should integrate RCFile with the column pruning results from Hive to make the reading faster.

Issue Links

• is blocked by: HIVE-352 Make Hive support column based storage
• is blocked by: HIVE-460 Improve ColumnPruner to prune more aggressively and keep column information for input tables