[HIVE-2097] Explore mechanisms for better compression with RC Files

• Type: Improvement
• Status: Open
• Priority: Minor
• Resolution: Unresolved
• Component/s: Query Processor, Serializers/Deserializers
• Assignee: Krishna Kumar
• Reporter: Krishna Kumar
• Created: 06/Apr/11 17:26
• Updated: 14/Aug/12 06:00

Description

Optimization of the compression mechanisms used by RC File to be explored.

Some initial ideas

1. More efficient serialization/deserialization based on type-specific and storage-specific knowledge.

For instance, storing sorted numeric values efficiently using some delta coding techniques

2. More efficient compression based on type-specific and storage-specific knowledge

Enable compression codecs to be specified based on types or individual columns

3. Reordering the on-disk storage for better compression efficiency.

Issue Links

• is related to: HIVE-352 Make Hive support column based storage

Sub-Tasks

• Enable/Add type-specific compression for rcfile: Closed
• Add UberCompressor Serde/Codec to contrib which allows per-column compression strategies: Patch Available
• Add Integer type compressors: Patch Available
• Add compressors for qualitative data types: Open