[HIVE-7859] Tune zlib compression in ORC to account for the encoding strategy

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.14.0
• Component/s: File Formats
• Assignee: Gopal Vijayaraghavan
• Reporter: Gopal Vijayaraghavan
• Created: 22/Aug/14 23:40
• Updated: 21/Jan/15 08:40
• Resolved: 11/Sep/14 18:18

Description

Currently ORC Zlib is slow because several compression strategies ZLib uses is already done by ORC in itself (dictionary, RLE, bit-packing).

We need to pick between Z_FILTERED, Z_HUFFMAN_ONLY, Z_RLE, Z_FIXED and Z_DEFAULT_STRATEGY according to column stream type.

For instance an RLE_V2 stream could a use Z_FILTERED compression without invoking the rest of the strategies.

The string streams can use Z_FIXED compression strategies and so on.

The core limitation to stick to retain compatibility with the default decompressor, so that these are automatically backward compatible.