[SPARK-1521] Take character set size into account when compressing in-memory string columns

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Won't Fix
• Affects Version/s: 1.1.0
• Component/s: SQL
• Labels: compression
• Assignee: Unassigned
• Reporter: Cheng Lian
• Created: 17/Apr/14 08:07
• Updated: 12/Jan/16 13:43
• Resolved: 12/Jan/16 13:43

Description

Quoted from a blog post from Facebook:

Strings dominate the largest tables in our warehouse and make up about 80% of the columns across the warehouse, so optimizing compression for string columns was important. By using a threshold on observed number of distinct column values per stripe, we modified the ORCFile writer to apply dictionary encoding to a stripe only when beneficial. Additionally, we sample the column values and take the character set of the column into account, since a small character set can be leveraged by codecs like Zlib for good compression and dictionary encoding then becomes unnecessary or sometimes even detrimental if applied.