[HBASE-2404] native fast compression codec

• Type: New Feature
• Status: Closed
• Priority: Major
• Resolution: Duplicate
• Assignee: Unassigned
• Reporter: Andrew Kyle Purtell
• Created: 02/Apr/10 17:08
• Updated: 11/Jun/22 23:22
• Resolved: 12/May/14 00:48

Description

We often recommend enabling LZO on tables, most users see big wins. LZO is roughly comparable to BigTable LZW, and we get something like prefix compression on keys. However, LZO is GPL licensed, so a series of install steps are required: http://wiki.apache.org/hadoop/UsingLzoCompression . It's easy to miss a step or get it wrong. If so, all writes on a table (re)configured to use LZO will fail.

Hadoop, well, Java, has native support for gzip compression but it is too slow generally; is a good option however for archival tables.

This issue is about considering bundling or creating a comparable alternate to LZO which is ASF 2.0 license compatible.

Issue Links

• relates to: HBASE-1797 HRegionServer crashes when trying to initialize LZO-Codec which doesn't exist. Instead error must be thrown while creating a HTable with LZO Compression enabled
• relates to: HBASE-3485 implement a Compressor based on LZF
• relates to: HADOOP-6389 Add support for LZF compression
• relates to: HADOOP-6349 Implement FastLZCodec for fastlz/lzo algorithm