[SPARK-2399] Add support for LZ4 compression

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.1.0
• Component/s: Spark Core
• Labels: compression lz4
• Target Version/s: 1.1.0
• Assignee: Reynold Xin
• Reporter: Greg Bowyer
• Created: 08/Jul/14 05:22
• Updated: 15/Jul/14 08:49
• Resolved: 15/Jul/14 08:49

Description

LZ4 is a compression codec of the same ideas as googles snappy, but has some advantages:

• It is faster than snappy with a similar compression ratio
• The implementation is Apache licensed and not GPL

It has shown promise in both the lucene and hadoop communities, and it looks like its a really easy add to spark io compression.

Attached is a patch that does this

Issue Links

• links to: Github pull request