LZ4 is a compression codec of the same ideas as googles snappy, but has some advantages:
- It is faster than snappy with a similar compression ratio
- The implementation is Apache licensed and not GPL
It has shown promise in both the lucene and hadoop communities, and it looks like its a really easy add to spark io compression.
Attached is a patch that does this