I've been testing Brotli, a new compression library based on LZ77 from Google. Google's brotli benchmarks look really good and we're also seeing a significant improvement in compression size, compression speed, or both.
[blue@work Downloads]$ time parquet from test.parquet -o test.snappy.parquet --compression-codec snappy --overwrite
[blue@work Downloads]$ time parquet from test.parquet -o test.br.parquet --compression-codec brotli --overwrite
[blue@work Downloads]$ time parquet from test.parquet -o test.gz.parquet --compression-codec gzip --overwrite
[blue@work Downloads]$ ls -l
-rw-r--r-- 1 blue blue 1068821936 May 10 11:06 test.br.parquet
-rw-r--r-- 1 blue blue 1421601880 May 10 11:10 test.gz.parquet
-rw-r--r-- 1 blue blue 2265950833 May 10 10:30 test.snappy.parquet
Brotli, at quality 1, is as fast as snappy and ends up smaller than gzip-9. Another test resulted in a slightly larger Brotli file than gzip produced, but Brotli was 4x faster. I'd like to get this compression codec into Hadoop.
Brotli is licensed with the MIT license, and the JNI library jbrotli is ALv2.