[AVRO-2195] Add Zstandard Codec

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.9.0
• Fix Version/s: 1.9.0
• Component/s: java
• Labels: patch
• Hadoop Flags: Incompatible change, Reviewed
• Assignee: Benson Qiu
• Reporter: Benson Qiu
• Created: 29/Jun/18 08:19
• Updated: 07/Feb/20 10:13
• Resolved: 30/Jul/18 08:20

Description

Inspired by AVRO-1373. The Zstandard algorithm is available in the commons-library, which Avro projects already depend on.

In a quick test that I did, Zstandard had a better compression ratio than deflate (compression level 9).

https://code.fb.com/core-data/smaller-and-faster-data-compression-with-zstandard/ 

Issue Links

• supercedes: AVRO-2162 Add Zstandard compression to avro file format