[BIGTOP-3921] Add ZSTD Codec Support for hadoop

• Type: Improvement
• Status: Open
• Priority: Minor
• Resolution: Unresolved
• Affects Version/s: 3.2.0
• Component/s: hadoop
• Assignee: Robin Zuo
• Reporter: Robin Zuo
• Created: 28/Mar/23 06:55
• Updated: 22/Apr/24 07:36

Description

Zstandard, or zstd as short version, is a fast lossless compression algorithm, targeting real-time compression scenarios at zlib-level and better compression ratios. It's backed by a very fast entropy stage, provided by Huff0 and FSE library.

 

ZSTD is famous for its better performance than other codecs.

When maintaining the same compression ratio, the decompression and compression rates are much higher than snappy; When maintaining the same decompression and compression rates, the compression ratio is about 50% higher than snappy.

 

For more detailed performance test result, please refer to zstd github repository:

https://github.com/facebook/zstd

 

So, I'd like add ZSTD codec compiled to hadoop native library by default.

Issue Links

• links to: GitHub pull request #1095