[HUDI-2928] Evaluate rebasing Hudi's default compression from Gzip to Zstd

• Type: Improvement
• Status: Open
• Priority: Critical
• Resolution: Unresolved
• Component/s: performance, storage-management
• Labels: pull-request-available
• Story Points: 2
• Epic Link: Performance Improvements
• Assignee: Alexey Kudinkin
• Reporter: Alexey Kudinkin
• Created: 03/Dec/21 20:48
• Updated: 10/Mar/23 01:50

Description

Currently, having Gzip as a default we prioritize Compression/Storage cost at the expense of

• Compute (on the write-path): about 30% of Compute burned during bulk-insert in local benchmarks on Amazon Reviews dataset is Gzip (see below) 
• Compute (on the read-path), as well as queries Latencies: queries scanning large datasets are likely to be compression-/CPU-bound (Gzip t/put is 3-4x less than Snappy, Zstd, EX)

P.S Spark switched its default compression algorithm to Snappy a while ago.

 

EDIT

We should actually evaluate putting in zstd instead of Snappy. It has compression ratios comparable to Gzip, while bringing in much better performance:

https://engineering.fb.com/2016/08/31/core-data/smaller-and-faster-data-compression-with-zstandard/

 

 

 

Issue Links

• is a child of: HUDI-2151 Make performant out-of-box configs
• is blocked by: HUDI-2811 Support Spark 3.2 and Parquet 1.12.x
• is related to: HUDI-2948 Hudi Clustering Performance
• links to: GitHub Pull Request #4214