[ARROW-6216] [C++] Allow user to select the compression level

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Fix Version/s: 0.15.0
• Component/s: C++
• Labels: pull-request-available
• Flags: Patch
• External issue URL: https://github.com/apache/arrow/issues/16843
• Assignee: Martin Radev
• Reporter: Martin Radev
• Created: 12/Aug/19 21:15
• Updated: 11/Jan/23 07:45
• Resolved: 26/Aug/19 03:09

Description

The compression level selected in Arrow for ZSTD is 1 which is the minimal compression level for the compressor. This leads to very high compression speed at the sacrifice of compression ratio.

The user should be allowed to select the compression level as both speed and ratio are data specific.

The proposed solution is to expose the knob via an environment variable such as ARROW_ZSTD_COMPRESSION_LEVEL.
Example:
export ARROW_ZSTD_COMPRESSION_LEVEL=10
./my_parquet_app

Here is a test run with compression levels of 1, 2 and 5:
Level   Time (s)   Size (mb)
1          13.02       181
2          13.10       177
5          19.44       148

Issue Links

• is related to: ARROW-2290 [C++/Python] Add ability to set codec options for lz4 codec
• links to: GitHub Pull Request #5071
• links to: GitHub Pull Request #5192