[PARQUET-2032] Deprecate LZ4, introduce new LZ4_RAW

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: parquet-mr
• Assignee: Unassigned
• Reporter: Gabor Szadovszky
• Created: 19/Apr/21 14:26
• Updated: 23/Jun/24 03:32

Description

The currently implemented LZ4 compression is based on the hadoop codec which is now deprecated (see details at PARQUET-1996). Also, a new, properly specified LZ4 compression (LZ4_RAW) has been introduced in the format.

The idea is to use the new LZ4_RAW compression for all cases when we currently use LZ4 and introduce a new configuration where the user would be able to switch to the deprecated behavior in case of the selected codec is LZ4.

For LZ4_RAW we will need to add the proper libs that will provide the officially specified LZ4 raw format.