[PARQUET-2436] More optimal memory usage in compression codecs

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.14.0
• Assignee: Gabor Szadovszky
• Reporter: Gabor Szadovszky
• Created: 21/Feb/24 13:37
• Updated: 23/Jun/24 03:33
• Resolved: 27/Feb/24 08:54

Description

Some of our current compressors/decompressors does unnecessary memory copies, suboptimal native calls (e.g. DirectZstd). The goal here is to use a ByteBufferAllocator wherever an allocation is necessary or rewrite if we can avoid memory copies.

Issue Links

• links to: GitHub Pull Request #1280