[PARQUET-2429] Direct buffer churn in NonBlockedDecompressor

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.14.0
• Assignee: Gian Merlino
• Reporter: Gian Merlino
• Created: 07/Feb/24 23:58
• Updated: 23/Jun/24 03:33
• Resolved: 23/Apr/24 05:06

Description

Input buffers for NonBlockedDecompressor (and NonBlockedCompressor) are grown one chunk at a time as the class receives successive setInput calls. When decompressing a 64MB block using a 4KB chunk size, this leads to thousands of allocations and deallocations totaling GBs of memory. This can be avoided by doubling the buffer each time rather than adding on a minimal amount of new space.

In a practical scenario I ran into, the time taken to read a 140MB Parquet file was reduced from 35s to <2s.

PR: https://github.com/apache/parquet-mr/pull/1270

Issue Links

• links to: GitHub Pull Request #1270