[SPARK-41952] Upgrade Parquet to fix off-heap memory leaks in Zstd codec

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 3.1.3, 3.3.1, 3.2.3
• Fix Version/s: 3.2.4, 3.3.3, 3.4.0
• Component/s: Input/Output
• Labels: pull-request-available
• Assignee: Cheng Pan
• Reporter: Alexey Kudinkin
• Created: 09/Jan/23 23:48
• Updated: 20/Jun/24 06:39
• Resolved: 20/Feb/23 17:45

Description

Recently, native memory leak have been discovered in Parquet in conjunction of it using Zstd decompressor from luben/zstd-jni library (PARQUET-2160).

This is very problematic to a point where we can't use Parquet w/ Zstd due to pervasive OOMs taking down our executors and disrupting our jobs.

Luckily fix addressing this had already landed in Parquet:
https://github.com/apache/parquet-mr/pull/982

 

Now, we just need to

1. Updated version of Parquet is released in a timely manner
2. Spark is upgraded onto this new version in the upcoming release

 

Issue Links

• links to: [Github] Pull Request #40091 (pan3793)
• links to: GitHub Pull Request #40091