[PARQUET-1485] Snappy Decompressor/Compressor may cause direct memory leak

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.8.1
• Fix Version/s: 1.11.0
• Component/s: parquet-mr
• Labels: pull-request-available

Environment

parquet-1.8.1

spark2.1

• Flags: Important
• Assignee: liupengcheng
• Reporter: liupengcheng
• Created: 02/Jan/19 06:33
• Updated: 20/Nov/19 22:56
• Resolved: 12/Feb/19 10:33

Description

In out production environment, we encountered a direct memory oom issues caused by the direct buffer not released in time.

After carefully checked the code, it seems that the some methods of SnappyDecompressor/SnappyCompressor would not release the direct buffer manually. If too much direct memory allocated and no GC happens, this bug may result in direct memory oom.

Moreover, if the `-XX:+DisableImplicitGC` jvm option is specified, the direct memory oom would happen easily for large datasets.

Seems that the problem still exist in the latest code.

Issue Links

• is duplicated by: PARQUET-1320 Fast clean unused direct memory when decompress
• is related to: PARQUET-1533 TestSnappy() throws OOM exception with Parquet-1485 change
• links to: GitHub Pull Request #581