[PARQUET-353] Compressors not getting recycled while writing parquet files, causing memory leak

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.6.0, 1.7.0, 1.8.0
• Fix Version/s: 1.9.0, 1.8.2
• Component/s: parquet-mr
• Assignee: Nitin Goyal
• Reporter: Nitin Goyal
• Created: 01/Aug/15 17:37
• Updated: 21/Apr/18 12:39
• Resolved: 11/Dec/15 21:19

Description

Compressors are not getting recycled while writing parquet files. This is causing native/physical memory leak in my spark app which is parquet write intensive since its creating new compressors everytime i write parquet files.

The actual code issue is that we are creating 'codecFactory' in 'getRecordWriter' method of ParquetOutputFormat.java but not calling codecFactory.release() which is responsible for recycling compressors.

Issue Links

• blocks: PARQUET-392 Release Parquet-mr 1.9.0
• links to: PR #282
• links to: PR #295