[HADOOP-12007] GzipCodec native CodecPool leaks memory

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 2.7.0
• Labels: pull-request-available
• Assignee: Unassigned
• Reporter: Yejun Yang
• Created: 20/May/15 07:23
• Updated: 01/Aug/22 13:53

Description

org/apache/hadoop/io/compress/GzipCodec.java call CompressionCodec.Util.createOutputStreamWithCodecPool to use CodecPool. But compressor objects are actually never returned to pool which cause memory leak.

HADOOP-10591 uses CompressionOutputStream.close() to return Compressor object to pool. But CompressionCodec.Util.createOutputStreamWithCodecPool actually returns a CompressorStream which overrides close().

This cause CodecPool.returnCompressor never being called. In my log file I can see lots of "Got brand-new compressor [.gz]" but no "Got recycled compressor".

Issue Links

• is broken by: HADOOP-10591 Compression codecs must used pooled direct buffers or deallocate direct buffers when stream is closed