[HADOOP-10591] Compression codecs must used pooled direct buffers or deallocate direct buffers when stream is closed

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.2.0
• Fix Version/s: 2.6.0
• Target Version/s: 2.6.0
• Assignee: Colin McCabe
• Reporter: Hari Shreedharan
• Created: 09/May/14 07:23
• Updated: 20/May/15 14:39
• Resolved: 17/Jul/14 18:21

Description

Currently direct buffers allocated by compression codecs like Gzip (which allocates 2 direct buffers per instance) are not deallocated when the stream is closed. Eventually for long running processes which create a huge number of files, these direct buffers are left hanging till a full gc, which may or may not happen in a reasonable amount of time - especially if the process does not use a whole lot of heap.

Either these buffers should be pooled or they should be deallocated when the stream is closed.

Issue Links

• breaks: HADOOP-12007 GzipCodec native CodecPool leaks memory
• breaks: FLUME-2416 Use CodecPool in compressed stream to prevent leak of direct buffers