[HADOOP-8522] ResetableGzipOutputStream creates invalid gzip files when finish() and resetState() are used

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.0.3, 2.0.0-alpha
• Fix Version/s: 3.0.0
• Component/s: io
• Labels: BB2015-05-TBR
• Hadoop Flags: Reviewed
• Assignee: Mike Percy
• Reporter: Mike Percy
• Created: 21/Jun/12 07:59
• Updated: 11/Nov/17 02:04
• Resolved: 11/Nov/17 01:46

Description

ResetableGzipOutputStream creates invalid gzip files when finish() and resetState() are used. The issue is that finish() flushes the compressor buffer and writes the gzip CRC32 + data length trailer. After that, resetState() does not repeat the gzip header, but simply starts writing more deflate-compressed data. The resultant files are not readable by the Linux "gunzip" tool. ResetableGzipOutputStream should write valid multi-member gzip files.

The gzip format is specified in RFC 1952.

Issue Links

• blocks: FLUME-2967 Corrupted gzip files generated when writting to S3
• is duplicated by: HADOOP-8625 Use GzipCodec to decompress data in ResetableGzipOutputStream test
• is duplicated by: HADOOP-6799 GzipCodec/CompressionOutputStream resetState() fails to reset gzip header and CRC