[HADOOP-6835] Support concatenated gzip files

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.20.2
• Fix Version/s: 0.22.0
• Component/s: io
• Hadoop Flags: Incompatible change, Reviewed
• Release Note: Processing of concatenated gzip files formerly stopped (quietly) at the end of the first substream/"member"; now processing will continue to the end of the concatenated stream, like gzip(1) does. (bzip2 support is unaffected by this patch.)
• Assignee: Greg Roelofs
• Reporter: Thomas White
• Created: 12/Jan/09 13:16
• Updated: 12/Dec/11 06:19
• Resolved: 07/Jul/10 23:23

Description

When running MapReduce with concatenated gzip files as input only the first part is read, which is confusing, to say the least. Concatenated gzip is described in http://www.gnu.org/software/gzip/manual/gzip.html#Advanced-usage and in http://www.ietf.org/rfc/rfc1952.txt. (See original report at http://www.nabble.com/Problem-with-Hadoop-and-concatenated-gzip-files-to21383097.html)

Issue Links

• blocks: HADOOP-7386 Support concatenated bzip2 files
• is related to: HADOOP-6335 Support reading of concatenated gzip and bzip2 files
• is related to: PIG-42 Pig should be able to split Gzip files like it can split Bzip files
• relates to: MAPREDUCE-1795 add error option if file-based record-readers fail to consume all input (e.g., concatenated gzip, bzip2)