[HADOOP-7076] Splittable Gzip

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Later
• Component/s: io
• Target Version/s: 0.23.2
• Release Note: Make Gzipped input splittable by offering a tradeoff between "Spent resources" and "Wall clock time"
• Assignee: Niels Basjes
• Reporter: Niels Basjes
• Created: 23/Dec/10 10:09
• Updated: 19/Sep/19 00:46
• Resolved: 05/Apr/12 21:23

Description

Files compressed with the gzip codec are not splittable due to the nature of the codec.
This limits the options you have scaling out when reading large gzipped input files.

Given the fact that gunzipping a 1GiB file usually takes only 2 minutes I figured that for some use cases wasting some resources may result in a shorter job time under certain conditions.
So reading the entire input file from the start for each split (wasting resources!!) may lead to additional scalability.

Issue Links

• is related to: HADOOP-7909 Implement a generic splittable signature-based compression format
• is related to: HADOOP-6153 RAgzip: multiple map tasks for a large gzipped file
• is related to: SPARK-29102 Read gzipped file into multiple partitions without full gzip expansion on a single-node