[PIG-2746] Pig doesn't detect all forms of compression extensions properly

• Type: Bug
• Status: Reopened
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 0.8.1
• Hadoop Flags: Reviewed
• Assignee: Unassigned
• Reporter: Harsh J
• Created: 11/Jun/12 06:35
• Updated: 08/Mar/17 12:28

Description

The PigStorage has the following snippet.

private void setCompression(Path path, Job job) {
     	String location=path.getName();
        if (location.endsWith(".bz2") || location.endsWith(".bz")) {
            FileOutputFormat.setCompressOutput(job, true);
            FileOutputFormat.setOutputCompressorClass(job,  BZip2Codec.class);
        }  else if (location.endsWith(".gz")) {
            FileOutputFormat.setCompressOutput(job, true);
            FileOutputFormat.setOutputCompressorClass(job, GzipCodec.class);
        } else {
            FileOutputFormat.setCompressOutput( job, false);
        }
    }

This limits it to only work with STORE filenames provided as 'output.gz' or 'output.bz2' and for the rest (like LZO) one has to specify codecs and manually enable compression.

Ideally Pig can rely on Hadoop's extension-to-codec detector instead of having this ladder.

Issue Links

• depends upon: HADOOP-8570 Bzip2Codec should accept .bz files too