[HBASE-2225] Enable compression in HBase Export

• Type: Improvement
• Status: Closed
• Priority: Minor
• Resolution: Duplicate
• Affects Version/s: 0.20.1
• Fix Version/s: 0.90.0
• Component/s: util

Environment

OS agnostic

• Release Note: HBase Export tool now uses GzipCodec to compress its output.
• Assignee: Kannan Muthukkaruppan
• Reporter: Ted Yu
• Created: 13/Feb/10 15:46
• Updated: 20/Nov/15 13:01
• Resolved: 28/Oct/10 23:45

Description

org.apache.hadoop.hbase.mapreduce.Export should set compression codec

In createSubmittableJob(), the following should be added:
FileOutputFormat.setCompressOutput(job, true);
FileOutputFormat.setOutputCompressorClass(job, org.apache.hadoop.io.compress.GzipCodec.class);

From my experiment, 10% to 50% reduction in Export output has been observed.

SequenceFileInputFormat used by the Import tool is able to detect GzipCodec - there is no change for Import class.

Issue Links

• is depended upon by: HBASE-2434 Add scanner caching option to Export and write buffer option for Import
• is duplicated by: HBASE-3166 HBase exporter should compress output files by default (or at least allow this as an option)