[HADOOP-13114] DistCp should have option to compress data on write

• Type: Improvement
• Status: Patch Available
• Priority: Minor
• Resolution: Unresolved
• Affects Version/s: 2.8.0, 2.7.3, 3.0.0-alpha1
• Component/s: tools/distcp
• Labels: distcp
• Assignee: Suraj Nayak
• Reporter: Suraj Nayak
• Created: 07/May/16 09:14
• Updated: 25/Apr/17 10:17

Description

DistCp utility should have capability to store data in user specified compression format. This avoids one hop of compressing data after transfer. Backup strategies to different cluster also get benefit of saving one IO operation to and from HDFS, thus saving resources, time and effort.

• Create an option -compressOutput defaulting to org.apache.hadoop.io.compress.BZip2Codec.
• Users will be able to change codec with -D mapreduce.output.fileoutputformat.compress.codec=org.apache.hadoop.io.compress.GzipCodec
• If distcp compression is enabled, suffix the filenames with default codec extension to indicate the file is compressed. Thus users can be aware of what codec was used to compress the data.

Issue Links

• is related to: HADOOP-8065 distcp should have an option to compress data while copying.