[HADOOP-14254] Add a Distcp option to preserve Erasure Coding attributes

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.0.0-alpha4
• Fix Version/s: 3.4.0
• Component/s: tools/distcp
• Target Version/s: 3.4.0, 3.3.2
• Hadoop Flags: Reviewed
• Assignee: Ayush Saxena
• Reporter: Wei-Chiu Chuang
• Created: 29/Mar/17 17:23
• Updated: 24/May/21 02:55
• Resolved: 13/May/20 19:03

Description

Currently Distcp does not preserve the erasure coding attributes properly. I propose we add a "-pe" switch to ensure erasure coded files at source are copied as erasure coded files at destination.

For example, if the src cluster has the following directories and files that are copied to dest cluster
hdfs://src/ root directory is replicated
hdfs://src/foo erasure code enabled directory
hdfs://src/foo/bar erasure coded file

after distcp, hdfs://dest/foo and hdfs://dest/foo/bar will not be erasure coded.

It may be useful to add such capability. One potential use is for disaster recovery. The other use is for out-of-place cluster upgrade.

Issue Links

• relates to: HDFS-10971 Distcp should not copy replication factor if source file is erasure coded