[HDFS-10971] Distcp should not copy replication factor if source file is erasure coded

• Type: Sub-task
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 3.0.0-alpha1
• Fix Version/s: 3.0.0-alpha4
• Component/s: distcp
• Labels: hdfs-ec-3.0-must-do
• Target Version/s: 3.0.0-alpha4
• Assignee: Manoj Govindassamy
• Reporter: Wei-Chiu Chuang
• Created: 06/Oct/16 18:34
• Updated: 02/Oct/19 17:14
• Resolved: 29/Mar/17 05:14

Description

The current erasure coding implementation uses replication factor field to store erasure coding policy.

Distcp copies the source file's replication factor to the destination if -pr is specified. However, if the source file is EC, the replication factor (which is EC policy) should not be replicated to the destination file. When a HdfsFileStatus is converted to FileStatus, the replication factor is set to 0 if it's an EC file.

In fact, I will attach a test case that shows trying to replicate the replication factor of an EC file results in an IOException: "Requested replication factor of 0 is less than the required minimum of 1 for /tmp/dst/dest2"

Issue Links

• is blocked by: HADOOP-13715 Add isErasureCoded() API to FileStatus class
• is related to: HADOOP-14254 Add a Distcp option to preserve Erasure Coding attributes