Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-8031 Follow-on work for erasure coding phase I (striping layout)
  3. HDFS-10971

Distcp should not copy replication factor if source file is erasure coded

    XMLWordPrintableJSON

    Details

      Description

      The current erasure coding implementation uses replication factor field to store erasure coding policy.

      Distcp copies the source file's replication factor to the destination if -pr is specified. However, if the source file is EC, the replication factor (which is EC policy) should not be replicated to the destination file. When a HdfsFileStatus is converted to FileStatus, the replication factor is set to 0 if it's an EC file.

      In fact, I will attach a test case that shows trying to replicate the replication factor of an EC file results in an IOException: "Requested replication factor of 0 is less than the required minimum of 1 for /tmp/dst/dest2"

        Attachments

        1. HDFS-10971.01.patch
          8 kB
          Manoj Govindassamy
        2. HDFS-10971.02.patch
          11 kB
          Manoj Govindassamy
        3. HDFS-10971.testcase.patch
          3 kB
          Wei-Chiu Chuang

          Issue Links

            Activity

              People

              • Assignee:
                manojg Manoj Govindassamy
                Reporter:
                weichiu Wei-Chiu Chuang
              • Votes:
                0 Vote for this issue
                Watchers:
                11 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: