Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-10974

Document replication factor for EC files.

    Details

      Description

      The replication factor of an EC file is 0 from a user perspective. This affects all user visible commands and APIs, such as hdfs dfs -ls, hdfs dfs -stat, webhdfs, httpfs...etc.

      In addition, FileSystem#setReplication API for an EC file is ignored. So is the command hdfs dfs -setrep

      1. HDFS-10974.001.patch
        4 kB
        Yiqun Lin
      2. HDFS-10974.002.patch
        5 kB
        Yiqun Lin
      3. HDFS-10974.003.patch
        4 kB
        Yiqun Lin
      4. HDFS-10974.004.patch
        6 kB
        Yiqun Lin

        Activity

        Hide
        linyiqun Yiqun Lin added a comment -

        Hi Wei-Chiu Chuang, I would like to attach a initial patch for this. But before that, I want to make a right understanding for this.

        The replication factor of an EC file is 0 from a user perspective. This affects all user visible commands and APIs, such as hdfs dfs -ls, hdfs dfs -stat, webhdfs, httpfs...etc.

        Does this means that we should document the replication factor of 0 for the EC file in some commands and APIs that will print the replicaion factor?
        I'd like to see your response, Wei-Chiu Chuang. Thanks!

        Show
        linyiqun Yiqun Lin added a comment - Hi Wei-Chiu Chuang , I would like to attach a initial patch for this. But before that, I want to make a right understanding for this. The replication factor of an EC file is 0 from a user perspective. This affects all user visible commands and APIs, such as hdfs dfs -ls, hdfs dfs -stat, webhdfs, httpfs...etc. Does this means that we should document the replication factor of 0 for the EC file in some commands and APIs that will print the replicaion factor? I'd like to see your response, Wei-Chiu Chuang . Thanks!
        Hide
        jojochuang Wei-Chiu Chuang added a comment -

        I think the easiest approach is to add this note somewhere in the HDFS Erasure Coding doc so that admins do not panic Most tools like -ls, -stat use FileStatus which is an abstraction for all file systems. Some of the file systems do not support replication factor (e.g. S3) so this is not unexpected.

        Show
        jojochuang Wei-Chiu Chuang added a comment - I think the easiest approach is to add this note somewhere in the HDFS Erasure Coding doc so that admins do not panic Most tools like -ls, -stat use FileStatus which is an abstraction for all file systems. Some of the file systems do not support replication factor (e.g. S3) so this is not unexpected.
        Hide
        linyiqun Yiqun Lin added a comment -

        Thanks for your comments, Wei-Chiu Chuang! Attach a initial patch for this.

        Show
        linyiqun Yiqun Lin added a comment - Thanks for your comments, Wei-Chiu Chuang ! Attach a initial patch for this.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 21s Docker mode activated.
        +1 @author 0m 1s The patch does not contain any @author tags.
        -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        0 mvndep 0m 16s Maven dependency ordering for branch
        +1 mvninstall 7m 22s trunk passed
        +1 compile 7m 9s trunk passed
        +1 checkstyle 1m 27s trunk passed
        +1 mvnsite 1m 47s trunk passed
        +1 mvneclipse 0m 25s trunk passed
        +1 findbugs 2m 57s trunk passed
        +1 javadoc 1m 20s trunk passed
        0 mvndep 0m 14s Maven dependency ordering for patch
        +1 mvninstall 1m 23s the patch passed
        +1 compile 7m 9s the patch passed
        +1 javac 7m 9s the patch passed
        -0 checkstyle 1m 31s root: The patch generated 2 new + 142 unchanged - 1 fixed = 144 total (was 143)
        +1 mvnsite 2m 1s the patch passed
        +1 mvneclipse 0m 27s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 3m 22s the patch passed
        +1 javadoc 1m 30s the patch passed
        -1 unit 8m 56s hadoop-common in the patch failed.
        +1 unit 76m 25s hadoop-hdfs in the patch passed.
        +1 asflicense 0m 23s The patch does not generate ASF License warnings.
        127m 14s



        Reason Tests
        Failed junit tests hadoop.cli.TestCLI
          hadoop.ipc.TestIPC



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:9560f25
        JIRA Issue HDFS-10974
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12832263/HDFS-10974.001.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 1212131e0cc2 3.13.0-96-generic #143-Ubuntu SMP Mon Aug 29 20:15:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 4d10621
        Default Java 1.8.0_101
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17069/artifact/patchprocess/diff-checkstyle-root.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/17069/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17069/testReport/
        modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: .
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17069/console
        Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 21s Docker mode activated. +1 @author 0m 1s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. 0 mvndep 0m 16s Maven dependency ordering for branch +1 mvninstall 7m 22s trunk passed +1 compile 7m 9s trunk passed +1 checkstyle 1m 27s trunk passed +1 mvnsite 1m 47s trunk passed +1 mvneclipse 0m 25s trunk passed +1 findbugs 2m 57s trunk passed +1 javadoc 1m 20s trunk passed 0 mvndep 0m 14s Maven dependency ordering for patch +1 mvninstall 1m 23s the patch passed +1 compile 7m 9s the patch passed +1 javac 7m 9s the patch passed -0 checkstyle 1m 31s root: The patch generated 2 new + 142 unchanged - 1 fixed = 144 total (was 143) +1 mvnsite 2m 1s the patch passed +1 mvneclipse 0m 27s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 3m 22s the patch passed +1 javadoc 1m 30s the patch passed -1 unit 8m 56s hadoop-common in the patch failed. +1 unit 76m 25s hadoop-hdfs in the patch passed. +1 asflicense 0m 23s The patch does not generate ASF License warnings. 127m 14s Reason Tests Failed junit tests hadoop.cli.TestCLI   hadoop.ipc.TestIPC Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Issue HDFS-10974 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12832263/HDFS-10974.001.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 1212131e0cc2 3.13.0-96-generic #143-Ubuntu SMP Mon Aug 29 20:15:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 4d10621 Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17069/artifact/patchprocess/diff-checkstyle-root.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/17069/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17069/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: . Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17069/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        drankye Kai Zheng added a comment -

        The replication factor of an EC file is 0 from a user perspective. This affects all user visible commands and APIs, such as hdfs dfs -ls, hdfs dfs -stat, webhdfs, httpfs...etc.

        I'm a little confused. Why is it 0 as the replication factor for striped files? As discussed in HDFS-10971, the better option should be 1 instead. Could we refine the value?

        Show
        drankye Kai Zheng added a comment - The replication factor of an EC file is 0 from a user perspective. This affects all user visible commands and APIs, such as hdfs dfs -ls, hdfs dfs -stat, webhdfs, httpfs...etc. I'm a little confused. Why is it 0 as the replication factor for striped files? As discussed in HDFS-10971 , the better option should be 1 instead. Could we refine the value?
        Hide
        jojochuang Wei-Chiu Chuang added a comment -

        Kai Zheng Good catch. This jira was filed before we made that consensus. So we should hold off this patch until a new behavior is committed.

        Show
        jojochuang Wei-Chiu Chuang added a comment - Kai Zheng Good catch. This jira was filed before we made that consensus. So we should hold off this patch until a new behavior is committed.
        Hide
        linyiqun Yiqun Lin added a comment -

        Thanks Kai Zheng for the comments Attach a new patch to redefined the replication factor and fix checkstyle warnings.

        Show
        linyiqun Yiqun Lin added a comment - Thanks Kai Zheng for the comments Attach a new patch to redefined the replication factor and fix checkstyle warnings.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 19s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        0 mvndep 1m 38s Maven dependency ordering for branch
        +1 mvninstall 7m 30s trunk passed
        +1 compile 7m 16s trunk passed
        +1 checkstyle 1m 34s trunk passed
        +1 mvnsite 1m 59s trunk passed
        +1 mvneclipse 0m 26s trunk passed
        +1 findbugs 3m 23s trunk passed
        +1 javadoc 1m 24s trunk passed
        0 mvndep 0m 14s Maven dependency ordering for patch
        +1 mvninstall 1m 40s the patch passed
        +1 compile 9m 8s the patch passed
        +1 javac 9m 8s the patch passed
        +1 checkstyle 1m 33s root: The patch generated 0 new + 137 unchanged - 6 fixed = 137 total (was 143)
        +1 mvnsite 2m 18s the patch passed
        +1 mvneclipse 0m 28s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 4m 24s the patch passed
        +1 javadoc 1m 27s the patch passed
        -1 unit 7m 55s hadoop-common in the patch failed.
        -1 unit 76m 20s hadoop-hdfs in the patch failed.
        +1 asflicense 0m 24s The patch does not generate ASF License warnings.
        132m 20s



        Reason Tests
        Failed junit tests hadoop.cli.TestCLI
          hadoop.hdfs.server.namenode.TestNameNodeMetadataConsistency



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:9560f25
        JIRA Issue HDFS-10974
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12832815/HDFS-10974.002.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 8fd08d4e9156 3.13.0-96-generic #143-Ubuntu SMP Mon Aug 29 20:15:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 7ba7092
        Default Java 1.8.0_101
        findbugs v3.0.0
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/17118/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/17118/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17118/testReport/
        modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: .
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17118/console
        Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 19s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. 0 mvndep 1m 38s Maven dependency ordering for branch +1 mvninstall 7m 30s trunk passed +1 compile 7m 16s trunk passed +1 checkstyle 1m 34s trunk passed +1 mvnsite 1m 59s trunk passed +1 mvneclipse 0m 26s trunk passed +1 findbugs 3m 23s trunk passed +1 javadoc 1m 24s trunk passed 0 mvndep 0m 14s Maven dependency ordering for patch +1 mvninstall 1m 40s the patch passed +1 compile 9m 8s the patch passed +1 javac 9m 8s the patch passed +1 checkstyle 1m 33s root: The patch generated 0 new + 137 unchanged - 6 fixed = 137 total (was 143) +1 mvnsite 2m 18s the patch passed +1 mvneclipse 0m 28s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 4m 24s the patch passed +1 javadoc 1m 27s the patch passed -1 unit 7m 55s hadoop-common in the patch failed. -1 unit 76m 20s hadoop-hdfs in the patch failed. +1 asflicense 0m 24s The patch does not generate ASF License warnings. 132m 20s Reason Tests Failed junit tests hadoop.cli.TestCLI   hadoop.hdfs.server.namenode.TestNameNodeMetadataConsistency Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Issue HDFS-10974 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12832815/HDFS-10974.002.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 8fd08d4e9156 3.13.0-96-generic #143-Ubuntu SMP Mon Aug 29 20:15:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 7ba7092 Default Java 1.8.0_101 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HDFS-Build/17118/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/17118/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17118/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: . Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17118/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 0s Docker mode activated.
        -1 patch 0m 5s HDFS-10974 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help.



        Subsystem Report/Notes
        JIRA Issue HDFS-10974
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12832815/HDFS-10974.002.patch
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/18221/console
        Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. -1 patch 0m 5s HDFS-10974 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. Subsystem Report/Notes JIRA Issue HDFS-10974 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12832815/HDFS-10974.002.patch Console output https://builds.apache.org/job/PreCommit-HDFS-Build/18221/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        linyiqun Yiqun Lin added a comment -

        Hi Wei-Chiu Chuang, is this JIRA still making sense now? As I see HDFS-10971 has been resolved, is that means we can go ahead in this JIRA?
        Attach the new patch to rebase the patch. If I am not documenting accurately, Please let me know. Thanks.

        Show
        linyiqun Yiqun Lin added a comment - Hi Wei-Chiu Chuang , is this JIRA still making sense now? As I see HDFS-10971 has been resolved, is that means we can go ahead in this JIRA? Attach the new patch to rebase the patch. If I am not documenting accurately, Please let me know. Thanks.
        Hide
        jojochuang Wei-Chiu Chuang added a comment - - edited

        Hi Yiqun Lin thanks for the patch.

        A doc patch to describe operations for EC files is essential. Your patch looks good overall with a few nits:

        Now the replication factor of an EC file is regarded as 1.

        I think you can rephrase it as "Replication factor of an EC file is meaningless. It is always 1 and cannot be changed via -setrep command. "

        Also, could you also add update DistCp.md.vm and include a statement saying -pr (preserve replication factor) is only valid if both source and target directory are not erasure coded? I regret I didn't recommend updating this statement in HDFS-10971.

        Thanks!

        Show
        jojochuang Wei-Chiu Chuang added a comment - - edited Hi Yiqun Lin thanks for the patch. A doc patch to describe operations for EC files is essential. Your patch looks good overall with a few nits: Now the replication factor of an EC file is regarded as 1. I think you can rephrase it as "Replication factor of an EC file is meaningless. It is always 1 and cannot be changed via -setrep command. " Also, could you also add update DistCp.md.vm and include a statement saying -pr (preserve replication factor) is only valid if both source and target directory are not erasure coded? I regret I didn't recommend updating this statement in HDFS-10971 . Thanks!
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 14s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        0 mvndep 0m 44s Maven dependency ordering for branch
        +1 mvninstall 14m 32s trunk passed
        +1 compile 17m 49s trunk passed
        +1 checkstyle 1m 59s trunk passed
        +1 mvnsite 2m 13s trunk passed
        +1 mvneclipse 0m 40s trunk passed
        +1 findbugs 3m 57s trunk passed
        +1 javadoc 1m 48s trunk passed
        0 mvndep 0m 17s Maven dependency ordering for patch
        +1 mvninstall 1m 42s the patch passed
        +1 compile 16m 15s the patch passed
        +1 javac 16m 15s the patch passed
        +1 checkstyle 2m 10s root: The patch generated 0 new + 10 unchanged - 6 fixed = 10 total (was 16)
        +1 mvnsite 2m 25s the patch passed
        +1 mvneclipse 0m 43s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 4m 14s the patch passed
        +1 javadoc 1m 46s the patch passed
        -1 unit 7m 52s hadoop-common in the patch failed.
        -1 unit 76m 53s hadoop-hdfs in the patch failed.
        +1 asflicense 0m 54s The patch does not generate ASF License warnings.
        160m 28s



        Reason Tests
        Failed junit tests hadoop.cli.TestCLI
          hadoop.security.TestShellBasedUnixGroupsMapping
          hadoop.hdfs.server.namenode.TestNamenodeCapacityReport



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:a9ad5d6
        JIRA Issue HDFS-10974
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12861014/HDFS-10974.003.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 64ec9a2e98f2 3.13.0-107-generic #154-Ubuntu SMP Tue Dec 20 09:57:27 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 84d787b
        Default Java 1.8.0_121
        findbugs v3.0.0
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/18886/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/18886/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/18886/testReport/
        modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: .
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/18886/console
        Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 14s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. 0 mvndep 0m 44s Maven dependency ordering for branch +1 mvninstall 14m 32s trunk passed +1 compile 17m 49s trunk passed +1 checkstyle 1m 59s trunk passed +1 mvnsite 2m 13s trunk passed +1 mvneclipse 0m 40s trunk passed +1 findbugs 3m 57s trunk passed +1 javadoc 1m 48s trunk passed 0 mvndep 0m 17s Maven dependency ordering for patch +1 mvninstall 1m 42s the patch passed +1 compile 16m 15s the patch passed +1 javac 16m 15s the patch passed +1 checkstyle 2m 10s root: The patch generated 0 new + 10 unchanged - 6 fixed = 10 total (was 16) +1 mvnsite 2m 25s the patch passed +1 mvneclipse 0m 43s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 4m 14s the patch passed +1 javadoc 1m 46s the patch passed -1 unit 7m 52s hadoop-common in the patch failed. -1 unit 76m 53s hadoop-hdfs in the patch failed. +1 asflicense 0m 54s The patch does not generate ASF License warnings. 160m 28s Reason Tests Failed junit tests hadoop.cli.TestCLI   hadoop.security.TestShellBasedUnixGroupsMapping   hadoop.hdfs.server.namenode.TestNamenodeCapacityReport Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HDFS-10974 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12861014/HDFS-10974.003.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 64ec9a2e98f2 3.13.0-107-generic #154-Ubuntu SMP Tue Dec 20 09:57:27 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 84d787b Default Java 1.8.0_121 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HDFS-Build/18886/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/18886/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/18886/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: . Console output https://builds.apache.org/job/PreCommit-HDFS-Build/18886/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        linyiqun Yiqun Lin added a comment -

        Thanks Wei-Chiu Chuang for the comments. The failed test TestCLI is related.
        Attach the new patch to address the comments and fix the test failure.

        Show
        linyiqun Yiqun Lin added a comment - Thanks Wei-Chiu Chuang for the comments. The failed test TestCLI is related. Attach the new patch to address the comments and fix the test failure.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 15s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        0 mvndep 1m 42s Maven dependency ordering for branch
        +1 mvninstall 12m 55s trunk passed
        +1 compile 16m 45s trunk passed
        +1 checkstyle 1m 47s trunk passed
        +1 mvnsite 2m 16s trunk passed
        +1 mvneclipse 1m 44s trunk passed
        +1 findbugs 4m 12s trunk passed
        +1 javadoc 1m 44s trunk passed
        0 mvndep 0m 13s Maven dependency ordering for patch
        +1 mvninstall 1m 44s the patch passed
        +1 compile 13m 45s the patch passed
        +1 javac 13m 45s the patch passed
        +1 checkstyle 1m 43s root: The patch generated 0 new + 10 unchanged - 6 fixed = 10 total (was 16)
        +1 mvnsite 2m 24s the patch passed
        +1 mvneclipse 1m 6s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 xml 0m 1s The patch has no ill-formed XML file.
        +1 findbugs 3m 58s the patch passed
        +1 javadoc 2m 0s the patch passed
        +1 unit 7m 14s hadoop-common in the patch passed.
        -1 unit 61m 13s hadoop-hdfs in the patch failed.
        +1 unit 12m 22s hadoop-distcp in the patch passed.
        +1 asflicense 0m 26s The patch does not generate ASF License warnings.
        152m 51s



        Reason Tests
        Failed junit tests hadoop.hdfs.web.TestWebHdfsTimeouts
          hadoop.hdfs.server.datanode.checker.TestThrottledAsyncCheckerTimeout



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:a9ad5d6
        JIRA Issue HDFS-10974
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12861053/HDFS-10974.004.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
        uname Linux 2a3c26468ccb 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12 13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 84d787b
        Default Java 1.8.0_121
        findbugs v3.0.0
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/18890/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/18890/testReport/
        modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs hadoop-tools/hadoop-distcp U: .
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/18890/console
        Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 15s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 1m 42s Maven dependency ordering for branch +1 mvninstall 12m 55s trunk passed +1 compile 16m 45s trunk passed +1 checkstyle 1m 47s trunk passed +1 mvnsite 2m 16s trunk passed +1 mvneclipse 1m 44s trunk passed +1 findbugs 4m 12s trunk passed +1 javadoc 1m 44s trunk passed 0 mvndep 0m 13s Maven dependency ordering for patch +1 mvninstall 1m 44s the patch passed +1 compile 13m 45s the patch passed +1 javac 13m 45s the patch passed +1 checkstyle 1m 43s root: The patch generated 0 new + 10 unchanged - 6 fixed = 10 total (was 16) +1 mvnsite 2m 24s the patch passed +1 mvneclipse 1m 6s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. +1 findbugs 3m 58s the patch passed +1 javadoc 2m 0s the patch passed +1 unit 7m 14s hadoop-common in the patch passed. -1 unit 61m 13s hadoop-hdfs in the patch failed. +1 unit 12m 22s hadoop-distcp in the patch passed. +1 asflicense 0m 26s The patch does not generate ASF License warnings. 152m 51s Reason Tests Failed junit tests hadoop.hdfs.web.TestWebHdfsTimeouts   hadoop.hdfs.server.datanode.checker.TestThrottledAsyncCheckerTimeout Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HDFS-10974 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12861053/HDFS-10974.004.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux 2a3c26468ccb 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12 13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 84d787b Default Java 1.8.0_121 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HDFS-Build/18890/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/18890/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs hadoop-tools/hadoop-distcp U: . Console output https://builds.apache.org/job/PreCommit-HDFS-Build/18890/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        jojochuang Wei-Chiu Chuang added a comment -

        Hello Yiqun Lin thanks for incorporating my comments in the new patch.

        I think the patch is good. +1

        Looking at code again, I think we can improve the handling of EC files better:

        • In SetReplication#processPath, can we check if the file is EC before invoking setReplication() on it?
        • -setrep also has a command switch -w to wait for the file to be replicated. Can we ignore this if the file is erasure coded?

        Let's file a new jira as these improvements is out of scope of the original intent.

        Show
        jojochuang Wei-Chiu Chuang added a comment - Hello Yiqun Lin thanks for incorporating my comments in the new patch. I think the patch is good. +1 Looking at code again, I think we can improve the handling of EC files better: In SetReplication#processPath , can we check if the file is EC before invoking setReplication() on it? -setrep also has a command switch -w to wait for the file to be replicated. Can we ignore this if the file is erasure coded? Let's file a new jira as these improvements is out of scope of the original intent.
        Hide
        jojochuang Wei-Chiu Chuang added a comment -

        Committed the patch 004 to trunk.

        Thanks very much Yiqun Lin for the patch and Kai Zheng for precious comments!

        Show
        jojochuang Wei-Chiu Chuang added a comment - Committed the patch 004 to trunk. Thanks very much Yiqun Lin for the patch and Kai Zheng for precious comments!
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11500 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11500/)
        HDFS-10974. Document replication factor for EC files. Contributed by (weichiu: rev 8c591b8d199325f49b5bba29240ca25610cf80a0)

        • (edit) hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HDFSErasureCoding.md
        • (edit) hadoop-common-project/hadoop-common/src/test/resources/testConf.xml
        • (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/SetReplication.java
        • (edit) hadoop-tools/hadoop-distcp/src/site/markdown/DistCp.md.vm
        • (edit) hadoop-common-project/hadoop-common/src/site/markdown/FileSystemShell.md
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11500 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11500/ ) HDFS-10974 . Document replication factor for EC files. Contributed by (weichiu: rev 8c591b8d199325f49b5bba29240ca25610cf80a0) (edit) hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HDFSErasureCoding.md (edit) hadoop-common-project/hadoop-common/src/test/resources/testConf.xml (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/SetReplication.java (edit) hadoop-tools/hadoop-distcp/src/site/markdown/DistCp.md.vm (edit) hadoop-common-project/hadoop-common/src/site/markdown/FileSystemShell.md
        Hide
        linyiqun Yiqun Lin added a comment -

        Thanks a lot for the review and commit, Wei-Chiu Chuang!

        Looking at code again, I think we can improve the handling of EC files better:
        ...

        It's a good proposal! I will take some time to work for this improvement these days.

        Show
        linyiqun Yiqun Lin added a comment - Thanks a lot for the review and commit, Wei-Chiu Chuang ! Looking at code again, I think we can improve the handling of EC files better: ... It's a good proposal! I will take some time to work for this improvement these days.

          People

          • Assignee:
            linyiqun Yiqun Lin
            Reporter:
            jojochuang Wei-Chiu Chuang
          • Votes:
            0 Vote for this issue
            Watchers:
            11 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development