Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-11416

Refactor out system default erasure coding policy

    Details

      Description

      As discussed on HDFS-7859, the system default EC policy is mostly a relic from development when the system only supported a single global policy. Now, we support multiple policies, and the system default policy is mostly used by tests.

      We should refactor to remove this concept.

      1. HDFS-11416.001.patch
        9 kB
        Andrew Wang
      2. HDFS-11416.002.patch
        10 kB
        Andrew Wang
      3. HDFS-11416.003.patch
        10 kB
        Andrew Wang
      4. HDFS-11416.004.patch
        10 kB
        Andrew Wang
      5. HDFS-11416.005.patch
        10 kB
        Andrew Wang
      6. HDFS-11416.006.patch
        11 kB
        Rakesh R

        Activity

        Hide
        andrew.wang Andrew Wang added a comment -

        Thanks for the review and commit Rakesh, thanks also Wei-chiu for reviews!

        Show
        andrew.wang Andrew Wang added a comment - Thanks for the review and commit Rakesh, thanks also Wei-chiu for reviews!
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11336 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11336/)
        HDFS-11416. Refactor out system default erasure coding policy. (rakeshr: rev 3749152b661d0359b3b941ab1d17177230f3b8dc)

        • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogLoader.java
        • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NamenodeFsck.java
        • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFile.java
        • (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/StringUtils.java
        • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirErasureCodingOp.java
        • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestStripedINodeFile.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11336 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11336/ ) HDFS-11416 . Refactor out system default erasure coding policy. (rakeshr: rev 3749152b661d0359b3b941ab1d17177230f3b8dc) (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogLoader.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NamenodeFsck.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFile.java (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/StringUtils.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirErasureCodingOp.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestStripedINodeFile.java
        Hide
        rakeshr Rakesh R added a comment -

        Also, attaching the committed patch, where I've fixed one checkstyle warning - removed unused import org.apache.hadoop.test.GenericTestUtils.

        Show
        rakeshr Rakesh R added a comment - Also, attaching the committed patch, where I've fixed one checkstyle warning - removed unused import org.apache.hadoop.test.GenericTestUtils .
        Hide
        rakeshr Rakesh R added a comment -

        Thanks Andrew Wang for the contribution. Thanks Wei-Chiu Chuang for the reviews.

        Committed to trunk.

        Show
        rakeshr Rakesh R added a comment - Thanks Andrew Wang for the contribution. Thanks Wei-Chiu Chuang for the reviews. Committed to trunk.
        Hide
        rakeshr Rakesh R added a comment - - edited

        Thanks Andrew Wang for the updates. +1 latest patch looks good to me. I'll commit this shortly.

        Note: Compile errors are due to yarn_ui changes and not related to this patch

        /testptch/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/target/src/main/webapp/node_modules/merge-trees/index.js:33
        class MergeTrees {
        ^^^^^
        Unexpected reserved word
        SyntaxError: Unexpected reserved word
        
        Show
        rakeshr Rakesh R added a comment - - edited Thanks Andrew Wang for the updates. +1 latest patch looks good to me. I'll commit this shortly. Note: Compile errors are due to yarn_ui changes and not related to this patch /testptch/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/target/src/main/webapp/node_modules/merge-trees/index.js:33 class MergeTrees { ^^^^^ Unexpected reserved word SyntaxError: Unexpected reserved word
        Hide
        andrew.wang Andrew Wang added a comment -

        I have an unused import I think can be removed at commit time, build failures are in javascript?

        Show
        andrew.wang Andrew Wang added a comment - I have an unused import I think can be removed at commit time, build failures are in javascript?
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 21s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        0 mvndep 0m 19s Maven dependency ordering for branch
        +1 mvninstall 13m 45s trunk passed
        -1 compile 11m 50s root in trunk failed.
        +1 checkstyle 2m 18s trunk passed
        +1 mvnsite 2m 25s trunk passed
        +1 mvneclipse 0m 36s trunk passed
        +1 findbugs 3m 18s trunk passed
        +1 javadoc 1m 34s trunk passed
        0 mvndep 0m 16s Maven dependency ordering for patch
        +1 mvninstall 1m 40s the patch passed
        -1 compile 10m 2s root in the patch failed.
        -1 javac 10m 2s root in the patch failed.
        -0 checkstyle 2m 17s root: The patch generated 1 new + 298 unchanged - 1 fixed = 299 total (was 299)
        +1 mvnsite 2m 21s the patch passed
        +1 mvneclipse 0m 43s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 4m 3s the patch passed
        +1 javadoc 1m 31s the patch passed
        +1 unit 8m 14s hadoop-common in the patch passed.
        -1 unit 97m 40s hadoop-hdfs in the patch failed.
        +1 asflicense 0m 37s The patch does not generate ASF License warnings.
        167m 0s



        Reason Tests
        Failed junit tests hadoop.hdfs.TestMaintenanceState
          hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:a9ad5d6
        JIRA Issue HDFS-11416
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12855679/HDFS-11416.005.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux aef40284fdcf 3.13.0-108-generic #155-Ubuntu SMP Wed Jan 11 16:58:52 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 747bafa
        Default Java 1.8.0_121
        compile https://builds.apache.org/job/PreCommit-HDFS-Build/18517/artifact/patchprocess/branch-compile-root.txt
        findbugs v3.0.0
        compile https://builds.apache.org/job/PreCommit-HDFS-Build/18517/artifact/patchprocess/patch-compile-root.txt
        javac https://builds.apache.org/job/PreCommit-HDFS-Build/18517/artifact/patchprocess/patch-compile-root.txt
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/18517/artifact/patchprocess/diff-checkstyle-root.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/18517/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/18517/testReport/
        modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: .
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/18517/console
        Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 21s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 0m 19s Maven dependency ordering for branch +1 mvninstall 13m 45s trunk passed -1 compile 11m 50s root in trunk failed. +1 checkstyle 2m 18s trunk passed +1 mvnsite 2m 25s trunk passed +1 mvneclipse 0m 36s trunk passed +1 findbugs 3m 18s trunk passed +1 javadoc 1m 34s trunk passed 0 mvndep 0m 16s Maven dependency ordering for patch +1 mvninstall 1m 40s the patch passed -1 compile 10m 2s root in the patch failed. -1 javac 10m 2s root in the patch failed. -0 checkstyle 2m 17s root: The patch generated 1 new + 298 unchanged - 1 fixed = 299 total (was 299) +1 mvnsite 2m 21s the patch passed +1 mvneclipse 0m 43s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 4m 3s the patch passed +1 javadoc 1m 31s the patch passed +1 unit 8m 14s hadoop-common in the patch passed. -1 unit 97m 40s hadoop-hdfs in the patch failed. +1 asflicense 0m 37s The patch does not generate ASF License warnings. 167m 0s Reason Tests Failed junit tests hadoop.hdfs.TestMaintenanceState   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HDFS-11416 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12855679/HDFS-11416.005.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux aef40284fdcf 3.13.0-108-generic #155-Ubuntu SMP Wed Jan 11 16:58:52 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 747bafa Default Java 1.8.0_121 compile https://builds.apache.org/job/PreCommit-HDFS-Build/18517/artifact/patchprocess/branch-compile-root.txt findbugs v3.0.0 compile https://builds.apache.org/job/PreCommit-HDFS-Build/18517/artifact/patchprocess/patch-compile-root.txt javac https://builds.apache.org/job/PreCommit-HDFS-Build/18517/artifact/patchprocess/patch-compile-root.txt checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/18517/artifact/patchprocess/diff-checkstyle-root.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/18517/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/18517/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: . Console output https://builds.apache.org/job/PreCommit-HDFS-Build/18517/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        andrew.wang Andrew Wang added a comment -

        Thanks for reviewing Rakesh, new patch attached to address your review comments.

        Show
        andrew.wang Andrew Wang added a comment - Thanks for reviewing Rakesh, new patch attached to address your review comments.
        Hide
        rakeshr Rakesh R added a comment -

        Thanks Andrew Wang for the patch. I've few minor comments,

        1. Please add javadoc for the newly added fnction StringUtils#byteToHexString(byte b)
        2. Please include fail statement for the test case TestStripedINodeFile#testInvalidECPolicy
             try{
                //.....
                //.....
                fail("Should throw IllegalArgumentException on invalid ec policy!");
              } catch (IllegalArgumentException e) {
                GenericTestUtils.assertExceptionContains("Could not find EC policy with" +
                    " ID 0xbb", e);
              }
          
        Show
        rakeshr Rakesh R added a comment - Thanks Andrew Wang for the patch. I've few minor comments, Please add javadoc for the newly added fnction StringUtils#byteToHexString(byte b) Please include fail statement for the test case TestStripedINodeFile#testInvalidECPolicy try { //..... //..... fail( "Should throw IllegalArgumentException on invalid ec policy!" ); } catch (IllegalArgumentException e) { GenericTestUtils.assertExceptionContains( "Could not find EC policy with" + " ID 0xbb" , e); }
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 25s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        0 mvndep 0m 15s Maven dependency ordering for branch
        +1 mvninstall 13m 29s trunk passed
        +1 compile 14m 10s trunk passed
        +1 checkstyle 2m 5s trunk passed
        +1 mvnsite 2m 7s trunk passed
        +1 mvneclipse 0m 40s trunk passed
        +1 findbugs 3m 36s trunk passed
        +1 javadoc 1m 47s trunk passed
        0 mvndep 0m 16s Maven dependency ordering for patch
        +1 mvninstall 1m 40s the patch passed
        +1 compile 13m 29s the patch passed
        +1 javac 13m 29s the patch passed
        +1 checkstyle 2m 14s root: The patch generated 0 new + 298 unchanged - 1 fixed = 298 total (was 299)
        +1 mvnsite 2m 29s the patch passed
        +1 mvneclipse 0m 39s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 4m 36s the patch passed
        +1 javadoc 1m 50s the patch passed
        -1 unit 9m 11s hadoop-common in the patch failed.
        -1 unit 80m 39s hadoop-hdfs in the patch failed.
        +1 asflicense 0m 37s The patch does not generate ASF License warnings.
        157m 33s



        Reason Tests
        Failed junit tests hadoop.fs.sftp.TestSFTPFileSystem
          hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting
        Timed out junit tests org.apache.hadoop.hdfs.server.blockmanagement.TestBlockStatsMXBean



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:a9ad5d6
        JIRA Issue HDFS-11416
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12855556/HDFS-11416.004.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux e12d92941186 3.13.0-105-generic #152-Ubuntu SMP Fri Dec 2 15:37:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 4e14ead
        Default Java 1.8.0_121
        findbugs v3.0.0
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/18511/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/18511/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/18511/testReport/
        modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: .
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/18511/console
        Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 25s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 0m 15s Maven dependency ordering for branch +1 mvninstall 13m 29s trunk passed +1 compile 14m 10s trunk passed +1 checkstyle 2m 5s trunk passed +1 mvnsite 2m 7s trunk passed +1 mvneclipse 0m 40s trunk passed +1 findbugs 3m 36s trunk passed +1 javadoc 1m 47s trunk passed 0 mvndep 0m 16s Maven dependency ordering for patch +1 mvninstall 1m 40s the patch passed +1 compile 13m 29s the patch passed +1 javac 13m 29s the patch passed +1 checkstyle 2m 14s root: The patch generated 0 new + 298 unchanged - 1 fixed = 298 total (was 299) +1 mvnsite 2m 29s the patch passed +1 mvneclipse 0m 39s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 4m 36s the patch passed +1 javadoc 1m 50s the patch passed -1 unit 9m 11s hadoop-common in the patch failed. -1 unit 80m 39s hadoop-hdfs in the patch failed. +1 asflicense 0m 37s The patch does not generate ASF License warnings. 157m 33s Reason Tests Failed junit tests hadoop.fs.sftp.TestSFTPFileSystem   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting Timed out junit tests org.apache.hadoop.hdfs.server.blockmanagement.TestBlockStatsMXBean Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HDFS-11416 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12855556/HDFS-11416.004.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux e12d92941186 3.13.0-105-generic #152-Ubuntu SMP Fri Dec 2 15:37:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 4e14ead Default Java 1.8.0_121 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HDFS-Build/18511/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/18511/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/18511/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: . Console output https://builds.apache.org/job/PreCommit-HDFS-Build/18511/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        andrew.wang Andrew Wang added a comment -

        Missed updating a different test case in that test class.

        Show
        andrew.wang Andrew Wang added a comment - Missed updating a different test case in that test class.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 11s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        0 mvndep 1m 52s Maven dependency ordering for branch
        +1 mvninstall 13m 59s trunk passed
        +1 compile 14m 41s trunk passed
        +1 checkstyle 2m 1s trunk passed
        +1 mvnsite 2m 15s trunk passed
        +1 mvneclipse 0m 37s trunk passed
        +1 findbugs 3m 38s trunk passed
        +1 javadoc 1m 37s trunk passed
        0 mvndep 0m 14s Maven dependency ordering for patch
        +1 mvninstall 1m 36s the patch passed
        +1 compile 13m 6s the patch passed
        +1 javac 13m 6s the patch passed
        +1 checkstyle 2m 13s root: The patch generated 0 new + 298 unchanged - 1 fixed = 298 total (was 299)
        +1 mvnsite 2m 16s the patch passed
        +1 mvneclipse 0m 40s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 4m 20s the patch passed
        +1 javadoc 1m 45s the patch passed
        -1 unit 8m 17s hadoop-common in the patch failed.
        -1 unit 77m 4s hadoop-hdfs in the patch failed.
        +1 asflicense 0m 43s The patch does not generate ASF License warnings.
        154m 29s



        Reason Tests
        Failed junit tests hadoop.ipc.TestRPCWaitForProxy
          hadoop.hdfs.TestMaintenanceState
          hadoop.hdfs.server.namenode.TestStripedINodeFile
        Timed out junit tests org.apache.hadoop.hdfs.server.blockmanagement.TestBlockStatsMXBean



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:a9ad5d6
        JIRA Issue HDFS-11416
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12855521/HDFS-11416.003.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 0024e22bd652 3.13.0-107-generic #154-Ubuntu SMP Tue Dec 20 09:57:27 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 6f6dfe0
        Default Java 1.8.0_121
        findbugs v3.0.0
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/18507/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/18507/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/18507/testReport/
        modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: .
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/18507/console
        Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 11s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 1m 52s Maven dependency ordering for branch +1 mvninstall 13m 59s trunk passed +1 compile 14m 41s trunk passed +1 checkstyle 2m 1s trunk passed +1 mvnsite 2m 15s trunk passed +1 mvneclipse 0m 37s trunk passed +1 findbugs 3m 38s trunk passed +1 javadoc 1m 37s trunk passed 0 mvndep 0m 14s Maven dependency ordering for patch +1 mvninstall 1m 36s the patch passed +1 compile 13m 6s the patch passed +1 javac 13m 6s the patch passed +1 checkstyle 2m 13s root: The patch generated 0 new + 298 unchanged - 1 fixed = 298 total (was 299) +1 mvnsite 2m 16s the patch passed +1 mvneclipse 0m 40s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 4m 20s the patch passed +1 javadoc 1m 45s the patch passed -1 unit 8m 17s hadoop-common in the patch failed. -1 unit 77m 4s hadoop-hdfs in the patch failed. +1 asflicense 0m 43s The patch does not generate ASF License warnings. 154m 29s Reason Tests Failed junit tests hadoop.ipc.TestRPCWaitForProxy   hadoop.hdfs.TestMaintenanceState   hadoop.hdfs.server.namenode.TestStripedINodeFile Timed out junit tests org.apache.hadoop.hdfs.server.blockmanagement.TestBlockStatsMXBean Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HDFS-11416 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12855521/HDFS-11416.003.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 0024e22bd652 3.13.0-107-generic #154-Ubuntu SMP Tue Dec 20 09:57:27 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 6f6dfe0 Default Java 1.8.0_121 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HDFS-Build/18507/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/18507/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/18507/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: . Console output https://builds.apache.org/job/PreCommit-HDFS-Build/18507/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        andrew.wang Andrew Wang added a comment -

        Tiny rev to revert the change of Preconditions.checkArgument to checkNotNull, which caused the failing TestStripedINodeFile test.

        Show
        andrew.wang Andrew Wang added a comment - Tiny rev to revert the change of Preconditions.checkArgument to checkNotNull, which caused the failing TestStripedINodeFile test.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 13s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        0 mvndep 1m 58s Maven dependency ordering for branch
        +1 mvninstall 13m 39s trunk passed
        +1 compile 14m 42s trunk passed
        +1 checkstyle 2m 0s trunk passed
        +1 mvnsite 2m 17s trunk passed
        +1 mvneclipse 0m 35s trunk passed
        +1 findbugs 3m 53s trunk passed
        +1 javadoc 1m 38s trunk passed
        0 mvndep 0m 16s Maven dependency ordering for patch
        +1 mvninstall 1m 43s the patch passed
        +1 compile 12m 7s the patch passed
        +1 javac 12m 7s the patch passed
        +1 checkstyle 1m 58s root: The patch generated 0 new + 298 unchanged - 1 fixed = 298 total (was 299)
        +1 mvnsite 2m 3s the patch passed
        +1 mvneclipse 0m 36s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 3m 37s the patch passed
        +1 javadoc 1m 34s the patch passed
        -1 unit 7m 58s hadoop-common in the patch failed.
        -1 unit 65m 54s hadoop-hdfs in the patch failed.
        +1 asflicense 0m 35s The patch does not generate ASF License warnings.
        140m 29s



        Reason Tests
        Failed junit tests hadoop.ha.TestZKFailoverController
          hadoop.hdfs.TestDFSClientRetries
          hadoop.hdfs.server.namenode.TestStripedINodeFile



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:a9ad5d6
        JIRA Issue HDFS-11416
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12855468/HDFS-11416.002.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux e536d4de17ce 3.13.0-107-generic #154-Ubuntu SMP Tue Dec 20 09:57:27 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 899d5c4
        Default Java 1.8.0_121
        findbugs v3.0.0
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/18498/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/18498/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/18498/testReport/
        modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: .
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/18498/console
        Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 13s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 1m 58s Maven dependency ordering for branch +1 mvninstall 13m 39s trunk passed +1 compile 14m 42s trunk passed +1 checkstyle 2m 0s trunk passed +1 mvnsite 2m 17s trunk passed +1 mvneclipse 0m 35s trunk passed +1 findbugs 3m 53s trunk passed +1 javadoc 1m 38s trunk passed 0 mvndep 0m 16s Maven dependency ordering for patch +1 mvninstall 1m 43s the patch passed +1 compile 12m 7s the patch passed +1 javac 12m 7s the patch passed +1 checkstyle 1m 58s root: The patch generated 0 new + 298 unchanged - 1 fixed = 298 total (was 299) +1 mvnsite 2m 3s the patch passed +1 mvneclipse 0m 36s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 3m 37s the patch passed +1 javadoc 1m 34s the patch passed -1 unit 7m 58s hadoop-common in the patch failed. -1 unit 65m 54s hadoop-hdfs in the patch failed. +1 asflicense 0m 35s The patch does not generate ASF License warnings. 140m 29s Reason Tests Failed junit tests hadoop.ha.TestZKFailoverController   hadoop.hdfs.TestDFSClientRetries   hadoop.hdfs.server.namenode.TestStripedINodeFile Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HDFS-11416 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12855468/HDFS-11416.002.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux e536d4de17ce 3.13.0-107-generic #154-Ubuntu SMP Tue Dec 20 09:57:27 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 899d5c4 Default Java 1.8.0_121 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HDFS-Build/18498/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/18498/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/18498/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: . Console output https://builds.apache.org/job/PreCommit-HDFS-Build/18498/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        andrew.wang Andrew Wang added a comment -

        I think this needed a slight rebase after HDFS-11382, attached.

        Show
        andrew.wang Andrew Wang added a comment - I think this needed a slight rebase after HDFS-11382 , attached.
        Hide
        jojochuang Wei-Chiu Chuang added a comment -

        This patch still applies to trunk with no conflicts after HDFS-11428, so I retriggered jenkins to see how it goes.

        Show
        jojochuang Wei-Chiu Chuang added a comment - This patch still applies to trunk with no conflicts after HDFS-11428 , so I retriggered jenkins to see how it goes.
        Hide
        jojochuang Wei-Chiu Chuang added a comment -

        The intent looks reasonable to me. I quickly reviewed the patch and it looks fine to me. Let's review this patch after HDFS-11428 is in. Thanks!

        Show
        jojochuang Wei-Chiu Chuang added a comment - The intent looks reasonable to me. I quickly reviewed the patch and it looks fine to me. Let's review this patch after HDFS-11428 is in. Thanks!
        Hide
        andrew.wang Andrew Wang added a comment -

        Forgot that this depends on HDFS-11428, which is why all the unit tests failed. Need to get that in first, though I think this is still ready for review.

        Show
        andrew.wang Andrew Wang added a comment - Forgot that this depends on HDFS-11428 , which is why all the unit tests failed. Need to get that in first, though I think this is still ready for review.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 22s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        0 mvndep 0m 14s Maven dependency ordering for branch
        +1 mvninstall 12m 46s trunk passed
        +1 compile 12m 59s trunk passed
        +1 checkstyle 1m 51s trunk passed
        +1 mvnsite 1m 59s trunk passed
        +1 mvneclipse 0m 38s trunk passed
        +1 findbugs 3m 14s trunk passed
        +1 javadoc 1m 33s trunk passed
        0 mvndep 0m 14s Maven dependency ordering for patch
        +1 mvninstall 1m 24s the patch passed
        +1 compile 10m 32s the patch passed
        +1 javac 10m 32s the patch passed
        +1 checkstyle 1m 52s the patch passed
        +1 mvnsite 1m 56s the patch passed
        +1 mvneclipse 0m 37s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 3m 31s the patch passed
        +1 javadoc 1m 32s the patch passed
        -1 unit 8m 15s hadoop-common in the patch failed.
        -1 unit 98m 55s hadoop-hdfs in the patch failed.
        +1 asflicense 0m 38s The patch does not generate ASF License warnings.
        166m 19s



        Reason Tests
        Failed junit tests hadoop.net.TestDNS
          hadoop.hdfs.server.mover.TestMover
          hadoop.hdfs.TestDecommissionWithStriped
          hadoop.hdfs.TestReadStripedFileWithMissingBlocks
          hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewerWithStripedBlocks
          hadoop.hdfs.server.namenode.TestAddStripedBlockInFBR
          hadoop.hdfs.server.namenode.TestFsck
          hadoop.hdfs.server.namenode.TestStripedINodeFile
          hadoop.hdfs.TestReadStripedFileWithDecoding
          hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped
          hadoop.hdfs.server.namenode.TestNameNodeMXBean
          hadoop.hdfs.server.namenode.TestReconstructStripedBlocks
          hadoop.hdfs.server.blockmanagement.TestReconstructStripedBlocksWithRackAwareness
          hadoop.hdfs.server.balancer.TestBalancer
          hadoop.hdfs.TestSafeModeWithStripedFile
          hadoop.hdfs.TestWriteReadStripedFile
          hadoop.hdfs.TestReconstructStripedFile
          hadoop.hdfs.server.namenode.TestAddStripedBlocks
          hadoop.hdfs.server.datanode.TestDataNodeErasureCodingMetrics
          hadoop.hdfs.TestErasureCodingPolicies
          hadoop.hdfs.server.blockmanagement.TestSequentialBlockGroupId
          hadoop.hdfs.server.namenode.TestAddOverReplicatedStripedBlocks
          hadoop.hdfs.TestFileChecksum
        Timed out junit tests org.apache.hadoop.hdfs.TestLeaseRecovery2



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:a9ad5d6
        JIRA Issue HDFS-11416
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12854561/HDFS-11416.001.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 984e96e3dfa1 3.13.0-107-generic #154-Ubuntu SMP Tue Dec 20 09:57:27 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 289bc50
        Default Java 1.8.0_121
        findbugs v3.0.0
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/18440/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/18440/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/18440/testReport/
        modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: .
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/18440/console
        Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 22s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 0m 14s Maven dependency ordering for branch +1 mvninstall 12m 46s trunk passed +1 compile 12m 59s trunk passed +1 checkstyle 1m 51s trunk passed +1 mvnsite 1m 59s trunk passed +1 mvneclipse 0m 38s trunk passed +1 findbugs 3m 14s trunk passed +1 javadoc 1m 33s trunk passed 0 mvndep 0m 14s Maven dependency ordering for patch +1 mvninstall 1m 24s the patch passed +1 compile 10m 32s the patch passed +1 javac 10m 32s the patch passed +1 checkstyle 1m 52s the patch passed +1 mvnsite 1m 56s the patch passed +1 mvneclipse 0m 37s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 3m 31s the patch passed +1 javadoc 1m 32s the patch passed -1 unit 8m 15s hadoop-common in the patch failed. -1 unit 98m 55s hadoop-hdfs in the patch failed. +1 asflicense 0m 38s The patch does not generate ASF License warnings. 166m 19s Reason Tests Failed junit tests hadoop.net.TestDNS   hadoop.hdfs.server.mover.TestMover   hadoop.hdfs.TestDecommissionWithStriped   hadoop.hdfs.TestReadStripedFileWithMissingBlocks   hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewerWithStripedBlocks   hadoop.hdfs.server.namenode.TestAddStripedBlockInFBR   hadoop.hdfs.server.namenode.TestFsck   hadoop.hdfs.server.namenode.TestStripedINodeFile   hadoop.hdfs.TestReadStripedFileWithDecoding   hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped   hadoop.hdfs.server.namenode.TestNameNodeMXBean   hadoop.hdfs.server.namenode.TestReconstructStripedBlocks   hadoop.hdfs.server.blockmanagement.TestReconstructStripedBlocksWithRackAwareness   hadoop.hdfs.server.balancer.TestBalancer   hadoop.hdfs.TestSafeModeWithStripedFile   hadoop.hdfs.TestWriteReadStripedFile   hadoop.hdfs.TestReconstructStripedFile   hadoop.hdfs.server.namenode.TestAddStripedBlocks   hadoop.hdfs.server.datanode.TestDataNodeErasureCodingMetrics   hadoop.hdfs.TestErasureCodingPolicies   hadoop.hdfs.server.blockmanagement.TestSequentialBlockGroupId   hadoop.hdfs.server.namenode.TestAddOverReplicatedStripedBlocks   hadoop.hdfs.TestFileChecksum Timed out junit tests org.apache.hadoop.hdfs.TestLeaseRecovery2 Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HDFS-11416 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12854561/HDFS-11416.001.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 984e96e3dfa1 3.13.0-107-generic #154-Ubuntu SMP Tue Dec 20 09:57:27 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 289bc50 Default Java 1.8.0_121 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HDFS-Build/18440/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/18440/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/18440/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: . Console output https://builds.apache.org/job/PreCommit-HDFS-Build/18440/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        andrew.wang Andrew Wang added a comment -

        Patch attached. This just removes production usages of getSystemDefaultPolicy. I'll file another JIRA to move this function to somewhere like DFSTestUtil which will also update all the references in test code.

        Show
        andrew.wang Andrew Wang added a comment - Patch attached. This just removes production usages of getSystemDefaultPolicy. I'll file another JIRA to move this function to somewhere like DFSTestUtil which will also update all the references in test code.

          People

          • Assignee:
            andrew.wang Andrew Wang
            Reporter:
            andrew.wang Andrew Wang
          • Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development