Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.6.0
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: None
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      Currently considerLoad avoids choosing nodes that are too active, so it helps level the HDFS load across the cluster. Under normal conditions, this is desired. However, when a cluster has a large percentage of nearly full nodes, this can make it difficult to find good targets because the placement policy wants to avoid the full nodes, but considerLoad wants to avoid the busy less-full nodes.

      1. HDFS-9318-v2.patch
        7 kB
        Kuhu Shukla
      2. HDFS-9318-v1.patch
        7 kB
        Kuhu Shukla

        Activity

        Hide
        hudson Hudson added a comment -

        ABORTED: Integrated in Hadoop-Hdfs-trunk-Java8 #579 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/579/)
        HDFS-9318. considerLoad factor can be improved. Contributed by Kuhu (kihwal: rev bf6aa30a156b3c5cac5469014a5989e0dfdc7256)

        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicyConsiderLoad.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
        Show
        hudson Hudson added a comment - ABORTED: Integrated in Hadoop-Hdfs-trunk-Java8 #579 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/579/ ) HDFS-9318 . considerLoad factor can be improved. Contributed by Kuhu (kihwal: rev bf6aa30a156b3c5cac5469014a5989e0dfdc7256) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicyConsiderLoad.java hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Hdfs-trunk #2518 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2518/)
        HDFS-9318. considerLoad factor can be improved. Contributed by Kuhu (kihwal: rev bf6aa30a156b3c5cac5469014a5989e0dfdc7256)

        • hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java
        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicyConsiderLoad.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #2518 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2518/ ) HDFS-9318 . considerLoad factor can be improved. Contributed by Kuhu (kihwal: rev bf6aa30a156b3c5cac5469014a5989e0dfdc7256) hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicyConsiderLoad.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Mapreduce-trunk #2578 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2578/)
        HDFS-9318. considerLoad factor can be improved. Contributed by Kuhu (kihwal: rev bf6aa30a156b3c5cac5469014a5989e0dfdc7256)

        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicyConsiderLoad.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java
        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #2578 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2578/ ) HDFS-9318 . considerLoad factor can be improved. Contributed by Kuhu (kihwal: rev bf6aa30a156b3c5cac5469014a5989e0dfdc7256) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicyConsiderLoad.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #638 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/638/)
        HDFS-9318. considerLoad factor can be improved. Contributed by Kuhu (kihwal: rev bf6aa30a156b3c5cac5469014a5989e0dfdc7256)

        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicyConsiderLoad.java
        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #638 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/638/ ) HDFS-9318 . considerLoad factor can be improved. Contributed by Kuhu (kihwal: rev bf6aa30a156b3c5cac5469014a5989e0dfdc7256) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicyConsiderLoad.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Yarn-trunk #1371 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1371/)
        HDFS-9318. considerLoad factor can be improved. Contributed by Kuhu (kihwal: rev bf6aa30a156b3c5cac5469014a5989e0dfdc7256)

        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicyConsiderLoad.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #1371 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1371/ ) HDFS-9318 . considerLoad factor can be improved. Contributed by Kuhu (kihwal: rev bf6aa30a156b3c5cac5469014a5989e0dfdc7256) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicyConsiderLoad.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #648 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/648/)
        HDFS-9318. considerLoad factor can be improved. Contributed by Kuhu (kihwal: rev bf6aa30a156b3c5cac5469014a5989e0dfdc7256)

        • hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java
        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicyConsiderLoad.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #648 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/648/ ) HDFS-9318 . considerLoad factor can be improved. Contributed by Kuhu (kihwal: rev bf6aa30a156b3c5cac5469014a5989e0dfdc7256) hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicyConsiderLoad.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-trunk-Commit #8769 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8769/)
        HDFS-9318. considerLoad factor can be improved. Contributed by Kuhu (kihwal: rev bf6aa30a156b3c5cac5469014a5989e0dfdc7256)

        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicyConsiderLoad.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #8769 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8769/ ) HDFS-9318 . considerLoad factor can be improved. Contributed by Kuhu (kihwal: rev bf6aa30a156b3c5cac5469014a5989e0dfdc7256) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicyConsiderLoad.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
        Hide
        kihwal Kihwal Lee added a comment -

        Thanks for working on this, Kuhu.

        Show
        kihwal Kihwal Lee added a comment - Thanks for working on this, Kuhu.
        Hide
        kihwal Kihwal Lee added a comment -

        +1 lgtm

        Show
        kihwal Kihwal Lee added a comment - +1 lgtm
        Hide
        kshukla Kuhu Shukla added a comment -

        The test failures are not seen when run locally and the reason for failure does not relate to this change. The ASF warnings are coming from testptch directory, again unrelated to the change AFAIK. The checkstyle for adding getter and setter may be an overkill considering the value comes from a config and used only in that one file. Requesting Kihwal Lee to review and share comments. Thank you.

        Show
        kshukla Kuhu Shukla added a comment - The test failures are not seen when run locally and the reason for failure does not relate to this change. The ASF warnings are coming from testptch directory, again unrelated to the change AFAIK. The checkstyle for adding getter and setter may be an overkill considering the value comes from a config and used only in that one file. Requesting Kihwal Lee to review and share comments. Thank you.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 5s docker + precommit patch detected.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        +1 mvninstall 3m 15s trunk passed
        +1 compile 0m 34s trunk passed with JDK v1.8.0_60
        +1 compile 0m 32s trunk passed with JDK v1.7.0_79
        +1 checkstyle 0m 17s trunk passed
        +1 mvneclipse 0m 14s trunk passed
        -1 findbugs 2m 1s hadoop-hdfs-project/hadoop-hdfs in trunk cannot run convertXmlToText from findbugs
        +1 javadoc 1m 9s trunk passed with JDK v1.8.0_60
        +1 javadoc 1m 52s trunk passed with JDK v1.7.0_79
        +1 mvninstall 0m 39s the patch passed
        +1 compile 0m 34s the patch passed with JDK v1.8.0_60
        +1 javac 0m 34s the patch passed
        +1 compile 0m 34s the patch passed with JDK v1.7.0_79
        +1 javac 0m 34s the patch passed
        -1 checkstyle 0m 16s Patch generated 1 new checkstyle issues in hadoop-hdfs-project/hadoop-hdfs (total was 423, now 424).
        +1 mvneclipse 0m 14s the patch passed
        +1 whitespace 0m 0s Patch has no whitespace issues.
        +1 xml 0m 1s The patch has no ill-formed XML file.
        +1 findbugs 2m 10s the patch passed
        +1 javadoc 1m 12s the patch passed with JDK v1.8.0_60
        +1 javadoc 1m 53s the patch passed with JDK v1.7.0_79
        -1 unit 51m 8s hadoop-hdfs in the patch failed with JDK v1.8.0_60.
        -1 unit 50m 10s hadoop-hdfs in the patch failed with JDK v1.7.0_79.
        -1 asflicense 0m 19s Patch generated 58 ASF License warnings.
        121m 50s



        Reason Tests
        JDK v1.8.0_60 Failed junit tests hadoop.hdfs.server.namenode.ha.TestHAStateTransitions
          hadoop.hdfs.server.balancer.TestBalancerWithMultipleNameNodes
          hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes
        JDK v1.7.0_79 Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes
          hadoop.hdfs.shortcircuit.TestShortCircuitCache
          hadoop.hdfs.TestDFSUpgradeFromImage
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure



        Subsystem Report/Notes
        Docker Client=1.7.1 Server=1.7.1 Image:test-patch-base-hadoop-date2015-11-04
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12770649/HDFS-9318-v2.patch
        JIRA Issue HDFS-9318
        Optional Tests asflicense javac javadoc mvninstall unit findbugs checkstyle compile xml
        uname Linux 87de32576c35 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /home/jenkins/jenkins-slave/workspace/PreCommit-HDFS-Build/patchprocess/apache-yetus-d0f6847/precommit/personality/hadoop.sh
        git revision trunk / 5667129
        Default Java 1.7.0_79
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_60 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_79
        findbugs v3.0.0
        findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/13386/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs.html
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/13386/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/13386/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_60.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/13386/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_79.txt
        unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/13386/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_60.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13386/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_79.txt
        JDK v1.7.0_79 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13386/testReport/
        asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/13386/artifact/patchprocess/patch-asflicense-problems.txt
        modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
        Max memory used 226MB
        Powered by Apache Yetus http://yetus.apache.org
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13386/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 5s docker + precommit patch detected. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 3m 15s trunk passed +1 compile 0m 34s trunk passed with JDK v1.8.0_60 +1 compile 0m 32s trunk passed with JDK v1.7.0_79 +1 checkstyle 0m 17s trunk passed +1 mvneclipse 0m 14s trunk passed -1 findbugs 2m 1s hadoop-hdfs-project/hadoop-hdfs in trunk cannot run convertXmlToText from findbugs +1 javadoc 1m 9s trunk passed with JDK v1.8.0_60 +1 javadoc 1m 52s trunk passed with JDK v1.7.0_79 +1 mvninstall 0m 39s the patch passed +1 compile 0m 34s the patch passed with JDK v1.8.0_60 +1 javac 0m 34s the patch passed +1 compile 0m 34s the patch passed with JDK v1.7.0_79 +1 javac 0m 34s the patch passed -1 checkstyle 0m 16s Patch generated 1 new checkstyle issues in hadoop-hdfs-project/hadoop-hdfs (total was 423, now 424). +1 mvneclipse 0m 14s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. +1 findbugs 2m 10s the patch passed +1 javadoc 1m 12s the patch passed with JDK v1.8.0_60 +1 javadoc 1m 53s the patch passed with JDK v1.7.0_79 -1 unit 51m 8s hadoop-hdfs in the patch failed with JDK v1.8.0_60. -1 unit 50m 10s hadoop-hdfs in the patch failed with JDK v1.7.0_79. -1 asflicense 0m 19s Patch generated 58 ASF License warnings. 121m 50s Reason Tests JDK v1.8.0_60 Failed junit tests hadoop.hdfs.server.namenode.ha.TestHAStateTransitions   hadoop.hdfs.server.balancer.TestBalancerWithMultipleNameNodes   hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes JDK v1.7.0_79 Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes   hadoop.hdfs.shortcircuit.TestShortCircuitCache   hadoop.hdfs.TestDFSUpgradeFromImage   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure Subsystem Report/Notes Docker Client=1.7.1 Server=1.7.1 Image:test-patch-base-hadoop-date2015-11-04 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12770649/HDFS-9318-v2.patch JIRA Issue HDFS-9318 Optional Tests asflicense javac javadoc mvninstall unit findbugs checkstyle compile xml uname Linux 87de32576c35 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /home/jenkins/jenkins-slave/workspace/PreCommit-HDFS-Build/patchprocess/apache-yetus-d0f6847/precommit/personality/hadoop.sh git revision trunk / 5667129 Default Java 1.7.0_79 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_60 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_79 findbugs v3.0.0 findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/13386/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs.html checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/13386/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/13386/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_60.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/13386/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_79.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/13386/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_60.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13386/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_79.txt JDK v1.7.0_79 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13386/testReport/ asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/13386/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Max memory used 226MB Powered by Apache Yetus http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13386/console This message was automatically generated.
        Hide
        kshukla Kuhu Shukla added a comment -

        Thank you for the comments Kihwal. Updated the patch.

        Show
        kshukla Kuhu Shukla added a comment - Thank you for the comments Kihwal. Updated the patch.
        Hide
        kihwal Kihwal Lee added a comment -

        The change itself looks good, but please add the new config to hdfs-default.xml along with documentation in the description field.

        Show
        kihwal Kihwal Lee added a comment - The change itself looks good, but please add the new config to hdfs-default.xml along with documentation in the description field.
        Hide
        kshukla Kuhu Shukla added a comment -

        None of the test failures are reproducible locally and pass successfully with the patch. Kihwal Lee, request for review/comments. Thanks.

        Show
        kshukla Kuhu Shukla added a comment - None of the test failures are reproducible locally and pass successfully with the patch. Kihwal Lee , request for review/comments. Thanks.
        Hide
        kihwal Kihwal Lee added a comment -

        The precommit ran but failed to post the result because of this error: JIRA Plugin: no credentials provided to write a comment.
        https://builds.apache.org/job/PreCommit-HDFS-Build/13243/console

        Here is the result:

        -1 overall

        Vote Subsystem Runtime Comment
        0 reexec 0m 10s docker + precommit patch detected.
        +1 @author 0m 0s The patch does not contain any @author
              tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or
              modified test files.
        +1 mvninstall 3m 17s trunk passed
        +1 compile 0m 35s trunk passed with JDK v1.8.0_60
        +1 compile 0m 33s trunk passed with JDK v1.7.0_79
        +1 checkstyle 0m 15s trunk passed
        +1 mvneclipse 0m 15s trunk passed
        -1 findbugs 1m 58s hadoop-hdfs-project/hadoop-hdfs in trunk
              cannot run convertXmlToText from findbugs
        +1 javadoc 1m 11s trunk passed with JDK v1.8.0_60
        +1 javadoc 1m 52s trunk passed with JDK v1.7.0_79
        +1 mvninstall 0m 40s the patch passed
        +1 compile 0m 33s the patch passed with JDK v1.8.0_60
        +1 javac 0m 33s the patch passed
        +1 compile 0m 33s the patch passed with JDK v1.7.0_79
        +1 javac 0m 33s the patch passed
        -1 checkstyle 0m 16s Patch generated 1 new checkstyle issues
              in hadoop-hdfs-project/hadoop-hdfs (total
              was 426, now 427).
        +1 mvneclipse 0m 15s the patch passed
        +1 whitespace 0m 0s Patch has no whitespace issues.
        +1 findbugs 2m 9s the patch passed
        +1 javadoc 1m 11s the patch passed with JDK v1.8.0_60
        +1 javadoc 1m 53s the patch passed with JDK v1.7.0_79
        -1 unit 56m 43s hadoop-hdfs in the patch failed with JDK
              v1.8.0_60.
        -1 unit 52m 17s hadoop-hdfs in the patch failed with JDK
              v1.7.0_79.
        -1 asflicense 0m 19s Patch generated 56 ASF License warnings.
            129m 48s
        Reason Tests
        JDK v1.7.0_79 Failed junit tests hadoop.hdfs.server.blockmanagement.TestNodeCount
          hadoop.hdfs.server.datanode.TestBlockScanner
          hadoop.hdfs.server.datanode.fsdataset.impl.TestScrLazyPersistFiles
          hadoop.hdfs.server.blockmanagement.TestPendingInvalidateBlock
          hadoop.hdfs.util.TestByteArrayManager
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure010
        Subsystem Report/Notes
        Docker Client=1.7.1 Server=1.7.1 Image:test-patch-base-hadoop-date2015-10-28
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12769181/HDFS-9318-v1.patch
        JIRA Issue HDFS-9318
        Optional Tests asflicense javac javadoc mvninstall unit findbugs checkstyle compile
        uname Linux 670c555545bc 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /home/jenkins/jenkins-slave/workspace/PreCommit-HDFS-Build/patchprocess/apache-yetus-06655ab/dev-support/personality/hadoop.sh
        git revision trunk / 6ff6663
        Default Java 1.7.0_79
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_60 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_79
        findbugs v3.0.0
        findbugs /testptch/hadoop/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs.html
        checkstyle /testptch/hadoop/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
        unit /testptch/hadoop/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_60.txt
        unit /testptch/hadoop/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_79.txt
        unit test logs /testptch/hadoop/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_60.txt /testptch/hadoop/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_79.txt
        JDK v1.7.0_79 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13243/testReport/
        asflicense /testptch/hadoop/patchprocess/patch-asflicense-problems.txt
        modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
        Max memory used 223MB
        Powered by Apache Yetus http://yetus.apache.org
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13243/console
        Show
        kihwal Kihwal Lee added a comment - The precommit ran but failed to post the result because of this error: JIRA Plugin: no credentials provided to write a comment. https://builds.apache.org/job/PreCommit-HDFS-Build/13243/console Here is the result: -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 10s docker + precommit patch detected. +1 @author 0m 0s The patch does not contain any @author       tags. +1 test4tests 0m 0s The patch appears to include 1 new or       modified test files. +1 mvninstall 3m 17s trunk passed +1 compile 0m 35s trunk passed with JDK v1.8.0_60 +1 compile 0m 33s trunk passed with JDK v1.7.0_79 +1 checkstyle 0m 15s trunk passed +1 mvneclipse 0m 15s trunk passed -1 findbugs 1m 58s hadoop-hdfs-project/hadoop-hdfs in trunk       cannot run convertXmlToText from findbugs +1 javadoc 1m 11s trunk passed with JDK v1.8.0_60 +1 javadoc 1m 52s trunk passed with JDK v1.7.0_79 +1 mvninstall 0m 40s the patch passed +1 compile 0m 33s the patch passed with JDK v1.8.0_60 +1 javac 0m 33s the patch passed +1 compile 0m 33s the patch passed with JDK v1.7.0_79 +1 javac 0m 33s the patch passed -1 checkstyle 0m 16s Patch generated 1 new checkstyle issues       in hadoop-hdfs-project/hadoop-hdfs (total       was 426, now 427). +1 mvneclipse 0m 15s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 2m 9s the patch passed +1 javadoc 1m 11s the patch passed with JDK v1.8.0_60 +1 javadoc 1m 53s the patch passed with JDK v1.7.0_79 -1 unit 56m 43s hadoop-hdfs in the patch failed with JDK       v1.8.0_60. -1 unit 52m 17s hadoop-hdfs in the patch failed with JDK       v1.7.0_79. -1 asflicense 0m 19s Patch generated 56 ASF License warnings.     129m 48s Reason Tests JDK v1.7.0_79 Failed junit tests hadoop.hdfs.server.blockmanagement.TestNodeCount   hadoop.hdfs.server.datanode.TestBlockScanner   hadoop.hdfs.server.datanode.fsdataset.impl.TestScrLazyPersistFiles   hadoop.hdfs.server.blockmanagement.TestPendingInvalidateBlock   hadoop.hdfs.util.TestByteArrayManager   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure010 Subsystem Report/Notes Docker Client=1.7.1 Server=1.7.1 Image:test-patch-base-hadoop-date2015-10-28 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12769181/HDFS-9318-v1.patch JIRA Issue HDFS-9318 Optional Tests asflicense javac javadoc mvninstall unit findbugs checkstyle compile uname Linux 670c555545bc 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /home/jenkins/jenkins-slave/workspace/PreCommit-HDFS-Build/patchprocess/apache-yetus-06655ab/dev-support/personality/hadoop.sh git revision trunk / 6ff6663 Default Java 1.7.0_79 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_60 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_79 findbugs v3.0.0 findbugs /testptch/hadoop/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs.html checkstyle /testptch/hadoop/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt unit /testptch/hadoop/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_60.txt unit /testptch/hadoop/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_79.txt unit test logs /testptch/hadoop/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_60.txt /testptch/hadoop/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_79.txt JDK v1.7.0_79 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13243/testReport/ asflicense /testptch/hadoop/patchprocess/patch-asflicense-problems.txt modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Max memory used 223MB Powered by Apache Yetus http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13243/console
        Hide
        kshukla Kuhu Shukla added a comment -

        The simpler way is to make this threshold check have a configurable factor that multiplies to the loadAverage instead of the fixed 2x.

        Show
        kshukla Kuhu Shukla added a comment - The simpler way is to make this threshold check have a configurable factor that multiplies to the loadAverage instead of the fixed 2x.

          People

          • Assignee:
            kshukla Kuhu Shukla
            Reporter:
            kshukla Kuhu Shukla
          • Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development