Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-6870

Fix floating point inaccuracies in resource availability check in AllocationBasedResourceUtilizationTracker

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.9.0, 3.0.0-beta1
    • Component/s: api, nodemanager
    • Labels:
      None

      Description

      We have seen issues on our clusters where the current way of computing CPU usage is having float-arithmetic inaccuracies (the bug is still there in trunk)

      Simple program to illustrate:

      Bar.java
            public static void main(String[] args) throws Exception {
                  float result = 0.0f;
                  for (int i = 0; i < 7; i++) {
                        if (i == 6) {
                              result += (float) 4 / (float)18;
                        } else {
                              result += (float) 2 / (float)18;
                        }
                  }
                  for (int i = 0; i < 7; i++) {
                        if (i == 6) {
                              result -= (float) 4 / (float)18;
                        } else {
                              result -= (float) 2 / (float)18;
                        }                 
                  }
                  System.out.println(result);
            }
      

      // Printed
      4.4703484E-8

      2017-04-12 05:43:24,014 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Not enough cpu for [container_e3295_1491978508342_0467_01_000030], Current CPU Allocation: [0.8888891], Requested CPU Allocation: [0.11111111]

      There are a few places with this issue:
      1. ResourceUtilization.java - set/getCPU both use float. When ContainerScheduler calls ContainersMonitor.increase/decreaseResourceUtilization, this may lead to issues.

      2. AllocationBasedResourceUtilizationTracker.java - hasResourcesAvailable uses float as well for CPU computation.

      1. YARN-6870-v3.patch
        9 kB
        Brook Zhou
      2. YARN-6870-v2.patch
        9 kB
        Brook Zhou
      3. YARN-6870-v1.patch
        9 kB
        Brook Zhou
      4. YARN-6870-v0.patch
        8 kB
        Brook Zhou

        Activity

        Hide
        asuresh Arun Suresh added a comment -

        Thanks for raising this Brook Zhou. Agreed, this can be a problem.. do you have a patch for this (preferably one for trunk and branch-2 as well) ?

        Show
        asuresh Arun Suresh added a comment - Thanks for raising this Brook Zhou . Agreed, this can be a problem.. do you have a patch for this (preferably one for trunk and branch-2 as well) ?
        Hide
        brookz Brook Zhou added a comment - - edited

        Arun Suresh Sure. For the scope of this patch, I want to check with you whether we need to change the ResourceUtilization proto and downstream objects to use integral values instead of [0,1.0], or whether we only need to fix AllocationBasedResourceUtilizationTracker.hasResourcesAvailable/ContainerScheduler.hasSufficientResources to convert from float to int for cpu calculations.

        Based on what I see, only the latter is the actual bug. The former is nice to have, but currently doesn't impact this JIRA. Thoughts?

        Show
        brookz Brook Zhou added a comment - - edited Arun Suresh Sure. For the scope of this patch, I want to check with you whether we need to change the ResourceUtilization proto and downstream objects to use integral values instead of [0,1.0] , or whether we only need to fix AllocationBasedResourceUtilizationTracker.hasResourcesAvailable/ContainerScheduler.hasSufficientResources to convert from float to int for cpu calculations. Based on what I see, only the latter is the actual bug. The former is nice to have, but currently doesn't impact this JIRA. Thoughts?
        Hide
        asuresh Arun Suresh added a comment - - edited

        Brook Zhou, I think we should start with addressing the bug in this jira. We can probably make a change in the internal protobuf representation if we find it useful for other downstream features like YARN-1011.

        Show
        asuresh Arun Suresh added a comment - - edited Brook Zhou , I think we should start with addressing the bug in this jira. We can probably make a change in the internal protobuf representation if we find it useful for other downstream features like YARN-1011 .
        Hide
        brookz Brook Zhou added a comment -

        Attached patch against trunk.

        Show
        brookz Brook Zhou added a comment - Attached patch against trunk.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 19s Docker mode activated.
              Prechecks
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
              trunk Compile Tests
        +1 mvninstall 14m 56s trunk passed
        +1 compile 0m 32s trunk passed
        +1 checkstyle 0m 18s trunk passed
        +1 mvnsite 0m 28s trunk passed
        -1 findbugs 0m 44s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager in trunk has 5 extant Findbugs warnings.
        +1 javadoc 0m 17s trunk passed
              Patch Compile Tests
        +1 mvninstall 0m 25s the patch passed
        +1 compile 0m 26s the patch passed
        +1 javac 0m 26s the patch passed
        -0 checkstyle 0m 15s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager: The patch generated 3 new + 0 unchanged - 0 fixed = 3 total (was 0)
        +1 mvnsite 0m 25s the patch passed
        -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
        +1 findbugs 0m 49s the patch passed
        -1 javadoc 0m 16s hadoop-yarn-server-nodemanager in the patch failed.
              Other Tests
        +1 unit 13m 49s hadoop-yarn-server-nodemanager in the patch passed.
        -1 asflicense 0m 17s The patch generated 1 ASF License warnings.
        35m 32s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:14b5c93
        JIRA Issue YARN-6870
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12879073/YARN-6870-v0.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux c1485119efc5 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 27a1a5f
        Default Java 1.8.0_131
        findbugs v3.1.0-RC1
        findbugs https://builds.apache.org/job/PreCommit-YARN-Build/16567/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-warnings.html
        checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/16567/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
        whitespace https://builds.apache.org/job/PreCommit-YARN-Build/16567/artifact/patchprocess/whitespace-eol.txt
        javadoc https://builds.apache.org/job/PreCommit-YARN-Build/16567/artifact/patchprocess/patch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/16567/testReport/
        asflicense https://builds.apache.org/job/PreCommit-YARN-Build/16567/artifact/patchprocess/patch-asflicense-problems.txt
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/16567/console
        Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 19s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.       trunk Compile Tests +1 mvninstall 14m 56s trunk passed +1 compile 0m 32s trunk passed +1 checkstyle 0m 18s trunk passed +1 mvnsite 0m 28s trunk passed -1 findbugs 0m 44s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager in trunk has 5 extant Findbugs warnings. +1 javadoc 0m 17s trunk passed       Patch Compile Tests +1 mvninstall 0m 25s the patch passed +1 compile 0m 26s the patch passed +1 javac 0m 26s the patch passed -0 checkstyle 0m 15s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager: The patch generated 3 new + 0 unchanged - 0 fixed = 3 total (was 0) +1 mvnsite 0m 25s the patch passed -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply +1 findbugs 0m 49s the patch passed -1 javadoc 0m 16s hadoop-yarn-server-nodemanager in the patch failed.       Other Tests +1 unit 13m 49s hadoop-yarn-server-nodemanager in the patch passed. -1 asflicense 0m 17s The patch generated 1 ASF License warnings. 35m 32s Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue YARN-6870 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12879073/YARN-6870-v0.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux c1485119efc5 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 27a1a5f Default Java 1.8.0_131 findbugs v3.1.0-RC1 findbugs https://builds.apache.org/job/PreCommit-YARN-Build/16567/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-warnings.html checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/16567/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt whitespace https://builds.apache.org/job/PreCommit-YARN-Build/16567/artifact/patchprocess/whitespace-eol.txt javadoc https://builds.apache.org/job/PreCommit-YARN-Build/16567/artifact/patchprocess/patch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/16567/testReport/ asflicense https://builds.apache.org/job/PreCommit-YARN-Build/16567/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/16567/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 22s Docker mode activated.
              Prechecks
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
              trunk Compile Tests
        +1 mvninstall 12m 47s trunk passed
        +1 compile 0m 29s trunk passed
        +1 checkstyle 0m 18s trunk passed
        +1 mvnsite 0m 27s trunk passed
        -1 findbugs 0m 41s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager in trunk has 5 extant Findbugs warnings.
        +1 javadoc 0m 18s trunk passed
              Patch Compile Tests
        +1 mvninstall 0m 23s the patch passed
        +1 compile 0m 25s the patch passed
        +1 javac 0m 25s the patch passed
        -0 checkstyle 0m 14s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager: The patch generated 3 new + 0 unchanged - 0 fixed = 3 total (was 0)
        +1 mvnsite 0m 23s the patch passed
        -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
        +1 findbugs 0m 45s the patch passed
        -1 javadoc 0m 15s hadoop-yarn-server-nodemanager in the patch failed.
              Other Tests
        +1 unit 13m 32s hadoop-yarn-server-nodemanager in the patch passed.
        -1 asflicense 0m 16s The patch generated 1 ASF License warnings.
        32m 55s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:14b5c93
        JIRA Issue YARN-6870
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12879073/YARN-6870-v0.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 5f5e643ebf32 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12 13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 27a1a5f
        Default Java 1.8.0_131
        findbugs v3.1.0-RC1
        findbugs https://builds.apache.org/job/PreCommit-YARN-Build/16568/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-warnings.html
        checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/16568/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
        whitespace https://builds.apache.org/job/PreCommit-YARN-Build/16568/artifact/patchprocess/whitespace-eol.txt
        javadoc https://builds.apache.org/job/PreCommit-YARN-Build/16568/artifact/patchprocess/patch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/16568/testReport/
        asflicense https://builds.apache.org/job/PreCommit-YARN-Build/16568/artifact/patchprocess/patch-asflicense-problems.txt
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/16568/console
        Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 22s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.       trunk Compile Tests +1 mvninstall 12m 47s trunk passed +1 compile 0m 29s trunk passed +1 checkstyle 0m 18s trunk passed +1 mvnsite 0m 27s trunk passed -1 findbugs 0m 41s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager in trunk has 5 extant Findbugs warnings. +1 javadoc 0m 18s trunk passed       Patch Compile Tests +1 mvninstall 0m 23s the patch passed +1 compile 0m 25s the patch passed +1 javac 0m 25s the patch passed -0 checkstyle 0m 14s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager: The patch generated 3 new + 0 unchanged - 0 fixed = 3 total (was 0) +1 mvnsite 0m 23s the patch passed -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply +1 findbugs 0m 45s the patch passed -1 javadoc 0m 15s hadoop-yarn-server-nodemanager in the patch failed.       Other Tests +1 unit 13m 32s hadoop-yarn-server-nodemanager in the patch passed. -1 asflicense 0m 16s The patch generated 1 ASF License warnings. 32m 55s Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue YARN-6870 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12879073/YARN-6870-v0.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 5f5e643ebf32 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12 13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 27a1a5f Default Java 1.8.0_131 findbugs v3.1.0-RC1 findbugs https://builds.apache.org/job/PreCommit-YARN-Build/16568/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-warnings.html checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/16568/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt whitespace https://builds.apache.org/job/PreCommit-YARN-Build/16568/artifact/patchprocess/whitespace-eol.txt javadoc https://builds.apache.org/job/PreCommit-YARN-Build/16568/artifact/patchprocess/patch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/16568/testReport/ asflicense https://builds.apache.org/job/PreCommit-YARN-Build/16568/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/16568/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 18s Docker mode activated.
              Prechecks
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
              trunk Compile Tests
        +1 mvninstall 13m 24s trunk passed
        +1 compile 0m 29s trunk passed
        +1 checkstyle 0m 18s trunk passed
        +1 mvnsite 0m 27s trunk passed
        -1 findbugs 0m 43s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager in trunk has 5 extant Findbugs warnings.
        +1 javadoc 0m 17s trunk passed
              Patch Compile Tests
        +1 mvninstall 0m 24s the patch passed
        +1 compile 0m 26s the patch passed
        +1 javac 0m 26s the patch passed
        +1 checkstyle 0m 15s the patch passed
        +1 mvnsite 0m 25s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 0m 48s the patch passed
        -1 javadoc 0m 15s hadoop-yarn-server-nodemanager in the patch failed.
              Other Tests
        +1 unit 13m 6s hadoop-yarn-server-nodemanager in the patch passed.
        +1 asflicense 0m 15s The patch does not generate ASF License warnings.
        33m 3s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:14b5c93
        JIRA Issue YARN-6870
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12879383/YARN-6870-v1.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux cb46ad6eb560 3.13.0-119-generic #166-Ubuntu SMP Wed May 3 12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 9ea01fd
        Default Java 1.8.0_131
        findbugs v3.1.0-RC1
        findbugs https://builds.apache.org/job/PreCommit-YARN-Build/16590/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-warnings.html
        javadoc https://builds.apache.org/job/PreCommit-YARN-Build/16590/artifact/patchprocess/patch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/16590/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/16590/console
        Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 18s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.       trunk Compile Tests +1 mvninstall 13m 24s trunk passed +1 compile 0m 29s trunk passed +1 checkstyle 0m 18s trunk passed +1 mvnsite 0m 27s trunk passed -1 findbugs 0m 43s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager in trunk has 5 extant Findbugs warnings. +1 javadoc 0m 17s trunk passed       Patch Compile Tests +1 mvninstall 0m 24s the patch passed +1 compile 0m 26s the patch passed +1 javac 0m 26s the patch passed +1 checkstyle 0m 15s the patch passed +1 mvnsite 0m 25s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 0m 48s the patch passed -1 javadoc 0m 15s hadoop-yarn-server-nodemanager in the patch failed.       Other Tests +1 unit 13m 6s hadoop-yarn-server-nodemanager in the patch passed. +1 asflicense 0m 15s The patch does not generate ASF License warnings. 33m 3s Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue YARN-6870 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12879383/YARN-6870-v1.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux cb46ad6eb560 3.13.0-119-generic #166-Ubuntu SMP Wed May 3 12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 9ea01fd Default Java 1.8.0_131 findbugs v3.1.0-RC1 findbugs https://builds.apache.org/job/PreCommit-YARN-Build/16590/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-warnings.html javadoc https://builds.apache.org/job/PreCommit-YARN-Build/16590/artifact/patchprocess/patch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/16590/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/16590/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        arun.suresh@gmail.com -Arun Suresh- added a comment -

        Thanks for the patch Brook,
        Looks like the javadoc issue is due to this:

        [ERROR] /testptch/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/scheduler/AllocationBasedResourceUtilizationTracker.java:145:
        error: malformed HTML
        [ERROR] * @return True if currentAllocation*totalCores + coresRequested <=
        
        

        Just replace the * with an x

        Show
        arun.suresh@gmail.com -Arun Suresh- added a comment - Thanks for the patch Brook, Looks like the javadoc issue is due to this: [ERROR] /testptch/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/scheduler/AllocationBasedResourceUtilizationTracker.java:145: error: malformed HTML [ERROR] * @return True if currentAllocation*totalCores + coresRequested <= Just replace the * with an x
        Hide
        asuresh Arun Suresh added a comment -

        +1 pending the javadoc fix

        Show
        asuresh Arun Suresh added a comment - +1 pending the javadoc fix
        Hide
        brookz Brook Zhou added a comment -

        Thanks, made the change.

        Show
        brookz Brook Zhou added a comment - Thanks, made the change.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 14s Docker mode activated.
              Prechecks
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
              trunk Compile Tests
        +1 mvninstall 12m 37s trunk passed
        +1 compile 0m 26s trunk passed
        +1 checkstyle 0m 15s trunk passed
        +1 mvnsite 0m 25s trunk passed
        -1 findbugs 0m 39s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager in trunk has 5 extant Findbugs warnings.
        +1 javadoc 0m 17s trunk passed
              Patch Compile Tests
        +1 mvninstall 0m 22s the patch passed
        +1 compile 0m 23s the patch passed
        +1 javac 0m 23s the patch passed
        +1 checkstyle 0m 15s the patch passed
        +1 mvnsite 0m 23s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 0m 44s the patch passed
        -1 javadoc 0m 13s hadoop-yarn-server-nodemanager in the patch failed.
              Other Tests
        +1 unit 13m 11s hadoop-yarn-server-nodemanager in the patch passed.
        +1 asflicense 0m 14s The patch does not generate ASF License warnings.
        31m 50s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:14b5c93
        JIRA Issue YARN-6870
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12879394/YARN-6870-v2.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 2d8f2c7a6fcc 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12 13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 77791e4
        Default Java 1.8.0_131
        findbugs v3.1.0-RC1
        findbugs https://builds.apache.org/job/PreCommit-YARN-Build/16593/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-warnings.html
        javadoc https://builds.apache.org/job/PreCommit-YARN-Build/16593/artifact/patchprocess/patch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/16593/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/16593/console
        Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 14s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.       trunk Compile Tests +1 mvninstall 12m 37s trunk passed +1 compile 0m 26s trunk passed +1 checkstyle 0m 15s trunk passed +1 mvnsite 0m 25s trunk passed -1 findbugs 0m 39s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager in trunk has 5 extant Findbugs warnings. +1 javadoc 0m 17s trunk passed       Patch Compile Tests +1 mvninstall 0m 22s the patch passed +1 compile 0m 23s the patch passed +1 javac 0m 23s the patch passed +1 checkstyle 0m 15s the patch passed +1 mvnsite 0m 23s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 0m 44s the patch passed -1 javadoc 0m 13s hadoop-yarn-server-nodemanager in the patch failed.       Other Tests +1 unit 13m 11s hadoop-yarn-server-nodemanager in the patch passed. +1 asflicense 0m 14s The patch does not generate ASF License warnings. 31m 50s Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue YARN-6870 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12879394/YARN-6870-v2.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 2d8f2c7a6fcc 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12 13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 77791e4 Default Java 1.8.0_131 findbugs v3.1.0-RC1 findbugs https://builds.apache.org/job/PreCommit-YARN-Build/16593/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-warnings.html javadoc https://builds.apache.org/job/PreCommit-YARN-Build/16593/artifact/patchprocess/patch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/16593/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/16593/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 21s Docker mode activated.
              Prechecks
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
              trunk Compile Tests
        +1 mvninstall 13m 37s trunk passed
        +1 compile 0m 30s trunk passed
        +1 checkstyle 0m 18s trunk passed
        +1 mvnsite 0m 27s trunk passed
        -1 findbugs 0m 43s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager in trunk has 5 extant Findbugs warnings.
        +1 javadoc 0m 18s trunk passed
              Patch Compile Tests
        +1 mvninstall 0m 24s the patch passed
        +1 compile 0m 27s the patch passed
        +1 javac 0m 27s the patch passed
        +1 checkstyle 0m 15s the patch passed
        +1 mvnsite 0m 25s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 0m 47s the patch passed
        +1 javadoc 0m 15s the patch passed
              Other Tests
        +1 unit 13m 43s hadoop-yarn-server-nodemanager in the patch passed.
        +1 asflicense 0m 15s The patch does not generate ASF License warnings.
        34m 0s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:14b5c93
        JIRA Issue YARN-6870
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12879438/YARN-6870-v3.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 5bf022fc80e6 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 713349a
        Default Java 1.8.0_131
        findbugs v3.1.0-RC1
        findbugs https://builds.apache.org/job/PreCommit-YARN-Build/16602/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-warnings.html
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/16602/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/16602/console
        Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 21s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.       trunk Compile Tests +1 mvninstall 13m 37s trunk passed +1 compile 0m 30s trunk passed +1 checkstyle 0m 18s trunk passed +1 mvnsite 0m 27s trunk passed -1 findbugs 0m 43s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager in trunk has 5 extant Findbugs warnings. +1 javadoc 0m 18s trunk passed       Patch Compile Tests +1 mvninstall 0m 24s the patch passed +1 compile 0m 27s the patch passed +1 javac 0m 27s the patch passed +1 checkstyle 0m 15s the patch passed +1 mvnsite 0m 25s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 0m 47s the patch passed +1 javadoc 0m 15s the patch passed       Other Tests +1 unit 13m 43s hadoop-yarn-server-nodemanager in the patch passed. +1 asflicense 0m 15s The patch does not generate ASF License warnings. 34m 0s Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue YARN-6870 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12879438/YARN-6870-v3.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 5bf022fc80e6 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 713349a Default Java 1.8.0_131 findbugs v3.1.0-RC1 findbugs https://builds.apache.org/job/PreCommit-YARN-Build/16602/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-warnings.html Test Results https://builds.apache.org/job/PreCommit-YARN-Build/16602/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/16602/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        brookz Brook Zhou added a comment -

        Findbugs warnings are from existing code not in this patch.

        Show
        brookz Brook Zhou added a comment - Findbugs warnings are from existing code not in this patch.
        Hide
        asuresh Arun Suresh added a comment -

        Thanks Brook Zhou. will commit this shortly

        Show
        asuresh Arun Suresh added a comment - Thanks Brook Zhou . will commit this shortly
        Hide
        asuresh Arun Suresh added a comment -

        Committed this to trunk and branch-2

        Show
        asuresh Arun Suresh added a comment - Committed this to trunk and branch-2
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #12074 (See https://builds.apache.org/job/Hadoop-trunk-Commit/12074/)
        YARN-6870. Fix floating point inaccuracies in resource availability (arun suresh: rev 890e14c02a612c772cecd5dff2411060efd418a3)

        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/scheduler/ContainerScheduler.java
        • (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/scheduler/TestAllocationBasedResourceUtilizationTracker.java
        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/scheduler/AllocationBasedResourceUtilizationTracker.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #12074 (See https://builds.apache.org/job/Hadoop-trunk-Commit/12074/ ) YARN-6870 . Fix floating point inaccuracies in resource availability (arun suresh: rev 890e14c02a612c772cecd5dff2411060efd418a3) (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/scheduler/ContainerScheduler.java (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/scheduler/TestAllocationBasedResourceUtilizationTracker.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/scheduler/AllocationBasedResourceUtilizationTracker.java

          People

          • Assignee:
            brookz Brook Zhou
            Reporter:
            brookz Brook Zhou
          • Votes:
            0 Vote for this issue
            Watchers:
            9 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development