Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-3432

Cluster metrics have wrong Total Memory when there is reserved memory on CS

    Details

    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      I noticed that when reservations happen when using the Capacity Scheduler, the UI and web services report the wrong total memory.

      For example. I have a 300GB of total memory in my cluster. I allocate 50 and I reserve 10. The cluster metrics for total memory get reported as 290GB.

      This was broken by https://issues.apache.org/jira/browse/YARN-656 so perhaps there is a difference between fair scheduler and capacity scheduler.

      1. YARN-3432-003.patch
        2 kB
        Brahma Reddy Battula
      2. YARN-3432-002.patch
        2 kB
        Brahma Reddy Battula
      3. YARN-3432.patch
        3 kB
        Brahma Reddy Battula

        Issue Links

          Activity

          Hide
          brahmareddy Brahma Reddy Battula added a comment -

          Reverting of YARN-656 should be fine I think..

          Show
          brahmareddy Brahma Reddy Battula added a comment - Reverting of YARN-656 should be fine I think..
          Hide
          tgraves Thomas Graves added a comment -

          that will fix it for the capacity scheduler, we need to see if that breaks the FairScheduler though.

          Show
          tgraves Thomas Graves added a comment - that will fix it for the capacity scheduler, we need to see if that breaks the FairScheduler though.
          Hide
          ajisakaa Akira Ajisaka added a comment -

          I'm thinking changing how to calculate total memory when the scheduler is CapacityScheduler is straightforward.

            if (rs instanceof CapacityScheduler) {
              this.totalMB = availableMB + allocatedMB + reservedMB;
            } else {
              this.totalMB = availableMB + allocatedMB;
            }
          

          Hi Thomas Graves and Brahma Reddy Battula, what do you think?

          Show
          ajisakaa Akira Ajisaka added a comment - I'm thinking changing how to calculate total memory when the scheduler is CapacityScheduler is straightforward. if (rs instanceof CapacityScheduler) { this .totalMB = availableMB + allocatedMB + reservedMB; } else { this .totalMB = availableMB + allocatedMB; } Hi Thomas Graves and Brahma Reddy Battula , what do you think?
          Hide
          ajisakaa Akira Ajisaka added a comment -

          Rethinking this, it's better to change how to calculate availableMB in CapacityScheduler. availableMB should include reservedMB for consistency.

          Show
          ajisakaa Akira Ajisaka added a comment - Rethinking this, it's better to change how to calculate availableMB in CapacityScheduler. availableMB should include reservedMB for consistency.
          Hide
          brahmareddy Brahma Reddy Battula added a comment -

          Akira Ajisaka thanks for taking a look into this issue.. It's better consisent between all the schdulers..Attached the patch..Kindly review..

          Show
          brahmareddy Brahma Reddy Battula added a comment - Akira Ajisaka thanks for taking a look into this issue.. It's better consisent between all the schdulers..Attached the patch..Kindly review..
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 15m 12s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          +1 javac 7m 43s There were no new javac warning messages.
          +1 javadoc 9m 48s There were no new javadoc warning messages.
          +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings.
          +1 checkstyle 0m 49s There were no new checkstyle issues.
          -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix.
          +1 install 1m 33s mvn install still works.
          +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
          +1 findbugs 1m 16s The patch does not introduce any new Findbugs (version 2.0.3) warnings.
          +1 yarn tests 52m 24s Tests passed in hadoop-yarn-server-resourcemanager.
              89m 43s  



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12731450/YARN-3432.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 6f62267
          whitespace https://builds.apache.org/job/PreCommit-YARN-Build/7808/artifact/patchprocess/whitespace.txt
          hadoop-yarn-server-resourcemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/7808/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/7808/testReport/
          Java 1.7.0_55
          uname Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/7808/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 15m 12s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 7m 43s There were no new javac warning messages. +1 javadoc 9m 48s There were no new javadoc warning messages. +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 0m 49s There were no new checkstyle issues. -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 install 1m 33s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. +1 findbugs 1m 16s The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 yarn tests 52m 24s Tests passed in hadoop-yarn-server-resourcemanager.     89m 43s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12731450/YARN-3432.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 6f62267 whitespace https://builds.apache.org/job/PreCommit-YARN-Build/7808/artifact/patchprocess/whitespace.txt hadoop-yarn-server-resourcemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/7808/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/7808/testReport/ Java 1.7.0_55 uname Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/7808/console This message was automatically generated.
          Hide
          ajisakaa Akira Ajisaka added a comment -

          Thanks Brahma Reddy Battula for taking this issue. The patch seems to revert YARN-656. I'm thinking that's not fine because it will break FairScheduler. This issue should fix CapacityScheduler only.

          Show
          ajisakaa Akira Ajisaka added a comment - Thanks Brahma Reddy Battula for taking this issue. The patch seems to revert YARN-656 . I'm thinking that's not fine because it will break FairScheduler. This issue should fix CapacityScheduler only.
          Hide
          brahmareddy Brahma Reddy Battula added a comment -

          Kindly review the attached patch!!!

          Show
          brahmareddy Brahma Reddy Battula added a comment - Kindly review the attached patch!!!
          Hide
          skaterxu qiang xu added a comment -

          I think we need to think about the vcore too.
          In current code, it is:
          this.totalMB = availableMB + allocatedMB;
          this.totalVirtualCores = availableVirtualCores + allocatedVirtualCores;

          Do we need to make this.totalVirtualCores = availableVirtualCores + allocatedVirtualCores+containersReserved;

          Show
          skaterxu qiang xu added a comment - I think we need to think about the vcore too. In current code, it is: this.totalMB = availableMB + allocatedMB; this.totalVirtualCores = availableVirtualCores + allocatedVirtualCores; Do we need to make this.totalVirtualCores = availableVirtualCores + allocatedVirtualCores+containersReserved;
          Hide
          skaterxu qiang xu added a comment -

          Sorry, should be
          this.totalVirtualCores = availableVirtualCores + allocatedVirtualCores+reservedVirtualCores

          Show
          skaterxu qiang xu added a comment - Sorry, should be this.totalVirtualCores = availableVirtualCores + allocatedVirtualCores+reservedVirtualCores
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 10s docker + precommit patch detected.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 mvninstall 3m 15s trunk passed
          +1 compile 0m 23s trunk passed with JDK v1.8.0_60
          +1 compile 0m 24s trunk passed with JDK v1.7.0_79
          +1 checkstyle 0m 11s trunk passed
          +1 mvneclipse 0m 15s trunk passed
          +1 findbugs 1m 8s trunk passed
          +1 javadoc 0m 22s trunk passed with JDK v1.8.0_60
          +1 javadoc 0m 26s trunk passed with JDK v1.7.0_79
          +1 mvninstall 0m 27s the patch passed
          +1 compile 0m 20s the patch passed with JDK v1.8.0_60
          +1 javac 0m 20s the patch passed
          +1 compile 0m 25s the patch passed with JDK v1.7.0_79
          +1 javac 0m 25s the patch passed
          +1 checkstyle 0m 11s the patch passed
          +1 mvneclipse 0m 15s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 1m 18s the patch passed
          +1 javadoc 0m 20s the patch passed with JDK v1.8.0_60
          +1 javadoc 0m 26s the patch passed with JDK v1.7.0_79
          -1 unit 58m 1s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.8.0_60.
          -1 unit 59m 1s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.7.0_79.
          +1 asflicense 0m 22s Patch does not generate ASF License warnings.
          128m 42s



          Reason Tests
          JDK v1.8.0_60 Failed junit tests hadoop.yarn.server.resourcemanager.TestAMAuthorization
            hadoop.yarn.server.resourcemanager.TestClientRMTokens
          JDK v1.7.0_79 Failed junit tests hadoop.yarn.server.resourcemanager.TestAMAuthorization
            hadoop.yarn.server.resourcemanager.TestClientRMTokens



          Subsystem Report/Notes
          Docker Client=1.7.1 Server=1.7.1 Image:test-patch-base-hadoop-date2015-11-04
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12737159/YARN-3432-002.patch
          JIRA Issue YARN-3432
          Optional Tests asflicense javac javadoc mvninstall unit findbugs checkstyle compile
          uname Linux 75c368b9f110 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /home/jenkins/jenkins-slave/workspace/PreCommit-YARN-Build/patchprocess/apache-yetus-d0f6847/precommit/personality/hadoop.sh
          git revision trunk / 3e1745d
          Default Java 1.7.0_79
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_60 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_79
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-YARN-Build/9628/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_60.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/9628/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_79.txt
          unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/9628/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_60.txt https://builds.apache.org/job/PreCommit-YARN-Build/9628/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_79.txt
          JDK v1.7.0_79 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9628/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
          Max memory used 227MB
          Powered by Apache Yetus http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/9628/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 10s docker + precommit patch detected. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 3m 15s trunk passed +1 compile 0m 23s trunk passed with JDK v1.8.0_60 +1 compile 0m 24s trunk passed with JDK v1.7.0_79 +1 checkstyle 0m 11s trunk passed +1 mvneclipse 0m 15s trunk passed +1 findbugs 1m 8s trunk passed +1 javadoc 0m 22s trunk passed with JDK v1.8.0_60 +1 javadoc 0m 26s trunk passed with JDK v1.7.0_79 +1 mvninstall 0m 27s the patch passed +1 compile 0m 20s the patch passed with JDK v1.8.0_60 +1 javac 0m 20s the patch passed +1 compile 0m 25s the patch passed with JDK v1.7.0_79 +1 javac 0m 25s the patch passed +1 checkstyle 0m 11s the patch passed +1 mvneclipse 0m 15s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 1m 18s the patch passed +1 javadoc 0m 20s the patch passed with JDK v1.8.0_60 +1 javadoc 0m 26s the patch passed with JDK v1.7.0_79 -1 unit 58m 1s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.8.0_60. -1 unit 59m 1s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.7.0_79. +1 asflicense 0m 22s Patch does not generate ASF License warnings. 128m 42s Reason Tests JDK v1.8.0_60 Failed junit tests hadoop.yarn.server.resourcemanager.TestAMAuthorization   hadoop.yarn.server.resourcemanager.TestClientRMTokens JDK v1.7.0_79 Failed junit tests hadoop.yarn.server.resourcemanager.TestAMAuthorization   hadoop.yarn.server.resourcemanager.TestClientRMTokens Subsystem Report/Notes Docker Client=1.7.1 Server=1.7.1 Image:test-patch-base-hadoop-date2015-11-04 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12737159/YARN-3432-002.patch JIRA Issue YARN-3432 Optional Tests asflicense javac javadoc mvninstall unit findbugs checkstyle compile uname Linux 75c368b9f110 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /home/jenkins/jenkins-slave/workspace/PreCommit-YARN-Build/patchprocess/apache-yetus-d0f6847/precommit/personality/hadoop.sh git revision trunk / 3e1745d Default Java 1.7.0_79 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_60 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_79 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-YARN-Build/9628/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_60.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/9628/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_79.txt unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/9628/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_60.txt https://builds.apache.org/job/PreCommit-YARN-Build/9628/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_79.txt JDK v1.7.0_79 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9628/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager Max memory used 227MB Powered by Apache Yetus http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-YARN-Build/9628/console This message was automatically generated.
          Hide
          ajisakaa Akira Ajisaka added a comment -

          Rethinking this, it's better to change how to calculate availableMB in CapacityScheduler. availableMB should include reservedMB for consistency.

          or to change how to calculate availableMB in FairScheduler to exclude reservedMB. As qiang xu commented, we need to take care of vcores as well.

          Show
          ajisakaa Akira Ajisaka added a comment - Rethinking this, it's better to change how to calculate availableMB in CapacityScheduler. availableMB should include reservedMB for consistency. or to change how to calculate availableMB in FairScheduler to exclude reservedMB. As qiang xu commented, we need to take care of vcores as well.
          Hide
          brahmareddy Brahma Reddy Battula added a comment -

          Akira Ajisaka thanks a lot for taking a look into this issue..I feel bettter to include in TotalMB/Containers as reservedMB/Container we are giving as metric and it will be clear to user also(even code change looks good since both are at same place).let me know your thought on this..

          Show
          brahmareddy Brahma Reddy Battula added a comment - Akira Ajisaka thanks a lot for taking a look into this issue..I feel bettter to include in TotalMB/Containers as reservedMB/Container we are giving as metric and it will be clear to user also(even code change looks good since both are at same place).let me know your thought on this..
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 mvninstall 9m 4s trunk passed
          +1 compile 0m 37s trunk passed with JDK v1.8.0_66
          +1 compile 0m 35s trunk passed with JDK v1.7.0_91
          +1 checkstyle 0m 16s trunk passed
          +1 mvnsite 0m 42s trunk passed
          +1 mvneclipse 0m 15s trunk passed
          +1 findbugs 1m 25s trunk passed
          -1 javadoc 0m 25s hadoop-yarn-server-resourcemanager in trunk failed with JDK v1.8.0_66.
          +1 javadoc 0m 32s trunk passed with JDK v1.7.0_91
          +1 mvninstall 0m 36s the patch passed
          +1 compile 0m 29s the patch passed with JDK v1.8.0_66
          +1 javac 0m 29s the patch passed
          +1 compile 0m 32s the patch passed with JDK v1.7.0_91
          +1 javac 0m 32s the patch passed
          +1 checkstyle 0m 15s the patch passed
          +1 mvnsite 0m 39s the patch passed
          +1 mvneclipse 0m 15s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 1m 29s the patch passed
          -1 javadoc 0m 24s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.8.0_66.
          +1 javadoc 0m 29s the patch passed with JDK v1.7.0_91
          -1 unit 60m 46s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.8.0_66.
          -1 unit 61m 33s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.7.0_91.
          +1 asflicense 0m 26s Patch does not generate ASF License warnings.
          143m 1s



          Reason Tests
          JDK v1.8.0_66 Failed junit tests hadoop.yarn.server.resourcemanager.TestAMAuthorization
            hadoop.yarn.server.resourcemanager.TestClientRMTokens
          JDK v1.7.0_91 Failed junit tests hadoop.yarn.server.resourcemanager.TestAMAuthorization
            hadoop.yarn.server.resourcemanager.TestClientRMTokens



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12777063/YARN-3432-003.patch
          JIRA Issue YARN-3432
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 621ec19efb19 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / a483017
          findbugs v3.0.0
          javadoc https://builds.apache.org/job/PreCommit-YARN-Build/9932/artifact/patchprocess/branch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_66.txt
          javadoc https://builds.apache.org/job/PreCommit-YARN-Build/9932/artifact/patchprocess/patch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_66.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/9932/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_66.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/9932/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_91.txt
          unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/9932/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-YARN-Build/9932/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_91.txt
          JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9932/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
          Max memory used 76MB
          Powered by Apache Yetus 0.1.0-SNAPSHOT http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/9932/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 9m 4s trunk passed +1 compile 0m 37s trunk passed with JDK v1.8.0_66 +1 compile 0m 35s trunk passed with JDK v1.7.0_91 +1 checkstyle 0m 16s trunk passed +1 mvnsite 0m 42s trunk passed +1 mvneclipse 0m 15s trunk passed +1 findbugs 1m 25s trunk passed -1 javadoc 0m 25s hadoop-yarn-server-resourcemanager in trunk failed with JDK v1.8.0_66. +1 javadoc 0m 32s trunk passed with JDK v1.7.0_91 +1 mvninstall 0m 36s the patch passed +1 compile 0m 29s the patch passed with JDK v1.8.0_66 +1 javac 0m 29s the patch passed +1 compile 0m 32s the patch passed with JDK v1.7.0_91 +1 javac 0m 32s the patch passed +1 checkstyle 0m 15s the patch passed +1 mvnsite 0m 39s the patch passed +1 mvneclipse 0m 15s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 1m 29s the patch passed -1 javadoc 0m 24s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.8.0_66. +1 javadoc 0m 29s the patch passed with JDK v1.7.0_91 -1 unit 60m 46s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.8.0_66. -1 unit 61m 33s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.7.0_91. +1 asflicense 0m 26s Patch does not generate ASF License warnings. 143m 1s Reason Tests JDK v1.8.0_66 Failed junit tests hadoop.yarn.server.resourcemanager.TestAMAuthorization   hadoop.yarn.server.resourcemanager.TestClientRMTokens JDK v1.7.0_91 Failed junit tests hadoop.yarn.server.resourcemanager.TestAMAuthorization   hadoop.yarn.server.resourcemanager.TestClientRMTokens Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12777063/YARN-3432-003.patch JIRA Issue YARN-3432 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 621ec19efb19 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / a483017 findbugs v3.0.0 javadoc https://builds.apache.org/job/PreCommit-YARN-Build/9932/artifact/patchprocess/branch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_66.txt javadoc https://builds.apache.org/job/PreCommit-YARN-Build/9932/artifact/patchprocess/patch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/9932/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/9932/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_91.txt unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/9932/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-YARN-Build/9932/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_91.txt JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9932/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager Max memory used 76MB Powered by Apache Yetus 0.1.0-SNAPSHOT http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-YARN-Build/9932/console This message was automatically generated.
          Hide
          ajisakaa Akira Ajisaka added a comment -

          Thanks Brahma Reddy Battula for the comment and updating the patch.

          I feel bettter to include in TotalMB/Containers as reservedMB/Container

          Do you mean you want to add "total containers" metric in ClusterMetrics? It should be discussed in a separate jira.

          I'm thinking "Total MB = Available MB + Allocated MB + Reserved MB" seems fine, so I'd like to change how to calculate the metrics in FairScheduler. However, it is an incompatible change.
          Therefore, I'm okay with your (compatible) approach, but you need to fix the following:

          +    if (rs instanceof CapacityScheduler) {
          +      this.totalMB = availableMB + allocatedMB + reservedMB;
          +      this.totalVirtualCores = availableVirtualCores + allocatedVirtualCores
          +          + containersReserved;
          

          containersReserved should be reservedVirtualCores.
          In addition, I'm thinking it's better to add a regression test to verify that totalMB and totalVirtualCores are unchanged when a resource is reserved in both CapacityScheduler and FairScheduler.

          Show
          ajisakaa Akira Ajisaka added a comment - Thanks Brahma Reddy Battula for the comment and updating the patch. I feel bettter to include in TotalMB/Containers as reservedMB/Container Do you mean you want to add "total containers" metric in ClusterMetrics? It should be discussed in a separate jira. I'm thinking "Total MB = Available MB + Allocated MB + Reserved MB" seems fine, so I'd like to change how to calculate the metrics in FairScheduler. However, it is an incompatible change. Therefore, I'm okay with your (compatible) approach, but you need to fix the following: + if (rs instanceof CapacityScheduler) { + this .totalMB = availableMB + allocatedMB + reservedMB; + this .totalVirtualCores = availableVirtualCores + allocatedVirtualCores + + containersReserved; containersReserved should be reservedVirtualCores . In addition, I'm thinking it's better to add a regression test to verify that totalMB and totalVirtualCores are unchanged when a resource is reserved in both CapacityScheduler and FairScheduler.
          Hide
          nroberts Nathan Roberts added a comment -

          Recently ran into this issue again. Just seems wrong that totalMB fluctuates significantly due to reservedMB moving around. Since we can't reserve beyond the size of the cluster, it seems fine to have Total MB = Available MB + Allocated MB + Reserved MB for the capacity scheduler.

          I guess this could be an incompatible change to anyone who's worked around the problem by adding reservedMB to totalMB. Vinod Kumar Vavilapalli, Karthik Kambatla, others have comments on this aspect?

          Show
          nroberts Nathan Roberts added a comment - Recently ran into this issue again. Just seems wrong that totalMB fluctuates significantly due to reservedMB moving around. Since we can't reserve beyond the size of the cluster, it seems fine to have Total MB = Available MB + Allocated MB + Reserved MB for the capacity scheduler. I guess this could be an incompatible change to anyone who's worked around the problem by adding reservedMB to totalMB. Vinod Kumar Vavilapalli , Karthik Kambatla , others have comments on this aspect?
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 17s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 mvninstall 6m 49s trunk passed
          +1 compile 0m 32s trunk passed
          +1 checkstyle 0m 20s trunk passed
          +1 mvnsite 0m 39s trunk passed
          +1 mvneclipse 0m 16s trunk passed
          +1 findbugs 1m 0s trunk passed
          +1 javadoc 0m 20s trunk passed
          +1 mvninstall 0m 30s the patch passed
          +1 compile 0m 30s the patch passed
          +1 javac 0m 30s the patch passed
          +1 checkstyle 0m 17s the patch passed
          +1 mvnsite 0m 35s the patch passed
          +1 mvneclipse 0m 14s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 1m 5s the patch passed
          +1 javadoc 0m 19s the patch passed
          +1 unit 38m 21s hadoop-yarn-server-resourcemanager in the patch passed.
          +1 asflicense 0m 15s The patch does not generate ASF License warnings.
          52m 55s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12777063/YARN-3432-003.patch
          JIRA Issue YARN-3432
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux b98b75af083b 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 3069df7
          Default Java 1.8.0_101
          findbugs v3.0.0
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/12997/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/12997/console
          Powered by Apache Yetus 0.3.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 17s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 6m 49s trunk passed +1 compile 0m 32s trunk passed +1 checkstyle 0m 20s trunk passed +1 mvnsite 0m 39s trunk passed +1 mvneclipse 0m 16s trunk passed +1 findbugs 1m 0s trunk passed +1 javadoc 0m 20s trunk passed +1 mvninstall 0m 30s the patch passed +1 compile 0m 30s the patch passed +1 javac 0m 30s the patch passed +1 checkstyle 0m 17s the patch passed +1 mvnsite 0m 35s the patch passed +1 mvneclipse 0m 14s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 5s the patch passed +1 javadoc 0m 19s the patch passed +1 unit 38m 21s hadoop-yarn-server-resourcemanager in the patch passed. +1 asflicense 0m 15s The patch does not generate ASF License warnings. 52m 55s Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12777063/YARN-3432-003.patch JIRA Issue YARN-3432 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux b98b75af083b 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 3069df7 Default Java 1.8.0_101 findbugs v3.0.0 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/12997/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/12997/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
          Hide
          kasha Karthik Kambatla added a comment -

          The last patch here seems reasonable. Yufei Gu has offered to confirm the same.

          Show
          kasha Karthik Kambatla added a comment - The last patch here seems reasonable. Yufei Gu has offered to confirm the same.
          Hide
          yufeigu Yufei Gu added a comment -

          The last patch seems good to me generally. I wonder if we can add unit tests for these since the different logics between FS and CS and these logic might change sometimes.

          Show
          yufeigu Yufei Gu added a comment - The last patch seems good to me generally. I wonder if we can add unit tests for these since the different logics between FS and CS and these logic might change sometimes.
          Hide
          ebadger Eric Badger added a comment -

          Brahma Reddy Battula, are you still working on this? I would be happy to pick up where the latest patch left off and add a unit test so that we can get this committed.

          Show
          ebadger Eric Badger added a comment - Brahma Reddy Battula , are you still working on this? I would be happy to pick up where the latest patch left off and add a unit test so that we can get this committed.
          Hide
          brahmareddy Brahma Reddy Battula added a comment -

          Oh sorry, missed this..didn't get time to work.. will try to update the unit testcase on next week.

          Show
          brahmareddy Brahma Reddy Battula added a comment - Oh sorry, missed this..didn't get time to work.. will try to update the unit testcase on next week.
          Hide
          curino Carlo Curino added a comment -

          Thanks Brahma Reddy Battula for the patch and folks for reviewing. During the bugbash we discussed and decide it was ok to commit as is. If you can add testcase later on would be good as well. Committed to trunk, branch-2, branch-2.8

          Show
          curino Carlo Curino added a comment - Thanks Brahma Reddy Battula for the patch and folks for reviewing. During the bugbash we discussed and decide it was ok to commit as is. If you can add testcase later on would be good as well. Committed to trunk, branch-2, branch-2.8
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10711 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10711/)
          YARN-3432. Cluster metrics have wrong Total Memory when there is (carlo curino: rev 892a8348fceb42069ea9877251c413fe33415e16)

          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/ClusterMetricsInfo.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10711 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10711/ ) YARN-3432 . Cluster metrics have wrong Total Memory when there is (carlo curino: rev 892a8348fceb42069ea9877251c413fe33415e16) (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/ClusterMetricsInfo.java
          Hide
          jhung Jonathan Hung added a comment -

          Hi Carlo Curino, this patch applies cleanly to branch-2.7, can we commit there as well? Thanks!

          Show
          jhung Jonathan Hung added a comment - Hi Carlo Curino , this patch applies cleanly to branch-2.7, can we commit there as well? Thanks!
          Hide
          brahmareddy Brahma Reddy Battula added a comment -

          Carlo Curino thanks for commit..and thanks to all for additional review.

          Jonathan Hung Pushed to branch-2.7.

          In the following code, containersReserved should be reservedVirtualCores,Even Akira mentioned same.

          +    if (rs instanceof CapacityScheduler) {
          +      this.totalMB = availableMB + allocatedMB + reservedMB;
          +      this.totalVirtualCores = availableVirtualCores + allocatedVirtualCores
          +          + containersReserved;
          

          As I am planing to raise separate issue for testcase(little tricky,as there are no way to get total metrics),I will handle as part of that.

          As testcase for this can be can be done only through webService..So I feel, Firstly we need to parameterize TestRMWebServices.java ,then we can add.

          Show
          brahmareddy Brahma Reddy Battula added a comment - Carlo Curino thanks for commit..and thanks to all for additional review. Jonathan Hung Pushed to branch-2.7. In the following code, containersReserved should be reservedVirtualCores ,Even Akira mentioned same. + if (rs instanceof CapacityScheduler) { + this .totalMB = availableMB + allocatedMB + reservedMB; + this .totalVirtualCores = availableVirtualCores + allocatedVirtualCores + + containersReserved; As I am planing to raise separate issue for testcase(little tricky,as there are no way to get total metrics),I will handle as part of that. As testcase for this can be can be done only through webService..So I feel, Firstly we need to parameterize TestRMWebServices.java ,then we can add.

            People

            • Assignee:
              brahmareddy Brahma Reddy Battula
              Reporter:
              tgraves Thomas Graves
            • Votes:
              0 Vote for this issue
              Watchers:
              18 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development