Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-5533

JMX AM Used metrics for queue wrong when app submited to nodelabel partition

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.9.0, 3.0.0-alpha1
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      1. Configure cluster with node label
      2. Configure default and root queue with label capacity 100
      3. Submit application to labelx and default queue
      4. Wait for application completion
      5. Check queue jmx metrics
       "name" : "Hadoop:service=ResourceManager,name=QueueMetrics,q0=root,q1=default",
          "modelerType" : "QueueMetrics,q0=root,q1=default",
          "tag.Queue" : "root.default",
          "tag.Context" : "yarn",
          "tag.Hostname" : "localhost",
          "running_0" : 0,
          "running_60" : 0,
          "running_300" : 0,
          "running_1440" : 0,
          "AMResourceLimitMB" : 512,
          "AMResourceLimitVCores" : 1,
          "UsedAMResourceMB" : 3072,
          "UsedAMResourceVCores" : 2,
      
      1. YARN-5533.0001.patch
        1 kB
        Bibin A Chundatt
      2. YARN-5533.0002.patch
        3 kB
        Bibin A Chundatt
      3. YARN-5553.0003.patch
        3 kB
        Bibin A Chundatt

        Activity

        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 0s Docker mode activated.
        -1 patch 0m 6s YARN-5533 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help.



        Subsystem Report/Notes
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12824553/YARN-5553.0003.patch
        JIRA Issue YARN-5533
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/12831/console
        Powered by Apache Yetus 0.3.0 http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. -1 patch 0m 6s YARN-5533 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. Subsystem Report/Notes JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12824553/YARN-5553.0003.patch JIRA Issue YARN-5533 Console output https://builds.apache.org/job/PreCommit-YARN-Build/12831/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10308 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10308/)
        YARN-5533. JMX AM Used metrics for queue wrong when app submited to (varunsaxena: rev 091dd19e8636b8bd5668dbb3bcd8bbce7c952274)

        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestNodeLabelContainerAllocation.java
        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10308 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10308/ ) YARN-5533 . JMX AM Used metrics for queue wrong when app submited to (varunsaxena: rev 091dd19e8636b8bd5668dbb3bcd8bbce7c952274) (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestNodeLabelContainerAllocation.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10307 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10307/)
        Revert "YARN-5533. JMX AM Used metrics for queue wrong when app submited (varunsaxena: rev 8aed374182f22c31e886a9a602fb907bbccc4309)

        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java
        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestNodeLabelContainerAllocation.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10307 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10307/ ) Revert " YARN-5533 . JMX AM Used metrics for queue wrong when app submited (varunsaxena: rev 8aed374182f22c31e886a9a602fb907bbccc4309) (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestNodeLabelContainerAllocation.java
        Hide
        varun_saxena Varun Saxena added a comment -

        Bibin A Chundatt, I have reopened the issue as a minor fix is required in the test case.

        assertEquals(0 * GB, leafQueue.getMetrics().getUsedAMResourceVCores());
        

        This should be 0, not 0 * GB.

        Show
        varun_saxena Varun Saxena added a comment - Bibin A Chundatt , I have reopened the issue as a minor fix is required in the test case. assertEquals(0 * GB, leafQueue.getMetrics().getUsedAMResourceVCores()); This should be 0, not 0 * GB.
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10306 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10306/)
        YARN-5533. JMX AM Used metrics for queue wrong when app submited to (varunsaxena: rev 59557e85a40fb91c7106e8ff3bfe958ffa244b29)

        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java
        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestNodeLabelContainerAllocation.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10306 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10306/ ) YARN-5533 . JMX AM Used metrics for queue wrong when app submited to (varunsaxena: rev 59557e85a40fb91c7106e8ff3bfe958ffa244b29) (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestNodeLabelContainerAllocation.java
        Hide
        varun_saxena Varun Saxena added a comment -

        Committed to trunk, branch-2.
        Thanks Bibin A Chundatt for your contribution

        Show
        varun_saxena Varun Saxena added a comment - Committed to trunk, branch-2. Thanks Bibin A Chundatt for your contribution
        Hide
        hadoopqa Hadoop QA added a comment -
        +1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 19s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        +1 mvninstall 7m 9s trunk passed
        +1 compile 0m 34s trunk passed
        +1 checkstyle 0m 22s trunk passed
        +1 mvnsite 0m 40s trunk passed
        +1 mvneclipse 0m 18s trunk passed
        +1 findbugs 1m 3s trunk passed
        +1 javadoc 0m 21s trunk passed
        +1 mvninstall 0m 31s the patch passed
        +1 compile 0m 31s the patch passed
        +1 javac 0m 31s the patch passed
        +1 checkstyle 0m 20s the patch passed
        +1 mvnsite 0m 37s the patch passed
        +1 mvneclipse 0m 15s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 1m 5s the patch passed
        +1 javadoc 0m 18s the patch passed
        +1 unit 37m 50s hadoop-yarn-server-resourcemanager in the patch passed.
        +1 asflicense 0m 15s The patch does not generate ASF License warnings.
        53m 8s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:9560f25
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12824478/YARN-5533.0002.patch
        JIRA Issue YARN-5533
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux a0bea8b68d2b 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / c5c3e81
        Default Java 1.8.0_101
        findbugs v3.0.0
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/12826/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/12826/console
        Powered by Apache Yetus 0.3.0 http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 19s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 7m 9s trunk passed +1 compile 0m 34s trunk passed +1 checkstyle 0m 22s trunk passed +1 mvnsite 0m 40s trunk passed +1 mvneclipse 0m 18s trunk passed +1 findbugs 1m 3s trunk passed +1 javadoc 0m 21s trunk passed +1 mvninstall 0m 31s the patch passed +1 compile 0m 31s the patch passed +1 javac 0m 31s the patch passed +1 checkstyle 0m 20s the patch passed +1 mvnsite 0m 37s the patch passed +1 mvneclipse 0m 15s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 5s the patch passed +1 javadoc 0m 18s the patch passed +1 unit 37m 50s hadoop-yarn-server-resourcemanager in the patch passed. +1 asflicense 0m 15s The patch does not generate ASF License warnings. 53m 8s Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12824478/YARN-5533.0002.patch JIRA Issue YARN-5533 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux a0bea8b68d2b 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / c5c3e81 Default Java 1.8.0_101 findbugs v3.0.0 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/12826/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/12826/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
        Hide
        bibinchundatt Bibin A Chundatt added a comment -

        Attaching patch after updating testcase

        Show
        bibinchundatt Bibin A Chundatt added a comment - Attaching patch after updating testcase
        Hide
        bibinchundatt Bibin A Chundatt added a comment -

        Varun Saxena
        Thank you for looking in to patch will update patch with testcase soon.

        Show
        bibinchundatt Bibin A Chundatt added a comment - Varun Saxena Thank you for looking in to patch will update patch with testcase soon.
        Hide
        varun_saxena Varun Saxena added a comment -

        BTW if its not very difficult, can we add a test case for this ?
        Say somewhere in TestNodeLabelContainerAllocation, probably add a few assertions in current tests after adding app attempt removed event.

        Show
        varun_saxena Varun Saxena added a comment - BTW if its not very difficult, can we add a test case for this ? Say somewhere in TestNodeLabelContainerAllocation, probably add a few assertions in current tests after adding app attempt removed event.
        Hide
        varun_saxena Varun Saxena added a comment - - edited

        Thanks Bibin A Chundatt for the patch.
        Changes LGTM.
        Will commit it later today unless there are further comments.

        Show
        varun_saxena Varun Saxena added a comment - - edited Thanks Bibin A Chundatt for the patch. Changes LGTM. Will commit it later today unless there are further comments.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 17s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        +1 mvninstall 6m 41s trunk passed
        +1 compile 0m 31s trunk passed
        +1 checkstyle 0m 21s trunk passed
        +1 mvnsite 0m 37s trunk passed
        +1 mvneclipse 0m 17s trunk passed
        +1 findbugs 0m 57s trunk passed
        +1 javadoc 0m 20s trunk passed
        +1 mvninstall 0m 31s the patch passed
        +1 compile 0m 29s the patch passed
        +1 javac 0m 29s the patch passed
        +1 checkstyle 0m 17s the patch passed
        +1 mvnsite 0m 35s the patch passed
        +1 mvneclipse 0m 15s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 1m 2s the patch passed
        +1 javadoc 0m 18s the patch passed
        +1 unit 37m 38s hadoop-yarn-server-resourcemanager in the patch passed.
        +1 asflicense 0m 15s The patch does not generate ASF License warnings.
        51m 57s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:9560f25
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12824198/YARN-5533.0001.patch
        JIRA Issue YARN-5533
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 41de3868c5b6 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / f80a729
        Default Java 1.8.0_101
        findbugs v3.0.0
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/12808/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/12808/console
        Powered by Apache Yetus 0.3.0 http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 17s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 6m 41s trunk passed +1 compile 0m 31s trunk passed +1 checkstyle 0m 21s trunk passed +1 mvnsite 0m 37s trunk passed +1 mvneclipse 0m 17s trunk passed +1 findbugs 0m 57s trunk passed +1 javadoc 0m 20s trunk passed +1 mvninstall 0m 31s the patch passed +1 compile 0m 29s the patch passed +1 javac 0m 29s the patch passed +1 checkstyle 0m 17s the patch passed +1 mvnsite 0m 35s the patch passed +1 mvneclipse 0m 15s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 2s the patch passed +1 javadoc 0m 18s the patch passed +1 unit 37m 38s hadoop-yarn-server-resourcemanager in the patch passed. +1 asflicense 0m 15s The patch does not generate ASF License warnings. 51m 57s Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12824198/YARN-5533.0001.patch JIRA Issue YARN-5533 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 41de3868c5b6 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / f80a729 Default Java 1.8.0_101 findbugs v3.0.0 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/12808/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/12808/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
        Hide
        bibinchundatt Bibin A Chundatt added a comment -

        During activate application the metrics is updated as below

              queueUsage.incAMUsed(partitionName,
                  application.getAMResource(partitionName));
              user.getResourceUsage().incAMUsed(partitionName,
                  application.getAMResource(partitionName));
              user.getResourceUsage().setAMLimit(partitionName, userAMLimit);
              metrics.incAMUsed(application.getUser(),
                  application.getAMResource(partitionName));
        

        When applcation is removed the partition is missed

              queueUsage.decAMUsed(partitionName,
                  application.getAMResource(partitionName));
              user.getResourceUsage().decAMUsed(partitionName,
                  application.getAMResource(partitionName));
              metrics.decAMUsed(application.getUser(), application.getAMResource());
        

        Should be application.getAMResource(partitionName)

        Show
        bibinchundatt Bibin A Chundatt added a comment - During activate application the metrics is updated as below queueUsage.incAMUsed(partitionName, application.getAMResource(partitionName)); user.getResourceUsage().incAMUsed(partitionName, application.getAMResource(partitionName)); user.getResourceUsage().setAMLimit(partitionName, userAMLimit); metrics.incAMUsed(application.getUser(), application.getAMResource(partitionName)); When applcation is removed the partition is missed queueUsage.decAMUsed(partitionName, application.getAMResource(partitionName)); user.getResourceUsage().decAMUsed(partitionName, application.getAMResource(partitionName)); metrics.decAMUsed(application.getUser(), application.getAMResource()); Should be application.getAMResource(partitionName)

          People

          • Assignee:
            bibinchundatt Bibin A Chundatt
            Reporter:
            bibinchundatt Bibin A Chundatt
          • Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development