Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-6195

Export UsedCapacity and AbsoluteUsedCapacity to JMX

    Details

      Description

      `usedCapacity` and `absoluteUsedCapacity` are currently not available as JMX.

      1. YARN-6195.001.patch
        19 kB
        Benson Qiu
      2. YARN-6195.002.patch
        18 kB
        Benson Qiu
      3. YARN-6195.003.patch
        24 kB
        Benson Qiu
      4. YARN-6195.004.patch
        23 kB
        Benson Qiu
      5. YARN-6195.005.patch
        24 kB
        Benson Qiu

        Issue Links

          Activity

          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11591 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11591/)
          YARN-6195. Export UsedCapacity and AbsoluteUsedCapacity to JMX. (jlowe: rev 0e065f2edead41f6f894c9d70def389af1c0f239)

          • (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/lib/MutableMetricsFactory.java
          • (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/lib/MetricsRegistry.java
          • (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/metrics2/lib/TestMetricsAnnotations.java
          • (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/metrics2/lib/TestMutableMetrics.java
          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/ReservationQueue.java
          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CSQueueMetrics.java
          • (add) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/lib/MutableGaugeFloat.java
          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CSQueue.java
          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java
          • (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/metrics2/lib/TestMetricsRegistry.java
          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/ParentQueue.java
          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/AbstractCSQueue.java
          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CSQueueUtils.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11591 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11591/ ) YARN-6195 . Export UsedCapacity and AbsoluteUsedCapacity to JMX. (jlowe: rev 0e065f2edead41f6f894c9d70def389af1c0f239) (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/lib/MutableMetricsFactory.java (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/lib/MetricsRegistry.java (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/metrics2/lib/TestMetricsAnnotations.java (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/metrics2/lib/TestMutableMetrics.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/ReservationQueue.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CSQueueMetrics.java (add) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/lib/MutableGaugeFloat.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CSQueue.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/metrics2/lib/TestMetricsRegistry.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/ParentQueue.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/AbstractCSQueue.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CSQueueUtils.java
          Hide
          benson.qiu Benson Qiu added a comment - - edited

          Thanks so much Jason Lowe for all of your help throughout this process, and thanks Naganarasimha G R, Wangda Tan, and Sunil G for taking the time to review my patch as well! I really appreciate it!

          Show
          benson.qiu Benson Qiu added a comment - - edited Thanks so much Jason Lowe for all of your help throughout this process, and thanks Naganarasimha G R , Wangda Tan , and Sunil G for taking the time to review my patch as well! I really appreciate it!
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11567 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11567/)
          YARN-6195. Export UsedCapacity and AbsoluteUsedCapacity to JMX. (jlowe: rev 0e065f2edead41f6f894c9d70def389af1c0f239)

          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/ParentQueue.java
          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CSQueueUtils.java
          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CSQueue.java
          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/ReservationQueue.java
          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CSQueueMetrics.java
          • (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/metrics2/lib/TestMetricsAnnotations.java
          • (add) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/lib/MutableGaugeFloat.java
          • (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/metrics2/lib/TestMetricsRegistry.java
          • (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/lib/MetricsRegistry.java
          • (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/metrics2/lib/TestMutableMetrics.java
          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java
          • (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/lib/MutableMetricsFactory.java
          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/AbstractCSQueue.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11567 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11567/ ) YARN-6195 . Export UsedCapacity and AbsoluteUsedCapacity to JMX. (jlowe: rev 0e065f2edead41f6f894c9d70def389af1c0f239) (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/ParentQueue.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CSQueueUtils.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CSQueue.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/ReservationQueue.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CSQueueMetrics.java (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/metrics2/lib/TestMetricsAnnotations.java (add) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/lib/MutableGaugeFloat.java (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/metrics2/lib/TestMetricsRegistry.java (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/lib/MetricsRegistry.java (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/metrics2/lib/TestMutableMetrics.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/lib/MutableMetricsFactory.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/AbstractCSQueue.java
          Hide
          jlowe Jason Lowe added a comment -

          Thanks to Benson Qiu for the contribution and to Naganarasimha G R, Wangda Tan, and Sunil G for additional review! I committed this to trunk and branch-2.

          Show
          jlowe Jason Lowe added a comment - Thanks to Benson Qiu for the contribution and to Naganarasimha G R , Wangda Tan , and Sunil G for additional review! I committed this to trunk and branch-2.
          Hide
          jlowe Jason Lowe added a comment -

          I do not think a separate HADOOP JIRA is necessary here. Committing this.

          Show
          jlowe Jason Lowe added a comment - I do not think a separate HADOOP JIRA is necessary here. Committing this.
          Hide
          benson.qiu Benson Qiu added a comment -

          Thanks Jason and Naga for taking a look at my latest patch!

          Jason Lowe: Do you happen to have the answer to Naga's question about making modifications to both hadoop-yarn-project and hadoop-common-project?

          Show
          benson.qiu Benson Qiu added a comment - Thanks Jason and Naga for taking a look at my latest patch! Jason Lowe : Do you happen to have the answer to Naga's question about making modifications to both hadoop-yarn-project and hadoop-common-project?
          Hide
          Naganarasimha Naganarasimha G R added a comment -

          Thanks for working on the new approach, One main query i have is, do we require to do commons modification in HADOOP project (as a separate jira) ? Other than this everything else is fine. Also for other metrics like allocatedMB/allocatedVCores/allocatedContainers etc still updates for all partitions, will raise a new jira for the same.

          Few very minor nits (not compulsary to be fixed but if you plan to make new patch...) :

          1. MetricsRegistry,ln no 448: empty line not required
          2. CSQueueUtils, ln no 182: we can rename childQueue to just queue as it can be parent or leaf queue.
          Show
          Naganarasimha Naganarasimha G R added a comment - Thanks for working on the new approach, One main query i have is, do we require to do commons modification in HADOOP project (as a separate jira) ? Other than this everything else is fine. Also for other metrics like allocatedMB/allocatedVCores/allocatedContainers etc still updates for all partitions, will raise a new jira for the same. Few very minor nits (not compulsary to be fixed but if you plan to make new patch...) : MetricsRegistry,ln no 448: empty line not required CSQueueUtils, ln no 182: we can rename childQueue to just queue as it can be parent or leaf queue.
          Hide
          jlowe Jason Lowe added a comment -

          Thanks for updating the patch! At first I thought we had a potential for an NPE in CSQueueUtils since there's this comment at the top:

            /**
             * Update partitioned resource usage, if nodePartition == null, will update
             * used resource for all partitions of this queue.
             */
            public static void updateUsedCapacity(final ResourceCalculator rc,
          

          However in practice all the callers translate a null label to the no label enum so we're good. Looks like we'd have NPE problems even before this patch if nodePartition really was null, so that's a bad comment unrelated to this patch.

          +1 for the latest patch. I'll commit this early next week if there are no objections.

          Show
          jlowe Jason Lowe added a comment - Thanks for updating the patch! At first I thought we had a potential for an NPE in CSQueueUtils since there's this comment at the top: /** * Update partitioned resource usage, if nodePartition == null , will update * used resource for all partitions of this queue. */ public static void updateUsedCapacity( final ResourceCalculator rc, However in practice all the callers translate a null label to the no label enum so we're good. Looks like we'd have NPE problems even before this patch if nodePartition really was null, so that's a bad comment unrelated to this patch. +1 for the latest patch. I'll commit this early next week if there are no objections.
          Hide
          benson.qiu Benson Qiu added a comment -

          Version 5 outputs usedCapacity and absoluteUsedCapacity for the default partition.

          Show
          benson.qiu Benson Qiu added a comment - Version 5 outputs usedCapacity and absoluteUsedCapacity for the default partition.
          Hide
          hadoopqa Hadoop QA added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 24s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 3 new or modified test files.
          0 mvndep 0m 14s Maven dependency ordering for branch
          +1 mvninstall 12m 58s trunk passed
          +1 compile 16m 24s trunk passed
          +1 checkstyle 2m 1s trunk passed
          +1 mvnsite 1m 49s trunk passed
          +1 mvneclipse 0m 41s trunk passed
          +1 findbugs 3m 0s trunk passed
          +1 javadoc 1m 24s trunk passed
          0 mvndep 0m 17s Maven dependency ordering for patch
          +1 mvninstall 1m 13s the patch passed
          +1 compile 14m 0s the patch passed
          +1 javac 14m 0s the patch passed
          -0 checkstyle 2m 3s root: The patch generated 7 new + 267 unchanged - 5 fixed = 274 total (was 272)
          +1 mvnsite 1m 50s the patch passed
          +1 mvneclipse 0m 47s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 3m 3s the patch passed
          +1 javadoc 1m 26s the patch passed
          +1 unit 7m 31s hadoop-common in the patch passed.
          +1 unit 39m 7s hadoop-yarn-server-resourcemanager in the patch passed.
          +1 asflicense 0m 39s The patch does not generate ASF License warnings.
          135m 21s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue YARN-6195
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12862149/YARN-6195.005.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 80c9a9a148fe 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 7d963c4
          Default Java 1.8.0_121
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/15534/artifact/patchprocess/diff-checkstyle-root.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/15534/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: .
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/15534/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 24s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 3 new or modified test files. 0 mvndep 0m 14s Maven dependency ordering for branch +1 mvninstall 12m 58s trunk passed +1 compile 16m 24s trunk passed +1 checkstyle 2m 1s trunk passed +1 mvnsite 1m 49s trunk passed +1 mvneclipse 0m 41s trunk passed +1 findbugs 3m 0s trunk passed +1 javadoc 1m 24s trunk passed 0 mvndep 0m 17s Maven dependency ordering for patch +1 mvninstall 1m 13s the patch passed +1 compile 14m 0s the patch passed +1 javac 14m 0s the patch passed -0 checkstyle 2m 3s root: The patch generated 7 new + 267 unchanged - 5 fixed = 274 total (was 272) +1 mvnsite 1m 50s the patch passed +1 mvneclipse 0m 47s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 3m 3s the patch passed +1 javadoc 1m 26s the patch passed +1 unit 7m 31s hadoop-common in the patch passed. +1 unit 39m 7s hadoop-yarn-server-resourcemanager in the patch passed. +1 asflicense 0m 39s The patch does not generate ASF License warnings. 135m 21s Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue YARN-6195 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12862149/YARN-6195.005.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 80c9a9a148fe 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 7d963c4 Default Java 1.8.0_121 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/15534/artifact/patchprocess/diff-checkstyle-root.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/15534/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: . Console output https://builds.apache.org/job/PreCommit-YARN-Build/15534/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          benson.qiu Benson Qiu added a comment -

          I will fix the findbugs issue from HadoopQA.

          Show
          benson.qiu Benson Qiu added a comment - I will fix the findbugs issue from HadoopQA.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 21s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 3 new or modified test files.
          0 mvndep 0m 18s Maven dependency ordering for branch
          +1 mvninstall 16m 29s trunk passed
          +1 compile 16m 48s trunk passed
          +1 checkstyle 2m 8s trunk passed
          +1 mvnsite 1m 59s trunk passed
          +1 mvneclipse 0m 45s trunk passed
          +1 findbugs 3m 32s trunk passed
          +1 javadoc 1m 37s trunk passed
          0 mvndep 0m 20s Maven dependency ordering for patch
          +1 mvninstall 1m 45s the patch passed
          +1 compile 17m 11s the patch passed
          +1 javac 17m 11s the patch passed
          -0 checkstyle 2m 11s root: The patch generated 7 new + 267 unchanged - 4 fixed = 274 total (was 271)
          +1 mvnsite 2m 1s the patch passed
          +1 mvneclipse 0m 51s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          -1 findbugs 1m 29s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
          +1 javadoc 1m 36s the patch passed
          +1 unit 8m 27s hadoop-common in the patch passed.
          -1 unit 41m 42s hadoop-yarn-server-resourcemanager in the patch failed.
          +1 asflicense 0m 43s The patch does not generate ASF License warnings.
          149m 0s



          Reason Tests
          FindBugs module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
            Dead store to nodeLabels in org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CSQueueUtils.updateUsedCapacity(ResourceCalculator, Resource, String, AbstractCSQueue) At CSQueueUtils.java:org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CSQueueUtils.updateUsedCapacity(ResourceCalculator, Resource, String, AbstractCSQueue) At CSQueueUtils.java:[line 190]
          Failed junit tests hadoop.yarn.server.resourcemanager.TestRMRestart



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue YARN-6195
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12862102/YARN-6195.004.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux d29b7f7bcb8d 3.13.0-105-generic #152-Ubuntu SMP Fri Dec 2 15:37:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 34ab8e7
          Default Java 1.8.0_121
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/15528/artifact/patchprocess/diff-checkstyle-root.txt
          findbugs https://builds.apache.org/job/PreCommit-YARN-Build/15528/artifact/patchprocess/new-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.html
          unit https://builds.apache.org/job/PreCommit-YARN-Build/15528/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/15528/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: .
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/15528/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 21s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 3 new or modified test files. 0 mvndep 0m 18s Maven dependency ordering for branch +1 mvninstall 16m 29s trunk passed +1 compile 16m 48s trunk passed +1 checkstyle 2m 8s trunk passed +1 mvnsite 1m 59s trunk passed +1 mvneclipse 0m 45s trunk passed +1 findbugs 3m 32s trunk passed +1 javadoc 1m 37s trunk passed 0 mvndep 0m 20s Maven dependency ordering for patch +1 mvninstall 1m 45s the patch passed +1 compile 17m 11s the patch passed +1 javac 17m 11s the patch passed -0 checkstyle 2m 11s root: The patch generated 7 new + 267 unchanged - 4 fixed = 274 total (was 271) +1 mvnsite 2m 1s the patch passed +1 mvneclipse 0m 51s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. -1 findbugs 1m 29s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) +1 javadoc 1m 36s the patch passed +1 unit 8m 27s hadoop-common in the patch passed. -1 unit 41m 42s hadoop-yarn-server-resourcemanager in the patch failed. +1 asflicense 0m 43s The patch does not generate ASF License warnings. 149m 0s Reason Tests FindBugs module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager   Dead store to nodeLabels in org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CSQueueUtils.updateUsedCapacity(ResourceCalculator, Resource, String, AbstractCSQueue) At CSQueueUtils.java:org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CSQueueUtils.updateUsedCapacity(ResourceCalculator, Resource, String, AbstractCSQueue) At CSQueueUtils.java: [line 190] Failed junit tests hadoop.yarn.server.resourcemanager.TestRMRestart Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue YARN-6195 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12862102/YARN-6195.004.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux d29b7f7bcb8d 3.13.0-105-generic #152-Ubuntu SMP Fri Dec 2 15:37:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 34ab8e7 Default Java 1.8.0_121 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/15528/artifact/patchprocess/diff-checkstyle-root.txt findbugs https://builds.apache.org/job/PreCommit-YARN-Build/15528/artifact/patchprocess/new-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.html unit https://builds.apache.org/job/PreCommit-YARN-Build/15528/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/15528/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: . Console output https://builds.apache.org/job/PreCommit-YARN-Build/15528/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          jlowe Jason Lowe added a comment -

          I'm totally OK with just reporting the default partition's stats in these new fields if everyone agrees that's a reasonable thing to do until QueueMetrics is partitioned. As a bonus, it's significantly cheaper to compute when there are many queues configured.

          Show
          jlowe Jason Lowe added a comment - I'm totally OK with just reporting the default partition's stats in these new fields if everyone agrees that's a reasonable thing to do until QueueMetrics is partitioned. As a bonus, it's significantly cheaper to compute when there are many queues configured.
          Hide
          sunilg Sunil G added a comment -

          I also feel that we can go for #2 suggestion by Wangda for now.
          abs/usedCapacity = abs/usedCapacity from default partition

          Because ideally we expect per-partition metrics in future. And if we take a combined used/absCapacity across all partitions, I do not think it will be very useful enough.

          Show
          sunilg Sunil G added a comment - I also feel that we can go for #2 suggestion by Wangda for now. abs/usedCapacity = abs/usedCapacity from default partition Because ideally we expect per-partition metrics in future. And if we take a combined used/absCapacity across all partitions, I do not think it will be very useful enough.
          Hide
          benson.qiu Benson Qiu added a comment - - edited

          Thanks Naganarasimha G R and Wangda Tan for the comments. Your explanations for why you prefer the default partition over the max across all partitions makes sense. The max across all partitions might be less useful since we don't have a way of reporting the label associated with the max value.

          Jason Lowe: What are your thoughts on this?

          Show
          benson.qiu Benson Qiu added a comment - - edited Thanks Naganarasimha G R and Wangda Tan for the comments. Your explanations for why you prefer the default partition over the max across all partitions makes sense. The max across all partitions might be less useful since we don't have a way of reporting the label associated with the max value. Jason Lowe : What are your thoughts on this?
          Hide
          leftnoteasy Wangda Tan added a comment -

          Jason Lowe/Benson Qiu, Sorry for jumping in late, since we need to squash multi vals to a single field, I don't think we can find a perfect solution here.

          Few options:
          #1 Behavior of this patch, usedCapacity = max(label, usedCapacity), absUsedCapacity = max(label, absUsedCapacity);
          #2 abs/usedCapacity = abs/usedCapacity from default partition
          #3 usedCapacity = total-used-resource / sum-of-configured-resource-for-all-partitions, absUsedCapacity=total-used-resource / sum-of-all-partition-resources.

          Considering issue pointed by Naga, I also prefer #2 since it looks simpler and has best compatibility once we have per-partition-metrics. #3 is also a reasonable solution to me however it needs more computation and changes semantics of (abs)usedCapacity.

          Thoughts?

          Show
          leftnoteasy Wangda Tan added a comment - Jason Lowe / Benson Qiu , Sorry for jumping in late, since we need to squash multi vals to a single field, I don't think we can find a perfect solution here. Few options: #1 Behavior of this patch, usedCapacity = max(label, usedCapacity), absUsedCapacity = max(label, absUsedCapacity); #2 abs/usedCapacity = abs/usedCapacity from default partition #3 usedCapacity = total-used-resource / sum-of-configured-resource-for-all-partitions, absUsedCapacity=total-used-resource / sum-of-all-partition-resources. Considering issue pointed by Naga, I also prefer #2 since it looks simpler and has best compatibility once we have per-partition-metrics. #3 is also a reasonable solution to me however it needs more computation and changes semantics of (abs)usedCapacity. Thoughts?
          Hide
          Naganarasimha Naganarasimha G R added a comment -

          Hi Benson Qiu & Jason Lowe, Sorry for pitching in late!
          IIUC currently we are not considering partition while updating queue metrics like allocatedMB / allocatedVCores / pendingMB / pendingVCores / reservedMB / reservedVCores... So as Jason Lowe mentioned many of the metrics in QueueMetrics needs to consider partition. Or at least it should be always giving Default Partition's statistics (for better compatability).
          IMHO i would opt to show default partition used/absolute-used metrics than/ showing max of used/absolute-used among the partition for following reasons :

          1. it will not be in sync with other existing metrics captured (which also needs to be converted to provide only for default partition)
          2. we might show used metric of one partition and absolute-used of other partition, as we are considering the max.

          Agree with Jason Lowe the best solution is per partition stats but max across partition would loose the significance of its value and admin cannot deduce anything from it. Also it would be better to have QueueMetrics Per partition similar to what we have QueueMetrics for users currently, but just that all fields might not make sense for PartitionQueueMetrics.

          Show
          Naganarasimha Naganarasimha G R added a comment - Hi Benson Qiu & Jason Lowe , Sorry for pitching in late! IIUC currently we are not considering partition while updating queue metrics like allocatedMB / allocatedVCores / pendingMB / pendingVCores / reservedMB / reservedVCores... So as Jason Lowe mentioned many of the metrics in QueueMetrics needs to consider partition. Or at least it should be always giving Default Partition 's statistics (for better compatability). IMHO i would opt to show default partition used/absolute-used metrics than/ showing max of used/absolute-used among the partition for following reasons : it will not be in sync with other existing metrics captured (which also needs to be converted to provide only for default partition) we might show used metric of one partition and absolute-used of other partition, as we are considering the max. Agree with Jason Lowe the best solution is per partition stats but max across partition would loose the significance of its value and admin cannot deduce anything from it. Also it would be better to have QueueMetrics Per partition similar to what we have QueueMetrics for users currently, but just that all fields might not make sense for PartitionQueueMetrics.
          Hide
          jlowe Jason Lowe added a comment -

          This seems like a reasonable approach to take until the node label dimensions are added to the QueueMetrics, but it would be nice to hear the thoughts from Wangda Tan or Sunil G.

          I'll commit this by the end of this week if I don't hear any objections.

          Show
          jlowe Jason Lowe added a comment - This seems like a reasonable approach to take until the node label dimensions are added to the QueueMetrics, but it would be nice to hear the thoughts from Wangda Tan or Sunil G . I'll commit this by the end of this week if I don't hear any objections.
          Hide
          benson.qiu Benson Qiu added a comment -

          Jason Lowe: Thanks for your most recent review. If nobody has any objections, is it okay to proceed with this patch?

          Show
          benson.qiu Benson Qiu added a comment - Jason Lowe : Thanks for your most recent review. If nobody has any objections, is it okay to proceed with this patch?
          Hide
          jlowe Jason Lowe added a comment -

          Latest patch lgtm, with the caveat that I don't think we can really support used capacity and absolute used capacity in the queue metrics without having per-partition queue metrics. Using a max across partitions seems like a reasonable value to report given we're trying to squash multiple values into a single field.

          Wangda Tan do you have any concerns about using a max-across-partitions approach here? If not then I think this is ready to go.

          Show
          jlowe Jason Lowe added a comment - Latest patch lgtm, with the caveat that I don't think we can really support used capacity and absolute used capacity in the queue metrics without having per-partition queue metrics. Using a max across partitions seems like a reasonable value to report given we're trying to squash multiple values into a single field. Wangda Tan do you have any concerns about using a max-across-partitions approach here? If not then I think this is ready to go.
          Hide
          benson.qiu Benson Qiu added a comment -

          Jason Lowe, Sunil Govind, Wangda Tan: Mind taking a look at the latest patch?

          Show
          benson.qiu Benson Qiu added a comment - Jason Lowe , Sunil Govind , Wangda Tan : Mind taking a look at the latest patch?
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 18s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 3 new or modified test files.
          0 mvndep 0m 16s Maven dependency ordering for branch
          +1 mvninstall 13m 4s trunk passed
          +1 compile 11m 3s trunk passed
          +1 checkstyle 2m 35s trunk passed
          +1 mvnsite 2m 8s trunk passed
          +1 mvneclipse 0m 47s trunk passed
          +1 findbugs 2m 59s trunk passed
          +1 javadoc 1m 32s trunk passed
          0 mvndep 0m 18s Maven dependency ordering for patch
          +1 mvninstall 1m 26s the patch passed
          +1 compile 11m 43s the patch passed
          +1 javac 11m 43s the patch passed
          -0 checkstyle 2m 13s root: The patch generated 7 new + 268 unchanged - 4 fixed = 275 total (was 272)
          +1 mvnsite 2m 5s the patch passed
          +1 mvneclipse 0m 54s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 3m 22s the patch passed
          +1 javadoc 1m 37s the patch passed
          +1 unit 8m 41s hadoop-common in the patch passed.
          -1 unit 44m 30s hadoop-yarn-server-resourcemanager in the patch failed.
          +1 asflicense 0m 54s The patch does not generate ASF License warnings.
          137m 6s



          Reason Tests
          Failed junit tests hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue YARN-6195
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12858968/YARN-6195.003.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 7eeaa24dff8e 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 615ac09
          Default Java 1.8.0_121
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/15294/artifact/patchprocess/diff-checkstyle-root.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/15294/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/15294/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: .
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/15294/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 18s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 3 new or modified test files. 0 mvndep 0m 16s Maven dependency ordering for branch +1 mvninstall 13m 4s trunk passed +1 compile 11m 3s trunk passed +1 checkstyle 2m 35s trunk passed +1 mvnsite 2m 8s trunk passed +1 mvneclipse 0m 47s trunk passed +1 findbugs 2m 59s trunk passed +1 javadoc 1m 32s trunk passed 0 mvndep 0m 18s Maven dependency ordering for patch +1 mvninstall 1m 26s the patch passed +1 compile 11m 43s the patch passed +1 javac 11m 43s the patch passed -0 checkstyle 2m 13s root: The patch generated 7 new + 268 unchanged - 4 fixed = 275 total (was 272) +1 mvnsite 2m 5s the patch passed +1 mvneclipse 0m 54s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 3m 22s the patch passed +1 javadoc 1m 37s the patch passed +1 unit 8m 41s hadoop-common in the patch passed. -1 unit 44m 30s hadoop-yarn-server-resourcemanager in the patch failed. +1 asflicense 0m 54s The patch does not generate ASF License warnings. 137m 6s Reason Tests Failed junit tests hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue YARN-6195 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12858968/YARN-6195.003.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 7eeaa24dff8e 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 615ac09 Default Java 1.8.0_121 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/15294/artifact/patchprocess/diff-checkstyle-root.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/15294/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/15294/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: . Console output https://builds.apache.org/job/PreCommit-YARN-Build/15294/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          benson.qiu Benson Qiu added a comment - - edited

          Wangda Tan, Sunil G: Here's the approach I have right now, based on Jason Lowe's comments. Would appreciate any thoughts.

          CSQueueUtils#updateUsedCapacity
              Set<String> nodeLabels = childQueue.getNodeLabelsForQueue();
          
              // {some code omitted}
          
              // QueueMetrics does not support per-label capacities,
              // so we report the maximum across all labels.
              for (String nodeLabel : nodeLabels) {
                usedCapacity =
                    Math.max(queueCapacities.getUsedCapacity(nodeLabel), usedCapacity);
                absoluteUsedCapacity =
                    Math.max(queueCapacities.getAbsoluteUsedCapacity(nodeLabel),
                        absoluteUsedCapacity);
              }
              queueMetrics.setUsedCapacity(usedCapacity);
              queueMetrics.setAbsoluteUsedCapacity(absoluteUsedCapacity);
          
          Show
          benson.qiu Benson Qiu added a comment - - edited Wangda Tan , Sunil G : Here's the approach I have right now, based on Jason Lowe 's comments. Would appreciate any thoughts. CSQueueUtils#updateUsedCapacity Set< String > nodeLabels = childQueue.getNodeLabelsForQueue(); // {some code omitted} // QueueMetrics does not support per-label capacities, // so we report the maximum across all labels. for ( String nodeLabel : nodeLabels) { usedCapacity = Math .max(queueCapacities.getUsedCapacity(nodeLabel), usedCapacity); absoluteUsedCapacity = Math .max(queueCapacities.getAbsoluteUsedCapacity(nodeLabel), absoluteUsedCapacity); } queueMetrics.setUsedCapacity(usedCapacity); queueMetrics.setAbsoluteUsedCapacity(absoluteUsedCapacity);
          Hide
          jlowe Jason Lowe added a comment -

          Pinging Wangda Tan and Sunil G to see if there's an opinion on how best to accommodate used capacity metrics before the metrics are split along label partitions.

          Show
          jlowe Jason Lowe added a comment - Pinging Wangda Tan and Sunil G to see if there's an opinion on how best to accommodate used capacity metrics before the metrics are split along label partitions.
          Hide
          jlowe Jason Lowe added a comment -

          Thanks for updating the patch!

          I discovered the queue metrics are doing something somewhat sane for labels before this change, see YARN-4484. We probably should do something similar, reporting the maximum used% like available is reporting the maximum available.

          The findbugs complaint is valid and needs to be fixed.

          Nit: We could make the CSQueueUtils call even simpler by retrieving the minimum allocation and queue usage from the queue rather than passing them separately.

          Show
          jlowe Jason Lowe added a comment - Thanks for updating the patch! I discovered the queue metrics are doing something somewhat sane for labels before this change, see YARN-4484 . We probably should do something similar, reporting the maximum used% like available is reporting the maximum available. The findbugs complaint is valid and needs to be fixed. Nit: We could make the CSQueueUtils call even simpler by retrieving the minimum allocation and queue usage from the queue rather than passing them separately.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 13s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 3 new or modified test files.
          0 mvndep 0m 14s Maven dependency ordering for branch
          +1 mvninstall 12m 30s trunk passed
          -1 compile 7m 0s root in trunk failed.
          +1 checkstyle 2m 2s trunk passed
          +1 mvnsite 1m 53s trunk passed
          +1 mvneclipse 0m 50s trunk passed
          +1 findbugs 2m 44s trunk passed
          +1 javadoc 1m 30s trunk passed
          0 mvndep 0m 16s Maven dependency ordering for patch
          +1 mvninstall 1m 11s the patch passed
          -1 compile 6m 35s root in the patch failed.
          -1 javac 6m 35s root in the patch failed.
          -0 checkstyle 2m 5s root: The patch generated 8 new + 197 unchanged - 3 fixed = 205 total (was 200)
          +1 mvnsite 1m 53s the patch passed
          +1 mvneclipse 0m 55s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          -1 findbugs 1m 22s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
          +1 javadoc 1m 33s the patch passed
          +1 unit 8m 47s hadoop-common in the patch passed.
          -1 unit 39m 54s hadoop-yarn-server-resourcemanager in the patch failed.
          +1 asflicense 0m 48s The patch does not generate ASF License warnings.
          121m 7s



          Reason Tests
          FindBugs module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
            Comparison of String parameter using == or != in org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CSQueueUtils.updateUsedCapacity(ResourceCalculator, Resource, Resource, ResourceUsage, String, AbstractCSQueue) At CSQueueUtils.java:== or != in org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CSQueueUtils.updateUsedCapacity(ResourceCalculator, Resource, Resource, ResourceUsage, String, AbstractCSQueue) At CSQueueUtils.java:[line 232]
          Failed junit tests hadoop.yarn.server.resourcemanager.TestRMRestart



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue YARN-6195
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12857580/YARN-6195.002.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux fba4c44ceb2c 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 04a5f5a
          Default Java 1.8.0_121
          compile https://builds.apache.org/job/PreCommit-YARN-Build/15242/artifact/patchprocess/branch-compile-root.txt
          findbugs v3.0.0
          compile https://builds.apache.org/job/PreCommit-YARN-Build/15242/artifact/patchprocess/patch-compile-root.txt
          javac https://builds.apache.org/job/PreCommit-YARN-Build/15242/artifact/patchprocess/patch-compile-root.txt
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/15242/artifact/patchprocess/diff-checkstyle-root.txt
          findbugs https://builds.apache.org/job/PreCommit-YARN-Build/15242/artifact/patchprocess/new-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.html
          unit https://builds.apache.org/job/PreCommit-YARN-Build/15242/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/15242/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: .
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/15242/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 13s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 3 new or modified test files. 0 mvndep 0m 14s Maven dependency ordering for branch +1 mvninstall 12m 30s trunk passed -1 compile 7m 0s root in trunk failed. +1 checkstyle 2m 2s trunk passed +1 mvnsite 1m 53s trunk passed +1 mvneclipse 0m 50s trunk passed +1 findbugs 2m 44s trunk passed +1 javadoc 1m 30s trunk passed 0 mvndep 0m 16s Maven dependency ordering for patch +1 mvninstall 1m 11s the patch passed -1 compile 6m 35s root in the patch failed. -1 javac 6m 35s root in the patch failed. -0 checkstyle 2m 5s root: The patch generated 8 new + 197 unchanged - 3 fixed = 205 total (was 200) +1 mvnsite 1m 53s the patch passed +1 mvneclipse 0m 55s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. -1 findbugs 1m 22s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) +1 javadoc 1m 33s the patch passed +1 unit 8m 47s hadoop-common in the patch passed. -1 unit 39m 54s hadoop-yarn-server-resourcemanager in the patch failed. +1 asflicense 0m 48s The patch does not generate ASF License warnings. 121m 7s Reason Tests FindBugs module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager   Comparison of String parameter using == or != in org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CSQueueUtils.updateUsedCapacity(ResourceCalculator, Resource, Resource, ResourceUsage, String, AbstractCSQueue) At CSQueueUtils.java:== or != in org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CSQueueUtils.updateUsedCapacity(ResourceCalculator, Resource, Resource, ResourceUsage, String, AbstractCSQueue) At CSQueueUtils.java: [line 232] Failed junit tests hadoop.yarn.server.resourcemanager.TestRMRestart Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue YARN-6195 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12857580/YARN-6195.002.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux fba4c44ceb2c 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 04a5f5a Default Java 1.8.0_121 compile https://builds.apache.org/job/PreCommit-YARN-Build/15242/artifact/patchprocess/branch-compile-root.txt findbugs v3.0.0 compile https://builds.apache.org/job/PreCommit-YARN-Build/15242/artifact/patchprocess/patch-compile-root.txt javac https://builds.apache.org/job/PreCommit-YARN-Build/15242/artifact/patchprocess/patch-compile-root.txt checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/15242/artifact/patchprocess/diff-checkstyle-root.txt findbugs https://builds.apache.org/job/PreCommit-YARN-Build/15242/artifact/patchprocess/new-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.html unit https://builds.apache.org/job/PreCommit-YARN-Build/15242/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/15242/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: . Console output https://builds.apache.org/job/PreCommit-YARN-Build/15242/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          jlowe Jason Lowe added a comment -

          What really needs to happen is a metrics-per-partition similar to how node labels were addressed in the scheduler web UI. In the short term I'd probably be OK with the metrics just reflecting the no label case until the per-label metrics work is done. Of course there needs to be TODO comments indicating it's not correct when labels are used.

          Looking at the patch closer I think the APIs need to be fixed. Before this patch, CSQueue's setUsedCapacity and setAbsoluteUsedCapacity methods were never called. That makes sense, given that the used capacity of a queue is not something an external entity should be telling the queue. The used capacity naturally falls out of what the queue is doing internally via container allocations and releases, and not because some external entity tells it the used capacity should be X%. I realize it was just a hack to get to the queue's metrics, but it's confusing at best. Given the interfaces aren't used, we should rip those out and eliminate that confusion rather than build on it.

          Instead of passing the queue to the CSQueueUtils updating method, I think we can simply pass the queue metrics instead. Then that can update both the QueueCapacities and the QueueMetrics in the same update. The metrics will still be broken for labels as they are today, but we can add a TODO and file a JIRA to fix that going forward. Actually instead of passing the QueueMetrics as an additional parameter, we could pass the CSQueue instead of the QueueCapacities argument and retrieve both the capacities and metrics objects via trivial accessors on the queue. That's what other CSQueueUtils methods are already doing, e.g.: updateQueueStatistics.

          Show
          jlowe Jason Lowe added a comment - What really needs to happen is a metrics-per-partition similar to how node labels were addressed in the scheduler web UI. In the short term I'd probably be OK with the metrics just reflecting the no label case until the per-label metrics work is done. Of course there needs to be TODO comments indicating it's not correct when labels are used. Looking at the patch closer I think the APIs need to be fixed. Before this patch, CSQueue's setUsedCapacity and setAbsoluteUsedCapacity methods were never called. That makes sense, given that the used capacity of a queue is not something an external entity should be telling the queue. The used capacity naturally falls out of what the queue is doing internally via container allocations and releases, and not because some external entity tells it the used capacity should be X%. I realize it was just a hack to get to the queue's metrics, but it's confusing at best. Given the interfaces aren't used, we should rip those out and eliminate that confusion rather than build on it. Instead of passing the queue to the CSQueueUtils updating method, I think we can simply pass the queue metrics instead. Then that can update both the QueueCapacities and the QueueMetrics in the same update. The metrics will still be broken for labels as they are today, but we can add a TODO and file a JIRA to fix that going forward. Actually instead of passing the QueueMetrics as an additional parameter, we could pass the CSQueue instead of the QueueCapacities argument and retrieve both the capacities and metrics objects via trivial accessors on the queue. That's what other CSQueueUtils methods are already doing, e.g.: updateQueueStatistics.
          Hide
          benson.qiu Benson Qiu added a comment -

          Thanks for the comments Jason Lowe!

          The current JMX QueueMetrics don't seem to take into account node labels - can we do the same for usedCapacity and absoluteUsedCapacity?

          The usedCapacity case would look like this:

          AbstractCSQueue.java
            @Override
            public void setUsedCapacity(float usedCapacity) {
              queueCapacities.setUsedCapacity(usedCapacity);
              metrics.setUsedCapacity(usedCapacity);
            }
            
            @Override
            public void setUsedCapacity(String label, float usedCapacity) {
              queueCapacities.setUsedCapacity(label, usedCapacity);
              if (label == null || label == CommonNodeLabelsManager.NO_LABEL) {
                metrics.setUsedCapacity(usedCapacity);
              }
            }
          
          Show
          benson.qiu Benson Qiu added a comment - Thanks for the comments Jason Lowe ! The current JMX QueueMetrics don't seem to take into account node labels - can we do the same for usedCapacity and absoluteUsedCapacity? The usedCapacity case would look like this: AbstractCSQueue.java @Override public void setUsedCapacity( float usedCapacity) { queueCapacities.setUsedCapacity(usedCapacity); metrics.setUsedCapacity(usedCapacity); } @Override public void setUsedCapacity( String label, float usedCapacity) { queueCapacities.setUsedCapacity(label, usedCapacity); if (label == null || label == CommonNodeLabelsManager.NO_LABEL) { metrics.setUsedCapacity(usedCapacity); } }
          Hide
          jlowe Jason Lowe added a comment -

          Thanks for the patch, Benson Qiu!

          I'm confused about how labels are supposed to interact with the JMX metrics. There's only one metric for used capacity in a queue even if there are labels? If the used proportion of a capacity is updated, it updates the same metric as for other labels or no label? The metric will always reflect the last update given at the time the metric is examined, meaning there could be wildly different results from reading to reading even if things aren't moving that much on the cluster in reality, e.g.: one reading gets the "fizgig" node label's usage while another gets the default/no label usage.

          MetricsRegistry.java: "mutable long float" and "mutable float integer" should both be "mutable float"

          The tabs flagged by the whitespace check need to be removed.

          Show
          jlowe Jason Lowe added a comment - Thanks for the patch, Benson Qiu ! I'm confused about how labels are supposed to interact with the JMX metrics. There's only one metric for used capacity in a queue even if there are labels? If the used proportion of a capacity is updated, it updates the same metric as for other labels or no label? The metric will always reflect the last update given at the time the metric is examined, meaning there could be wildly different results from reading to reading even if things aren't moving that much on the cluster in reality, e.g.: one reading gets the "fizgig" node label's usage while another gets the default/no label usage. MetricsRegistry.java: "mutable long float" and "mutable float integer" should both be "mutable float" The tabs flagged by the whitespace check need to be removed.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 18s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 3 new or modified test files.
          0 mvndep 0m 14s Maven dependency ordering for branch
          +1 mvninstall 13m 6s trunk passed
          +1 compile 13m 14s trunk passed
          +1 checkstyle 1m 56s trunk passed
          +1 mvnsite 1m 42s trunk passed
          +1 mvneclipse 0m 39s trunk passed
          +1 findbugs 2m 41s trunk passed
          +1 javadoc 1m 17s trunk passed
          0 mvndep 0m 16s Maven dependency ordering for patch
          +1 mvninstall 1m 12s the patch passed
          +1 compile 11m 30s the patch passed
          +1 javac 11m 30s the patch passed
          +1 checkstyle 2m 4s the patch passed
          +1 mvnsite 1m 55s the patch passed
          +1 mvneclipse 0m 46s the patch passed
          -1 whitespace 0m 0s The patch 8 line(s) with tabs.
          +1 findbugs 3m 24s the patch passed
          +1 javadoc 1m 24s the patch passed
          +1 unit 8m 46s hadoop-common in the patch passed.
          -1 unit 44m 47s hadoop-yarn-server-resourcemanager in the patch failed.
          +1 asflicense 0m 37s The patch does not generate ASF License warnings.
          136m 23s



          Reason Tests
          Failed junit tests hadoop.yarn.server.resourcemanager.TestRMRestart
            hadoop.yarn.server.resourcemanager.scheduler.fair.TestFSAppStarvation



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue YARN-6195
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12852846/YARN-6195.001.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 004c679b35af 3.13.0-105-generic #152-Ubuntu SMP Fri Dec 2 15:37:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 0fc6f38
          Default Java 1.8.0_121
          findbugs v3.0.0
          whitespace https://builds.apache.org/job/PreCommit-YARN-Build/14961/artifact/patchprocess/whitespace-tabs.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/14961/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/14961/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: .
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/14961/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 18s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 3 new or modified test files. 0 mvndep 0m 14s Maven dependency ordering for branch +1 mvninstall 13m 6s trunk passed +1 compile 13m 14s trunk passed +1 checkstyle 1m 56s trunk passed +1 mvnsite 1m 42s trunk passed +1 mvneclipse 0m 39s trunk passed +1 findbugs 2m 41s trunk passed +1 javadoc 1m 17s trunk passed 0 mvndep 0m 16s Maven dependency ordering for patch +1 mvninstall 1m 12s the patch passed +1 compile 11m 30s the patch passed +1 javac 11m 30s the patch passed +1 checkstyle 2m 4s the patch passed +1 mvnsite 1m 55s the patch passed +1 mvneclipse 0m 46s the patch passed -1 whitespace 0m 0s The patch 8 line(s) with tabs. +1 findbugs 3m 24s the patch passed +1 javadoc 1m 24s the patch passed +1 unit 8m 46s hadoop-common in the patch passed. -1 unit 44m 47s hadoop-yarn-server-resourcemanager in the patch failed. +1 asflicense 0m 37s The patch does not generate ASF License warnings. 136m 23s Reason Tests Failed junit tests hadoop.yarn.server.resourcemanager.TestRMRestart   hadoop.yarn.server.resourcemanager.scheduler.fair.TestFSAppStarvation Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue YARN-6195 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12852846/YARN-6195.001.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 004c679b35af 3.13.0-105-generic #152-Ubuntu SMP Fri Dec 2 15:37:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 0fc6f38 Default Java 1.8.0_121 findbugs v3.0.0 whitespace https://builds.apache.org/job/PreCommit-YARN-Build/14961/artifact/patchprocess/whitespace-tabs.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/14961/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/14961/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: . Console output https://builds.apache.org/job/PreCommit-YARN-Build/14961/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.

            People

            • Assignee:
              benson.qiu Benson Qiu
              Reporter:
              benson.qiu Benson Qiu
            • Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development