Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 2.7.1
    • 2.8.0, 3.0.0-alpha1
    • None
    • None
    • Reviewed

    Description

      Currently when label-queue mappings are not available, there are few assumptions taken in UI and in metrics.

      In above case where labels are enabled and available in cluster but without any queue mappings, UI displays queues under labels. This is not correct.
      Currently labels enabled check and availability of labels are considered to render scheduler UI. Henceforth we also need to check whether

      • queue-mappings are available
      • nodes are mapped with labels with proper exclusivity flags on

      This ticket also will try to see the default configurations in queue when labels are not mapped.

      Attachments

        1. 0005-YARN-4634.patch
          2 kB
          Sunil G
        2. 0004-YARN-4634.patch
          2 kB
          Sunil G
        3. 0003-YARN-4634.patch
          2 kB
          Sunil G
        4. 0002-YARN-4634.patch
          8 kB
          Sunil G
        5. 0001-YARN-4634.patch
          4 kB
          Sunil G

        Activity

          andrew.wang Andrew Wang made changes -
          Fix Version/s 3.0.0-alpha1 [ 12335735 ]
          sunilg Sunil G added a comment -

          Thank you very much for the review and commit Wangda Tan!

          sunilg Sunil G added a comment - Thank you very much for the review and commit Wangda Tan !
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #9537 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9537/)
          YARN-4634. Scheduler UI/Metrics need to consider cases like non-queue (wangda: rev 12b11e2e688158404feeb3ded37eb6cccad4ea5c)

          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/CapacitySchedulerPage.java
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #9537 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9537/ ) YARN-4634 . Scheduler UI/Metrics need to consider cases like non-queue (wangda: rev 12b11e2e688158404feeb3ded37eb6cccad4ea5c) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/CapacitySchedulerPage.java
          leftnoteasy Wangda Tan made changes -
          Fix Version/s 2.8.0 [ 12329059 ]
          Hadoop Flags Reviewed [ 10343 ]
          Resolution Fixed [ 1 ]
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          leftnoteasy Wangda Tan added a comment -

          Committed to trunk/branch-2/branch-2.8,

          Thanks Sunil G.

          leftnoteasy Wangda Tan added a comment - Committed to trunk/branch-2/branch-2.8, Thanks Sunil G .
          sunilg Sunil G added a comment -

          Test case failures are known. Wangda Tan Could you pls take a look as jenkins passed.

          sunilg Sunil G added a comment - Test case failures are known. Wangda Tan Could you pls take a look as jenkins passed.
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 11s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 mvninstall 6m 35s trunk passed
          +1 compile 0m 26s trunk passed with JDK v1.8.0_74
          +1 compile 0m 28s trunk passed with JDK v1.7.0_95
          +1 checkstyle 0m 17s trunk passed
          +1 mvnsite 0m 34s trunk passed
          +1 mvneclipse 0m 15s trunk passed
          +1 findbugs 1m 5s trunk passed
          +1 javadoc 0m 21s trunk passed with JDK v1.8.0_74
          +1 javadoc 0m 27s trunk passed with JDK v1.7.0_95
          +1 mvninstall 0m 29s the patch passed
          +1 compile 0m 24s the patch passed with JDK v1.8.0_74
          +1 javac 0m 24s the patch passed
          +1 compile 0m 25s the patch passed with JDK v1.7.0_95
          +1 javac 0m 25s the patch passed
          +1 checkstyle 0m 15s the patch passed
          +1 mvnsite 0m 32s the patch passed
          +1 mvneclipse 0m 11s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 1m 16s the patch passed
          +1 javadoc 0m 19s the patch passed with JDK v1.8.0_74
          +1 javadoc 0m 24s the patch passed with JDK v1.7.0_95
          -1 unit 66m 23s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.8.0_74.
          -1 unit 67m 41s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.7.0_95.
          +1 asflicense 0m 16s Patch does not generate ASF License warnings.
          150m 10s



          Reason Tests
          JDK v1.8.0_74 Failed junit tests hadoop.yarn.server.resourcemanager.TestAMAuthorization
            hadoop.yarn.server.resourcemanager.TestClientRMTokens
          JDK v1.7.0_95 Failed junit tests hadoop.yarn.server.resourcemanager.TestAMAuthorization
            hadoop.yarn.server.resourcemanager.TestClientRMTokens



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12792296/0005-YARN-4634.patch
          JIRA Issue YARN-4634
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 798120fc6e08 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 2e040d3
          Default Java 1.7.0_95
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_74 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-YARN-Build/10741/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_74.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/10741/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_95.txt
          unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/10741/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_74.txt https://builds.apache.org/job/PreCommit-YARN-Build/10741/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_95.txt
          JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/10741/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/10741/console
          Powered by Apache Yetus 0.2.0 http://yetus.apache.org

          This message was automatically generated.

          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 11s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 6m 35s trunk passed +1 compile 0m 26s trunk passed with JDK v1.8.0_74 +1 compile 0m 28s trunk passed with JDK v1.7.0_95 +1 checkstyle 0m 17s trunk passed +1 mvnsite 0m 34s trunk passed +1 mvneclipse 0m 15s trunk passed +1 findbugs 1m 5s trunk passed +1 javadoc 0m 21s trunk passed with JDK v1.8.0_74 +1 javadoc 0m 27s trunk passed with JDK v1.7.0_95 +1 mvninstall 0m 29s the patch passed +1 compile 0m 24s the patch passed with JDK v1.8.0_74 +1 javac 0m 24s the patch passed +1 compile 0m 25s the patch passed with JDK v1.7.0_95 +1 javac 0m 25s the patch passed +1 checkstyle 0m 15s the patch passed +1 mvnsite 0m 32s the patch passed +1 mvneclipse 0m 11s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 1m 16s the patch passed +1 javadoc 0m 19s the patch passed with JDK v1.8.0_74 +1 javadoc 0m 24s the patch passed with JDK v1.7.0_95 -1 unit 66m 23s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.8.0_74. -1 unit 67m 41s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.7.0_95. +1 asflicense 0m 16s Patch does not generate ASF License warnings. 150m 10s Reason Tests JDK v1.8.0_74 Failed junit tests hadoop.yarn.server.resourcemanager.TestAMAuthorization   hadoop.yarn.server.resourcemanager.TestClientRMTokens JDK v1.7.0_95 Failed junit tests hadoop.yarn.server.resourcemanager.TestAMAuthorization   hadoop.yarn.server.resourcemanager.TestClientRMTokens Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12792296/0005-YARN-4634.patch JIRA Issue YARN-4634 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 798120fc6e08 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 2e040d3 Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_74 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-YARN-Build/10741/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_74.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/10741/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_95.txt unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/10741/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_74.txt https://builds.apache.org/job/PreCommit-YARN-Build/10741/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_95.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/10741/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/10741/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
          leftnoteasy Wangda Tan added a comment -

          Looks good, +1.

          leftnoteasy Wangda Tan added a comment - Looks good, +1.
          sunilg Sunil G made changes -
          Attachment 0005-YARN-4634.patch [ 12792296 ]
          sunilg Sunil G added a comment -

          Attaching an updated patch as discussed. Wangda Tan, could you please take a look.

          sunilg Sunil G added a comment - Attaching an updated patch as discussed. Wangda Tan , could you please take a look.
          sunilg Sunil G added a comment -

          Yes. This looks fine. Bcz all other cases will be handle with this. I ll update the patch.

          sunilg Sunil G added a comment - Yes. This looks fine. Bcz all other cases will be handle with this. I ll update the patch.
          leftnoteasy Wangda Tan added a comment -

          Sunil G,

          It seems:

                   if (null == nodeLabelsInfo || (nodeLabelsInfo.size() == 1
          +            && nodeLabelsInfo.get(0).getLabelName().isEmpty())
          +            || !hasAnyLabelLinkedToNM) {
          

          Can be simplified to:

          if (!hasAnyLabelLinkedToNM) ...
          

          Correct?

          leftnoteasy Wangda Tan added a comment - Sunil G , It seems: if ( null == nodeLabelsInfo || (nodeLabelsInfo.size() == 1 + && nodeLabelsInfo.get(0).getLabelName().isEmpty()) + || !hasAnyLabelLinkedToNM) { Can be simplified to: if (!hasAnyLabelLinkedToNM) ... Correct?
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 16s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 mvninstall 6m 42s trunk passed
          +1 compile 0m 28s trunk passed with JDK v1.8.0_72
          +1 compile 0m 29s trunk passed with JDK v1.7.0_95
          +1 checkstyle 0m 18s trunk passed
          +1 mvnsite 0m 34s trunk passed
          +1 mvneclipse 0m 15s trunk passed
          +1 findbugs 1m 6s trunk passed
          +1 javadoc 0m 21s trunk passed with JDK v1.8.0_72
          +1 javadoc 0m 26s trunk passed with JDK v1.7.0_95
          +1 mvninstall 0m 30s the patch passed
          +1 compile 0m 23s the patch passed with JDK v1.8.0_72
          +1 javac 0m 23s the patch passed
          +1 compile 0m 27s the patch passed with JDK v1.7.0_95
          +1 javac 0m 27s the patch passed
          +1 checkstyle 0m 15s the patch passed
          +1 mvnsite 0m 32s the patch passed
          +1 mvneclipse 0m 12s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 1m 15s the patch passed
          +1 javadoc 0m 19s the patch passed with JDK v1.8.0_72
          +1 javadoc 0m 24s the patch passed with JDK v1.7.0_95
          -1 unit 70m 25s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.8.0_72.
          -1 unit 71m 54s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.7.0_95.
          +1 asflicense 0m 18s Patch does not generate ASF License warnings.
          158m 47s



          Reason Tests
          JDK v1.8.0_72 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens
            hadoop.yarn.server.resourcemanager.TestAMAuthorization
          JDK v1.7.0_95 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens
            hadoop.yarn.server.resourcemanager.TestAMAuthorization



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12790948/0004-YARN-4634.patch
          JIRA Issue YARN-4634
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 26d69ab54967 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 27e0681
          Default Java 1.7.0_95
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_72 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-YARN-Build/10686/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_72.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/10686/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_95.txt
          unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/10686/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_72.txt https://builds.apache.org/job/PreCommit-YARN-Build/10686/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_95.txt
          JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/10686/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/10686/console
          Powered by Apache Yetus 0.3.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 16s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 6m 42s trunk passed +1 compile 0m 28s trunk passed with JDK v1.8.0_72 +1 compile 0m 29s trunk passed with JDK v1.7.0_95 +1 checkstyle 0m 18s trunk passed +1 mvnsite 0m 34s trunk passed +1 mvneclipse 0m 15s trunk passed +1 findbugs 1m 6s trunk passed +1 javadoc 0m 21s trunk passed with JDK v1.8.0_72 +1 javadoc 0m 26s trunk passed with JDK v1.7.0_95 +1 mvninstall 0m 30s the patch passed +1 compile 0m 23s the patch passed with JDK v1.8.0_72 +1 javac 0m 23s the patch passed +1 compile 0m 27s the patch passed with JDK v1.7.0_95 +1 javac 0m 27s the patch passed +1 checkstyle 0m 15s the patch passed +1 mvnsite 0m 32s the patch passed +1 mvneclipse 0m 12s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 1m 15s the patch passed +1 javadoc 0m 19s the patch passed with JDK v1.8.0_72 +1 javadoc 0m 24s the patch passed with JDK v1.7.0_95 -1 unit 70m 25s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.8.0_72. -1 unit 71m 54s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.7.0_95. +1 asflicense 0m 18s Patch does not generate ASF License warnings. 158m 47s Reason Tests JDK v1.8.0_72 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens   hadoop.yarn.server.resourcemanager.TestAMAuthorization JDK v1.7.0_95 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens   hadoop.yarn.server.resourcemanager.TestAMAuthorization Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12790948/0004-YARN-4634.patch JIRA Issue YARN-4634 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 26d69ab54967 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 27e0681 Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_72 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-YARN-Build/10686/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_72.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/10686/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_95.txt unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/10686/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_72.txt https://builds.apache.org/job/PreCommit-YARN-Build/10686/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_95.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/10686/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/10686/console Powered by Apache Yetus 0.3.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          sunilg Sunil G made changes -
          Attachment 0004-YARN-4634.patch [ 12790948 ]
          sunilg Sunil G added a comment -

          Updating patch correcting findbugs warning.

          sunilg Sunil G added a comment - Updating patch correcting findbugs warning.
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 18s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 mvninstall 6m 57s trunk passed
          +1 compile 0m 26s trunk passed with JDK v1.8.0_72
          +1 compile 0m 28s trunk passed with JDK v1.7.0_95
          +1 checkstyle 0m 18s trunk passed
          +1 mvnsite 0m 34s trunk passed
          +1 mvneclipse 0m 15s trunk passed
          +1 findbugs 1m 4s trunk passed
          +1 javadoc 0m 20s trunk passed with JDK v1.8.0_72
          +1 javadoc 0m 26s trunk passed with JDK v1.7.0_95
          +1 mvninstall 0m 29s the patch passed
          +1 compile 0m 25s the patch passed with JDK v1.8.0_72
          +1 javac 0m 25s the patch passed
          +1 compile 0m 26s the patch passed with JDK v1.7.0_95
          +1 javac 0m 26s the patch passed
          -1 checkstyle 0m 16s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: patch generated 1 new + 55 unchanged - 0 fixed = 56 total (was 55)
          +1 mvnsite 0m 32s the patch passed
          +1 mvneclipse 0m 12s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          -1 findbugs 1m 20s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
          +1 javadoc 0m 22s the patch passed with JDK v1.8.0_72
          +1 javadoc 0m 24s the patch passed with JDK v1.7.0_95
          -1 unit 71m 4s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.8.0_72.
          -1 unit 72m 1s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.7.0_95.
          +1 asflicense 0m 19s Patch does not generate ASF License warnings.
          159m 58s



          Reason Tests
          FindBugs module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
            Nullcheck of CapacitySchedulerPage$QueuesBlock.nodeLabelsInfo at line 419 of value previously dereferenced in org.apache.hadoop.yarn.server.resourcemanager.webapp.CapacitySchedulerPage$QueuesBlock.render(HtmlBlock$Block) At CapacitySchedulerPage.java:419 of value previously dereferenced in org.apache.hadoop.yarn.server.resourcemanager.webapp.CapacitySchedulerPage$QueuesBlock.render(HtmlBlock$Block) At CapacitySchedulerPage.java:[line 409]
          JDK v1.8.0_72 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens
            hadoop.yarn.server.resourcemanager.TestAMAuthorization
          JDK v1.7.0_95 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens
            hadoop.yarn.server.resourcemanager.TestAMAuthorization



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12790754/0003-YARN-4634.patch
          JIRA Issue YARN-4634
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux d7cbcceaf109 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 44d9bac
          Default Java 1.7.0_95
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_72 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/10679/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
          findbugs https://builds.apache.org/job/PreCommit-YARN-Build/10679/artifact/patchprocess/new-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.html
          unit https://builds.apache.org/job/PreCommit-YARN-Build/10679/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_72.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/10679/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_95.txt
          unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/10679/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_72.txt https://builds.apache.org/job/PreCommit-YARN-Build/10679/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_95.txt
          JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/10679/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/10679/console
          Powered by Apache Yetus 0.3.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 18s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 6m 57s trunk passed +1 compile 0m 26s trunk passed with JDK v1.8.0_72 +1 compile 0m 28s trunk passed with JDK v1.7.0_95 +1 checkstyle 0m 18s trunk passed +1 mvnsite 0m 34s trunk passed +1 mvneclipse 0m 15s trunk passed +1 findbugs 1m 4s trunk passed +1 javadoc 0m 20s trunk passed with JDK v1.8.0_72 +1 javadoc 0m 26s trunk passed with JDK v1.7.0_95 +1 mvninstall 0m 29s the patch passed +1 compile 0m 25s the patch passed with JDK v1.8.0_72 +1 javac 0m 25s the patch passed +1 compile 0m 26s the patch passed with JDK v1.7.0_95 +1 javac 0m 26s the patch passed -1 checkstyle 0m 16s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: patch generated 1 new + 55 unchanged - 0 fixed = 56 total (was 55) +1 mvnsite 0m 32s the patch passed +1 mvneclipse 0m 12s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. -1 findbugs 1m 20s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) +1 javadoc 0m 22s the patch passed with JDK v1.8.0_72 +1 javadoc 0m 24s the patch passed with JDK v1.7.0_95 -1 unit 71m 4s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.8.0_72. -1 unit 72m 1s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.7.0_95. +1 asflicense 0m 19s Patch does not generate ASF License warnings. 159m 58s Reason Tests FindBugs module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager   Nullcheck of CapacitySchedulerPage$QueuesBlock.nodeLabelsInfo at line 419 of value previously dereferenced in org.apache.hadoop.yarn.server.resourcemanager.webapp.CapacitySchedulerPage$QueuesBlock.render(HtmlBlock$Block) At CapacitySchedulerPage.java:419 of value previously dereferenced in org.apache.hadoop.yarn.server.resourcemanager.webapp.CapacitySchedulerPage$QueuesBlock.render(HtmlBlock$Block) At CapacitySchedulerPage.java: [line 409] JDK v1.8.0_72 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens   hadoop.yarn.server.resourcemanager.TestAMAuthorization JDK v1.7.0_95 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens   hadoop.yarn.server.resourcemanager.TestAMAuthorization Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12790754/0003-YARN-4634.patch JIRA Issue YARN-4634 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux d7cbcceaf109 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 44d9bac Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_72 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/10679/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt findbugs https://builds.apache.org/job/PreCommit-YARN-Build/10679/artifact/patchprocess/new-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.html unit https://builds.apache.org/job/PreCommit-YARN-Build/10679/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_72.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/10679/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_95.txt unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/10679/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_72.txt https://builds.apache.org/job/PreCommit-YARN-Build/10679/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_95.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/10679/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/10679/console Powered by Apache Yetus 0.3.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          sunilg Sunil G made changes -
          Attachment 0003-YARN-4634.patch [ 12790754 ]
          sunilg Sunil G added a comment -

          Updating patch as per the above mentioned comments.

          sunilg Sunil G added a comment - Updating patch as per the above mentioned comments.
          leftnoteasy Wangda Tan added a comment -

          Sunil G, sounds good.

          leftnoteasy Wangda Tan added a comment - Sunil G , sounds good.
          sunilg Sunil G added a comment -

          Thanks Wangda Tan for the comments.

          I agree that we are considering too many variables to decide upon whether to render with labels or not in earlier patch. Yes, it adds the complexity. Meantime, current system has some corner cases where label-queue mappings are not present too. So i think such cases can be handled by assuming that we render UI with labels.

          So I will try to consolidate the idea here, if cluster has labels other than DEFAULT_LABEL and at least one such label has >0 active NMs (other than DEFAULT_LABEL), then we will render UI with labels. Is this fine?

          sunilg Sunil G added a comment - Thanks Wangda Tan for the comments. I agree that we are considering too many variables to decide upon whether to render with labels or not in earlier patch. Yes, it adds the complexity. Meantime, current system has some corner cases where label-queue mappings are not present too. So i think such cases can be handled by assuming that we render UI with labels. So I will try to consolidate the idea here, if cluster has labels other than DEFAULT_LABEL and at least one such label has >0 active NMs (other than DEFAULT_LABEL), then we will render UI with labels. Is this fine?
          leftnoteasy Wangda Tan added a comment -

          Hi Sunil G,

          Thanks for updating, however, I will hesitate to add new state labelToQueueMappingAvailable to RMNodeLabelsManager and AbstractCSQueue needs to update that state as well.

          I suggest to check show labels hierarchy only if:

          • There's label other than DEFAULT_LABEL has >0 activeNMs.

          And let's keep queues page as simple as possible.

          You can check RMNodeLabelsManager#pullRMNodeLabelsInfo for details (And NodeLabelsPage as an example).

          Thoughts?

          leftnoteasy Wangda Tan added a comment - Hi Sunil G , Thanks for updating, however, I will hesitate to add new state labelToQueueMappingAvailable to RMNodeLabelsManager and AbstractCSQueue needs to update that state as well. I suggest to check show labels hierarchy only if: There's label other than DEFAULT_LABEL has >0 activeNMs. And let's keep queues page as simple as possible. You can check RMNodeLabelsManager#pullRMNodeLabelsInfo for details (And NodeLabelsPage as an example). Thoughts?
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 10s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 mvninstall 6m 41s trunk passed
          +1 compile 0m 25s trunk passed with JDK v1.8.0_72
          +1 compile 0m 28s trunk passed with JDK v1.7.0_95
          +1 checkstyle 0m 20s trunk passed
          +1 mvnsite 0m 35s trunk passed
          +1 mvneclipse 0m 14s trunk passed
          +1 findbugs 1m 4s trunk passed
          +1 javadoc 0m 21s trunk passed with JDK v1.8.0_72
          +1 javadoc 0m 25s trunk passed with JDK v1.7.0_95
          +1 mvninstall 0m 29s the patch passed
          +1 compile 0m 22s the patch passed with JDK v1.8.0_72
          +1 javac 0m 22s the patch passed
          +1 compile 0m 26s the patch passed with JDK v1.7.0_95
          +1 javac 0m 26s the patch passed
          -1 checkstyle 0m 18s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: patch generated 2 new + 200 unchanged - 0 fixed = 202 total (was 200)
          +1 mvnsite 0m 31s the patch passed
          +1 mvneclipse 0m 12s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 1m 15s the patch passed
          +1 javadoc 0m 18s the patch passed with JDK v1.8.0_72
          +1 javadoc 0m 22s the patch passed with JDK v1.7.0_95
          -1 unit 66m 0s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.8.0_72.
          -1 unit 67m 11s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.7.0_95.
          +1 asflicense 0m 18s Patch does not generate ASF License warnings.
          149m 24s



          Reason Tests
          JDK v1.8.0_72 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens
            hadoop.yarn.server.resourcemanager.TestAMAuthorization
          JDK v1.7.0_95 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens
            hadoop.yarn.server.resourcemanager.TestAMAuthorization



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12789216/0002-YARN-4634.patch
          JIRA Issue YARN-4634
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 052f91d7d4e4 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 3369a4f
          Default Java 1.7.0_95
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_72 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/10620/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/10620/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_72.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/10620/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_95.txt
          unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/10620/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_72.txt https://builds.apache.org/job/PreCommit-YARN-Build/10620/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_95.txt
          JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/10620/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/10620/console
          Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 10s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 6m 41s trunk passed +1 compile 0m 25s trunk passed with JDK v1.8.0_72 +1 compile 0m 28s trunk passed with JDK v1.7.0_95 +1 checkstyle 0m 20s trunk passed +1 mvnsite 0m 35s trunk passed +1 mvneclipse 0m 14s trunk passed +1 findbugs 1m 4s trunk passed +1 javadoc 0m 21s trunk passed with JDK v1.8.0_72 +1 javadoc 0m 25s trunk passed with JDK v1.7.0_95 +1 mvninstall 0m 29s the patch passed +1 compile 0m 22s the patch passed with JDK v1.8.0_72 +1 javac 0m 22s the patch passed +1 compile 0m 26s the patch passed with JDK v1.7.0_95 +1 javac 0m 26s the patch passed -1 checkstyle 0m 18s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: patch generated 2 new + 200 unchanged - 0 fixed = 202 total (was 200) +1 mvnsite 0m 31s the patch passed +1 mvneclipse 0m 12s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 1m 15s the patch passed +1 javadoc 0m 18s the patch passed with JDK v1.8.0_72 +1 javadoc 0m 22s the patch passed with JDK v1.7.0_95 -1 unit 66m 0s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.8.0_72. -1 unit 67m 11s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.7.0_95. +1 asflicense 0m 18s Patch does not generate ASF License warnings. 149m 24s Reason Tests JDK v1.8.0_72 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens   hadoop.yarn.server.resourcemanager.TestAMAuthorization JDK v1.7.0_95 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens   hadoop.yarn.server.resourcemanager.TestAMAuthorization Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12789216/0002-YARN-4634.patch JIRA Issue YARN-4634 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 052f91d7d4e4 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 3369a4f Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_72 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/10620/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/10620/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_72.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/10620/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_95.txt unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/10620/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_72.txt https://builds.apache.org/job/PreCommit-YARN-Build/10620/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_95.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/10620/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/10620/console Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          sunilg Sunil G made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          sunilg Sunil G made changes -
          Attachment 0002-YARN-4634.patch [ 12789216 ]
          sunilg Sunil G added a comment -

          Thank you Wangda Tan

          Addressing this case in this new patch. I tested few cases as well

          Base configuration is:
          a. Labels are added in cluster
          b. Few nodes were configured with label.

          Now ,
          scenario 1.
          c. accessible labels for queues were kept default in cs xml
          d. no capacity configurations for labels in any queue.
          Here Scheduler page will show queues direct (It wont be under any labels)

          scenario 2.
          c. accessible labels for queues were kept default in cs xml
          d. capacity configurations were added for some labels in one queue.
          Here Scheduler page will show queues under any labels.
          This covers the scenario which you mentioned..

          scenario 3.
          c. accessible labels for queues were configured in cs xml

          Here Scheduler page will show queues under any labels as we have some accessible label configurations.

          One observation:
          QueueCapacities#clearConfigurableFields only resets the capacity for all its labels to 0 during reinitialize. So getExistingNodeLabels will still have this labels. Now assume we removed some label from a queue, and did reinitialize. Because of above mentioned behavior, we can get this removed label in getExistingNodeLabels. Eventhough capacity is 0 for this label, i think its not correct.If you feel its a pblm, i think we can handle in another ticket. Thoughts?

          sunilg Sunil G added a comment - Thank you Wangda Tan Addressing this case in this new patch. I tested few cases as well Base configuration is: a. Labels are added in cluster b. Few nodes were configured with label. Now , scenario 1. c. accessible labels for queues were kept default in cs xml d. no capacity configurations for labels in any queue. Here Scheduler page will show queues direct (It wont be under any labels) scenario 2. c. accessible labels for queues were kept default in cs xml d. capacity configurations were added for some labels in one queue. Here Scheduler page will show queues under any labels. This covers the scenario which you mentioned.. scenario 3. c. accessible labels for queues were configured in cs xml Here Scheduler page will show queues under any labels as we have some accessible label configurations. One observation : QueueCapacities#clearConfigurableFields only resets the capacity for all its labels to 0 during reinitialize. So getExistingNodeLabels will still have this labels. Now assume we removed some label from a queue, and did reinitialize. Because of above mentioned behavior, we can get this removed label in getExistingNodeLabels . Eventhough capacity is 0 for this label, i think its not correct.If you feel its a pblm, i think we can handle in another ticket. Thoughts?
          sunilg Sunil G added a comment -

          Yes Wangda Tan, Thanks for pointing out this case.
          I think we need to check whether any capacity configurations has happened in cluster for any labels also. Correct?

          sunilg Sunil G added a comment - Yes Wangda Tan , Thanks for pointing out this case. I think we need to check whether any capacity configurations has happened in cluster for any labels also. Correct?
          leftnoteasy Wangda Tan added a comment -

          Sunil G,
          Thanks for explanations. Does this handle following case?

          • All queues have access to *
          • Some labels added to cluster
          • Some queues' capacities other than default partition is set

          In your existing logic, the isLabelToQueueMappingAvailable will be false, and partition structure will not be shown. Correct?

          leftnoteasy Wangda Tan added a comment - Sunil G , Thanks for explanations. Does this handle following case? All queues have access to * Some labels added to cluster Some queues' capacities other than default partition is set In your existing logic, the isLabelToQueueMappingAvailable will be false, and partition structure will not be shown. Correct?
          sunilg Sunil G added a comment -

          Hi Wangda Tan
          Thank you for the clarification. Pls correct me if I am wrong, I think it is also handled.

          RMNodeLabelsManager#reinitializeQueueLabels is invoked with queueToLabels as arguement. Here queueToLabels is actually CS#getQueueToLabels(). And this getQueueToLabels() will be collecting the labels configured per queue from queue.getAccessibleNodeLabels(), and this can be ANY or "*" in default case.

          Now in reinitializeQueueLabels, I am setting setLabelToQueueMappingAvailable() to true after the check with ANY.

                  Set<String> labels = entry.getValue();
                  if (labels.contains(ANY)) {
                    continue;
                  }
          
                  this.setLabelToQueueMappingAvailable(true);
          

          So isLabelToQueueMappingAvailable() will cover the default cause of ANY which will coming from getAccessibleNodeLabels from a queue. Pls help to share your thoughts on this. If this is not the case which you were mentionining, could you pls help to point the same.

          sunilg Sunil G added a comment - Hi Wangda Tan Thank you for the clarification. Pls correct me if I am wrong, I think it is also handled. RMNodeLabelsManager#reinitializeQueueLabels is invoked with queueToLabels as arguement. Here queueToLabels is actually CS#getQueueToLabels() . And this getQueueToLabels() will be collecting the labels configured per queue from queue.getAccessibleNodeLabels() , and this can be ANY or "*" in default case. Now in reinitializeQueueLabels , I am setting setLabelToQueueMappingAvailable() to true after the check with ANY . Set< String > labels = entry.getValue(); if (labels.contains(ANY)) { continue ; } this .setLabelToQueueMappingAvailable( true ); So isLabelToQueueMappingAvailable() will cover the default cause of ANY which will coming from getAccessibleNodeLabels from a queue. Pls help to share your thoughts on this. If this is not the case which you were mentionining, could you pls help to point the same.
          leftnoteasy Wangda Tan added a comment -

          Sunil G,

          My understanding of your proposal is, we will show partition only when:

          • nodeLabelManager.pullRMNodeLabelsInfo() has valid labels
          • AND isLabelToQueueMappingAvailable() is true.

          This makes sense to me, but please handle the case that ANY label is assigned to queue by default.

          leftnoteasy Wangda Tan added a comment - Sunil G , My understanding of your proposal is, we will show partition only when: nodeLabelManager.pullRMNodeLabelsInfo() has valid labels AND isLabelToQueueMappingAvailable() is true. This makes sense to me, but please handle the case that ANY label is assigned to queue by default.
          sunilg Sunil G added a comment -

          Hi Wangda Tan
          Yes, I understood the scenario which you have mentioned. And this was one of my doubt while I was fixing this pblm.

          We can have some queues as orphan ones (queues that are not part of any labels). If we assume these queues can be categorized under DEFULT_PARTITION, in a way its fine i think. Logically its somewhat fine if we show capacity as 0 under other labels.

          To me, we can ignore label only when it is a completely "orphan" label, which is not assigned to any node and queue.

          I agree to this. Due to this reason, I have placed this.setLabelToQueueMappingAvailable(true); in reinitializeQueueLabels.

                for (Entry<String, Set<String>> entry : queueToLabels.entrySet()) {
                  String queue = entry.getKey();
                  Queue q = new Queue();
                  this.queueCollections.put(queue, q);
          
                  Set<String> labels = entry.getValue();
                  if (labels.contains(ANY)) {
                    continue;
                  }
                  this.setLabelToQueueMappingAvailable(true);
          
                  q.accessibleNodeLabels.addAll(labels);
          

          For any queue, if labels are not configured, then accessableLabels is "*" or "ANY". Hence if there is one queue which has some real labels, then I set this variable to true and think that there is a queue which is under a label. Now eventhough other queues are not with any labels, still i can consider them as under "DEFAULT_PARTITION"
          As you mentioned, if none of the queues has any label, then this code wont hit.

          So i have now 2 informations
          1. isLabelToQueueMappingAvailable() ==> Gives information whether there is any label-queue is available.
          2. ( nodeLabelManager.pullRMNodeLabelsInfo() == null) or only ANY is available

          Second one give information that whether any labels are in cluster. So with these 2 entities, we can decide whether all queues are orphan OR no labels are in cluster.

          Is this ok?

          sunilg Sunil G added a comment - Hi Wangda Tan Yes, I understood the scenario which you have mentioned. And this was one of my doubt while I was fixing this pblm. We can have some queues as orphan ones (queues that are not part of any labels). If we assume these queues can be categorized under DEFULT_PARTITION, in a way its fine i think. Logically its somewhat fine if we show capacity as 0 under other labels. To me, we can ignore label only when it is a completely "orphan" label, which is not assigned to any node and queue. I agree to this. Due to this reason, I have placed this.setLabelToQueueMappingAvailable(true); in reinitializeQueueLabels . for (Entry< String , Set< String >> entry : queueToLabels.entrySet()) { String queue = entry.getKey(); Queue q = new Queue(); this .queueCollections.put(queue, q); Set< String > labels = entry.getValue(); if (labels.contains(ANY)) { continue ; } this .setLabelToQueueMappingAvailable( true ); q.accessibleNodeLabels.addAll(labels); For any queue, if labels are not configured, then accessableLabels is "*" or "ANY". Hence if there is one queue which has some real labels, then I set this variable to true and think that there is a queue which is under a label. Now eventhough other queues are not with any labels, still i can consider them as under "DEFAULT_PARTITION" As you mentioned, if none of the queues has any label, then this code wont hit. So i have now 2 informations 1. isLabelToQueueMappingAvailable() ==> Gives information whether there is any label-queue is available. 2. ( nodeLabelManager.pullRMNodeLabelsInfo() == null) or only ANY is available Second one give information that whether any labels are in cluster. So with these 2 entities, we can decide whether all queues are orphan OR no labels are in cluster. Is this ok?
          leftnoteasy Wangda Tan added a comment -

          Sunil G, is your patch able to handle following case?
          Labels assigned to nodes, labels are non-exclusive, no label assigned to queue. In this case, should we should queue hierarchy under labels or not? IAW, queues could use resources of other labels even if there's no queue-label mapping.
          To me, we can ignore label only when it is a completely "orphan" label, which is not assigned to any node and queue.

          leftnoteasy Wangda Tan added a comment - Sunil G , is your patch able to handle following case? Labels assigned to nodes, labels are non-exclusive, no label assigned to queue. In this case, should we should queue hierarchy under labels or not? IAW, queues could use resources of other labels even if there's no queue-label mapping. To me, we can ignore label only when it is a completely "orphan" label, which is not assigned to any node and queue.
          sunilg Sunil G made changes -
          Attachment 0001-YARN-4634.patch [ 12786839 ]
          sunilg Sunil G added a comment -

          Attaching an initial version of patch which fixes UI rendering.

          sunilg Sunil G added a comment - Attaching an initial version of patch which fixes UI rendering.
          sunilg Sunil G made changes -
          Field Original Value New Value
          Parent YARN-2492 [ 12738742 ]
          Issue Type Bug [ 1 ] Sub-task [ 7 ]
          sunilg Sunil G created issue -

          People

            sunilg Sunil G
            sunilg Sunil G
            Votes:
            0 Vote for this issue
            Watchers:
            Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack