Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-4751

In 2.7, Labeled queue usage not shown properly in capacity scheduler UI

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.7.3
    • Fix Version/s: 2.7.3
    • Component/s: capacity scheduler, yarn
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      In 2.6 and 2.7, the capacity scheduler UI does not have the queue graphs separated by partition. When applications are running on a labeled queue, no color is shown in the bar graph, and several of the "Used" metrics are zero.

      1. 2.7 CS UI No BarGraph.jpg
        349 kB
        Eric Payne
      2. YARH-4752-branch-2.7.001.patch
        9 kB
        Eric Payne
      3. YARH-4752-branch-2.7.002.patch
        9 kB
        Eric Payne
      4. YARN-4751-branch-2.7.003.patch
        32 kB
        Eric Payne
      5. YARN-4751-branch-2.7.004.patch
        12 kB
        Eric Payne
      6. YARN-4751-branch-2.7.005.patch
        13 kB
        Eric Payne

        Issue Links

          Activity

          Hide
          eepayne Eric Payne added a comment -

          In the attached screenshot, please note that the Used Capacity, Absolute Used Capacity, and Active User Info::Used Resources values are all zero even though Num Containers is 11. The application runs and completes successfully.

          Show
          eepayne Eric Payne added a comment - In the attached screenshot, please note that the Used Capacity , Absolute Used Capacity , and Active User Info::Used Resources values are all zero even though Num Containers is 11. The application runs and completes successfully.
          Hide
          sunilg Sunil G added a comment -

          Eric Payne, Thanks for updating this. YARN-4304 handled issue in few metrics. In Trunk, I could see this metric is coming correctly. So I think one UI ticket is not picked in 2.7.

          Show
          sunilg Sunil G added a comment - Eric Payne , Thanks for updating this. YARN-4304 handled issue in few metrics. In Trunk, I could see this metric is coming correctly. So I think one UI ticket is not picked in 2.7.
          Hide
          eepayne Eric Payne added a comment -

          Thanks, Sunil G for pointing out YARN-4304. I see that this revision has several JIRAs that would also need to be pulled back if YARN-4304 is cherry picked to 2.7, including YARN-1651 YARN-2003 YARN-3362 YARN-3463 YARN-3961 YARN-4082 YARN-4162. Is that correct? I think it would be better if we had a 2.7-specific patch for YARN-4304. Is that something you would be willing to provide?

          Show
          eepayne Eric Payne added a comment - Thanks, Sunil G for pointing out YARN-4304 . I see that this revision has several JIRAs that would also need to be pulled back if YARN-4304 is cherry picked to 2.7, including YARN-1651 YARN-2003 YARN-3362 YARN-3463 YARN-3961 YARN-4082 YARN-4162 . Is that correct? I think it would be better if we had a 2.7-specific patch for YARN-4304 . Is that something you would be willing to provide?
          Hide
          eepayne Eric Payne added a comment -

          Sunil G, perhaps we don't need the whole patch from YARN-4304. Might I suggest doing something simple, like YARH-4752-branch-2.7.001.patch?

          Show
          eepayne Eric Payne added a comment - Sunil G , perhaps we don't need the whole patch from YARN-4304 . Might I suggest doing something simple, like YARH-4752-branch-2.7.001.patch ?
          Hide
          sunilg Sunil G added a comment -

          Hi Eric Payne
          Extremely Sorry for the delay here . I somehow missed this notification, Yes, a full patch of YARN-4304 is not needed here.

          In that patch, I added one method in AbstractCSQueue named getNodeLabelsForQueue.
          This will get all valid labels for a queue. We may not need to go over all labels in cluster here. So could we take that code alone here, and use the same in the current uploaded patch. Does it make sense? thoughts?

          Show
          sunilg Sunil G added a comment - Hi Eric Payne Extremely Sorry for the delay here . I somehow missed this notification, Yes, a full patch of YARN-4304 is not needed here. In that patch, I added one method in AbstractCSQueue named getNodeLabelsForQueue . This will get all valid labels for a queue. We may not need to go over all labels in cluster here. So could we take that code alone here, and use the same in the current uploaded patch. Does it make sense? thoughts?
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 28s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 mvninstall 6m 15s branch-2.7 passed
          +1 compile 0m 27s branch-2.7 passed with JDK v1.8.0_74
          +1 compile 0m 26s branch-2.7 passed with JDK v1.7.0_95
          +1 checkstyle 0m 32s branch-2.7 passed
          +1 mvnsite 0m 32s branch-2.7 passed
          +1 mvneclipse 0m 14s branch-2.7 passed
          -1 findbugs 1m 4s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager in branch-2.7 has 1 extant Findbugs warnings.
          +1 javadoc 0m 18s branch-2.7 passed with JDK v1.8.0_74
          +1 javadoc 0m 24s branch-2.7 passed with JDK v1.7.0_95
          +1 mvninstall 0m 28s the patch passed
          +1 compile 0m 22s the patch passed with JDK v1.8.0_74
          +1 javac 0m 22s the patch passed
          +1 compile 0m 23s the patch passed with JDK v1.7.0_95
          +1 javac 0m 23s the patch passed
          -1 checkstyle 0m 32s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: patch generated 14 new + 1200 unchanged - 2 fixed = 1214 total (was 1202)
          +1 mvnsite 0m 29s the patch passed
          +1 mvneclipse 0m 12s the patch passed
          -1 whitespace 0m 0s The patch has 1122 line(s) that end in whitespace. Use git apply --whitespace=fix.
          -1 whitespace 0m 29s The patch has 97 line(s) with tabs.
          +1 findbugs 1m 10s the patch passed
          +1 javadoc 0m 16s the patch passed with JDK v1.8.0_74
          +1 javadoc 0m 21s the patch passed with JDK v1.7.0_95
          -1 unit 52m 14s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.8.0_74.
          -1 unit 53m 10s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.7.0_95.
          -1 asflicense 43m 30s Patch generated 77 ASF License warnings.
          165m 35s



          Reason Tests
          JDK v1.8.0_74 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens
            hadoop.yarn.server.resourcemanager.applicationsmanager.TestAMRestart
            hadoop.yarn.server.resourcemanager.TestAMAuthorization
          JDK v1.7.0_95 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens
            hadoop.yarn.server.resourcemanager.TestAMAuthorization



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:c420dfe
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12791487/YARH-4752-branch-2.7.001.patch
          JIRA Issue YARN-4751
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 2f79cc2eee23 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision branch-2.7 / 71e89e8
          Default Java 1.7.0_95
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_74 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
          findbugs v3.0.0
          findbugs https://builds.apache.org/job/PreCommit-YARN-Build/10710/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-warnings.html
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/10710/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
          whitespace https://builds.apache.org/job/PreCommit-YARN-Build/10710/artifact/patchprocess/whitespace-eol.txt
          whitespace https://builds.apache.org/job/PreCommit-YARN-Build/10710/artifact/patchprocess/whitespace-tabs.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/10710/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_74.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/10710/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_95.txt
          unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/10710/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_74.txt https://builds.apache.org/job/PreCommit-YARN-Build/10710/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_95.txt
          JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/10710/testReport/
          asflicense https://builds.apache.org/job/PreCommit-YARN-Build/10710/artifact/patchprocess/patch-asflicense-problems.txt
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/10710/console
          Powered by Apache Yetus 0.3.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 28s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 6m 15s branch-2.7 passed +1 compile 0m 27s branch-2.7 passed with JDK v1.8.0_74 +1 compile 0m 26s branch-2.7 passed with JDK v1.7.0_95 +1 checkstyle 0m 32s branch-2.7 passed +1 mvnsite 0m 32s branch-2.7 passed +1 mvneclipse 0m 14s branch-2.7 passed -1 findbugs 1m 4s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager in branch-2.7 has 1 extant Findbugs warnings. +1 javadoc 0m 18s branch-2.7 passed with JDK v1.8.0_74 +1 javadoc 0m 24s branch-2.7 passed with JDK v1.7.0_95 +1 mvninstall 0m 28s the patch passed +1 compile 0m 22s the patch passed with JDK v1.8.0_74 +1 javac 0m 22s the patch passed +1 compile 0m 23s the patch passed with JDK v1.7.0_95 +1 javac 0m 23s the patch passed -1 checkstyle 0m 32s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: patch generated 14 new + 1200 unchanged - 2 fixed = 1214 total (was 1202) +1 mvnsite 0m 29s the patch passed +1 mvneclipse 0m 12s the patch passed -1 whitespace 0m 0s The patch has 1122 line(s) that end in whitespace. Use git apply --whitespace=fix. -1 whitespace 0m 29s The patch has 97 line(s) with tabs. +1 findbugs 1m 10s the patch passed +1 javadoc 0m 16s the patch passed with JDK v1.8.0_74 +1 javadoc 0m 21s the patch passed with JDK v1.7.0_95 -1 unit 52m 14s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.8.0_74. -1 unit 53m 10s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.7.0_95. -1 asflicense 43m 30s Patch generated 77 ASF License warnings. 165m 35s Reason Tests JDK v1.8.0_74 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens   hadoop.yarn.server.resourcemanager.applicationsmanager.TestAMRestart   hadoop.yarn.server.resourcemanager.TestAMAuthorization JDK v1.7.0_95 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens   hadoop.yarn.server.resourcemanager.TestAMAuthorization Subsystem Report/Notes Docker Image:yetus/hadoop:c420dfe JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12791487/YARH-4752-branch-2.7.001.patch JIRA Issue YARN-4751 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 2f79cc2eee23 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision branch-2.7 / 71e89e8 Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_74 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 findbugs https://builds.apache.org/job/PreCommit-YARN-Build/10710/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-warnings.html checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/10710/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt whitespace https://builds.apache.org/job/PreCommit-YARN-Build/10710/artifact/patchprocess/whitespace-eol.txt whitespace https://builds.apache.org/job/PreCommit-YARN-Build/10710/artifact/patchprocess/whitespace-tabs.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/10710/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_74.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/10710/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_95.txt unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/10710/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_74.txt https://builds.apache.org/job/PreCommit-YARN-Build/10710/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_95.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/10710/testReport/ asflicense https://builds.apache.org/job/PreCommit-YARN-Build/10710/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/10710/console Powered by Apache Yetus 0.3.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          eepayne Eric Payne added a comment -

          In that patch, I added one method in AbstractCSQueue named getNodeLabelsForQueue.
          This will get all valid labels for a queue. We may not need to go over all labels in cluster here. So could we take that code alone here, and use the same in the current uploaded patch. Does it make sense? thoughts?

          Thanks, Sunil G, for pointing out getNodeLabelsForQueue from YARN-4304. Unfortunately, that method looks like it depends on QueueCapacities#getNodePartitionsSet and ResourceUsage#getNodePartitionsSet, which were implemented as part of YARN-2696. I am afraid that once we start pulling on that dependency chain, it will get complicated very quickly.

          AFAICT, including all cluster node labels as implemented in the current patch will produce the correct results, it just may traverse more labels than it needs to, and the non-relevant ones will result in 0. Including all relevant labels is especially important when calculating values for the used capacity bar graph to be displayed on the parent and root queues' bar graphs in the capacity scheduler UI page.

          Show
          eepayne Eric Payne added a comment - In that patch, I added one method in AbstractCSQueue named getNodeLabelsForQueue. This will get all valid labels for a queue. We may not need to go over all labels in cluster here. So could we take that code alone here, and use the same in the current uploaded patch. Does it make sense? thoughts? Thanks, Sunil G , for pointing out getNodeLabelsForQueue from YARN-4304 . Unfortunately, that method looks like it depends on QueueCapacities#getNodePartitionsSet and ResourceUsage#getNodePartitionsSet , which were implemented as part of YARN-2696 . I am afraid that once we start pulling on that dependency chain, it will get complicated very quickly. AFAICT, including all cluster node labels as implemented in the current patch will produce the correct results, it just may traverse more labels than it needs to, and the non-relevant ones will result in 0. Including all relevant labels is especially important when calculating values for the used capacity bar graph to be displayed on the parent and root queues' bar graphs in the capacity scheduler UI page.
          Hide
          eepayne Eric Payne added a comment -

          Attaching YARH-4752-branch-2.7.002.patch. It addresses some of the checkstyle warnings. The findbugs warning is regarding ZKRMStateStore, which version 001 of this patch didn't change. This false findbugs warning looks to be documented in YARN-4328.

          Regarding unit tests, I have been trying to decide the best approach. I would like to start up a MiniYARNCluster in order to get as close to a labeled environment so that this patch can be tested. However, starting up a MiniYARNCluster, configuring multiple nodes, configuring multiple queues, configuring labels, and launching containers on both labeled and non-labeled queues will be a lot of testing overhead for a patch that will only be integrated into 2.7. There are a couple of unit tests that could be used as models, but I don't see anything that already does what would be needed.

          Sunil G and Jason Lowe, I would appreciate your opinion on the best unit test approach.

          Show
          eepayne Eric Payne added a comment - Attaching YARH-4752-branch-2.7.002.patch . It addresses some of the checkstyle warnings. The findbugs warning is regarding ZKRMStateStore , which version 001 of this patch didn't change. This false findbugs warning looks to be documented in YARN-4328 . Regarding unit tests, I have been trying to decide the best approach. I would like to start up a MiniYARNCluster in order to get as close to a labeled environment so that this patch can be tested. However, starting up a MiniYARNCluster, configuring multiple nodes, configuring multiple queues, configuring labels, and launching containers on both labeled and non-labeled queues will be a lot of testing overhead for a patch that will only be integrated into 2.7. There are a couple of unit tests that could be used as models, but I don't see anything that already does what would be needed. Sunil G and Jason Lowe , I would appreciate your opinion on the best unit test approach.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 21s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 mvninstall 6m 6s branch-2.7 passed
          +1 compile 0m 25s branch-2.7 passed with JDK v1.8.0_74
          +1 compile 0m 26s branch-2.7 passed with JDK v1.7.0_95
          +1 checkstyle 0m 33s branch-2.7 passed
          +1 mvnsite 0m 35s branch-2.7 passed
          +1 mvneclipse 0m 13s branch-2.7 passed
          -1 findbugs 1m 3s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager in branch-2.7 has 1 extant Findbugs warnings.
          +1 javadoc 0m 21s branch-2.7 passed with JDK v1.8.0_74
          +1 javadoc 0m 24s branch-2.7 passed with JDK v1.7.0_95
          +1 mvninstall 0m 29s the patch passed
          +1 compile 0m 26s the patch passed with JDK v1.8.0_74
          +1 javac 0m 26s the patch passed
          +1 compile 0m 26s the patch passed with JDK v1.7.0_95
          +1 javac 0m 26s the patch passed
          -1 checkstyle 0m 30s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: patch generated 8 new + 1200 unchanged - 2 fixed = 1208 total (was 1202)
          +1 mvnsite 0m 31s the patch passed
          +1 mvneclipse 0m 12s the patch passed
          -1 whitespace 0m 0s The patch has 1499 line(s) that end in whitespace. Use git apply --whitespace=fix.
          -1 whitespace 0m 36s The patch has 249 line(s) with tabs.
          +1 findbugs 1m 11s the patch passed
          +1 javadoc 0m 17s the patch passed with JDK v1.8.0_74
          +1 javadoc 0m 22s the patch passed with JDK v1.7.0_95
          -1 unit 52m 54s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.8.0_74.
          -1 unit 53m 21s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.7.0_95.
          -1 asflicense 2m 13s Patch generated 61 ASF License warnings.
          125m 17s



          Reason Tests
          JDK v1.8.0_74 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens
            hadoop.yarn.server.resourcemanager.TestAMAuthorization
          JDK v1.7.0_95 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens
            hadoop.yarn.server.resourcemanager.TestAMAuthorization



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:c420dfe
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12793161/YARH-4752-branch-2.7.002.patch
          JIRA Issue YARN-4751
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux c43d6d39f483 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision branch-2.7 / b409ce8
          Default Java 1.7.0_95
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_74 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
          findbugs v3.0.0
          findbugs https://builds.apache.org/job/PreCommit-YARN-Build/10767/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-warnings.html
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/10767/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
          whitespace https://builds.apache.org/job/PreCommit-YARN-Build/10767/artifact/patchprocess/whitespace-eol.txt
          whitespace https://builds.apache.org/job/PreCommit-YARN-Build/10767/artifact/patchprocess/whitespace-tabs.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/10767/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_74.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/10767/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_95.txt
          unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/10767/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_74.txt https://builds.apache.org/job/PreCommit-YARN-Build/10767/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_95.txt
          JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/10767/testReport/
          asflicense https://builds.apache.org/job/PreCommit-YARN-Build/10767/artifact/patchprocess/patch-asflicense-problems.txt
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/10767/console
          Powered by Apache Yetus 0.2.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 21s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 6m 6s branch-2.7 passed +1 compile 0m 25s branch-2.7 passed with JDK v1.8.0_74 +1 compile 0m 26s branch-2.7 passed with JDK v1.7.0_95 +1 checkstyle 0m 33s branch-2.7 passed +1 mvnsite 0m 35s branch-2.7 passed +1 mvneclipse 0m 13s branch-2.7 passed -1 findbugs 1m 3s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager in branch-2.7 has 1 extant Findbugs warnings. +1 javadoc 0m 21s branch-2.7 passed with JDK v1.8.0_74 +1 javadoc 0m 24s branch-2.7 passed with JDK v1.7.0_95 +1 mvninstall 0m 29s the patch passed +1 compile 0m 26s the patch passed with JDK v1.8.0_74 +1 javac 0m 26s the patch passed +1 compile 0m 26s the patch passed with JDK v1.7.0_95 +1 javac 0m 26s the patch passed -1 checkstyle 0m 30s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: patch generated 8 new + 1200 unchanged - 2 fixed = 1208 total (was 1202) +1 mvnsite 0m 31s the patch passed +1 mvneclipse 0m 12s the patch passed -1 whitespace 0m 0s The patch has 1499 line(s) that end in whitespace. Use git apply --whitespace=fix. -1 whitespace 0m 36s The patch has 249 line(s) with tabs. +1 findbugs 1m 11s the patch passed +1 javadoc 0m 17s the patch passed with JDK v1.8.0_74 +1 javadoc 0m 22s the patch passed with JDK v1.7.0_95 -1 unit 52m 54s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.8.0_74. -1 unit 53m 21s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.7.0_95. -1 asflicense 2m 13s Patch generated 61 ASF License warnings. 125m 17s Reason Tests JDK v1.8.0_74 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens   hadoop.yarn.server.resourcemanager.TestAMAuthorization JDK v1.7.0_95 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens   hadoop.yarn.server.resourcemanager.TestAMAuthorization Subsystem Report/Notes Docker Image:yetus/hadoop:c420dfe JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12793161/YARH-4752-branch-2.7.002.patch JIRA Issue YARN-4751 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux c43d6d39f483 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision branch-2.7 / b409ce8 Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_74 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 findbugs https://builds.apache.org/job/PreCommit-YARN-Build/10767/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-warnings.html checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/10767/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt whitespace https://builds.apache.org/job/PreCommit-YARN-Build/10767/artifact/patchprocess/whitespace-eol.txt whitespace https://builds.apache.org/job/PreCommit-YARN-Build/10767/artifact/patchprocess/whitespace-tabs.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/10767/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_74.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/10767/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_95.txt unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/10767/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_74.txt https://builds.apache.org/job/PreCommit-YARN-Build/10767/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_95.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/10767/testReport/ asflicense https://builds.apache.org/job/PreCommit-YARN-Build/10767/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/10767/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
          Hide
          sunilg Sunil G added a comment -

          Thanks Eric Payne for pointing out the dependencies. Yes, its come with a chain of patches. We are trying to discuss in YARN-3216 about getting it ready for 2.7 line. If we can confirm that, I can make a clean patch with minimal changes needed to get AM Resource Percent for labels and it can get almost all new changes what we have done recently. But its more a difference approach as we cant cherry pick the needed patch, so it may be complex 2.7 line later. I would like to hear your thoughts on that also.

          Coming to this patch, I think we needed only to verify the resource usages in NodeLabel scenario. In trunk TestCapacitySchedulerNodeLabelUpdate covers various cases. And it uses NullRMNodeLabelsManager, so many events wont be done and it ll be good w.r.t unit test cases. Could we consider this test class?

          Show
          sunilg Sunil G added a comment - Thanks Eric Payne for pointing out the dependencies. Yes, its come with a chain of patches. We are trying to discuss in YARN-3216 about getting it ready for 2.7 line. If we can confirm that, I can make a clean patch with minimal changes needed to get AM Resource Percent for labels and it can get almost all new changes what we have done recently. But its more a difference approach as we cant cherry pick the needed patch, so it may be complex 2.7 line later. I would like to hear your thoughts on that also. Coming to this patch, I think we needed only to verify the resource usages in NodeLabel scenario. In trunk TestCapacitySchedulerNodeLabelUpdate covers various cases. And it uses NullRMNodeLabelsManager , so many events wont be done and it ll be good w.r.t unit test cases. Could we consider this test class?
          Hide
          eepayne Eric Payne added a comment -

          Thanks, Sunil G. I will look into trunk's version of TestCapacitySchedulerNodeLabelUpdate.

          Regarding the larger issue of how to fix this problem in 2.7, I am fine if you want to provide a backport of some of the patches you mentioned. However, my biggest concern is the time and effort that will take, along with the added risk. As I mentioned above, there seems to be a lot of inter-dependencies that involve adding more features and fixes than just the one documented by this JIRA. I'm not sure we want all of that complexity going back into 2.7.

          Show
          eepayne Eric Payne added a comment - Thanks, Sunil G . I will look into trunk's version of TestCapacitySchedulerNodeLabelUpdate . Regarding the larger issue of how to fix this problem in 2.7, I am fine if you want to provide a backport of some of the patches you mentioned. However, my biggest concern is the time and effort that will take, along with the added risk. As I mentioned above, there seems to be a lot of inter-dependencies that involve adding more features and fixes than just the one documented by this JIRA. I'm not sure we want all of that complexity going back into 2.7.
          Hide
          sunilg Sunil G added a comment -

          Yes, agreeing to the point that we have to aggregate and peek into multiple patches to get the functionality. If 2.7 doesnt need new features/enhancements for labels, we can bring in patches on use case basis. cc/Wangda Tan (No longer used)

          Show
          sunilg Sunil G added a comment - Yes, agreeing to the point that we have to aggregate and peek into multiple patches to get the functionality. If 2.7 doesnt need new features/enhancements for labels, we can bring in patches on use case basis. cc/ Wangda Tan (No longer used)
          Hide
          eepayne Eric Payne added a comment -

          Yes, agreeing to the point that we have to aggregate and peek into multiple patches to get the functionality. If 2.7 doesnt need new features/enhancements for labels, we can bring in patches on use case basis. cc/Wangda Tan (No longer used)

          Wangda Tan may not have seen this.

          Show
          eepayne Eric Payne added a comment - Yes, agreeing to the point that we have to aggregate and peek into multiple patches to get the functionality. If 2.7 doesnt need new features/enhancements for labels, we can bring in patches on use case basis. cc/Wangda Tan (No longer used) Wangda Tan may not have seen this.
          Hide
          leftnoteasy Wangda Tan added a comment -

          Hi Eric Payne, Sunil G.

          Quickly read discussions and looked at patch.

          Several questions / comments:
          1) The ultimate solution seems to be YARN-3362. Have you evaluated how hard to back port it?
          2) If you don't want to backport YARN-3362. IIUC, the computation of total-used-capacity-considers-all-labels seems wrong:
          In your patch it is Σ(queue.label.used_capacity), actually it should be Σ(queue.label.used_resource) / Σ(root.label.total_resource)

          Thoughts?

          Show
          leftnoteasy Wangda Tan added a comment - Hi Eric Payne , Sunil G . Quickly read discussions and looked at patch. Several questions / comments: 1) The ultimate solution seems to be YARN-3362 . Have you evaluated how hard to back port it? 2) If you don't want to backport YARN-3362 . IIUC, the computation of total-used-capacity-considers-all-labels seems wrong: In your patch it is Σ(queue.label.used_capacity), actually it should be Σ(queue.label.used_resource) / Σ(root.label.total_resource) Thoughts?
          Hide
          sunilg Sunil G added a comment -

          As Eric Payne mentioned, we might need below tickets also to get YARN-3362 in.
          YARN-3463 and YARN-4082.

          Along with few UI changes, it may look clean if we use this. But if those UI changes are not needed, new patch as updated here is fine (after fixing Wangda's comment).

          Show
          sunilg Sunil G added a comment - As Eric Payne mentioned, we might need below tickets also to get YARN-3362 in. YARN-3463 and YARN-4082 . Along with few UI changes, it may look clean if we use this. But if those UI changes are not needed, new patch as updated here is fine (after fixing Wangda's comment).
          Hide
          eepayne Eric Payne added a comment -

          Thank you very much, Wangda Tan!

          1) The ultimate solution seems to be YARN-3362. Have you evaluated how hard to back port it?

          My intent with this JIRA was simply to provide accurate visual feedback on the Capacity Scheduler Application Queues page for resources used by labeled queues. I will look deeper into what it would take to backport YARN-3362. My concern is that it would include more risk and features than we really want to backport into 2.7.

          2) If you don't want to backport YARN-3362. IIUC, the computation of total-used-capacity-considers-all-labels seems wrong:
          In your patch it is Σ(queue.label.used_capacity), actually it should be Σ(queue.label.used_resource) / S(root.label.total_resource)

          I think you are talking about getAllUsedCapacityConsideringLabels. IIUC, in this patch, used capacity is calculated correctly by CSQueueUtils#updateQueueStatistics when it is called by LeafQueue to allocate resources. updateQueueStatistics calls AbstractCSQueue#getAllUsedResourcesConsideringLabels to retrieve all resources consumed by the queue, including labeled resources. It then divides the consumed resources by the queue's limit:

            public static void updateQueueStatistics(
          ...
              Resource usedResources = childQueue.getAllUsedResourcesConsideringLabels();
          ...
                usedCapacity = 
                    Resources.equals(queueLimit, Resources.none()) ? 0 :
                    Resources.divide(calculator, clusterResource, 
                        usedResources, queueLimit);
          ...
              childQueue.setUsedCapacity(usedCapacity);
          

          Even still, this patch is, perhaps, not ideal. When used capacity, absolute used capacity, and available resources are set, these metrics are associated with NO_LABEL. So, although the metrics reflect all used resources for a queue (even at parent levels), they are not set per label but just gathered in with NO_LABEL. In order to associate the resources with a label, I think the interface to CSQueueUtils#updateQueueStatistics may need to change.

          Let me get a more precise view of what it would take to backport YARN-3362.
          Thanks again.

          Show
          eepayne Eric Payne added a comment - Thank you very much, Wangda Tan ! 1) The ultimate solution seems to be YARN-3362 . Have you evaluated how hard to back port it? My intent with this JIRA was simply to provide accurate visual feedback on the Capacity Scheduler Application Queues page for resources used by labeled queues. I will look deeper into what it would take to backport YARN-3362 . My concern is that it would include more risk and features than we really want to backport into 2.7. 2) If you don't want to backport YARN-3362 . IIUC, the computation of total-used-capacity-considers-all-labels seems wrong: In your patch it is Σ(queue.label.used_capacity), actually it should be Σ(queue.label.used_resource) / S(root.label.total_resource) I think you are talking about getAllUsedCapacityConsideringLabels . IIUC, in this patch, used capacity is calculated correctly by CSQueueUtils#updateQueueStatistics when it is called by LeafQueue to allocate resources. updateQueueStatistics calls AbstractCSQueue#getAllUsedResourcesConsideringLabels to retrieve all resources consumed by the queue, including labeled resources. It then divides the consumed resources by the queue's limit: public static void updateQueueStatistics( ... Resource usedResources = childQueue.getAllUsedResourcesConsideringLabels(); ... usedCapacity = Resources.equals(queueLimit, Resources.none()) ? 0 : Resources.divide(calculator, clusterResource, usedResources, queueLimit); ... childQueue.setUsedCapacity(usedCapacity); Even still, this patch is, perhaps, not ideal. When used capacity, absolute used capacity, and available resources are set, these metrics are associated with NO_LABEL . So, although the metrics reflect all used resources for a queue (even at parent levels), they are not set per label but just gathered in with NO_LABEL . In order to associate the resources with a label, I think the interface to CSQueueUtils#updateQueueStatistics may need to change. Let me get a more precise view of what it would take to backport YARN-3362 . Thanks again.
          Hide
          eepayne Eric Payne added a comment -

          Wangda Tan / Sunil G:

          YARN-3362 (Add node label usage in RM CapacityScheduler web UI) is dependent upon parts of both YARN-3361 (CapacityScheduler side changes to support non-exclusive node labels) and YARN-4162 (CapacityScheduler: Add resource usage by partition and queue capacity by partition to REST API). In turn, YARN-3361 is dependent on YARN-3356 (Capacity Scheduler FiCaSchedulerApp should use ResourceUsage to track used-resources-by-label.).

          The dependencies are as follows:

          • YARN-3362 depends on YARN-3361 for the new method AbstractCSQueue#accessibleToPartition
          • YARN-3362 depends on YARN-4162 for addition of the CapacityScheduler parameter to CapacitySchedulerInfo constructor
          • YARN-3361 depends on YARN-3356 for the modifications to LeafQueue#canAssignToUser

          I am not comfortable with pulling all of these changes back into 2.7, especially the changes for non-exclusive node labels. I fear that there is a risk of destabilization. It may be possible to rewrite YARN-3362 so that it is not dependent on those other patches, but that seems like it could also be a risk and a lot of time spent for the purpose of getting the GUI metrics correct.

          Please let me know what approach you would like to take. If the current patch is not an option, then we can close this JIRA as won'tfix.

          Show
          eepayne Eric Payne added a comment - Wangda Tan / Sunil G : YARN-3362 (Add node label usage in RM CapacityScheduler web UI) is dependent upon parts of both YARN-3361 (CapacityScheduler side changes to support non-exclusive node labels) and YARN-4162 (CapacityScheduler: Add resource usage by partition and queue capacity by partition to REST API). In turn, YARN-3361 is dependent on YARN-3356 (Capacity Scheduler FiCaSchedulerApp should use ResourceUsage to track used-resources-by-label.). The dependencies are as follows: YARN-3362 depends on YARN-3361 for the new method AbstractCSQueue#accessibleToPartition YARN-3362 depends on YARN-4162 for addition of the CapacityScheduler parameter to CapacitySchedulerInfo constructor YARN-3361 depends on YARN-3356 for the modifications to LeafQueue#canAssignToUser I am not comfortable with pulling all of these changes back into 2.7, especially the changes for non-exclusive node labels. I fear that there is a risk of destabilization. It may be possible to rewrite YARN-3362 so that it is not dependent on those other patches, but that seems like it could also be a risk and a lot of time spent for the purpose of getting the GUI metrics correct. Please let me know what approach you would like to take. If the current patch is not an option, then we can close this JIRA as won'tfix.
          Hide
          sunilg Sunil G added a comment -

          Hi Eric Payne
          Thanks for a very detailed view of the dependency of YARN-3362.

          Few of the dependent jiras also come up with some feature enhancement. so the question will be like whether 2.7 line needs the same. According to me, I think bug fixes need to be there in 2.7. Other enhancements may nt be needed. Being said that, may be its better to fix this directly here.
          Buf if we are going in this line, then porting any of these tickets in future release lines of 2.7 will be tough. Bcz this will be an independent patch and we may need to revert. I still we feel we can fix direct here. Will wait for Wangda Tan opinion also.

          One more comment in current patch:

          public final float getAllUsedCapacityConsideringLabels() {
          154	    float usedCapacity = 0.0f;
          155	    for (String nl : getAccessibleLabelSet()) {
          156	      usedCapacity += queueCapacities.getUsedCapacity(nl);
          157	    }
          

          As mentioned, we are not setting usedCapacity per label in 2.7 I think. So will above code work or I missed something? If its fine, may be can we calculate again by userd_resource/total_resource way.

          Show
          sunilg Sunil G added a comment - Hi Eric Payne Thanks for a very detailed view of the dependency of YARN-3362 . Few of the dependent jiras also come up with some feature enhancement. so the question will be like whether 2.7 line needs the same. According to me, I think bug fixes need to be there in 2.7. Other enhancements may nt be needed. Being said that, may be its better to fix this directly here. Buf if we are going in this line, then porting any of these tickets in future release lines of 2.7 will be tough. Bcz this will be an independent patch and we may need to revert. I still we feel we can fix direct here. Will wait for Wangda Tan opinion also. One more comment in current patch: public final float getAllUsedCapacityConsideringLabels() { 154 float usedCapacity = 0.0f; 155 for ( String nl : getAccessibleLabelSet()) { 156 usedCapacity += queueCapacities.getUsedCapacity(nl); 157 } As mentioned, we are not setting usedCapacity per label in 2.7 I think. So will above code work or I missed something? If its fine, may be can we calculate again by userd_resource/total_resource way.
          Hide
          leftnoteasy Wangda Tan added a comment -

          Eric Payne/Sunil G,

          I looked at YARN-3362 and dependencies, it seems to me that we only need minor rebases to make YARN-3362 works:

          • Additional CS parameter isn't required by YARN-3362.
          • accessibleToPartition could be added to rebased patch, it only has several lines.

          I agree that we should avoid pulling feature patches like YARN-3361, but it seems to me that rebasing YARN-3362 is a better and also possibly easier fix to 2.7.

          Thoughts?

          Show
          leftnoteasy Wangda Tan added a comment - Eric Payne / Sunil G , I looked at YARN-3362 and dependencies, it seems to me that we only need minor rebases to make YARN-3362 works: Additional CS parameter isn't required by YARN-3362 . accessibleToPartition could be added to rebased patch, it only has several lines. I agree that we should avoid pulling feature patches like YARN-3361 , but it seems to me that rebasing YARN-3362 is a better and also possibly easier fix to 2.7. Thoughts?
          Hide
          eepayne Eric Payne added a comment -

          As mentioned, we are not setting usedCapacity per label in 2.7 I think. So will above code work or I missed something? If its fine, may be can we calculate again by userd_resource/total_resource way.

          Thanks Sunil G. Yes, the code will work. However, as I mentioned above, it is not ideal because when CSQueueUtils#updateQueueStatistics updates used capacity, it overloads it into the NO_LABEL partition. It may be better to re-calculate. I will look into that.

          I looked at YARN-3362 and dependencies, it seems to me that we only need minor rebases to make YARN-3362 works:

          • accessibleToPartition could be added to rebased patch, it only has several lines.

          Thanks Wangda Tan. Will look into trying to pull in only YARN-3362, with modifications.

          Show
          eepayne Eric Payne added a comment - As mentioned, we are not setting usedCapacity per label in 2.7 I think. So will above code work or I missed something? If its fine, may be can we calculate again by userd_resource/total_resource way. Thanks Sunil G . Yes, the code will work. However, as I mentioned above , it is not ideal because when CSQueueUtils#updateQueueStatistics updates used capacity, it overloads it into the NO_LABEL partition. It may be better to re-calculate. I will look into that. I looked at YARN-3362 and dependencies, it seems to me that we only need minor rebases to make YARN-3362 works: accessibleToPartition could be added to rebased patch, it only has several lines. Thanks Wangda Tan . Will look into trying to pull in only YARN-3362 , with modifications.
          Hide
          eepayne Eric Payne added a comment -

          Thank you, Wangda Tan and Sunil G, for your reviews and helpful insights.

          I backported YARN-3362 to 2.7, but it did not provide all of the correct statistics for labeled queues all by itself.

          • Used Capacity was not calculated correctly for node labels because QueueCapacities#setUsedCapacity does not have a label-specific interface that could set the capacity per partition. In order to get the updated changes to CSQueueUtils#updateQueueStatistics that set used capacity per label, I would need to backport YARN-2696. I did not do that since it also had other dependencies and was more invasive than I felt was needed.
            • Instead, I added a getUsedCapacity method to AbstractCSQueue that takes a label and returns the computed used capacity for the queue for that label.
          • The RMNodeLabel class is named NodeLabel in 2.7.
          • The CapacitySchedulerInfo constructor doesn't have the CapacityScheduler parameter in 2.7, so I left it out (as you suggested previously).
          • The YARN-3362 patch has a dependency on AbstractCSQueue#accessibleToPartition which was not in 2.7. Rather than backport YARN-3361 as well, I just recrated accessibleToPartition in this patch (as you suggested previously).
          • One of the changes for CapacitySchedulerInfo#getQueues in YARN-3362 added a check to skip non-accessible queues. In 2.7, there are 2 loops that iterate over the queues whereas in 2.8, there is only 1. I put the check in the first loop because the first loop is filtering the queues to be processed by the second loop
          • I modified AbstractCSQueue#allocateResource and releaseResource to intersect with all labels in the passed nodeLabels set if the accessible node labels included the ANY selection. Without this, queue usage for root would not be preserved per-partiton and the root queue's bar graph would not display any usage at all for specific partitions.
          • Modified LeafQueue#getUser to sum all labeled and unlabeld used resources for a user on a queue. Without this, the Used Resource in the Active Users Info section would be 0.
          Show
          eepayne Eric Payne added a comment - Thank you, Wangda Tan and Sunil G , for your reviews and helpful insights. I backported YARN-3362 to 2.7, but it did not provide all of the correct statistics for labeled queues all by itself. Used Capacity was not calculated correctly for node labels because QueueCapacities#setUsedCapacity does not have a label-specific interface that could set the capacity per partition. In order to get the updated changes to CSQueueUtils#updateQueueStatistics that set used capacity per label, I would need to backport YARN-2696 . I did not do that since it also had other dependencies and was more invasive than I felt was needed. Instead, I added a getUsedCapacity method to AbstractCSQueue that takes a label and returns the computed used capacity for the queue for that label. The RMNodeLabel class is named NodeLabel in 2.7. The CapacitySchedulerInfo constructor doesn't have the CapacityScheduler parameter in 2.7, so I left it out (as you suggested previously). The YARN-3362 patch has a dependency on AbstractCSQueue#accessibleToPartition which was not in 2.7. Rather than backport YARN-3361 as well, I just recrated accessibleToPartition in this patch (as you suggested previously). One of the changes for CapacitySchedulerInfo#getQueues in YARN-3362 added a check to skip non-accessible queues. In 2.7, there are 2 loops that iterate over the queues whereas in 2.8, there is only 1. I put the check in the first loop because the first loop is filtering the queues to be processed by the second loop I modified AbstractCSQueue#allocateResource and releaseResource to intersect with all labels in the passed nodeLabels set if the accessible node labels included the ANY selection. Without this, queue usage for root would not be preserved per-partiton and the root queue's bar graph would not display any usage at all for specific partitions. Modified LeafQueue#getUser to sum all labeled and unlabeld used resources for a user on a queue. Without this, the Used Resource in the Active Users Info section would be 0.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 19s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 6m 5s branch-2.7 passed
          +1 compile 0m 24s branch-2.7 passed with JDK v1.8.0_77
          +1 compile 0m 27s branch-2.7 passed with JDK v1.7.0_95
          +1 checkstyle 0m 40s branch-2.7 passed
          +1 mvnsite 0m 33s branch-2.7 passed
          +1 mvneclipse 0m 15s branch-2.7 passed
          -1 findbugs 1m 2s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager in branch-2.7 has 1 extant Findbugs warnings.
          +1 javadoc 0m 20s branch-2.7 passed with JDK v1.8.0_77
          +1 javadoc 0m 24s branch-2.7 passed with JDK v1.7.0_95
          +1 mvninstall 0m 29s the patch passed
          +1 compile 0m 21s the patch passed with JDK v1.8.0_77
          +1 javac 0m 21s the patch passed
          +1 compile 0m 24s the patch passed with JDK v1.7.0_95
          +1 javac 0m 24s the patch passed
          -1 checkstyle 0m 37s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: patch generated 88 new + 1596 unchanged - 43 fixed = 1684 total (was 1639)
          +1 mvnsite 0m 30s the patch passed
          +1 mvneclipse 0m 12s the patch passed
          -1 whitespace 0m 0s The patch has 2046 line(s) that end in whitespace. Use git apply --whitespace=fix.
          -1 whitespace 0m 53s The patch has 291 line(s) with tabs.
          -1 findbugs 1m 15s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager generated 2 new + 1 unchanged - 0 fixed = 3 total (was 1)
          +1 javadoc 0m 16s the patch passed with JDK v1.8.0_77
          +1 javadoc 0m 21s the patch passed with JDK v1.7.0_95
          -1 unit 53m 18s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.8.0_77.
          -1 unit 54m 36s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.7.0_95.
          -1 asflicense 2m 22s Patch generated 61 ASF License warnings.
          127m 29s



          Reason Tests
          FindBugs module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
            Inconsistent synchronization of org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.AbstractCSQueue.cr; locked 57% of time Unsynchronized access at AbstractCSQueue.java:57% of time Unsynchronized access at AbstractCSQueue.java:[line 554]
            org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.AbstractCSQueue.getAbsoluteUsedCapacity(String) is unsynchronized, org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.AbstractCSQueue.setAbsoluteUsedCapacity(float) is synchronized At AbstractCSQueue.java:synchronized At AbstractCSQueue.java:[lines 554-559]
          JDK v1.8.0_77 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens
            hadoop.yarn.server.resourcemanager.TestAMAuthorization
          JDK v1.7.0_95 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens
            hadoop.yarn.server.resourcemanager.TestAMAuthorization



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:c420dfe
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12797881/YARN-4751-branch-2.7.003.patch
          JIRA Issue YARN-4751
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 0781118c1988 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision branch-2.7 / 801c95c
          Default Java 1.7.0_95
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_77 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
          findbugs v3.0.0
          findbugs https://builds.apache.org/job/PreCommit-YARN-Build/11010/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-warnings.html
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/11010/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
          whitespace https://builds.apache.org/job/PreCommit-YARN-Build/11010/artifact/patchprocess/whitespace-eol.txt
          whitespace https://builds.apache.org/job/PreCommit-YARN-Build/11010/artifact/patchprocess/whitespace-tabs.txt
          findbugs https://builds.apache.org/job/PreCommit-YARN-Build/11010/artifact/patchprocess/new-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.html
          unit https://builds.apache.org/job/PreCommit-YARN-Build/11010/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_77.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/11010/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_95.txt
          unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/11010/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_77.txt https://builds.apache.org/job/PreCommit-YARN-Build/11010/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_95.txt
          JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/11010/testReport/
          asflicense https://builds.apache.org/job/PreCommit-YARN-Build/11010/artifact/patchprocess/patch-asflicense-problems.txt
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/11010/console
          Powered by Apache Yetus 0.2.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 19s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 6m 5s branch-2.7 passed +1 compile 0m 24s branch-2.7 passed with JDK v1.8.0_77 +1 compile 0m 27s branch-2.7 passed with JDK v1.7.0_95 +1 checkstyle 0m 40s branch-2.7 passed +1 mvnsite 0m 33s branch-2.7 passed +1 mvneclipse 0m 15s branch-2.7 passed -1 findbugs 1m 2s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager in branch-2.7 has 1 extant Findbugs warnings. +1 javadoc 0m 20s branch-2.7 passed with JDK v1.8.0_77 +1 javadoc 0m 24s branch-2.7 passed with JDK v1.7.0_95 +1 mvninstall 0m 29s the patch passed +1 compile 0m 21s the patch passed with JDK v1.8.0_77 +1 javac 0m 21s the patch passed +1 compile 0m 24s the patch passed with JDK v1.7.0_95 +1 javac 0m 24s the patch passed -1 checkstyle 0m 37s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: patch generated 88 new + 1596 unchanged - 43 fixed = 1684 total (was 1639) +1 mvnsite 0m 30s the patch passed +1 mvneclipse 0m 12s the patch passed -1 whitespace 0m 0s The patch has 2046 line(s) that end in whitespace. Use git apply --whitespace=fix. -1 whitespace 0m 53s The patch has 291 line(s) with tabs. -1 findbugs 1m 15s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager generated 2 new + 1 unchanged - 0 fixed = 3 total (was 1) +1 javadoc 0m 16s the patch passed with JDK v1.8.0_77 +1 javadoc 0m 21s the patch passed with JDK v1.7.0_95 -1 unit 53m 18s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.8.0_77. -1 unit 54m 36s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.7.0_95. -1 asflicense 2m 22s Patch generated 61 ASF License warnings. 127m 29s Reason Tests FindBugs module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager   Inconsistent synchronization of org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.AbstractCSQueue.cr; locked 57% of time Unsynchronized access at AbstractCSQueue.java:57% of time Unsynchronized access at AbstractCSQueue.java: [line 554]   org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.AbstractCSQueue.getAbsoluteUsedCapacity(String) is unsynchronized, org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.AbstractCSQueue.setAbsoluteUsedCapacity(float) is synchronized At AbstractCSQueue.java:synchronized At AbstractCSQueue.java: [lines 554-559] JDK v1.8.0_77 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens   hadoop.yarn.server.resourcemanager.TestAMAuthorization JDK v1.7.0_95 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens   hadoop.yarn.server.resourcemanager.TestAMAuthorization Subsystem Report/Notes Docker Image:yetus/hadoop:c420dfe JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12797881/YARN-4751-branch-2.7.003.patch JIRA Issue YARN-4751 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 0781118c1988 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision branch-2.7 / 801c95c Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_77 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 findbugs https://builds.apache.org/job/PreCommit-YARN-Build/11010/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-warnings.html checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/11010/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt whitespace https://builds.apache.org/job/PreCommit-YARN-Build/11010/artifact/patchprocess/whitespace-eol.txt whitespace https://builds.apache.org/job/PreCommit-YARN-Build/11010/artifact/patchprocess/whitespace-tabs.txt findbugs https://builds.apache.org/job/PreCommit-YARN-Build/11010/artifact/patchprocess/new-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.html unit https://builds.apache.org/job/PreCommit-YARN-Build/11010/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_77.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/11010/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_95.txt unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/11010/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_77.txt https://builds.apache.org/job/PreCommit-YARN-Build/11010/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_95.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/11010/testReport/ asflicense https://builds.apache.org/job/PreCommit-YARN-Build/11010/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/11010/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
          Hide
          sunilg Sunil G added a comment -

          Hi Eric Payne
          I have a suggestion here. Is it better to have YARN-3362 2.7 version patch to be added in YARN-3362 and commit that. Then other changes to do here. I also feel its not very easy to do this, if its problematic I think we can do here too. What do u feel.

          Show
          sunilg Sunil G added a comment - Hi Eric Payne I have a suggestion here. Is it better to have YARN-3362 2.7 version patch to be added in YARN-3362 and commit that. Then other changes to do here. I also feel its not very easy to do this, if its problematic I think we can do here too. What do u feel.
          Hide
          eepayne Eric Payne added a comment -

          Thanks for the suggestion Sunil G:

          Is it better to have YARN-3362 2.7 version patch to be added in YARN-3362 and commit that.

          I'm okay with submitting the YARN-4751-branch-2.7.003.patch patch to YARN-3362.

          Then other changes to do here.

          What other changes would be done here?

          Show
          eepayne Eric Payne added a comment - Thanks for the suggestion Sunil G : Is it better to have YARN-3362 2.7 version patch to be added in YARN-3362 and commit that. I'm okay with submitting the YARN-4751 -branch-2.7.003.patch patch to YARN-3362 . Then other changes to do here. What other changes would be done here?
          Hide
          eepayne Eric Payne added a comment -

          Is it better to have YARN-3362 2.7 version patch to be added in YARN-3362 and commit that.

          Sunil G: Sorry, I think I understand now what you are saying. You mean, Just backport the changes from YARN-3362 and submit that as a patch there. Then, in this JIRA (YARN-4751), make additional changes to present the correct values for all of the labeled metrics. Is that correct?

          I guess I'm okay with that, but I don't really like the idea of submitting a patch to YARN-3362 that we know gives wrong metrics in the UI.

          Show
          eepayne Eric Payne added a comment - Is it better to have YARN-3362 2.7 version patch to be added in YARN-3362 and commit that. Sunil G : Sorry, I think I understand now what you are saying. You mean, Just backport the changes from YARN-3362 and submit that as a patch there. Then, in this JIRA ( YARN-4751 ), make additional changes to present the correct values for all of the labeled metrics. Is that correct? I guess I'm okay with that, but I don't really like the idea of submitting a patch to YARN-3362 that we know gives wrong metrics in the UI.
          Hide
          leftnoteasy Wangda Tan added a comment -

          Sunil G / Eric Payne,

          If I understand correctly, what Sunil G suggested is,

          • Uploaded a ver 2.7 patch to YARN-3362
          • Address other necessary changes in this patch. (If no much changes required, merging all changes to YARN-3362 is also fine to me).

          I agree with this approach in general. Ideally we should commit patch which addresses same issues with same JIRA number to different releases (regardless of if we modified patch or not), otherwise it causes issues while doing backport.

          Show
          leftnoteasy Wangda Tan added a comment - Sunil G / Eric Payne , If I understand correctly, what Sunil G suggested is, Uploaded a ver 2.7 patch to YARN-3362 Address other necessary changes in this patch. (If no much changes required, merging all changes to YARN-3362 is also fine to me). I agree with this approach in general. Ideally we should commit patch which addresses same issues with same JIRA number to different releases (regardless of if we modified patch or not), otherwise it causes issues while doing backport.
          Hide
          eepayne Eric Payne added a comment -

          f I understand correctly, what Sunil G suggested is,

          Uploaded a ver 2.7 patch to YARN-3362
          Address other necessary changes in this patch. (If no much changes required, merging all changes to YARN-3362 is also fine to me).

          I agree with this approach in general. Ideally we should commit patch which addresses same issues with same JIRA number to different releases (regardless of if we modified patch or not), otherwise it causes issues while doing backport.

          Thanks Wangda Tan. Yes, I understand how we would want to have all versions of a patch for a particular JIRA be in that same JIRA. Also, I can see how putting only the YARN-3362 backport piece into that JIRA would make it easier to review.

          My concern was that when I backported YARN-3362 to 2.7, most of the metrics were still incorrect on the labeled queues. If you think its fine to commit the YARN-3362 backported patch with several metrics producing inaccurate data and then soon after submit the fixes to those in this JIRA, I'm okay with it.

          Show
          eepayne Eric Payne added a comment - f I understand correctly, what Sunil G suggested is, Uploaded a ver 2.7 patch to YARN-3362 Address other necessary changes in this patch. (If no much changes required, merging all changes to YARN-3362 is also fine to me). I agree with this approach in general. Ideally we should commit patch which addresses same issues with same JIRA number to different releases (regardless of if we modified patch or not), otherwise it causes issues while doing backport. Thanks Wangda Tan . Yes, I understand how we would want to have all versions of a patch for a particular JIRA be in that same JIRA. Also, I can see how putting only the YARN-3362 backport piece into that JIRA would make it easier to review. My concern was that when I backported YARN-3362 to 2.7, most of the metrics were still incorrect on the labeled queues. If you think its fine to commit the YARN-3362 backported patch with several metrics producing inaccurate data and then soon after submit the fixes to those in this JIRA, I'm okay with it.
          Hide
          leftnoteasy Wangda Tan added a comment -

          My concern was that when I backported YARN-3362 to 2.7...

          My suggestion is: If Metrics released fixes are relatively small, we can put them together to YARN-3362-2.7.patch. If we need to modify more files, you can put fixes of metrics to this JIRA and update title/desc of this JIRA accordingly.

          Thoughts?

          Show
          leftnoteasy Wangda Tan added a comment - My concern was that when I backported YARN-3362 to 2.7... My suggestion is: If Metrics released fixes are relatively small, we can put them together to YARN-3362 -2.7.patch. If we need to modify more files, you can put fixes of metrics to this JIRA and update title/desc of this JIRA accordingly. Thoughts?
          Hide
          eepayne Eric Payne added a comment -

          Thanks Wangda Tan, Sunil G. I think there are enough changes needed to provide accurate methods, so I will submit the 2.7 patch for YARN-3362 there and then split out the rest of the changes for this JIRA.

          Show
          eepayne Eric Payne added a comment - Thanks Wangda Tan , Sunil G . I think there are enough changes needed to provide accurate methods, so I will submit the 2.7 patch for YARN-3362 there and then split out the rest of the changes for this JIRA.
          Hide
          eepayne Eric Payne added a comment -

          Wangda Tan / Sunil G. Thanks for your help on this issue. I am uploading version YARN-4751-branch-2.7.004.patch. It depends on the 2.7 patch for YARN-3362, and is just the fixes needed to provide accurate metrics.

          Show
          eepayne Eric Payne added a comment - Wangda Tan / Sunil G . Thanks for your help on this issue. I am uploading version YARN-4751 -branch-2.7.004.patch. It depends on the 2.7 patch for YARN-3362 , and is just the fixes needed to provide accurate metrics.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          -1 patch 0m 7s YARN-4751 does not apply to branch-2.7. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help.



          Subsystem Report/Notes
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12798877/YARN-4751-branch-2.7.004.patch
          JIRA Issue YARN-4751
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/11090/console
          Powered by Apache Yetus 0.2.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. -1 patch 0m 7s YARN-4751 does not apply to branch-2.7. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. Subsystem Report/Notes JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12798877/YARN-4751-branch-2.7.004.patch JIRA Issue YARN-4751 Console output https://builds.apache.org/job/PreCommit-YARN-Build/11090/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
          Hide
          eepayne Eric Payne added a comment -

          YARN-4751 does not apply to branch-2.7. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help.

          YARN-4751-branch-2.7.004.patch won't apply until the 2.7 patch for YARN-3362 is committed.

          Show
          eepayne Eric Payne added a comment - YARN-4751 does not apply to branch-2.7. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. YARN-4751 -branch-2.7.004.patch won't apply until the 2.7 patch for YARN-3362 is committed.
          Hide
          Naganarasimha Naganarasimha G R added a comment -

          Hi Eric Payne, Have triggered the jenkins build. Lets try to push this at the earliest.

          Show
          Naganarasimha Naganarasimha G R added a comment - Hi Eric Payne , Have triggered the jenkins build. Lets try to push this at the earliest.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 23m 19s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 mvninstall 9m 31s branch-2.7 passed
          +1 compile 0m 27s branch-2.7 passed with JDK v1.8.0_91
          +1 compile 0m 30s branch-2.7 passed with JDK v1.7.0_101
          +1 checkstyle 0m 43s branch-2.7 passed
          +1 mvnsite 0m 38s branch-2.7 passed
          +1 mvneclipse 0m 19s branch-2.7 passed
          -1 findbugs 1m 26s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager in branch-2.7 has 1 extant Findbugs warnings.
          +1 javadoc 0m 25s branch-2.7 passed with JDK v1.8.0_91
          +1 javadoc 0m 27s branch-2.7 passed with JDK v1.7.0_101
          +1 mvninstall 0m 29s the patch passed
          +1 compile 0m 25s the patch passed with JDK v1.8.0_91
          +1 javac 0m 25s the patch passed
          +1 compile 0m 26s the patch passed with JDK v1.7.0_101
          +1 javac 0m 26s the patch passed
          -1 checkstyle 0m 37s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: patch generated 21 new + 1299 unchanged - 3 fixed = 1320 total (was 1302)
          +1 mvnsite 0m 33s the patch passed
          +1 mvneclipse 0m 12s the patch passed
          -1 whitespace 0m 0s The patch has 2558 line(s) that end in whitespace. Use git apply --whitespace=fix.
          -1 whitespace 1m 4s The patch has 295 line(s) with tabs.
          -1 findbugs 1m 26s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager generated 2 new + 1 unchanged - 0 fixed = 3 total (was 1)
          +1 javadoc 0m 22s the patch passed with JDK v1.8.0_91
          +1 javadoc 0m 24s the patch passed with JDK v1.7.0_101
          -1 unit 55m 41s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.8.0_91.
          -1 unit 56m 15s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.7.0_101.
          +1 asflicense 0m 18s Patch does not generate ASF License warnings.
          157m 32s



          Reason Tests
          FindBugs module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
            Inconsistent synchronization of org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.AbstractCSQueue.cr; locked 57% of time Unsynchronized access at AbstractCSQueue.java:57% of time Unsynchronized access at AbstractCSQueue.java:[line 558]
            org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.AbstractCSQueue.getAbsoluteUsedCapacity(String) is unsynchronized, org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.AbstractCSQueue.setAbsoluteUsedCapacity(float) is synchronized At AbstractCSQueue.java:synchronized At AbstractCSQueue.java:[lines 558-563]
          JDK v1.8.0_91 Failed junit tests hadoop.yarn.server.resourcemanager.TestRMRestart
            hadoop.yarn.server.resourcemanager.TestClientRMTokens
            hadoop.yarn.server.resourcemanager.TestAMAuthorization
          JDK v1.7.0_101 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens
            hadoop.yarn.server.resourcemanager.TestAMAuthorization



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:c420dfe
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12798877/YARN-4751-branch-2.7.004.patch
          JIRA Issue YARN-4751
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 75aa6ab8f7f2 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision branch-2.7 / 8f17395
          Default Java 1.7.0_101
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_91 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_101
          findbugs v3.0.0
          findbugs https://builds.apache.org/job/PreCommit-YARN-Build/11480/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-warnings.html
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/11480/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
          whitespace https://builds.apache.org/job/PreCommit-YARN-Build/11480/artifact/patchprocess/whitespace-eol.txt
          whitespace https://builds.apache.org/job/PreCommit-YARN-Build/11480/artifact/patchprocess/whitespace-tabs.txt
          findbugs https://builds.apache.org/job/PreCommit-YARN-Build/11480/artifact/patchprocess/new-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.html
          unit https://builds.apache.org/job/PreCommit-YARN-Build/11480/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_91.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/11480/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_101.txt
          unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/11480/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_91.txt https://builds.apache.org/job/PreCommit-YARN-Build/11480/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_101.txt
          JDK v1.7.0_101 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/11480/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/11480/console
          Powered by Apache Yetus 0.2.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 23m 19s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 9m 31s branch-2.7 passed +1 compile 0m 27s branch-2.7 passed with JDK v1.8.0_91 +1 compile 0m 30s branch-2.7 passed with JDK v1.7.0_101 +1 checkstyle 0m 43s branch-2.7 passed +1 mvnsite 0m 38s branch-2.7 passed +1 mvneclipse 0m 19s branch-2.7 passed -1 findbugs 1m 26s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager in branch-2.7 has 1 extant Findbugs warnings. +1 javadoc 0m 25s branch-2.7 passed with JDK v1.8.0_91 +1 javadoc 0m 27s branch-2.7 passed with JDK v1.7.0_101 +1 mvninstall 0m 29s the patch passed +1 compile 0m 25s the patch passed with JDK v1.8.0_91 +1 javac 0m 25s the patch passed +1 compile 0m 26s the patch passed with JDK v1.7.0_101 +1 javac 0m 26s the patch passed -1 checkstyle 0m 37s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: patch generated 21 new + 1299 unchanged - 3 fixed = 1320 total (was 1302) +1 mvnsite 0m 33s the patch passed +1 mvneclipse 0m 12s the patch passed -1 whitespace 0m 0s The patch has 2558 line(s) that end in whitespace. Use git apply --whitespace=fix. -1 whitespace 1m 4s The patch has 295 line(s) with tabs. -1 findbugs 1m 26s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager generated 2 new + 1 unchanged - 0 fixed = 3 total (was 1) +1 javadoc 0m 22s the patch passed with JDK v1.8.0_91 +1 javadoc 0m 24s the patch passed with JDK v1.7.0_101 -1 unit 55m 41s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.8.0_91. -1 unit 56m 15s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.7.0_101. +1 asflicense 0m 18s Patch does not generate ASF License warnings. 157m 32s Reason Tests FindBugs module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager   Inconsistent synchronization of org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.AbstractCSQueue.cr; locked 57% of time Unsynchronized access at AbstractCSQueue.java:57% of time Unsynchronized access at AbstractCSQueue.java: [line 558]   org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.AbstractCSQueue.getAbsoluteUsedCapacity(String) is unsynchronized, org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.AbstractCSQueue.setAbsoluteUsedCapacity(float) is synchronized At AbstractCSQueue.java:synchronized At AbstractCSQueue.java: [lines 558-563] JDK v1.8.0_91 Failed junit tests hadoop.yarn.server.resourcemanager.TestRMRestart   hadoop.yarn.server.resourcemanager.TestClientRMTokens   hadoop.yarn.server.resourcemanager.TestAMAuthorization JDK v1.7.0_101 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens   hadoop.yarn.server.resourcemanager.TestAMAuthorization Subsystem Report/Notes Docker Image:yetus/hadoop:c420dfe JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12798877/YARN-4751-branch-2.7.004.patch JIRA Issue YARN-4751 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 75aa6ab8f7f2 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision branch-2.7 / 8f17395 Default Java 1.7.0_101 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_91 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_101 findbugs v3.0.0 findbugs https://builds.apache.org/job/PreCommit-YARN-Build/11480/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-warnings.html checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/11480/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt whitespace https://builds.apache.org/job/PreCommit-YARN-Build/11480/artifact/patchprocess/whitespace-eol.txt whitespace https://builds.apache.org/job/PreCommit-YARN-Build/11480/artifact/patchprocess/whitespace-tabs.txt findbugs https://builds.apache.org/job/PreCommit-YARN-Build/11480/artifact/patchprocess/new-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.html unit https://builds.apache.org/job/PreCommit-YARN-Build/11480/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_91.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/11480/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_101.txt unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/11480/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_91.txt https://builds.apache.org/job/PreCommit-YARN-Build/11480/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_101.txt JDK v1.7.0_101 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/11480/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/11480/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
          Hide
          Naganarasimha Naganarasimha G R added a comment -

          Hi Eric Payne, Can you please take a look at the findbugs, and checkstyle issues seems like some of them are related to the patch !

          Show
          Naganarasimha Naganarasimha G R added a comment - Hi Eric Payne , Can you please take a look at the findbugs, and checkstyle issues seems like some of them are related to the patch !
          Hide
          eepayne Eric Payne added a comment -

          Thanks, Naganarasimha G R. You are correct, most of the findbugs and checkstyle warnings are related to this patch. I am uploading new patch addressing them:
          YARN-4751-branch-2.7.005.patch

          Show
          eepayne Eric Payne added a comment - Thanks, Naganarasimha G R . You are correct, most of the findbugs and checkstyle warnings are related to this patch. I am uploading new patch addressing them: YARN-4751 -branch-2.7.005.patch
          Hide
          eepayne Eric Payne added a comment -

          I am cancelling and resubmitting the patch in an attempt to kick the branch-2.7 build.

          Show
          eepayne Eric Payne added a comment - I am cancelling and resubmitting the patch in an attempt to kick the branch-2.7 build.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 15s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 mvninstall 6m 22s branch-2.7 passed
          +1 compile 0m 25s branch-2.7 passed with JDK v1.8.0_91
          +1 compile 0m 27s branch-2.7 passed with JDK v1.7.0_101
          +1 checkstyle 0m 33s branch-2.7 passed
          +1 mvnsite 0m 34s branch-2.7 passed
          +1 mvneclipse 0m 14s branch-2.7 passed
          -1 findbugs 1m 5s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager in branch-2.7 has 1 extant Findbugs warnings.
          +1 javadoc 0m 19s branch-2.7 passed with JDK v1.8.0_91
          +1 javadoc 0m 23s branch-2.7 passed with JDK v1.7.0_101
          +1 mvninstall 0m 28s the patch passed
          +1 compile 0m 23s the patch passed with JDK v1.8.0_91
          +1 javac 0m 23s the patch passed
          +1 compile 0m 24s the patch passed with JDK v1.7.0_101
          +1 javac 0m 24s the patch passed
          -1 checkstyle 0m 30s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: patch generated 17 new + 1298 unchanged - 4 fixed = 1315 total (was 1302)
          +1 mvnsite 0m 33s the patch passed
          +1 mvneclipse 0m 11s the patch passed
          -1 whitespace 0m 1s The patch has 2933 line(s) that end in whitespace. Use git apply --whitespace=fix.
          -1 whitespace 1m 6s The patch has 295 line(s) with tabs.
          +1 findbugs 1m 17s the patch passed
          +1 javadoc 0m 18s the patch passed with JDK v1.8.0_91
          +1 javadoc 0m 22s the patch passed with JDK v1.7.0_101
          -1 unit 50m 21s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.8.0_91.
          -1 unit 51m 46s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.7.0_101.
          +1 asflicense 0m 16s Patch does not generate ASF License warnings.
          119m 56s



          Reason Tests
          JDK v1.8.0_91 Failed junit tests hadoop.yarn.server.resourcemanager.TestAMAuthorization
            hadoop.yarn.server.resourcemanager.TestClientRMTokens
          JDK v1.7.0_101 Failed junit tests hadoop.yarn.server.resourcemanager.TestAMAuthorization
            hadoop.yarn.server.resourcemanager.TestClientRMTokens



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:c420dfe
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12804758/YARN-4751-branch-2.7.005.patch
          JIRA Issue YARN-4751
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 5b2c3113459b 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision branch-2.7 / 0abb366
          Default Java 1.7.0_101
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_91 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_101
          findbugs v3.0.0
          findbugs https://builds.apache.org/job/PreCommit-YARN-Build/11596/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-warnings.html
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/11596/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
          whitespace https://builds.apache.org/job/PreCommit-YARN-Build/11596/artifact/patchprocess/whitespace-eol.txt
          whitespace https://builds.apache.org/job/PreCommit-YARN-Build/11596/artifact/patchprocess/whitespace-tabs.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/11596/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_91.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/11596/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_101.txt
          unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/11596/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_91.txt https://builds.apache.org/job/PreCommit-YARN-Build/11596/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_101.txt
          JDK v1.7.0_101 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/11596/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/11596/console
          Powered by Apache Yetus 0.2.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 15s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 6m 22s branch-2.7 passed +1 compile 0m 25s branch-2.7 passed with JDK v1.8.0_91 +1 compile 0m 27s branch-2.7 passed with JDK v1.7.0_101 +1 checkstyle 0m 33s branch-2.7 passed +1 mvnsite 0m 34s branch-2.7 passed +1 mvneclipse 0m 14s branch-2.7 passed -1 findbugs 1m 5s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager in branch-2.7 has 1 extant Findbugs warnings. +1 javadoc 0m 19s branch-2.7 passed with JDK v1.8.0_91 +1 javadoc 0m 23s branch-2.7 passed with JDK v1.7.0_101 +1 mvninstall 0m 28s the patch passed +1 compile 0m 23s the patch passed with JDK v1.8.0_91 +1 javac 0m 23s the patch passed +1 compile 0m 24s the patch passed with JDK v1.7.0_101 +1 javac 0m 24s the patch passed -1 checkstyle 0m 30s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: patch generated 17 new + 1298 unchanged - 4 fixed = 1315 total (was 1302) +1 mvnsite 0m 33s the patch passed +1 mvneclipse 0m 11s the patch passed -1 whitespace 0m 1s The patch has 2933 line(s) that end in whitespace. Use git apply --whitespace=fix. -1 whitespace 1m 6s The patch has 295 line(s) with tabs. +1 findbugs 1m 17s the patch passed +1 javadoc 0m 18s the patch passed with JDK v1.8.0_91 +1 javadoc 0m 22s the patch passed with JDK v1.7.0_101 -1 unit 50m 21s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.8.0_91. -1 unit 51m 46s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.7.0_101. +1 asflicense 0m 16s Patch does not generate ASF License warnings. 119m 56s Reason Tests JDK v1.8.0_91 Failed junit tests hadoop.yarn.server.resourcemanager.TestAMAuthorization   hadoop.yarn.server.resourcemanager.TestClientRMTokens JDK v1.7.0_101 Failed junit tests hadoop.yarn.server.resourcemanager.TestAMAuthorization   hadoop.yarn.server.resourcemanager.TestClientRMTokens Subsystem Report/Notes Docker Image:yetus/hadoop:c420dfe JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12804758/YARN-4751-branch-2.7.005.patch JIRA Issue YARN-4751 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 5b2c3113459b 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision branch-2.7 / 0abb366 Default Java 1.7.0_101 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_91 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_101 findbugs v3.0.0 findbugs https://builds.apache.org/job/PreCommit-YARN-Build/11596/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-warnings.html checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/11596/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt whitespace https://builds.apache.org/job/PreCommit-YARN-Build/11596/artifact/patchprocess/whitespace-eol.txt whitespace https://builds.apache.org/job/PreCommit-YARN-Build/11596/artifact/patchprocess/whitespace-tabs.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/11596/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_91.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/11596/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_101.txt unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/11596/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_91.txt https://builds.apache.org/job/PreCommit-YARN-Build/11596/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_101.txt JDK v1.7.0_101 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/11596/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/11596/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
          Hide
          Naganarasimha Naganarasimha G R added a comment -

          Hi Eric Payne,
          Latest patch LGTM and the findbugs and testcase failures is not related to this jira also have created the build and tested it locally. So will wait few hours before committing this for feedback.
          cc Sunil G

          Show
          Naganarasimha Naganarasimha G R added a comment - Hi Eric Payne , Latest patch LGTM and the findbugs and testcase failures is not related to this jira also have created the build and tested it locally. So will wait few hours before committing this for feedback. cc Sunil G
          Hide
          sunilg Sunil G added a comment -

          Patch looks fine for me. +1

          Show
          sunilg Sunil G added a comment - Patch looks fine for me. +1
          Hide
          Naganarasimha Naganarasimha G R added a comment -

          Thanks Sunil G for the final review, committing this patch in. Thanks for the contribution Eric Payne and Sunil G & Tan, Wangda for reviews. Committed this patch to 2.7 branch

          Show
          Naganarasimha Naganarasimha G R added a comment - Thanks Sunil G for the final review, committing this patch in. Thanks for the contribution Eric Payne and Sunil G & Tan, Wangda for reviews. Committed this patch to 2.7 branch
          Hide
          Naganarasimha Naganarasimha G R added a comment -

          Thanks Sunil G for the final review, committing this patch in. Thanks for the contribution Eric Payne and Sunil G & Tan, Wangda for reviews. Committed this patch to 2.7 branch

          Show
          Naganarasimha Naganarasimha G R added a comment - Thanks Sunil G for the final review, committing this patch in. Thanks for the contribution Eric Payne and Sunil G & Tan, Wangda for reviews. Committed this patch to 2.7 branch
          Hide
          eepayne Eric Payne added a comment -
          Show
          eepayne Eric Payne added a comment - Thanks Naganarasimha G R !
          Hide
          eepayne Eric Payne added a comment -

          Changing Fix Version to 2.7.3 since branch 2.7.3 has not yet been created.

          Show
          eepayne Eric Payne added a comment - Changing Fix Version to 2.7.3 since branch 2.7.3 has not yet been created.
          Hide
          vinodkv Vinod Kumar Vavilapalli added a comment -

          Closing the JIRA as part of 2.7.3 release.

          Show
          vinodkv Vinod Kumar Vavilapalli added a comment - Closing the JIRA as part of 2.7.3 release.

            People

            • Assignee:
              eepayne Eric Payne
              Reporter:
              eepayne Eric Payne
            • Votes:
              0 Vote for this issue
              Watchers:
              11 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development