Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-6164

Expose Queue Configurations per Node Label through YARN client api

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.7.2
    • Fix Version/s: 2.9.0, 3.0.0-alpha4
    • Component/s: None
    • Labels:
      None

      Description

      `yarn.scheduler.capacity.maximum-am-resource-percent` is exposed through the Cluster Scheduler API, but not through YarnClient.

      Since YarnClient and RM REST APIs depend on different ports (8032 vs 8088 by default), it would be nice to expose `maximum-am-resource-percent` in YarnClient as well.

      1. YARN-6164.001.patch
        14 kB
        Benson Qiu
      2. YARN-6164.002.patch
        9 kB
        Benson Qiu
      3. YARN-6164.003.patch
        9 kB
        Benson Qiu
      4. YARN-6164.004.patch
        20 kB
        Benson Qiu
      5. YARN-6164.005.patch
        21 kB
        Benson Qiu
      6. YARN-6164.006.patch
        28 kB
        Benson Qiu
      7. YARN-6164.007.patch
        31 kB
        Benson Qiu
      8. YARN-6164.008.patch
        33 kB
        Benson Qiu
      9. YARN-6164-branch-2.001.patch
        28 kB
        Benson Qiu

        Activity

        Hide
        benson.qiu Benson Qiu added a comment -

        Thanks Sunil G for committing this patch! Thanks Bibin A Chundatt, Wangda Tan, Rohith Sharma K S, churro morales!

        Show
        benson.qiu Benson Qiu added a comment - Thanks Sunil G for committing this patch! Thanks Bibin A Chundatt , Wangda Tan , Rohith Sharma K S , churro morales !
        Hide
        sunilg Sunil G added a comment -

        Committed to trunk/branch-2. Thanks for the contribution Benson Qiu. Thanks Rohith Sharma K S Wangda Tan Bibin A Chundatt for additional reviews and thoughts.

        Show
        sunilg Sunil G added a comment - Committed to trunk/branch-2. Thanks for the contribution Benson Qiu . Thanks Rohith Sharma K S Wangda Tan Bibin A Chundatt for additional reviews and thoughts.
        Hide
        benson.qiu Benson Qiu added a comment -

        Looks like the build is fixed now (YARN-6501).

        Show
        benson.qiu Benson Qiu added a comment - Looks like the build is fixed now ( YARN-6501 ).
        Hide
        sunilg Sunil G added a comment -

        Seems branch-2 build is broken now w/o this patch.
        I will raise a ticket and we will commit this patch once branch-2 is stabilized.

        Show
        sunilg Sunil G added a comment - Seems branch-2 build is broken now w/o this patch. I will raise a ticket and we will commit this patch once branch-2 is stabilized.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 15m 47s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 3 new or modified test files.
        0 mvndep 0m 40s Maven dependency ordering for branch
        -1 mvninstall 3m 50s root in branch-2 failed.
        +1 compile 1m 44s branch-2 passed with JDK v1.8.0_121
        -1 compile 1m 29s hadoop-yarn in branch-2 failed with JDK v1.7.0_121.
        +1 checkstyle 0m 40s branch-2 passed
        -1 mvnsite 0m 21s hadoop-yarn-server-resourcemanager in branch-2 failed.
        +1 mvneclipse 0m 57s branch-2 passed
        -1 findbugs 0m 18s hadoop-yarn-server-resourcemanager in branch-2 failed.
        +1 javadoc 1m 12s branch-2 passed with JDK v1.8.0_121
        +1 javadoc 1m 29s branch-2 passed with JDK v1.7.0_121
        0 mvndep 0m 10s Maven dependency ordering for patch
        -1 mvninstall 0m 20s hadoop-yarn-server-resourcemanager in the patch failed.
        +1 compile 1m 43s the patch passed with JDK v1.8.0_121
        +1 cc 1m 43s the patch passed
        +1 javac 1m 43s the patch passed
        -1 compile 1m 28s hadoop-yarn in the patch failed with JDK v1.7.0_121.
        -1 cc 1m 28s hadoop-yarn in the patch failed with JDK v1.7.0_121.
        -1 javac 1m 28s hadoop-yarn in the patch failed with JDK v1.7.0_121.
        -0 checkstyle 0m 41s hadoop-yarn-project/hadoop-yarn: The patch generated 6 new + 242 unchanged - 0 fixed = 248 total (was 242)
        -1 mvnsite 0m 23s hadoop-yarn-server-resourcemanager in the patch failed.
        +1 mvneclipse 0m 56s the patch passed
        +1 whitespace 0m 1s The patch has no whitespace issues.
        -1 findbugs 0m 19s hadoop-yarn-server-resourcemanager in the patch failed.
        +1 javadoc 1m 14s the patch passed with JDK v1.8.0_121
        +1 javadoc 1m 31s the patch passed with JDK v1.7.0_121
        +1 unit 0m 28s hadoop-yarn-api in the patch passed with JDK v1.7.0_121.
        +1 unit 2m 33s hadoop-yarn-common in the patch passed with JDK v1.7.0_121.
        -1 unit 0m 22s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.7.0_121.
        +1 unit 19m 57s hadoop-yarn-client in the patch passed with JDK v1.7.0_121.
        +1 asflicense 0m 22s The patch does not generate ASF License warnings.
        143m 41s



        Reason Tests
        JDK v1.8.0_121 Failed junit tests hadoop.yarn.server.resourcemanager.TestRMRestart
          hadoop.yarn.server.resourcemanager.TestResourceTrackerService



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:8515d35
        JIRA Issue YARN-6164
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12864200/YARN-6164-branch-2.001.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc
        uname Linux 261bd4a87e68 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12 13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision branch-2 / 6865746
        Default Java 1.7.0_121
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_121 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_121
        mvninstall https://builds.apache.org/job/PreCommit-YARN-Build/15688/artifact/patchprocess/branch-mvninstall-root.txt
        compile https://builds.apache.org/job/PreCommit-YARN-Build/15688/artifact/patchprocess/branch-compile-hadoop-yarn-project_hadoop-yarn-jdk1.7.0_121.txt
        mvnsite https://builds.apache.org/job/PreCommit-YARN-Build/15688/artifact/patchprocess/branch-mvnsite-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
        findbugs v3.0.0
        findbugs https://builds.apache.org/job/PreCommit-YARN-Build/15688/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
        mvninstall https://builds.apache.org/job/PreCommit-YARN-Build/15688/artifact/patchprocess/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
        compile https://builds.apache.org/job/PreCommit-YARN-Build/15688/artifact/patchprocess/patch-compile-hadoop-yarn-project_hadoop-yarn-jdk1.7.0_121.txt
        cc https://builds.apache.org/job/PreCommit-YARN-Build/15688/artifact/patchprocess/patch-compile-hadoop-yarn-project_hadoop-yarn-jdk1.7.0_121.txt
        javac https://builds.apache.org/job/PreCommit-YARN-Build/15688/artifact/patchprocess/patch-compile-hadoop-yarn-project_hadoop-yarn-jdk1.7.0_121.txt
        checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/15688/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt
        mvnsite https://builds.apache.org/job/PreCommit-YARN-Build/15688/artifact/patchprocess/patch-mvnsite-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
        findbugs https://builds.apache.org/job/PreCommit-YARN-Build/15688/artifact/patchprocess/patch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
        unit https://builds.apache.org/job/PreCommit-YARN-Build/15688/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_121.txt
        JDK v1.7.0_121 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/15688/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client U: hadoop-yarn-project/hadoop-yarn
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/15688/console
        Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 15m 47s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 3 new or modified test files. 0 mvndep 0m 40s Maven dependency ordering for branch -1 mvninstall 3m 50s root in branch-2 failed. +1 compile 1m 44s branch-2 passed with JDK v1.8.0_121 -1 compile 1m 29s hadoop-yarn in branch-2 failed with JDK v1.7.0_121. +1 checkstyle 0m 40s branch-2 passed -1 mvnsite 0m 21s hadoop-yarn-server-resourcemanager in branch-2 failed. +1 mvneclipse 0m 57s branch-2 passed -1 findbugs 0m 18s hadoop-yarn-server-resourcemanager in branch-2 failed. +1 javadoc 1m 12s branch-2 passed with JDK v1.8.0_121 +1 javadoc 1m 29s branch-2 passed with JDK v1.7.0_121 0 mvndep 0m 10s Maven dependency ordering for patch -1 mvninstall 0m 20s hadoop-yarn-server-resourcemanager in the patch failed. +1 compile 1m 43s the patch passed with JDK v1.8.0_121 +1 cc 1m 43s the patch passed +1 javac 1m 43s the patch passed -1 compile 1m 28s hadoop-yarn in the patch failed with JDK v1.7.0_121. -1 cc 1m 28s hadoop-yarn in the patch failed with JDK v1.7.0_121. -1 javac 1m 28s hadoop-yarn in the patch failed with JDK v1.7.0_121. -0 checkstyle 0m 41s hadoop-yarn-project/hadoop-yarn: The patch generated 6 new + 242 unchanged - 0 fixed = 248 total (was 242) -1 mvnsite 0m 23s hadoop-yarn-server-resourcemanager in the patch failed. +1 mvneclipse 0m 56s the patch passed +1 whitespace 0m 1s The patch has no whitespace issues. -1 findbugs 0m 19s hadoop-yarn-server-resourcemanager in the patch failed. +1 javadoc 1m 14s the patch passed with JDK v1.8.0_121 +1 javadoc 1m 31s the patch passed with JDK v1.7.0_121 +1 unit 0m 28s hadoop-yarn-api in the patch passed with JDK v1.7.0_121. +1 unit 2m 33s hadoop-yarn-common in the patch passed with JDK v1.7.0_121. -1 unit 0m 22s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.7.0_121. +1 unit 19m 57s hadoop-yarn-client in the patch passed with JDK v1.7.0_121. +1 asflicense 0m 22s The patch does not generate ASF License warnings. 143m 41s Reason Tests JDK v1.8.0_121 Failed junit tests hadoop.yarn.server.resourcemanager.TestRMRestart   hadoop.yarn.server.resourcemanager.TestResourceTrackerService Subsystem Report/Notes Docker Image:yetus/hadoop:8515d35 JIRA Issue YARN-6164 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12864200/YARN-6164-branch-2.001.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc uname Linux 261bd4a87e68 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12 13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision branch-2 / 6865746 Default Java 1.7.0_121 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_121 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_121 mvninstall https://builds.apache.org/job/PreCommit-YARN-Build/15688/artifact/patchprocess/branch-mvninstall-root.txt compile https://builds.apache.org/job/PreCommit-YARN-Build/15688/artifact/patchprocess/branch-compile-hadoop-yarn-project_hadoop-yarn-jdk1.7.0_121.txt mvnsite https://builds.apache.org/job/PreCommit-YARN-Build/15688/artifact/patchprocess/branch-mvnsite-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt findbugs v3.0.0 findbugs https://builds.apache.org/job/PreCommit-YARN-Build/15688/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt mvninstall https://builds.apache.org/job/PreCommit-YARN-Build/15688/artifact/patchprocess/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt compile https://builds.apache.org/job/PreCommit-YARN-Build/15688/artifact/patchprocess/patch-compile-hadoop-yarn-project_hadoop-yarn-jdk1.7.0_121.txt cc https://builds.apache.org/job/PreCommit-YARN-Build/15688/artifact/patchprocess/patch-compile-hadoop-yarn-project_hadoop-yarn-jdk1.7.0_121.txt javac https://builds.apache.org/job/PreCommit-YARN-Build/15688/artifact/patchprocess/patch-compile-hadoop-yarn-project_hadoop-yarn-jdk1.7.0_121.txt checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/15688/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt mvnsite https://builds.apache.org/job/PreCommit-YARN-Build/15688/artifact/patchprocess/patch-mvnsite-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt findbugs https://builds.apache.org/job/PreCommit-YARN-Build/15688/artifact/patchprocess/patch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/15688/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_121.txt JDK v1.7.0_121 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/15688/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client U: hadoop-yarn-project/hadoop-yarn Console output https://builds.apache.org/job/PreCommit-YARN-Build/15688/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        sunilg Sunil G added a comment -

        Benson Qiu Thanks for uploading branch-2 patch.
        I see that you have added YARN-6164.009.patch. Hence above jenkins came as a failure.
        I think it must be branch-2 and you have already done same. So you can delete this v9 patch to avoid confusion.

        Show
        sunilg Sunil G added a comment - Benson Qiu Thanks for uploading branch-2 patch. I see that you have added YARN-6164 .009.patch . Hence above jenkins came as a failure. I think it must be branch-2 and you have already done same. So you can delete this v9 patch to avoid confusion.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 0s Docker mode activated.
        -1 patch 0m 7s YARN-6164 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help.



        Subsystem Report/Notes
        JIRA Issue YARN-6164
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12864198/YARN-6164.009.patch
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/15687/console
        Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. -1 patch 0m 7s YARN-6164 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. Subsystem Report/Notes JIRA Issue YARN-6164 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12864198/YARN-6164.009.patch Console output https://builds.apache.org/job/PreCommit-YARN-Build/15687/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11605 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11605/)
        YARN-6164. Expose Queue Configurations per Node Label through YARN (sunilg: rev 9ff25408bc096dfc0a8522106eec54282303c7ab)

        • (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/QueueConfigurations.java
        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/QueueInfoPBImpl.java
        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/QueueInfo.java
        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/AbstractCSQueue.java
        • (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/QueueConfigurationsPBImpl.java
        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestClientRMService.java
        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/api/TestPBImplRecords.java
        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/yarn_protos.proto
        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/cli/TestYarnCLI.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11605 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11605/ ) YARN-6164 . Expose Queue Configurations per Node Label through YARN (sunilg: rev 9ff25408bc096dfc0a8522106eec54282303c7ab) (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/QueueConfigurations.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/QueueInfoPBImpl.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/QueueInfo.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/AbstractCSQueue.java (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/QueueConfigurationsPBImpl.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestClientRMService.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/api/TestPBImplRecords.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/yarn_protos.proto (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/cli/TestYarnCLI.java
        Hide
        sunilg Sunil G added a comment -

        Committed to trunk. Benson Qiu, however this patch is not getting cleanly applied to branch-2. There are some conflicts. Could you please give me a clean patch on branch-2. Thank you.

        Show
        sunilg Sunil G added a comment - Committed to trunk. Benson Qiu , however this patch is not getting cleanly applied to branch-2. There are some conflicts. Could you please give me a clean patch on branch-2. Thank you.
        Hide
        rohithsharma Rohith Sharma K S added a comment -

        I took a glance at the patch, +1 LGTM.

        Show
        rohithsharma Rohith Sharma K S added a comment - I took a glance at the patch, +1 LGTM.
        Hide
        sunilg Sunil G added a comment -

        Yes. I ll be committing the same later today.

        Show
        sunilg Sunil G added a comment - Yes. I ll be committing the same later today.
        Hide
        benson.qiu Benson Qiu added a comment -

        Sunil G: Is it okay to move forward with this patch?

        Show
        benson.qiu Benson Qiu added a comment - Sunil G : Is it okay to move forward with this patch?
        Hide
        sunilg Sunil G added a comment -

        I will commit tomorrow if there are no objections.

        Show
        sunilg Sunil G added a comment - I will commit tomorrow if there are no objections.
        Hide
        sunilg Sunil G added a comment -

        New patch looks good to me. I ll wait for Wangda Tan also to take a final look. After that I will commit the same.
        Thank you Benson Qiu.

        Show
        sunilg Sunil G added a comment - New patch looks good to me. I ll wait for Wangda Tan also to take a final look. After that I will commit the same. Thank you Benson Qiu .
        Hide
        benson.qiu Benson Qiu added a comment -

        Sunil G: Mind taking a look at the latest patch?

        Show
        benson.qiu Benson Qiu added a comment - Sunil G : Mind taking a look at the latest patch?
        Hide
        benson.qiu Benson Qiu added a comment -

        Thanks Sunil G for the review.

        Here are my the changes for version 8 based on your comments:

        1. Added Apache license header to QueueConfigurations and QueueConfigurationsPBImpl.
        2. I ran TestRMRestart.testFinishedAppRemovalAfterRMRestart several times locally, seems flaky.
        3. Fixed most of the checkstyle comments.
        4. Made QueueConfigurationsProto optional in yarn_protos.proto
        5. Uncommented setCapacity in QueueConfigurationPBImpl.java. Thanks for catching that.
        Show
        benson.qiu Benson Qiu added a comment - Thanks Sunil G for the review. Here are my the changes for version 8 based on your comments: Added Apache license header to QueueConfigurations and QueueConfigurationsPBImpl. I ran TestRMRestart.testFinishedAppRemovalAfterRMRestart several times locally, seems flaky. Fixed most of the checkstyle comments. Made QueueConfigurationsProto optional in yarn_protos.proto Uncommented setCapacity in QueueConfigurationPBImpl.java. Thanks for catching that.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 18s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 3 new or modified test files.
        0 mvndep 0m 11s Maven dependency ordering for branch
        +1 mvninstall 13m 19s trunk passed
        +1 compile 11m 31s trunk passed
        +1 checkstyle 0m 59s trunk passed
        +1 mvnsite 2m 28s trunk passed
        +1 mvneclipse 1m 21s trunk passed
        +1 findbugs 3m 53s trunk passed
        +1 javadoc 1m 46s trunk passed
        0 mvndep 0m 10s Maven dependency ordering for patch
        +1 mvninstall 1m 43s the patch passed
        +1 compile 8m 59s the patch passed
        +1 cc 8m 59s the patch passed
        +1 javac 8m 59s the patch passed
        -0 checkstyle 0m 55s hadoop-yarn-project/hadoop-yarn: The patch generated 6 new + 237 unchanged - 0 fixed = 243 total (was 237)
        +1 mvnsite 2m 21s the patch passed
        +1 mvneclipse 1m 19s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 5m 9s the patch passed
        +1 javadoc 1m 48s the patch passed
        +1 unit 0m 35s hadoop-yarn-api in the patch passed.
        +1 unit 2m 26s hadoop-yarn-common in the patch passed.
        -1 unit 38m 55s hadoop-yarn-server-resourcemanager in the patch failed.
        +1 unit 19m 32s hadoop-yarn-client in the patch passed.
        +1 asflicense 0m 38s The patch does not generate ASF License warnings.
        129m 12s



        Reason Tests
        Failed junit tests hadoop.yarn.server.resourcemanager.TestRMRestart



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:a9ad5d6
        JIRA Issue YARN-6164
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12861273/YARN-6164.008.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc
        uname Linux 92feee8b6808 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 7297e82
        Default Java 1.8.0_121
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/15440/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt
        unit https://builds.apache.org/job/PreCommit-YARN-Build/15440/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/15440/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client U: hadoop-yarn-project/hadoop-yarn
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/15440/console
        Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 18s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 3 new or modified test files. 0 mvndep 0m 11s Maven dependency ordering for branch +1 mvninstall 13m 19s trunk passed +1 compile 11m 31s trunk passed +1 checkstyle 0m 59s trunk passed +1 mvnsite 2m 28s trunk passed +1 mvneclipse 1m 21s trunk passed +1 findbugs 3m 53s trunk passed +1 javadoc 1m 46s trunk passed 0 mvndep 0m 10s Maven dependency ordering for patch +1 mvninstall 1m 43s the patch passed +1 compile 8m 59s the patch passed +1 cc 8m 59s the patch passed +1 javac 8m 59s the patch passed -0 checkstyle 0m 55s hadoop-yarn-project/hadoop-yarn: The patch generated 6 new + 237 unchanged - 0 fixed = 243 total (was 237) +1 mvnsite 2m 21s the patch passed +1 mvneclipse 1m 19s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 5m 9s the patch passed +1 javadoc 1m 48s the patch passed +1 unit 0m 35s hadoop-yarn-api in the patch passed. +1 unit 2m 26s hadoop-yarn-common in the patch passed. -1 unit 38m 55s hadoop-yarn-server-resourcemanager in the patch failed. +1 unit 19m 32s hadoop-yarn-client in the patch passed. +1 asflicense 0m 38s The patch does not generate ASF License warnings. 129m 12s Reason Tests Failed junit tests hadoop.yarn.server.resourcemanager.TestRMRestart Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue YARN-6164 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12861273/YARN-6164.008.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc uname Linux 92feee8b6808 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 7297e82 Default Java 1.8.0_121 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/15440/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/15440/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/15440/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client U: hadoop-yarn-project/hadoop-yarn Console output https://builds.apache.org/job/PreCommit-YARN-Build/15440/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        sunilg Sunil G added a comment -

        Thanks Benson Qiu for the updated patch.

        Few minor comments

        1. QueueConfigurations and QueueConfigurationsPBImpl does not have Apache license header. Please add the same. This was the reason for ASF License red failure in jenkins.
        2. Test failure seems random. YARN-5652 is already raised for that. Hence its wont be a problem from your patch.
        3. Could you please check checkstyle failures from jenkins. Some comments looks good to be fixed in that. (like unused imports). Whichever could be fixed and possible, you could give a try.
        4. In yarn_protos.proto for * QueueConfigurationsMapProto*, QueueConfigurationsProto could be made optional.
        5. In QueueConfigurationsPBImpl.java, setCapacity seems not override. Its commented, is it intended?

        I think rest all changes looks fine. Thank You.

        Show
        sunilg Sunil G added a comment - Thanks Benson Qiu for the updated patch. Few minor comments QueueConfigurations and QueueConfigurationsPBImpl does not have Apache license header. Please add the same. This was the reason for ASF License red failure in jenkins. Test failure seems random. YARN-5652 is already raised for that. Hence its wont be a problem from your patch. Could you please check checkstyle failures from jenkins. Some comments looks good to be fixed in that. (like unused imports). Whichever could be fixed and possible, you could give a try. In yarn_protos.proto for * QueueConfigurationsMapProto*, QueueConfigurationsProto could be made optional. In QueueConfigurationsPBImpl.java , setCapacity seems not override. Its commented, is it intended? I think rest all changes looks fine. Thank You.
        Hide
        benson.qiu Benson Qiu added a comment -

        Sunil G: Version 7 contains the changes you mentioned yesterday.

        TestRMAdminService passes for me locally.

        I will fix the whitespace error in the next patch.

        Please let me know if there are any other changes I should make.

        Show
        benson.qiu Benson Qiu added a comment - Sunil G : Version 7 contains the changes you mentioned yesterday. TestRMAdminService passes for me locally. I will fix the whitespace error in the next patch. Please let me know if there are any other changes I should make.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 19s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 3 new or modified test files.
        0 mvndep 0m 9s Maven dependency ordering for branch
        +1 mvninstall 13m 30s trunk passed
        +1 compile 16m 39s trunk passed
        +1 checkstyle 0m 53s trunk passed
        +1 mvnsite 2m 17s trunk passed
        +1 mvneclipse 1m 18s trunk passed
        +1 findbugs 3m 58s trunk passed
        +1 javadoc 1m 49s trunk passed
        0 mvndep 0m 9s Maven dependency ordering for patch
        +1 mvninstall 1m 42s the patch passed
        +1 compile 11m 24s the patch passed
        +1 cc 11m 24s the patch passed
        +1 javac 11m 24s the patch passed
        -0 checkstyle 0m 55s hadoop-yarn-project/hadoop-yarn: The patch generated 17 new + 237 unchanged - 0 fixed = 254 total (was 237)
        +1 mvnsite 2m 26s the patch passed
        +1 mvneclipse 1m 10s the patch passed
        -1 whitespace 0m 0s The patch has 2 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
        +1 findbugs 5m 3s the patch passed
        +1 javadoc 1m 47s the patch passed
        +1 unit 0m 34s hadoop-yarn-api in the patch passed.
        +1 unit 2m 39s hadoop-yarn-common in the patch passed.
        -1 unit 40m 13s hadoop-yarn-server-resourcemanager in the patch failed.
        +1 unit 20m 15s hadoop-yarn-client in the patch passed.
        -1 asflicense 0m 33s The patch generated 2 ASF License warnings.
        138m 37s



        Reason Tests
        Failed junit tests hadoop.yarn.server.resourcemanager.TestRMAdminService



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:a9ad5d6
        JIRA Issue YARN-6164
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12860020/YARN-6164.007.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc
        uname Linux 9c201554bcd0 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / f462e1f
        Default Java 1.8.0_121
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/15359/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt
        whitespace https://builds.apache.org/job/PreCommit-YARN-Build/15359/artifact/patchprocess/whitespace-eol.txt
        unit https://builds.apache.org/job/PreCommit-YARN-Build/15359/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/15359/testReport/
        asflicense https://builds.apache.org/job/PreCommit-YARN-Build/15359/artifact/patchprocess/patch-asflicense-problems.txt
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client U: hadoop-yarn-project/hadoop-yarn
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/15359/console
        Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 19s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 3 new or modified test files. 0 mvndep 0m 9s Maven dependency ordering for branch +1 mvninstall 13m 30s trunk passed +1 compile 16m 39s trunk passed +1 checkstyle 0m 53s trunk passed +1 mvnsite 2m 17s trunk passed +1 mvneclipse 1m 18s trunk passed +1 findbugs 3m 58s trunk passed +1 javadoc 1m 49s trunk passed 0 mvndep 0m 9s Maven dependency ordering for patch +1 mvninstall 1m 42s the patch passed +1 compile 11m 24s the patch passed +1 cc 11m 24s the patch passed +1 javac 11m 24s the patch passed -0 checkstyle 0m 55s hadoop-yarn-project/hadoop-yarn: The patch generated 17 new + 237 unchanged - 0 fixed = 254 total (was 237) +1 mvnsite 2m 26s the patch passed +1 mvneclipse 1m 10s the patch passed -1 whitespace 0m 0s The patch has 2 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply +1 findbugs 5m 3s the patch passed +1 javadoc 1m 47s the patch passed +1 unit 0m 34s hadoop-yarn-api in the patch passed. +1 unit 2m 39s hadoop-yarn-common in the patch passed. -1 unit 40m 13s hadoop-yarn-server-resourcemanager in the patch failed. +1 unit 20m 15s hadoop-yarn-client in the patch passed. -1 asflicense 0m 33s The patch generated 2 ASF License warnings. 138m 37s Reason Tests Failed junit tests hadoop.yarn.server.resourcemanager.TestRMAdminService Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue YARN-6164 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12860020/YARN-6164.007.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc uname Linux 9c201554bcd0 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / f462e1f Default Java 1.8.0_121 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/15359/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt whitespace https://builds.apache.org/job/PreCommit-YARN-Build/15359/artifact/patchprocess/whitespace-eol.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/15359/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/15359/testReport/ asflicense https://builds.apache.org/job/PreCommit-YARN-Build/15359/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client U: hadoop-yarn-project/hadoop-yarn Console output https://builds.apache.org/job/PreCommit-YARN-Build/15359/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        sunilg Sunil G added a comment -

        Yes. Makes sense. you can skip my comment #2.

        Show
        sunilg Sunil G added a comment - Yes. Makes sense. you can skip my comment #2.
        Hide
        benson.qiu Benson Qiu added a comment -

        Thanks for the comments Sunil G!

        I'm a bit confused about comment #2, placing setters in QueueConfigurationsPBImpl instead of QueueConfigurations. Wouldn't that break QueueConfigurations#newInstance and AbstractCSQueue#getQueueConfigurations?

        Show
        benson.qiu Benson Qiu added a comment - Thanks for the comments Sunil G ! I'm a bit confused about comment #2, placing setters in QueueConfigurationsPBImpl instead of QueueConfigurations . Wouldn't that break QueueConfigurations#newInstance and AbstractCSQueue#getQueueConfigurations?
        Hide
        sunilg Sunil G added a comment -

        Few comments:

        In QueueConfigurations
        1. All api's could be Unstable since its a fresh set of apis
        2. All setters could be placed on QueueConfigurationsPBImpl. We need not have to place in QueueConfigurations.

        QueueConfigurationsProto.proto
        3. I am not very sure whether we need to put all params as required. Its better to keep them as optional.
        4. In QueueConfigurationsMapProto, nodeLabel could be renamed as partitionName.

        5. Some more test cases from client side to see whether we are able to pull this data cleanly. You could add them in TestClientRMService or TestResourceMgrDelegate

        Show
        sunilg Sunil G added a comment - Few comments: In QueueConfigurations 1. All api's could be Unstable since its a fresh set of apis 2. All setters could be placed on QueueConfigurationsPBImpl . We need not have to place in QueueConfigurations. QueueConfigurationsProto.proto 3. I am not very sure whether we need to put all params as required . Its better to keep them as optional. 4. In QueueConfigurationsMapProto , nodeLabel could be renamed as partitionName. 5. Some more test cases from client side to see whether we are able to pull this data cleanly. You could add them in TestClientRMService or TestResourceMgrDelegate
        Hide
        sunilg Sunil G added a comment -

        Sorry for the delay. I will take a look.

        Show
        sunilg Sunil G added a comment - Sorry for the delay. I will take a look.
        Hide
        benson.qiu Benson Qiu added a comment -

        Sunil Govind, Wangda Tan, Bibin A Chundatt: Mind taking a look at the latest patch? Version 6 contains a per-label map of QueueConfigurations.

        Show
        benson.qiu Benson Qiu added a comment - Sunil Govind , Wangda Tan , Bibin A Chundatt : Mind taking a look at the latest patch? Version 6 contains a per-label map of QueueConfigurations .
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 16s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
        0 mvndep 0m 46s Maven dependency ordering for branch
        +1 mvninstall 13m 21s trunk passed
        -1 compile 5m 22s hadoop-yarn in trunk failed.
        +1 checkstyle 1m 3s trunk passed
        +1 mvnsite 3m 8s trunk passed
        +1 mvneclipse 1m 50s trunk passed
        +1 findbugs 4m 57s trunk passed
        +1 javadoc 2m 19s trunk passed
        0 mvndep 0m 11s Maven dependency ordering for patch
        +1 mvninstall 1m 56s the patch passed
        -1 compile 5m 3s hadoop-yarn in the patch failed.
        -1 cc 5m 3s hadoop-yarn in the patch failed.
        -1 javac 5m 3s hadoop-yarn in the patch failed.
        -0 checkstyle 1m 3s hadoop-yarn-project/hadoop-yarn: The patch generated 17 new + 199 unchanged - 0 fixed = 216 total (was 199)
        +1 mvnsite 2m 45s the patch passed
        +1 mvneclipse 1m 43s the patch passed
        -1 whitespace 0m 0s The patch has 2 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
        +1 findbugs 5m 9s the patch passed
        +1 javadoc 2m 15s the patch passed
        +1 unit 0m 41s hadoop-yarn-api in the patch passed.
        +1 unit 2m 47s hadoop-yarn-common in the patch passed.
        +1 unit 42m 12s hadoop-yarn-server-resourcemanager in the patch passed.
        +1 unit 20m 30s hadoop-yarn-client in the patch passed.
        -1 asflicense 0m 44s The patch generated 2 ASF License warnings.
        129m 48s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:a9ad5d6
        JIRA Issue YARN-6164
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12858847/YARN-6164.006.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc
        uname Linux bc7f7c37afa9 3.13.0-105-generic #152-Ubuntu SMP Fri Dec 2 15:37:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / cc1292e
        Default Java 1.8.0_121
        compile https://builds.apache.org/job/PreCommit-YARN-Build/15284/artifact/patchprocess/branch-compile-hadoop-yarn-project_hadoop-yarn.txt
        findbugs v3.0.0
        compile https://builds.apache.org/job/PreCommit-YARN-Build/15284/artifact/patchprocess/patch-compile-hadoop-yarn-project_hadoop-yarn.txt
        cc https://builds.apache.org/job/PreCommit-YARN-Build/15284/artifact/patchprocess/patch-compile-hadoop-yarn-project_hadoop-yarn.txt
        javac https://builds.apache.org/job/PreCommit-YARN-Build/15284/artifact/patchprocess/patch-compile-hadoop-yarn-project_hadoop-yarn.txt
        checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/15284/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt
        whitespace https://builds.apache.org/job/PreCommit-YARN-Build/15284/artifact/patchprocess/whitespace-eol.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/15284/testReport/
        asflicense https://builds.apache.org/job/PreCommit-YARN-Build/15284/artifact/patchprocess/patch-asflicense-problems.txt
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client U: hadoop-yarn-project/hadoop-yarn
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/15284/console
        Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 16s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. 0 mvndep 0m 46s Maven dependency ordering for branch +1 mvninstall 13m 21s trunk passed -1 compile 5m 22s hadoop-yarn in trunk failed. +1 checkstyle 1m 3s trunk passed +1 mvnsite 3m 8s trunk passed +1 mvneclipse 1m 50s trunk passed +1 findbugs 4m 57s trunk passed +1 javadoc 2m 19s trunk passed 0 mvndep 0m 11s Maven dependency ordering for patch +1 mvninstall 1m 56s the patch passed -1 compile 5m 3s hadoop-yarn in the patch failed. -1 cc 5m 3s hadoop-yarn in the patch failed. -1 javac 5m 3s hadoop-yarn in the patch failed. -0 checkstyle 1m 3s hadoop-yarn-project/hadoop-yarn: The patch generated 17 new + 199 unchanged - 0 fixed = 216 total (was 199) +1 mvnsite 2m 45s the patch passed +1 mvneclipse 1m 43s the patch passed -1 whitespace 0m 0s The patch has 2 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply +1 findbugs 5m 9s the patch passed +1 javadoc 2m 15s the patch passed +1 unit 0m 41s hadoop-yarn-api in the patch passed. +1 unit 2m 47s hadoop-yarn-common in the patch passed. +1 unit 42m 12s hadoop-yarn-server-resourcemanager in the patch passed. +1 unit 20m 30s hadoop-yarn-client in the patch passed. -1 asflicense 0m 44s The patch generated 2 ASF License warnings. 129m 48s Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue YARN-6164 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12858847/YARN-6164.006.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc uname Linux bc7f7c37afa9 3.13.0-105-generic #152-Ubuntu SMP Fri Dec 2 15:37:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / cc1292e Default Java 1.8.0_121 compile https://builds.apache.org/job/PreCommit-YARN-Build/15284/artifact/patchprocess/branch-compile-hadoop-yarn-project_hadoop-yarn.txt findbugs v3.0.0 compile https://builds.apache.org/job/PreCommit-YARN-Build/15284/artifact/patchprocess/patch-compile-hadoop-yarn-project_hadoop-yarn.txt cc https://builds.apache.org/job/PreCommit-YARN-Build/15284/artifact/patchprocess/patch-compile-hadoop-yarn-project_hadoop-yarn.txt javac https://builds.apache.org/job/PreCommit-YARN-Build/15284/artifact/patchprocess/patch-compile-hadoop-yarn-project_hadoop-yarn.txt checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/15284/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt whitespace https://builds.apache.org/job/PreCommit-YARN-Build/15284/artifact/patchprocess/whitespace-eol.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/15284/testReport/ asflicense https://builds.apache.org/job/PreCommit-YARN-Build/15284/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client U: hadoop-yarn-project/hadoop-yarn Console output https://builds.apache.org/job/PreCommit-YARN-Build/15284/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        sunilg Sunil G added a comment -

        Benson Qiu, Thanks for the taking interest to contribute. Really appreciate.
        This is a good addition to api layer to provide missing details. So I am not seeing any reason not to add this.
        Please prepare patch, and I will be able to help.

        Show
        sunilg Sunil G added a comment - Benson Qiu , Thanks for the taking interest to contribute. Really appreciate. This is a good addition to api layer to provide missing details. So I am not seeing any reason not to add this. Please prepare patch, and I will be able to help.
        Hide
        benson.qiu Benson Qiu added a comment -

        Sunil G, Wangda Tan: Sure, I can add QueueCapacities to QueueInfo as originally discussed.

        If I work on this, is it very likely that my patch will get accepted? I would be working on this patch on my own time (not for the company I work for).

        Show
        benson.qiu Benson Qiu added a comment - Sunil G , Wangda Tan : Sure, I can add QueueCapacities to QueueInfo as originally discussed. If I work on this, is it very likely that my patch will get accepted? I would be working on this patch on my own time (not for the company I work for).
        Hide
        sunilg Sunil G added a comment -

        Hi Benson Qiu

        I had offline discussed with Wangda Tan yesterday and we feel that we can have a class named QueueCapacities in QueueInfo. This could have fields like cap/max-cap/max-am-perc (all are per label based) etc.
        Could you please share your thoughts also?

        Show
        sunilg Sunil G added a comment - Hi Benson Qiu I had offline discussed with Wangda Tan yesterday and we feel that we can have a class named QueueCapacities in QueueInfo . This could have fields like cap/max-cap/max-am-perc (all are per label based) etc. Could you please share your thoughts also?
        Hide
        sunilg Sunil G added a comment - - edited

        Thanks Wangda Tan for sharing the thoughts.

        It makes sense by seeing the diversity of params we have in QueueInfo. On the same note in long run, we are looking for something like QueueCapacities, QueueConfigurations, and ResourceUsages inside QueueInfo. Correct? and MaxAMPercentage could be part of QueueCapacities itself.

        I guess this could generalized improvement task, and for simplicity we could use @Unstable/@Private version of am percentage as given in current patch. Do you see this as a fine approach at this point?

        Show
        sunilg Sunil G added a comment - - edited Thanks Wangda Tan for sharing the thoughts. It makes sense by seeing the diversity of params we have in QueueInfo. On the same note in long run, we are looking for something like QueueCapacities , QueueConfigurations , and ResourceUsages inside QueueInfo . Correct? and MaxAMPercentage could be part of QueueCapacities itself. I guess this could generalized improvement task, and for simplicity we could use @Unstable/@Private version of am percentage as given in current patch. Do you see this as a fine approach at this point?
        Hide
        leftnoteasy Wangda Tan added a comment -

        Adding QueueConfigurations to QueueInfo is a good idea to me, but since it is supposed to handle other options like capacity/preemption-disabled, etc. I would prefer to continue existing approach of this patch, and think about how to add QueueConfiguration separately.

        In my opinion MaxAMPercentage is subject to be changed, so would prefer to make it @Unstable/@Private for MaxAMPercentages.java (methods and class), and QueueInfo#getMaxAMPercentages.

        Thoughts?

        Show
        leftnoteasy Wangda Tan added a comment - Adding QueueConfigurations to QueueInfo is a good idea to me, but since it is supposed to handle other options like capacity/preemption-disabled, etc. I would prefer to continue existing approach of this patch, and think about how to add QueueConfiguration separately. In my opinion MaxAMPercentage is subject to be changed, so would prefer to make it @Unstable/@Private for MaxAMPercentages.java (methods and class), and QueueInfo#getMaxAMPercentages. Thoughts?
        Hide
        sunilg Sunil G added a comment -

        Benson Qiu
        I personally feel that having QueueConfigurations inside QueueInfo is a general approach here. And looks fine for me. Wangda Tan, do you mind add some thoughts here.

        Show
        sunilg Sunil G added a comment - Benson Qiu I personally feel that having QueueConfigurations inside QueueInfo is a general approach here. And looks fine for me. Wangda Tan , do you mind add some thoughts here.
        Hide
        benson.qiu Benson Qiu added a comment -

        Sunil G Bibin A Chundatt

        I'm planning to set aside some time this week to work on creating `QueueConfigurations`.

        I would like to apply this patch to 2.8.0 as well.

        If I complete the feature discussed above (add a per-label map of QueueConfigurations, what is the likelihood that my patch will be accepted?

        Show
        benson.qiu Benson Qiu added a comment - Sunil G Bibin A Chundatt I'm planning to set aside some time this week to work on creating `QueueConfigurations`. I would like to apply this patch to 2.8.0 as well. If I complete the feature discussed above (add a per-label map of QueueConfigurations , what is the likelihood that my patch will be accepted?
        Hide
        sunilg Sunil G added a comment -

        churro morales

        I understood the your point. However this is an api exposed to client side and it ll be better if we do it clean. Otherwise once its released, we have to support the same for future version till we can deprecate it with a major version.

        The change suggested here is not very major, rather QueueConfigurations will be similar to MaxAMPercentage class in the patch. It should be pretty straight forward as per me.

        Show
        sunilg Sunil G added a comment - churro morales I understood the your point. However this is an api exposed to client side and it ll be better if we do it clean. Otherwise once its released, we have to support the same for future version till we can deprecate it with a major version. The change suggested here is not very major, rather QueueConfigurations will be similar to MaxAMPercentage class in the patch. It should be pretty straight forward as per me.
        Hide
        churromorales churro morales added a comment -

        This looks like a new feature, what do you think about putting this in a separate ticket so we can work on this in parallel. Also what do you think about getting the existing work in? I could use this patch where I work as well.

        Thanks everyone.

        Show
        churromorales churro morales added a comment - This looks like a new feature, what do you think about putting this in a separate ticket so we can work on this in parallel. Also what do you think about getting the existing work in? I could use this patch where I work as well. Thanks everyone.
        Hide
        bibinchundatt Bibin A Chundatt added a comment -

        I think lets not deleting any existing configuration or fields from QueueInfo. We can add this new map of QueueConfigurations

        Yes. We should move this direction so that in future can be accommodate other parameters also easily.

        Show
        bibinchundatt Bibin A Chundatt added a comment - I think lets not deleting any existing configuration or fields from QueueInfo. We can add this new map of QueueConfigurations Yes. We should move this direction so that in future can be accommodate other parameters also easily.
        Hide
        sunilg Sunil G added a comment -

        I think lets not deleting any existing configuration or fields from QueueInfo. We can add this new map of QueueConfigurations, and once this is matured, we can deprecate others if needed.

        Show
        sunilg Sunil G added a comment - I think lets not deleting any existing configuration or fields from QueueInfo. We can add this new map of QueueConfigurations , and once this is matured, we can deprecate others if needed.
        Hide
        benson.qiu Benson Qiu added a comment -

        Thanks for the comments Sunil G.

        So QueueInfo would expose something like `Map<String,QueueConfigurations> getQueueConfigurations()`.

        `QueueInfo` already has `getCapacity()` and `getMaximumCapacity()`, which currently doesn't take into account node-labels. Are there any concerns with breaking backwards compatibility if we delete these two methods from the interface?

        Show
        benson.qiu Benson Qiu added a comment - Thanks for the comments Sunil G . So QueueInfo would expose something like `Map<String,QueueConfigurations> getQueueConfigurations()`. `QueueInfo` already has `getCapacity()` and `getMaximumCapacity()`, which currently doesn't take into account node-labels. Are there any concerns with breaking backwards compatibility if we delete these two methods from the interface?
        Hide
        sunilg Sunil G added a comment -

        Thanks Benson Qiu for the patch

        There are some more high level thoughts after i was checking various CLI and other PBImpl code.

        1. We are not sharing per-label Queue resource usage
        2. We are not sharing per-label Queue configurations (capacity etc).

        More or less I am thinking in line of thoughts shared by Bibin A Chundatt earlier.

        Since many informations related to configuration is already exposed by QueueInfo, Properties may not a become a better logical grouping unless we move all existing config items to new subclass.

        I still kind of feel above statement makes sense. However this is similar to a problem addressed in QueueCapacities etc. And in long run, this may be helpful.

        My proposal is something like below.

        In QueueInfo, lets have an api called setQueueConfigurations. A new api record could be added named QueueConfigurations and it can have below sub items

        1. capacity
        2. abs-capacity
        3. max-capacity
        4. abs-max-capacity
        5. max-am-perc

        If we have a map in QueueInfo per-label, then QueueConfigurations could be mapped against its accessable-labels.

        I would like to have discussion on this approach and we can make it more generic. Benson Qiu, Bibin A Chundatt and Rohith Sharma K S , thoughts?

        Show
        sunilg Sunil G added a comment - Thanks Benson Qiu for the patch There are some more high level thoughts after i was checking various CLI and other PBImpl code. We are not sharing per-label Queue resource usage We are not sharing per-label Queue configurations (capacity etc). More or less I am thinking in line of thoughts shared by Bibin A Chundatt earlier. Since many informations related to configuration is already exposed by QueueInfo, Properties may not a become a better logical grouping unless we move all existing config items to new subclass. I still kind of feel above statement makes sense. However this is similar to a problem addressed in QueueCapacities etc. And in long run, this may be helpful. My proposal is something like below. In QueueInfo , lets have an api called setQueueConfigurations . A new api record could be added named QueueConfigurations and it can have below sub items capacity abs-capacity max-capacity abs-max-capacity max-am-perc If we have a map in QueueInfo per-label, then QueueConfigurations could be mapped against its accessable-labels. I would like to have discussion on this approach and we can make it more generic. Benson Qiu , Bibin A Chundatt and Rohith Sharma K S , thoughts?
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 22s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
        0 mvndep 0m 48s Maven dependency ordering for branch
        +1 mvninstall 14m 1s trunk passed
        +1 compile 9m 33s trunk passed
        +1 checkstyle 0m 59s trunk passed
        +1 mvnsite 2m 46s trunk passed
        +1 mvneclipse 1m 19s trunk passed
        +1 findbugs 4m 51s trunk passed
        +1 javadoc 2m 0s trunk passed
        0 mvndep 0m 12s Maven dependency ordering for patch
        +1 mvninstall 2m 22s the patch passed
        +1 compile 7m 16s the patch passed
        +1 cc 7m 16s the patch passed
        +1 javac 7m 16s the patch passed
        +1 checkstyle 0m 55s the patch passed
        +1 mvnsite 2m 24s the patch passed
        +1 mvneclipse 1m 19s the patch passed
        -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
        -1 findbugs 1m 22s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
        +1 javadoc 1m 51s the patch passed
        +1 unit 0m 34s hadoop-yarn-api in the patch passed.
        +1 unit 2m 42s hadoop-yarn-common in the patch passed.
        -1 unit 42m 54s hadoop-yarn-server-resourcemanager in the patch failed.
        -1 unit 32m 5s hadoop-yarn-client in the patch failed.
        +1 asflicense 0m 33s The patch does not generate ASF License warnings.
        145m 49s



        Reason Tests
        FindBugs module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
          Comparison of String objects using == or != in org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.AbstractCSQueue.getMaxAMPercentages() At AbstractCSQueue.java:== or != in org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.AbstractCSQueue.getMaxAMPercentages() At AbstractCSQueue.java:[line 458]
        Failed junit tests hadoop.yarn.server.resourcemanager.TestOpportunisticContainerAllocatorAMService
        Timed out junit tests org.apache.hadoop.yarn.client.api.impl.TestOpportunisticContainerAllocation



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:a9ad5d6
        JIRA Issue YARN-6164
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12853686/YARN-6164.005.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc
        uname Linux 5df7a6595093 3.13.0-105-generic #152-Ubuntu SMP Fri Dec 2 15:37:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 6ba61d2
        Default Java 1.8.0_121
        findbugs v3.0.0
        whitespace https://builds.apache.org/job/PreCommit-YARN-Build/15019/artifact/patchprocess/whitespace-eol.txt
        findbugs https://builds.apache.org/job/PreCommit-YARN-Build/15019/artifact/patchprocess/new-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.html
        unit https://builds.apache.org/job/PreCommit-YARN-Build/15019/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
        unit https://builds.apache.org/job/PreCommit-YARN-Build/15019/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/15019/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client U: hadoop-yarn-project/hadoop-yarn
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/15019/console
        Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 22s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. 0 mvndep 0m 48s Maven dependency ordering for branch +1 mvninstall 14m 1s trunk passed +1 compile 9m 33s trunk passed +1 checkstyle 0m 59s trunk passed +1 mvnsite 2m 46s trunk passed +1 mvneclipse 1m 19s trunk passed +1 findbugs 4m 51s trunk passed +1 javadoc 2m 0s trunk passed 0 mvndep 0m 12s Maven dependency ordering for patch +1 mvninstall 2m 22s the patch passed +1 compile 7m 16s the patch passed +1 cc 7m 16s the patch passed +1 javac 7m 16s the patch passed +1 checkstyle 0m 55s the patch passed +1 mvnsite 2m 24s the patch passed +1 mvneclipse 1m 19s the patch passed -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply -1 findbugs 1m 22s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) +1 javadoc 1m 51s the patch passed +1 unit 0m 34s hadoop-yarn-api in the patch passed. +1 unit 2m 42s hadoop-yarn-common in the patch passed. -1 unit 42m 54s hadoop-yarn-server-resourcemanager in the patch failed. -1 unit 32m 5s hadoop-yarn-client in the patch failed. +1 asflicense 0m 33s The patch does not generate ASF License warnings. 145m 49s Reason Tests FindBugs module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager   Comparison of String objects using == or != in org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.AbstractCSQueue.getMaxAMPercentages() At AbstractCSQueue.java:== or != in org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.AbstractCSQueue.getMaxAMPercentages() At AbstractCSQueue.java: [line 458] Failed junit tests hadoop.yarn.server.resourcemanager.TestOpportunisticContainerAllocatorAMService Timed out junit tests org.apache.hadoop.yarn.client.api.impl.TestOpportunisticContainerAllocation Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue YARN-6164 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12853686/YARN-6164.005.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc uname Linux 5df7a6595093 3.13.0-105-generic #152-Ubuntu SMP Fri Dec 2 15:37:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 6ba61d2 Default Java 1.8.0_121 findbugs v3.0.0 whitespace https://builds.apache.org/job/PreCommit-YARN-Build/15019/artifact/patchprocess/whitespace-eol.txt findbugs https://builds.apache.org/job/PreCommit-YARN-Build/15019/artifact/patchprocess/new-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.html unit https://builds.apache.org/job/PreCommit-YARN-Build/15019/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/15019/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/15019/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client U: hadoop-yarn-project/hadoop-yarn Console output https://builds.apache.org/job/PreCommit-YARN-Build/15019/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        benson.qiu Benson Qiu added a comment -

        005: Fix org.apache.hadoop.yarn.api.TestPBImplRecords.testQueueInfoPBImpl

        The other failing tests from version 004 pass on my local machine.

        Show
        benson.qiu Benson Qiu added a comment - 005: Fix org.apache.hadoop.yarn.api.TestPBImplRecords.testQueueInfoPBImpl The other failing tests from version 004 pass on my local machine.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 20s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        0 mvndep 0m 25s Maven dependency ordering for branch
        +1 mvninstall 12m 37s trunk passed
        +1 compile 8m 4s trunk passed
        +1 checkstyle 0m 49s trunk passed
        +1 mvnsite 2m 10s trunk passed
        +1 mvneclipse 1m 14s trunk passed
        +1 findbugs 3m 46s trunk passed
        +1 javadoc 1m 42s trunk passed
        0 mvndep 0m 9s Maven dependency ordering for patch
        +1 mvninstall 1m 38s the patch passed
        +1 compile 6m 0s the patch passed
        +1 cc 6m 0s the patch passed
        +1 javac 6m 0s the patch passed
        +1 checkstyle 0m 47s the patch passed
        +1 mvnsite 2m 6s the patch passed
        +1 mvneclipse 1m 11s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        -1 findbugs 1m 14s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
        +1 javadoc 1m 40s the patch passed
        +1 unit 0m 32s hadoop-yarn-api in the patch passed.
        -1 unit 2m 27s hadoop-yarn-common in the patch failed.
        -1 unit 40m 6s hadoop-yarn-server-resourcemanager in the patch failed.
        +1 unit 17m 24s hadoop-yarn-client in the patch passed.
        +1 asflicense 0m 29s The patch does not generate ASF License warnings.
        119m 6s



        Reason Tests
        FindBugs module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
          Comparison of String objects using == or != in org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.AbstractCSQueue.getMaxAMPercentages() At AbstractCSQueue.java:== or != in org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.AbstractCSQueue.getMaxAMPercentages() At AbstractCSQueue.java:[line 458]
        Failed junit tests hadoop.yarn.api.TestPBImplRecords
          hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairSchedulerPreemption
          hadoop.yarn.server.resourcemanager.TestRMRestart
          hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer
          hadoop.yarn.server.resourcemanager.scheduler.fair.TestFSAppStarvation



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:a9ad5d6
        JIRA Issue YARN-6164
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12853510/YARN-6164.004.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc
        uname Linux 9fc0d5c372c2 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 8035749
        Default Java 1.8.0_121
        findbugs v3.0.0
        findbugs https://builds.apache.org/job/PreCommit-YARN-Build/15016/artifact/patchprocess/new-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.html
        unit https://builds.apache.org/job/PreCommit-YARN-Build/15016/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common.txt
        unit https://builds.apache.org/job/PreCommit-YARN-Build/15016/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/15016/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client U: hadoop-yarn-project/hadoop-yarn
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/15016/console
        Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 20s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 0m 25s Maven dependency ordering for branch +1 mvninstall 12m 37s trunk passed +1 compile 8m 4s trunk passed +1 checkstyle 0m 49s trunk passed +1 mvnsite 2m 10s trunk passed +1 mvneclipse 1m 14s trunk passed +1 findbugs 3m 46s trunk passed +1 javadoc 1m 42s trunk passed 0 mvndep 0m 9s Maven dependency ordering for patch +1 mvninstall 1m 38s the patch passed +1 compile 6m 0s the patch passed +1 cc 6m 0s the patch passed +1 javac 6m 0s the patch passed +1 checkstyle 0m 47s the patch passed +1 mvnsite 2m 6s the patch passed +1 mvneclipse 1m 11s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. -1 findbugs 1m 14s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) +1 javadoc 1m 40s the patch passed +1 unit 0m 32s hadoop-yarn-api in the patch passed. -1 unit 2m 27s hadoop-yarn-common in the patch failed. -1 unit 40m 6s hadoop-yarn-server-resourcemanager in the patch failed. +1 unit 17m 24s hadoop-yarn-client in the patch passed. +1 asflicense 0m 29s The patch does not generate ASF License warnings. 119m 6s Reason Tests FindBugs module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager   Comparison of String objects using == or != in org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.AbstractCSQueue.getMaxAMPercentages() At AbstractCSQueue.java:== or != in org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.AbstractCSQueue.getMaxAMPercentages() At AbstractCSQueue.java: [line 458] Failed junit tests hadoop.yarn.api.TestPBImplRecords   hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairSchedulerPreemption   hadoop.yarn.server.resourcemanager.TestRMRestart   hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer   hadoop.yarn.server.resourcemanager.scheduler.fair.TestFSAppStarvation Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue YARN-6164 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12853510/YARN-6164.004.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc uname Linux 9fc0d5c372c2 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 8035749 Default Java 1.8.0_121 findbugs v3.0.0 findbugs https://builds.apache.org/job/PreCommit-YARN-Build/15016/artifact/patchprocess/new-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.html unit https://builds.apache.org/job/PreCommit-YARN-Build/15016/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/15016/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/15016/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client U: hadoop-yarn-project/hadoop-yarn Console output https://builds.apache.org/job/PreCommit-YARN-Build/15016/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        benson.qiu Benson Qiu added a comment -

        Sunil G: I added per-label-per-queue AM resource percentages.

        Show
        benson.qiu Benson Qiu added a comment - Sunil G : I added per-label-per-queue AM resource percentages.
        Hide
        benson.qiu Benson Qiu added a comment - - edited

        Version 004: per-node-label max AM limits.

        Show
        benson.qiu Benson Qiu added a comment - - edited Version 004: per-node-label max AM limits.
        Hide
        sunilg Sunil G added a comment -

        Bibin A Chundatt, I am still doubtful about that approach. Since many informations related to configuration is already exposed by QueueInfo, Properties may not a become a better logical grouping unless we move all existing config items to new subclass. This will cause compatibility, and we wont be able to do that. For me, I feel we many confuse more if we add Properties sub section , as its not a single grouping for configuration related items.

        Show
        sunilg Sunil G added a comment - Bibin A Chundatt , I am still doubtful about that approach. Since many informations related to configuration is already exposed by QueueInfo, Properties may not a become a better logical grouping unless we move all existing config items to new subclass. This will cause compatibility, and we wont be able to do that. For me, I feel we many confuse more if we add Properties sub section , as its not a single grouping for configuration related items.
        Hide
        bibinchundatt Bibin A Chundatt added a comment -

        For every new property added in queue we are trying to handle separately in QueueInfo. If we expose Property object in QueueInfo and add all static configs to that no need to handle separately.

        already have mix set of informations such as capacity, application, preemption disable

        As you said since QueueInfo exposed all above we might have continue doing the same.

        Show
        bibinchundatt Bibin A Chundatt added a comment - For every new property added in queue we are trying to handle separately in QueueInfo. If we expose Property object in QueueInfo and add all static configs to that no need to handle separately. already have mix set of informations such as capacity, application, preemption disable As you said since QueueInfo exposed all above we might have continue doing the same.
        Hide
        sunilg Sunil G added a comment -

        Bibin A Chundatt

        org.apache.hadoop.yarn.api.records.QueueInfo already have mix set of informations such as capacity, application, preemption disable info etc, so I thought adding another param inside QueueInfo for am-percentage as well. I guess you are mentioning about a subclass inside QueueInfo and push all these under that? Could u please explain a bit more.

        Show
        sunilg Sunil G added a comment - Bibin A Chundatt org.apache.hadoop.yarn.api.records.QueueInfo already have mix set of informations such as capacity, application, preemption disable info etc, so I thought adding another param inside QueueInfo for am-percentage as well. I guess you are mentioning about a subclass inside QueueInfo and push all these under that? Could u please explain a bit more.
        Hide
        bibinchundatt Bibin A Chundatt added a comment - - edited

        Sunil G
        Thoughts on getting these as Properties ?? So that per queue all additional info could be send.

        Show
        bibinchundatt Bibin A Chundatt added a comment - - edited Sunil G Thoughts on getting these as Properties ?? So that per queue all additional info could be send.
        Hide
        sunilg Sunil G added a comment -

        Benson Qiu

        Am resource percentage could be configured per-label-per-queue as well. So instead of sending one AM resource percentage, I would like to send

        [{"default", 20%},{"labelA", 10%},{"labelB", 10%}] 

        . Thoughts?

        Show
        sunilg Sunil G added a comment - Benson Qiu Am resource percentage could be configured per-label-per-queue as well. So instead of sending one AM resource percentage, I would like to send [{"default", 20%},{"labelA", 10%},{"labelB", 10%}] . Thoughts?
        Hide
        benson.qiu Benson Qiu added a comment -

        Version 003: Fix test failures

        Show
        benson.qiu Benson Qiu added a comment - Version 003: Fix test failures
        Hide
        benson.qiu Benson Qiu added a comment -

        Version 002: Overloaded QueueInfo.newInstance instead of modifying the method signature. Thanks churro morales for the suggestion.

        Show
        benson.qiu Benson Qiu added a comment - Version 002: Overloaded QueueInfo.newInstance instead of modifying the method signature. Thanks churro morales for the suggestion.
        Hide
        BensonQiu Deactivate added a comment -

        I built the latest version of Hadoop and it appears that maxAMLimitPercentage has been added to the Cluster Scheduler API (/ws/v1/cluster/scheduler).

        I still don't believe this config is exposed through YarnClient or YarnCLI though.

        Show
        BensonQiu Deactivate added a comment - I built the latest version of Hadoop and it appears that maxAMLimitPercentage has been added to the Cluster Scheduler API (/ws/v1/cluster/scheduler). I still don't believe this config is exposed through YarnClient or YarnCLI though.

          People

          • Assignee:
            benson.qiu Benson Qiu
            Reporter:
            benson.qiu Benson Qiu
          • Votes:
            0 Vote for this issue
            Watchers:
            12 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development