Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 3.1.0
    • capacity scheduler
    • None
    • Reviewed

    Description

      This involves changes to queue mapping flow to pass along context information for auto queue creation. Auto creation of queues will be part of Capacity Scheduler flow while attempting to resolve queues during application submission. The leaf queues which do not exist are auto created under parent queues which have been explicitly enabled for auto queue creation . In order to determine which parent queue to create the leaf queues under - parent queues need to be specified in queue mapping configuration

      Attachments

        1. YARN-7419.1.patch
          142 kB
          Suma Shivaprasad
        2. YARN-7419.2.patch
          115 kB
          Suma Shivaprasad
        3. YARN-7419.3.patch
          116 kB
          Suma Shivaprasad
        4. YARN-7419.4.patch
          120 kB
          Suma Shivaprasad
        5. YARN-7419.5.patch
          122 kB
          Suma Shivaprasad
        6. YARN-7419.6.patch
          122 kB
          Suma Shivaprasad
        7. YARN-7419.7.patch
          124 kB
          Suma Shivaprasad
        8. YARN-7419.8.patch
          119 kB
          Suma Shivaprasad
        9. YARN-7419.9.patch
          124 kB
          Suma Shivaprasad
        10. YARN-7419.patch
          142 kB
          Suma Shivaprasad

        Issue Links

          Activity

            Attaching patch on top of code changes in YARN-7394

            suma.shivaprasad Suma Shivaprasad added a comment - Attaching patch on top of code changes in YARN-7394
            hadoopqa Hadoop QA added a comment -
            -1 overall



            Vote Subsystem Runtime Comment
            0 reexec 0m 12s Docker mode activated.
                  Prechecks
            +1 @author 0m 0s The patch does not contain any @author tags.
            +1 test4tests 0m 0s The patch appears to include 8 new or modified test files.
                  trunk Compile Tests
            +1 mvninstall 16m 19s trunk passed
            +1 compile 0m 36s trunk passed
            +1 checkstyle 0m 31s trunk passed
            +1 mvnsite 0m 38s trunk passed
            +1 shadedclient 10m 13s branch has no errors when building and testing our client artifacts.
            -1 findbugs 1m 5s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager in trunk has 2 extant Findbugs warnings.
            +1 javadoc 0m 21s trunk passed
                  Patch Compile Tests
            +1 mvninstall 0m 38s the patch passed
            +1 compile 0m 34s the patch passed
            +1 javac 0m 34s the patch passed
            -0 checkstyle 0m 31s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 159 new + 632 unchanged - 16 fixed = 791 total (was 648)
            +1 mvnsite 0m 37s the patch passed
            +1 whitespace 0m 0s The patch has no whitespace issues.
            +1 shadedclient 10m 13s patch has no errors when building and testing our client artifacts.
            -1 findbugs 1m 14s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager generated 3 new + 2 unchanged - 0 fixed = 5 total (was 2)
            -1 javadoc 0m 20s hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager generated 1 new + 4 unchanged - 0 fixed = 5 total (was 4)
                  Other Tests
            -1 unit 54m 49s hadoop-yarn-server-resourcemanager in the patch failed.
            +1 asflicense 0m 17s The patch does not generate ASF License warnings.
            98m 48s



            Reason Tests
            FindBugs module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
              Self assignment of field ApplicationPlacementContext.parentQueue in new org.apache.hadoop.yarn.server.resourcemanager.placement.ApplicationPlacementContext(String) At ApplicationPlacementContext.java:in new org.apache.hadoop.yarn.server.resourcemanager.placement.ApplicationPlacementContext(String) At ApplicationPlacementContext.java:[line 37]
              instanceof will always return true for all non-null values in org.apache.hadoop.yarn.server.resourcemanager.placement.UserGroupMappingPlacementRule.validateParentQueue(CSQueue, String, String), since all org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ManagedParentQueue are instances of org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ManagedParentQueue At UserGroupMappingPlacementRule.java:for all non-null values in org.apache.hadoop.yarn.server.resourcemanager.placement.UserGroupMappingPlacementRule.validateParentQueue(CSQueue, String, String), since all org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ManagedParentQueue are instances of org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ManagedParentQueue At UserGroupMappingPlacementRule.java:[line 385]
              Unread field:field be static? At UserGroupMappingPlacementRule.java:[line 82]
            Failed junit tests hadoop.yarn.server.resourcemanager.reservation.TestCapacityOverTimePolicy



            Subsystem Report/Notes
            Docker Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639
            JIRA Issue YARN-7419
            JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12895282/YARN-7419.patch
            Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
            uname Linux c4d6daf6e611 3.13.0-119-generic #166-Ubuntu SMP Wed May 3 12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
            Build tool maven
            Personality /testptch/patchprocess/precommit/personality/provided.sh
            git revision trunk / d57dba9
            maven version: Apache Maven 3.3.9
            Default Java 1.8.0_131
            findbugs v3.1.0-RC1
            findbugs https://builds.apache.org/job/PreCommit-YARN-Build/18293/artifact/out/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-warnings.html
            checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/18293/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
            findbugs https://builds.apache.org/job/PreCommit-YARN-Build/18293/artifact/out/new-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.html
            javadoc https://builds.apache.org/job/PreCommit-YARN-Build/18293/artifact/out/diff-javadoc-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
            unit https://builds.apache.org/job/PreCommit-YARN-Build/18293/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
            Test Results https://builds.apache.org/job/PreCommit-YARN-Build/18293/testReport/
            modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
            Console output https://builds.apache.org/job/PreCommit-YARN-Build/18293/console
            Powered by Apache Yetus 0.7.0-SNAPSHOT http://yetus.apache.org

            This message was automatically generated.

            hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 12s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 8 new or modified test files.       trunk Compile Tests +1 mvninstall 16m 19s trunk passed +1 compile 0m 36s trunk passed +1 checkstyle 0m 31s trunk passed +1 mvnsite 0m 38s trunk passed +1 shadedclient 10m 13s branch has no errors when building and testing our client artifacts. -1 findbugs 1m 5s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager in trunk has 2 extant Findbugs warnings. +1 javadoc 0m 21s trunk passed       Patch Compile Tests +1 mvninstall 0m 38s the patch passed +1 compile 0m 34s the patch passed +1 javac 0m 34s the patch passed -0 checkstyle 0m 31s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 159 new + 632 unchanged - 16 fixed = 791 total (was 648) +1 mvnsite 0m 37s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 shadedclient 10m 13s patch has no errors when building and testing our client artifacts. -1 findbugs 1m 14s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager generated 3 new + 2 unchanged - 0 fixed = 5 total (was 2) -1 javadoc 0m 20s hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager generated 1 new + 4 unchanged - 0 fixed = 5 total (was 4)       Other Tests -1 unit 54m 49s hadoop-yarn-server-resourcemanager in the patch failed. +1 asflicense 0m 17s The patch does not generate ASF License warnings. 98m 48s Reason Tests FindBugs module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager   Self assignment of field ApplicationPlacementContext.parentQueue in new org.apache.hadoop.yarn.server.resourcemanager.placement.ApplicationPlacementContext(String) At ApplicationPlacementContext.java:in new org.apache.hadoop.yarn.server.resourcemanager.placement.ApplicationPlacementContext(String) At ApplicationPlacementContext.java: [line 37]   instanceof will always return true for all non-null values in org.apache.hadoop.yarn.server.resourcemanager.placement.UserGroupMappingPlacementRule.validateParentQueue(CSQueue, String, String), since all org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ManagedParentQueue are instances of org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ManagedParentQueue At UserGroupMappingPlacementRule.java:for all non-null values in org.apache.hadoop.yarn.server.resourcemanager.placement.UserGroupMappingPlacementRule.validateParentQueue(CSQueue, String, String), since all org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ManagedParentQueue are instances of org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ManagedParentQueue At UserGroupMappingPlacementRule.java: [line 385]   Unread field:field be static? At UserGroupMappingPlacementRule.java: [line 82] Failed junit tests hadoop.yarn.server.resourcemanager.reservation.TestCapacityOverTimePolicy Subsystem Report/Notes Docker Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 JIRA Issue YARN-7419 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12895282/YARN-7419.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle uname Linux c4d6daf6e611 3.13.0-119-generic #166-Ubuntu SMP Wed May 3 12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/patchprocess/precommit/personality/provided.sh git revision trunk / d57dba9 maven version: Apache Maven 3.3.9 Default Java 1.8.0_131 findbugs v3.1.0-RC1 findbugs https://builds.apache.org/job/PreCommit-YARN-Build/18293/artifact/out/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-warnings.html checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/18293/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt findbugs https://builds.apache.org/job/PreCommit-YARN-Build/18293/artifact/out/new-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.html javadoc https://builds.apache.org/job/PreCommit-YARN-Build/18293/artifact/out/diff-javadoc-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/18293/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/18293/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/18293/console Powered by Apache Yetus 0.7.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.

            Attached patch with checkstyle, findbugs fixed

            suma.shivaprasad Suma Shivaprasad added a comment - Attached patch with checkstyle, findbugs fixed
            hadoopqa Hadoop QA added a comment -
            -1 overall



            Vote Subsystem Runtime Comment
            0 reexec 0m 11s Docker mode activated.
                  Prechecks
            +1 @author 0m 0s The patch does not contain any @author tags.
            +1 test4tests 0m 0s The patch appears to include 8 new or modified test files.
                  trunk Compile Tests
            +1 mvninstall 17m 11s trunk passed
            +1 compile 0m 40s trunk passed
            +1 checkstyle 0m 37s trunk passed
            +1 mvnsite 0m 44s trunk passed
            +1 shadedclient 11m 17s branch has no errors when building and testing our client artifacts.
            -1 findbugs 1m 3s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager in trunk has 2 extant Findbugs warnings.
            +1 javadoc 0m 22s trunk passed
                  Patch Compile Tests
            +1 mvninstall 0m 38s the patch passed
            +1 compile 0m 33s the patch passed
            +1 javac 0m 33s the patch passed
            -0 checkstyle 0m 34s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 157 new + 632 unchanged - 16 fixed = 789 total (was 648)
            +1 mvnsite 0m 36s the patch passed
            +1 whitespace 0m 0s The patch has no whitespace issues.
            +1 shadedclient 11m 13s patch has no errors when building and testing our client artifacts.
            +1 findbugs 1m 12s the patch passed
            -1 javadoc 0m 21s hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager generated 1 new + 4 unchanged - 0 fixed = 5 total (was 4)
                  Other Tests
            -1 unit 56m 12s hadoop-yarn-server-resourcemanager in the patch failed.
            +1 asflicense 0m 20s The patch does not generate ASF License warnings.
            103m 21s



            Reason Tests
            Failed junit tests hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacitySchedulerNodeLabelUpdate
              hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacitySchedulerAutoQueueCreation



            Subsystem Report/Notes
            Docker Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639
            JIRA Issue YARN-7419
            JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12895324/YARN-7419.1.patch
            Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
            uname Linux 87bf4629c223 3.13.0-119-generic #166-Ubuntu SMP Wed May 3 12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
            Build tool maven
            Personality /testptch/patchprocess/precommit/personality/provided.sh
            git revision trunk / 0cc98ae
            maven version: Apache Maven 3.3.9
            Default Java 1.8.0_131
            findbugs v3.1.0-RC1
            findbugs https://builds.apache.org/job/PreCommit-YARN-Build/18296/artifact/out/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-warnings.html
            checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/18296/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
            javadoc https://builds.apache.org/job/PreCommit-YARN-Build/18296/artifact/out/diff-javadoc-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
            unit https://builds.apache.org/job/PreCommit-YARN-Build/18296/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
            Test Results https://builds.apache.org/job/PreCommit-YARN-Build/18296/testReport/
            modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
            Console output https://builds.apache.org/job/PreCommit-YARN-Build/18296/console
            Powered by Apache Yetus 0.7.0-SNAPSHOT http://yetus.apache.org

            This message was automatically generated.

            hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 11s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 8 new or modified test files.       trunk Compile Tests +1 mvninstall 17m 11s trunk passed +1 compile 0m 40s trunk passed +1 checkstyle 0m 37s trunk passed +1 mvnsite 0m 44s trunk passed +1 shadedclient 11m 17s branch has no errors when building and testing our client artifacts. -1 findbugs 1m 3s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager in trunk has 2 extant Findbugs warnings. +1 javadoc 0m 22s trunk passed       Patch Compile Tests +1 mvninstall 0m 38s the patch passed +1 compile 0m 33s the patch passed +1 javac 0m 33s the patch passed -0 checkstyle 0m 34s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 157 new + 632 unchanged - 16 fixed = 789 total (was 648) +1 mvnsite 0m 36s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 shadedclient 11m 13s patch has no errors when building and testing our client artifacts. +1 findbugs 1m 12s the patch passed -1 javadoc 0m 21s hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager generated 1 new + 4 unchanged - 0 fixed = 5 total (was 4)       Other Tests -1 unit 56m 12s hadoop-yarn-server-resourcemanager in the patch failed. +1 asflicense 0m 20s The patch does not generate ASF License warnings. 103m 21s Reason Tests Failed junit tests hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacitySchedulerNodeLabelUpdate   hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacitySchedulerAutoQueueCreation Subsystem Report/Notes Docker Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 JIRA Issue YARN-7419 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12895324/YARN-7419.1.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle uname Linux 87bf4629c223 3.13.0-119-generic #166-Ubuntu SMP Wed May 3 12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/patchprocess/precommit/personality/provided.sh git revision trunk / 0cc98ae maven version: Apache Maven 3.3.9 Default Java 1.8.0_131 findbugs v3.1.0-RC1 findbugs https://builds.apache.org/job/PreCommit-YARN-Build/18296/artifact/out/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-warnings.html checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/18296/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt javadoc https://builds.apache.org/job/PreCommit-YARN-Build/18296/artifact/out/diff-javadoc-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/18296/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/18296/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/18296/console Powered by Apache Yetus 0.7.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
            leftnoteasy Wangda Tan added a comment -

            Thanks suma.shivaprasad for working on this.

            Some comments:

            1) Why changes of getClock are needed, seems nobody is using them?

            2) Following methods can reuse leaf queue's methods (just pass <parent-queue-path>.AUTO_CREATED_LEAF_QUEUE_TEMPLATE_PREFIX as queue path).

            getAutoCreatedLeafQueueTemplateMaxCapacity(String)
            getAutoCreatedQueuesMaxChildQueuesLimit(String)
            setAutoCreatedLeafQueueTemplateCapacity(String, float)
            setAutoCreatedLeafQueueTemplateMaxApplicationsPerQueue(String, int)
            setAutoCreatedLeafQueueTemplateMaxCapacity(String, float)
            

            And this config: AUTO_CREATE_QUEUE_MAX_QUEUES should be part of parent queue instead of leaf queue templete.

            3) Could you create a AutoCreatedLeafQueueTemplate class which encapsulate all leaf queue's properties and replace all getter/setters of leaf queue properties.

            4) sumOfChildCapacities/sumOfChildAbsCapacities can be moved to AbstractManagedParentQueue, correct?

            5) ApplicationPlacementContext, why keep PlacementRule as one of the fields?

            6) There're two rmApp.getApplicationQueuePlacementContext() calls inside CapacityScheduler, one of them can be removed. (Which we can avoid access RMApps map twice).

            leftnoteasy Wangda Tan added a comment - Thanks suma.shivaprasad for working on this. Some comments: 1) Why changes of getClock are needed, seems nobody is using them? 2) Following methods can reuse leaf queue's methods (just pass <parent-queue-path>.AUTO_CREATED_LEAF_QUEUE_TEMPLATE_PREFIX as queue path). getAutoCreatedLeafQueueTemplateMaxCapacity( String ) getAutoCreatedQueuesMaxChildQueuesLimit( String ) setAutoCreatedLeafQueueTemplateCapacity( String , float ) setAutoCreatedLeafQueueTemplateMaxApplicationsPerQueue( String , int ) setAutoCreatedLeafQueueTemplateMaxCapacity( String , float ) And this config: AUTO_CREATE_QUEUE_MAX_QUEUES should be part of parent queue instead of leaf queue templete. 3) Could you create a AutoCreatedLeafQueueTemplate class which encapsulate all leaf queue's properties and replace all getter/setters of leaf queue properties. 4) sumOfChildCapacities/sumOfChildAbsCapacities can be moved to AbstractManagedParentQueue, correct? 5) ApplicationPlacementContext, why keep PlacementRule as one of the fields? 6) There're two rmApp.getApplicationQueuePlacementContext() calls inside CapacityScheduler, one of them can be removed. (Which we can avoid access RMApps map twice).

            Thanks for the review wangda Attached patch with review comments addressed

            suma.shivaprasad Suma Shivaprasad added a comment - Thanks for the review wangda Attached patch with review comments addressed

            Had missed removing sumOfChildCapacities from ParentQueue. Fixed in this patch

            suma.shivaprasad Suma Shivaprasad added a comment - Had missed removing sumOfChildCapacities from ParentQueue. Fixed in this patch
            hadoopqa Hadoop QA added a comment -
            -1 overall



            Vote Subsystem Runtime Comment
            0 reexec 0m 9s Docker mode activated.
                  Prechecks
            +1 @author 0m 0s The patch does not contain any @author tags.
            +1 test4tests 0m 0s The patch appears to include 6 new or modified test files.
                  trunk Compile Tests
            +1 mvninstall 16m 43s trunk passed
            +1 compile 0m 47s trunk passed
            +1 checkstyle 0m 36s trunk passed
            +1 mvnsite 0m 45s trunk passed
            +1 shadedclient 11m 38s branch has no errors when building and testing our client artifacts.
            +1 findbugs 1m 17s trunk passed
            +1 javadoc 0m 29s trunk passed
                  Patch Compile Tests
            +1 mvninstall 0m 50s the patch passed
            +1 compile 0m 55s the patch passed
            +1 javac 0m 55s the patch passed
            -0 checkstyle 0m 37s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 167 new + 581 unchanged - 0 fixed = 748 total (was 581)
            +1 mvnsite 0m 42s the patch passed
            +1 whitespace 0m 0s The patch has no whitespace issues.
            +1 shadedclient 11m 35s patch has no errors when building and testing our client artifacts.
            +1 findbugs 1m 23s the patch passed
            +1 javadoc 0m 25s the patch passed
                  Other Tests
            -1 unit 57m 10s hadoop-yarn-server-resourcemanager in the patch failed.
            +1 asflicense 0m 22s The patch does not generate ASF License warnings.
            106m 6s



            Reason Tests
            Failed junit tests hadoop.yarn.server.resourcemanager.reservation.TestCapacityOverTimePolicy
              hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer



            Subsystem Report/Notes
            Docker Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639
            JIRA Issue YARN-7419
            JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12896522/YARN-7419.3.patch
            Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
            uname Linux 3c7f5c99d398 3.13.0-129-generic #178-Ubuntu SMP Fri Aug 11 12:48:20 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
            Build tool maven
            Personality /testptch/patchprocess/precommit/personality/provided.sh
            git revision trunk / 13fa2d4
            maven version: Apache Maven 3.3.9
            Default Java 1.8.0_131
            findbugs v3.1.0-RC1
            checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/18388/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
            unit https://builds.apache.org/job/PreCommit-YARN-Build/18388/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
            Test Results https://builds.apache.org/job/PreCommit-YARN-Build/18388/testReport/
            Max. process+thread count 872 (vs. ulimit of 5000)
            modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
            Console output https://builds.apache.org/job/PreCommit-YARN-Build/18388/console
            Powered by Apache Yetus 0.7.0-SNAPSHOT http://yetus.apache.org

            This message was automatically generated.

            hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 9s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 6 new or modified test files.       trunk Compile Tests +1 mvninstall 16m 43s trunk passed +1 compile 0m 47s trunk passed +1 checkstyle 0m 36s trunk passed +1 mvnsite 0m 45s trunk passed +1 shadedclient 11m 38s branch has no errors when building and testing our client artifacts. +1 findbugs 1m 17s trunk passed +1 javadoc 0m 29s trunk passed       Patch Compile Tests +1 mvninstall 0m 50s the patch passed +1 compile 0m 55s the patch passed +1 javac 0m 55s the patch passed -0 checkstyle 0m 37s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 167 new + 581 unchanged - 0 fixed = 748 total (was 581) +1 mvnsite 0m 42s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 shadedclient 11m 35s patch has no errors when building and testing our client artifacts. +1 findbugs 1m 23s the patch passed +1 javadoc 0m 25s the patch passed       Other Tests -1 unit 57m 10s hadoop-yarn-server-resourcemanager in the patch failed. +1 asflicense 0m 22s The patch does not generate ASF License warnings. 106m 6s Reason Tests Failed junit tests hadoop.yarn.server.resourcemanager.reservation.TestCapacityOverTimePolicy   hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer Subsystem Report/Notes Docker Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 JIRA Issue YARN-7419 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12896522/YARN-7419.3.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle uname Linux 3c7f5c99d398 3.13.0-129-generic #178-Ubuntu SMP Fri Aug 11 12:48:20 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/patchprocess/precommit/personality/provided.sh git revision trunk / 13fa2d4 maven version: Apache Maven 3.3.9 Default Java 1.8.0_131 findbugs v3.1.0-RC1 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/18388/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/18388/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/18388/testReport/ Max. process+thread count 872 (vs. ulimit of 5000) modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/18388/console Powered by Apache Yetus 0.7.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
            leftnoteasy Wangda Tan added a comment -

            Thanks suma.shivaprasad for updating the patch, more comments:

            1) CapacityScheduler:
            1.1 Instead of fetching ApplicationPlacementContext from RMApp (to avoid perf/locking issue and code flow becomes more clear), you can add the ApplicationPlacementContext to AppAddedSchedulerEvent. And getPlacementContext API can be removed from RMApp.

            1.2 Move following if ... to addApplication:

            if (placementContext != null) {
              // ...
            }
            

            Like,

            if (queue == null && placementContext != null) {
              //Could be a potential auto-created leaf queue
            }
            

            Only enter the autoCreateLeafQueue function when necessary.

            1.3 Following two catch exception can be merged by using YarnException | IOException e syntax.

            catch (YarnException e) {
                      LOG.error("Could not auto-create leaf queue due to : ", e);
                      final String message =
                          "Application " + applicationId + " submission by user " + user
                              + " to  queue: " + queueName + " failed : " + e.getMessage();
                      this.rmContext.getDispatcher().getEventHandler().handle(
                          new RMAppEvent(applicationId, RMAppEventType.APP_REJECTED,
                              message));
                    } catch (IOException e) {
                      final String message =
                          "Application " + applicationId + " submission by user " + user
                              + " to  queue: " + queueName + " failed : " + e.getMessage();
                      LOG.error("Could not auto-create leaf queue due to : ", e);
                      this.rmContext.getDispatcher().getEventHandler().handle(
                          new RMAppEvent(applicationId, RMAppEventType.APP_REJECTED,
                              message));
                    }
            

            1.4 Following message is not clear enough:

                      String message =
                          "Application " + applicationId + " submission by user " + user
                              + " to queue: " + queueName + " failed : "
                              + "Queue mapping does not exist for user";
            

            It should say directly specify a autocreated queue name is prohibited, it has to be automatically mapped, etc.

            1.5 I'm not sure if this check is necessary, I think previous logics should be enough to detect this correct?

            else if (!queue.getParent().getQueueName().equals(
                        placementContext.getParentQueue())) {
                      String message =
                          "Auto created Leaf queue " + placementContext.getQueue() + " already exists under " + queue
                              .getParent().getQueuePath()
                              + ".But Queue mapping has a different parent queue "
                              + placementContext.getParentQueue()
                              + " for the specified user : " + user;
                      this.rmContext.getDispatcher().getEventHandler().handle(
                          new RMAppEvent(applicationId, RMAppEventType.APP_REJECTED,
                              message));
                      return;
                    } 
            

            1.6 clock is still here, move to a separate patch?

            2) CapacitySchedulerConfiguration:

            • getQueuePlacementRules is unused.
            • Make sure all new added methods/fields are @Private
            • FAIL_AUTO_CREATION_ON_EXCEEDING_CAPACITY is this necessary? Should we just fail leaf queue creation when it exceeds parent queue's limit?

            Renames:

            • AutoCreatedLeafQueueTemplate.Builder#capacity => capacities

            Unecesssary changes:

            • CapacitySchedulerContext
            • AbstractCSQueue

            Miscs:

            • Did you accidentally included YARN-6124 in this patch? Could you revert that part?
            leftnoteasy Wangda Tan added a comment - Thanks suma.shivaprasad for updating the patch, more comments: 1) CapacityScheduler: 1.1 Instead of fetching ApplicationPlacementContext from RMApp (to avoid perf/locking issue and code flow becomes more clear), you can add the ApplicationPlacementContext to AppAddedSchedulerEvent . And getPlacementContext API can be removed from RMApp. 1.2 Move following if ... to addApplication: if (placementContext != null ) { // ... } Like, if (queue == null && placementContext != null ) { //Could be a potential auto-created leaf queue } Only enter the autoCreateLeafQueue function when necessary. 1.3 Following two catch exception can be merged by using YarnException | IOException e syntax. catch (YarnException e) { LOG.error( "Could not auto-create leaf queue due to : " , e); final String message = "Application " + applicationId + " submission by user " + user + " to queue: " + queueName + " failed : " + e.getMessage(); this .rmContext.getDispatcher().getEventHandler().handle( new RMAppEvent(applicationId, RMAppEventType.APP_REJECTED, message)); } catch (IOException e) { final String message = "Application " + applicationId + " submission by user " + user + " to queue: " + queueName + " failed : " + e.getMessage(); LOG.error( "Could not auto-create leaf queue due to : " , e); this .rmContext.getDispatcher().getEventHandler().handle( new RMAppEvent(applicationId, RMAppEventType.APP_REJECTED, message)); } 1.4 Following message is not clear enough: String message = "Application " + applicationId + " submission by user " + user + " to queue: " + queueName + " failed : " + "Queue mapping does not exist for user" ; It should say directly specify a autocreated queue name is prohibited, it has to be automatically mapped, etc. 1.5 I'm not sure if this check is necessary, I think previous logics should be enough to detect this correct? else if (!queue.getParent().getQueueName().equals( placementContext.getParentQueue())) { String message = "Auto created Leaf queue " + placementContext.getQueue() + " already exists under " + queue .getParent().getQueuePath() + ".But Queue mapping has a different parent queue " + placementContext.getParentQueue() + " for the specified user : " + user; this .rmContext.getDispatcher().getEventHandler().handle( new RMAppEvent(applicationId, RMAppEventType.APP_REJECTED, message)); return ; } 1.6 clock is still here, move to a separate patch? 2) CapacitySchedulerConfiguration: getQueuePlacementRules is unused. Make sure all new added methods/fields are @Private FAIL_AUTO_CREATION_ON_EXCEEDING_CAPACITY is this necessary? Should we just fail leaf queue creation when it exceeds parent queue's limit? Renames: AutoCreatedLeafQueueTemplate.Builder#capacity => capacities Unecesssary changes: CapacitySchedulerContext AbstractCSQueue Miscs: Did you accidentally included YARN-6124 in this patch? Could you revert that part?

            Thanks wangda Fixed most of the review comments.

            suma.shivaprasad Suma Shivaprasad added a comment - Thanks wangda Fixed most of the review comments.

            Thanks Suma Shivaprasad for updating the patch, more comments:
            1) CapacityScheduler:

            1.1 Instead of fetching ApplicationPlacementContext from RMApp (to avoid perf/locking issue and code flow becomes more clear), you can add the ApplicationPlacementContext to AppAddedSchedulerEvent. And getPlacementContext API can be removed from RMApp.

            Fixed for both new application submissions and recovery

            1.2 Move following if ... to addApplication:
            if (placementContext != null)

            Unknown macro: { // ... }

            Like,
            if (queue == null && placementContext != null)

            Unknown macro: { //Could be a potential auto-created leaf queue }

            Only enter the autoCreateLeafQueue function when necessary.

            Fixed

            1.3 Following two catch exception can be merged by using YarnException | IOException e syntax.
            catch (YarnException e)

            Unknown macro: { LOG.error("Could not auto-create leaf queue due to }

            catch (IOException e)

            Unknown macro: { final String message = "Application " + applicationId + " submission by user " + user + " to queue}

            Fixed

            1.4 Following message is not clear enough:
            String message =
            "Application " + applicationId + " submission by user " + user
            + " to queue: " + queueName + " failed : "
            + "Queue mapping does not exist for user";
            It should say directly specify a autocreated queue name is prohibited, it has to be automatically mapped, etc.

            Fixed

            1.5 I'm not sure if this check is necessary, I think previous logics should be enough to detect this correct?
            else if (!queue.getParent().getQueueName().equals(
            placementContext.getParentQueue()))

            Unknown macro: { String message = "Auto created Leaf queue " + placementContext.getQueue() + " already exists under " + queue .getParent().getQueuePath() + ".But Queue mapping has a different parent queue " + placementContext.getParentQueue() + " for the specified user }

            This is needed in the case where auto created leaf queue already exists but its parent queue has been updated in the queue mapping to a different queue which could lead to inconsistencies
            since we do not allow same leaf queue name under different hierarchies

            1.6 clock is still here, move to a separate patch?

            Fixed

            2) CapacitySchedulerConfiguration:
            getQueuePlacementRules is unused.
            Make sure all new added methods/fields are @Private

            Fixed
            FAIL_AUTO_CREATION_ON_EXCEEDING_CAPACITY is this necessary? Should we just fail leaf queue creation when it exceeds parent queue's limit?

            Renames:
            AutoCreatedLeafQueueTemplate.Builder#capacity => capacities
            Unecesssary changes:
            CapacitySchedulerContext
            AbstractCSQueue

            Fixed
            Miscs:
            Did you accidentally included YARN-6124 in this patch? Could you revert that part?

            suma.shivaprasad Suma Shivaprasad added a comment - Thanks Suma Shivaprasad for updating the patch, more comments: 1) CapacityScheduler: 1.1 Instead of fetching ApplicationPlacementContext from RMApp (to avoid perf/locking issue and code flow becomes more clear), you can add the ApplicationPlacementContext to AppAddedSchedulerEvent. And getPlacementContext API can be removed from RMApp. Fixed for both new application submissions and recovery 1.2 Move following if ... to addApplication: if (placementContext != null) Unknown macro: { // ... } Like, if (queue == null && placementContext != null) Unknown macro: { //Could be a potential auto-created leaf queue } Only enter the autoCreateLeafQueue function when necessary. Fixed 1.3 Following two catch exception can be merged by using YarnException | IOException e syntax. catch (YarnException e) Unknown macro: { LOG.error("Could not auto-create leaf queue due to } catch (IOException e) Unknown macro: { final String message = "Application " + applicationId + " submission by user " + user + " to queue} Fixed 1.4 Following message is not clear enough: String message = "Application " + applicationId + " submission by user " + user + " to queue: " + queueName + " failed : " + "Queue mapping does not exist for user"; It should say directly specify a autocreated queue name is prohibited, it has to be automatically mapped, etc. Fixed 1.5 I'm not sure if this check is necessary, I think previous logics should be enough to detect this correct? else if (!queue.getParent().getQueueName().equals( placementContext.getParentQueue())) Unknown macro: { String message = "Auto created Leaf queue " + placementContext.getQueue() + " already exists under " + queue .getParent().getQueuePath() + ".But Queue mapping has a different parent queue " + placementContext.getParentQueue() + " for the specified user } This is needed in the case where auto created leaf queue already exists but its parent queue has been updated in the queue mapping to a different queue which could lead to inconsistencies since we do not allow same leaf queue name under different hierarchies 1.6 clock is still here, move to a separate patch? Fixed 2) CapacitySchedulerConfiguration: getQueuePlacementRules is unused. Make sure all new added methods/fields are @Private Fixed FAIL_AUTO_CREATION_ON_EXCEEDING_CAPACITY is this necessary? Should we just fail leaf queue creation when it exceeds parent queue's limit? Renames: AutoCreatedLeafQueueTemplate.Builder#capacity => capacities Unecesssary changes: CapacitySchedulerContext AbstractCSQueue Fixed Miscs: Did you accidentally included YARN-6124 in this patch? Could you revert that part?
            hadoopqa Hadoop QA added a comment -
            -1 overall



            Vote Subsystem Runtime Comment
            0 reexec 0m 25s Docker mode activated.
                  Prechecks
            +1 @author 0m 0s The patch does not contain any @author tags.
            +1 test4tests 0m 0s The patch appears to include 6 new or modified test files.
                  trunk Compile Tests
            +1 mvninstall 24m 48s trunk passed
            +1 compile 1m 3s trunk passed
            +1 checkstyle 0m 46s trunk passed
            +1 mvnsite 1m 10s trunk passed
            +1 shadedclient 14m 35s branch has no errors when building and testing our client artifacts.
            +1 findbugs 2m 11s trunk passed
            +1 javadoc 0m 55s trunk passed
                  Patch Compile Tests
            +1 mvninstall 1m 25s the patch passed
            +1 compile 1m 18s the patch passed
            +1 javac 1m 18s the patch passed
            -0 checkstyle 1m 4s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 150 new + 625 unchanged - 2 fixed = 775 total (was 627)
            +1 mvnsite 1m 16s the patch passed
            -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
            +1 shadedclient 15m 17s patch has no errors when building and testing our client artifacts.
            +1 findbugs 2m 42s the patch passed
            +1 javadoc 1m 9s the patch passed
                  Other Tests
            -1 unit 2m 1s hadoop-yarn-server-resourcemanager in the patch failed.
            -1 asflicense 0m 55s The patch generated 15 ASF License warnings.
            73m 20s



            Reason Tests
            Failed junit tests hadoop.yarn.server.resourcemanager.TestRM



            Subsystem Report/Notes
            Docker Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639
            JIRA Issue YARN-7419
            JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12896970/YARN-7419.4.patch
            Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
            uname Linux 81d4e2a5abc1 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
            Build tool maven
            Personality /testptch/patchprocess/precommit/personality/provided.sh
            git revision trunk / 5eb7dbe
            maven version: Apache Maven 3.3.9
            Default Java 1.8.0_131
            findbugs v3.1.0-RC1
            checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/18423/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
            whitespace https://builds.apache.org/job/PreCommit-YARN-Build/18423/artifact/out/whitespace-eol.txt
            unit https://builds.apache.org/job/PreCommit-YARN-Build/18423/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
            Test Results https://builds.apache.org/job/PreCommit-YARN-Build/18423/testReport/
            asflicense https://builds.apache.org/job/PreCommit-YARN-Build/18423/artifact/out/patch-asflicense-problems.txt
            Max. process+thread count 296 (vs. ulimit of 5000)
            modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
            Console output https://builds.apache.org/job/PreCommit-YARN-Build/18423/console
            Powered by Apache Yetus 0.7.0-SNAPSHOT http://yetus.apache.org

            This message was automatically generated.

            hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 25s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 6 new or modified test files.       trunk Compile Tests +1 mvninstall 24m 48s trunk passed +1 compile 1m 3s trunk passed +1 checkstyle 0m 46s trunk passed +1 mvnsite 1m 10s trunk passed +1 shadedclient 14m 35s branch has no errors when building and testing our client artifacts. +1 findbugs 2m 11s trunk passed +1 javadoc 0m 55s trunk passed       Patch Compile Tests +1 mvninstall 1m 25s the patch passed +1 compile 1m 18s the patch passed +1 javac 1m 18s the patch passed -0 checkstyle 1m 4s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 150 new + 625 unchanged - 2 fixed = 775 total (was 627) +1 mvnsite 1m 16s the patch passed -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply +1 shadedclient 15m 17s patch has no errors when building and testing our client artifacts. +1 findbugs 2m 42s the patch passed +1 javadoc 1m 9s the patch passed       Other Tests -1 unit 2m 1s hadoop-yarn-server-resourcemanager in the patch failed. -1 asflicense 0m 55s The patch generated 15 ASF License warnings. 73m 20s Reason Tests Failed junit tests hadoop.yarn.server.resourcemanager.TestRM Subsystem Report/Notes Docker Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 JIRA Issue YARN-7419 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12896970/YARN-7419.4.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle uname Linux 81d4e2a5abc1 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/patchprocess/precommit/personality/provided.sh git revision trunk / 5eb7dbe maven version: Apache Maven 3.3.9 Default Java 1.8.0_131 findbugs v3.1.0-RC1 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/18423/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt whitespace https://builds.apache.org/job/PreCommit-YARN-Build/18423/artifact/out/whitespace-eol.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/18423/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/18423/testReport/ asflicense https://builds.apache.org/job/PreCommit-YARN-Build/18423/artifact/out/patch-asflicense-problems.txt Max. process+thread count 296 (vs. ulimit of 5000) modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/18423/console Powered by Apache Yetus 0.7.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.

            Fixed some checkstyle issues and added missing return for CapacityScheduler when queue is null

            suma.shivaprasad Suma Shivaprasad added a comment - Fixed some checkstyle issues and added missing return for CapacityScheduler when queue is null
            hadoopqa Hadoop QA added a comment -
            -1 overall



            Vote Subsystem Runtime Comment
            0 reexec 0m 12s Docker mode activated.
                  Prechecks
            +1 @author 0m 0s The patch does not contain any @author tags.
            +1 test4tests 0m 0s The patch appears to include 6 new or modified test files.
                  trunk Compile Tests
            +1 mvninstall 19m 33s trunk passed
            +1 compile 0m 36s trunk passed
            +1 checkstyle 0m 31s trunk passed
            +1 mvnsite 0m 40s trunk passed
            +1 shadedclient 10m 58s branch has no errors when building and testing our client artifacts.
            +1 findbugs 1m 7s trunk passed
            +1 javadoc 0m 25s trunk passed
                  Patch Compile Tests
            +1 mvninstall 0m 39s the patch passed
            +1 compile 0m 36s the patch passed
            +1 javac 0m 36s the patch passed
            -0 checkstyle 0m 31s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 109 new + 625 unchanged - 3 fixed = 734 total (was 628)
            +1 mvnsite 0m 38s the patch passed
            -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
            +1 shadedclient 10m 44s patch has no errors when building and testing our client artifacts.
            +1 findbugs 1m 9s the patch passed
            +1 javadoc 0m 22s the patch passed
                  Other Tests
            -1 unit 55m 5s hadoop-yarn-server-resourcemanager in the patch failed.
            +1 asflicense 0m 20s The patch does not generate ASF License warnings.
            103m 56s



            Reason Tests
            Failed junit tests hadoop.yarn.server.resourcemanager.scheduler.capacity.TestIncreaseAllocationExpirer



            Subsystem Report/Notes
            Docker Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639
            JIRA Issue YARN-7419
            JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12897001/YARN-7419.5.patch
            Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
            uname Linux e7df8630bbb6 3.13.0-133-generic #182-Ubuntu SMP Tue Sep 19 15:49:21 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
            Build tool maven
            Personality /testptch/patchprocess/precommit/personality/provided.sh
            git revision trunk / 10a1f55
            maven version: Apache Maven 3.3.9
            Default Java 1.8.0_151
            findbugs v3.1.0-RC1
            checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/18426/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
            whitespace https://builds.apache.org/job/PreCommit-YARN-Build/18426/artifact/out/whitespace-eol.txt
            unit https://builds.apache.org/job/PreCommit-YARN-Build/18426/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
            Test Results https://builds.apache.org/job/PreCommit-YARN-Build/18426/testReport/
            Max. process+thread count 849 (vs. ulimit of 5000)
            modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
            Console output https://builds.apache.org/job/PreCommit-YARN-Build/18426/console
            Powered by Apache Yetus 0.7.0-SNAPSHOT http://yetus.apache.org

            This message was automatically generated.

            hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 12s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 6 new or modified test files.       trunk Compile Tests +1 mvninstall 19m 33s trunk passed +1 compile 0m 36s trunk passed +1 checkstyle 0m 31s trunk passed +1 mvnsite 0m 40s trunk passed +1 shadedclient 10m 58s branch has no errors when building and testing our client artifacts. +1 findbugs 1m 7s trunk passed +1 javadoc 0m 25s trunk passed       Patch Compile Tests +1 mvninstall 0m 39s the patch passed +1 compile 0m 36s the patch passed +1 javac 0m 36s the patch passed -0 checkstyle 0m 31s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 109 new + 625 unchanged - 3 fixed = 734 total (was 628) +1 mvnsite 0m 38s the patch passed -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply +1 shadedclient 10m 44s patch has no errors when building and testing our client artifacts. +1 findbugs 1m 9s the patch passed +1 javadoc 0m 22s the patch passed       Other Tests -1 unit 55m 5s hadoop-yarn-server-resourcemanager in the patch failed. +1 asflicense 0m 20s The patch does not generate ASF License warnings. 103m 56s Reason Tests Failed junit tests hadoop.yarn.server.resourcemanager.scheduler.capacity.TestIncreaseAllocationExpirer Subsystem Report/Notes Docker Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 JIRA Issue YARN-7419 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12897001/YARN-7419.5.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle uname Linux e7df8630bbb6 3.13.0-133-generic #182-Ubuntu SMP Tue Sep 19 15:49:21 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/patchprocess/precommit/personality/provided.sh git revision trunk / 10a1f55 maven version: Apache Maven 3.3.9 Default Java 1.8.0_151 findbugs v3.1.0-RC1 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/18426/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt whitespace https://builds.apache.org/job/PreCommit-YARN-Build/18426/artifact/out/whitespace-eol.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/18426/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/18426/testReport/ Max. process+thread count 849 (vs. ulimit of 5000) modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/18426/console Powered by Apache Yetus 0.7.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
            leftnoteasy Wangda Tan added a comment -

            Thanks suma.shivaprasad for updating the patch, in general patch looks good. Could you help to fix following checkstyle issues if possible?

            HiddenField
            LineLength
            UnusedImports

            leftnoteasy Wangda Tan added a comment - Thanks suma.shivaprasad for updating the patch, in general patch looks good. Could you help to fix following checkstyle issues if possible? HiddenField LineLength UnusedImports

            Fixed some more checkstyle issues

            suma.shivaprasad Suma Shivaprasad added a comment - Fixed some more checkstyle issues
            hadoopqa Hadoop QA added a comment -
            -1 overall



            Vote Subsystem Runtime Comment
            0 reexec 12m 44s Docker mode activated.
                  Prechecks
            +1 @author 0m 0s The patch does not contain any @author tags.
            +1 test4tests 0m 0s The patch appears to include 5 new or modified test files.
                  trunk Compile Tests
            +1 mvninstall 33m 18s trunk passed
            +1 compile 1m 29s trunk passed
            +1 checkstyle 1m 5s trunk passed
            +1 mvnsite 1m 33s trunk passed
            +1 shadedclient 18m 6s branch has no errors when building and testing our client artifacts.
            +1 findbugs 2m 23s trunk passed
            +1 javadoc 0m 52s trunk passed
                  Patch Compile Tests
            +1 mvninstall 1m 35s the patch passed
            +1 compile 1m 29s the patch passed
            +1 javac 1m 29s the patch passed
            -0 checkstyle 1m 3s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 53 new + 611 unchanged - 4 fixed = 664 total (was 615)
            +1 mvnsite 1m 32s the patch passed
            -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
            +1 shadedclient 17m 32s patch has no errors when building and testing our client artifacts.
            +1 findbugs 1m 14s the patch passed
            +1 javadoc 0m 23s the patch passed
                  Other Tests
            +1 unit 55m 55s hadoop-yarn-server-resourcemanager in the patch passed.
            +1 asflicense 0m 28s The patch does not generate ASF License warnings.
            151m 27s



            Subsystem Report/Notes
            Docker Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639
            JIRA Issue YARN-7419
            JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12897132/YARN-7419.6.patch
            Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
            uname Linux 7d1a48856853 3.13.0-133-generic #182-Ubuntu SMP Tue Sep 19 15:49:21 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
            Build tool maven
            Personality /testptch/patchprocess/precommit/personality/provided.sh
            git revision trunk / 8a1bd9a
            maven version: Apache Maven 3.3.9
            Default Java 1.8.0_151
            findbugs v3.1.0-RC1
            checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/18433/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
            whitespace https://builds.apache.org/job/PreCommit-YARN-Build/18433/artifact/out/whitespace-eol.txt
            Test Results https://builds.apache.org/job/PreCommit-YARN-Build/18433/testReport/
            Max. process+thread count 812 (vs. ulimit of 5000)
            modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
            Console output https://builds.apache.org/job/PreCommit-YARN-Build/18433/console
            Powered by Apache Yetus 0.7.0-SNAPSHOT http://yetus.apache.org

            This message was automatically generated.

            hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 12m 44s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 5 new or modified test files.       trunk Compile Tests +1 mvninstall 33m 18s trunk passed +1 compile 1m 29s trunk passed +1 checkstyle 1m 5s trunk passed +1 mvnsite 1m 33s trunk passed +1 shadedclient 18m 6s branch has no errors when building and testing our client artifacts. +1 findbugs 2m 23s trunk passed +1 javadoc 0m 52s trunk passed       Patch Compile Tests +1 mvninstall 1m 35s the patch passed +1 compile 1m 29s the patch passed +1 javac 1m 29s the patch passed -0 checkstyle 1m 3s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 53 new + 611 unchanged - 4 fixed = 664 total (was 615) +1 mvnsite 1m 32s the patch passed -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply +1 shadedclient 17m 32s patch has no errors when building and testing our client artifacts. +1 findbugs 1m 14s the patch passed +1 javadoc 0m 23s the patch passed       Other Tests +1 unit 55m 55s hadoop-yarn-server-resourcemanager in the patch passed. +1 asflicense 0m 28s The patch does not generate ASF License warnings. 151m 27s Subsystem Report/Notes Docker Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 JIRA Issue YARN-7419 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12897132/YARN-7419.6.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle uname Linux 7d1a48856853 3.13.0-133-generic #182-Ubuntu SMP Tue Sep 19 15:49:21 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/patchprocess/precommit/personality/provided.sh git revision trunk / 8a1bd9a maven version: Apache Maven 3.3.9 Default Java 1.8.0_151 findbugs v3.1.0-RC1 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/18433/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt whitespace https://builds.apache.org/job/PreCommit-YARN-Build/18433/artifact/out/whitespace-eol.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/18433/testReport/ Max. process+thread count 812 (vs. ulimit of 5000) modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/18433/console Powered by Apache Yetus 0.7.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.

            Fixed remaining checkstyle issues

            suma.shivaprasad Suma Shivaprasad added a comment - Fixed remaining checkstyle issues
            hadoopqa Hadoop QA added a comment -
            -1 overall



            Vote Subsystem Runtime Comment
            0 reexec 0m 18s Docker mode activated.
                  Prechecks
            +1 @author 0m 0s The patch does not contain any @author tags.
            +1 test4tests 0m 0s The patch appears to include 5 new or modified test files.
                  trunk Compile Tests
            +1 mvninstall 14m 50s trunk passed
            +1 compile 0m 37s trunk passed
            +1 checkstyle 0m 34s trunk passed
            +1 mvnsite 0m 39s trunk passed
            +1 shadedclient 10m 26s branch has no errors when building and testing our client artifacts.
            +1 findbugs 0m 59s trunk passed
            +1 javadoc 0m 24s trunk passed
                  Patch Compile Tests
            +1 mvninstall 0m 36s the patch passed
            +1 compile 0m 33s the patch passed
            +1 javac 0m 33s the patch passed
            -0 checkstyle 0m 31s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 44 new + 609 unchanged - 6 fixed = 653 total (was 615)
            +1 mvnsite 0m 36s the patch passed
            -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
            +1 shadedclient 10m 16s patch has no errors when building and testing our client artifacts.
            +1 findbugs 1m 3s the patch passed
            +1 javadoc 0m 23s the patch passed
                  Other Tests
            -1 unit 60m 42s hadoop-yarn-server-resourcemanager in the patch failed.
            +1 asflicense 0m 20s The patch does not generate ASF License warnings.
            103m 39s



            Reason Tests
            Timed out junit tests org.apache.hadoop.yarn.server.resourcemanager.TestRMStoreCommands
              org.apache.hadoop.yarn.server.resourcemanager.TestSubmitApplicationWithRMHA
              org.apache.hadoop.yarn.server.resourcemanager.TestKillApplicationWithRMHA



            Subsystem Report/Notes
            Docker Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639
            JIRA Issue YARN-7419
            JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12897222/YARN-7419.7.patch
            Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
            uname Linux e1b98de310a9 4.4.0-64-generic #85-Ubuntu SMP Mon Feb 20 11:50:30 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
            Build tool maven
            Personality /testptch/patchprocess/precommit/personality/provided.sh
            git revision trunk / ff9f7fc
            maven version: Apache Maven 3.3.9
            Default Java 1.8.0_151
            findbugs v3.1.0-RC1
            checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/18448/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
            whitespace https://builds.apache.org/job/PreCommit-YARN-Build/18448/artifact/out/whitespace-eol.txt
            unit https://builds.apache.org/job/PreCommit-YARN-Build/18448/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
            Test Results https://builds.apache.org/job/PreCommit-YARN-Build/18448/testReport/
            Max. process+thread count 842 (vs. ulimit of 5000)
            modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
            Console output https://builds.apache.org/job/PreCommit-YARN-Build/18448/console
            Powered by Apache Yetus 0.7.0-SNAPSHOT http://yetus.apache.org

            This message was automatically generated.

            hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 18s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 5 new or modified test files.       trunk Compile Tests +1 mvninstall 14m 50s trunk passed +1 compile 0m 37s trunk passed +1 checkstyle 0m 34s trunk passed +1 mvnsite 0m 39s trunk passed +1 shadedclient 10m 26s branch has no errors when building and testing our client artifacts. +1 findbugs 0m 59s trunk passed +1 javadoc 0m 24s trunk passed       Patch Compile Tests +1 mvninstall 0m 36s the patch passed +1 compile 0m 33s the patch passed +1 javac 0m 33s the patch passed -0 checkstyle 0m 31s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 44 new + 609 unchanged - 6 fixed = 653 total (was 615) +1 mvnsite 0m 36s the patch passed -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply +1 shadedclient 10m 16s patch has no errors when building and testing our client artifacts. +1 findbugs 1m 3s the patch passed +1 javadoc 0m 23s the patch passed       Other Tests -1 unit 60m 42s hadoop-yarn-server-resourcemanager in the patch failed. +1 asflicense 0m 20s The patch does not generate ASF License warnings. 103m 39s Reason Tests Timed out junit tests org.apache.hadoop.yarn.server.resourcemanager.TestRMStoreCommands   org.apache.hadoop.yarn.server.resourcemanager.TestSubmitApplicationWithRMHA   org.apache.hadoop.yarn.server.resourcemanager.TestKillApplicationWithRMHA Subsystem Report/Notes Docker Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 JIRA Issue YARN-7419 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12897222/YARN-7419.7.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle uname Linux e1b98de310a9 4.4.0-64-generic #85-Ubuntu SMP Mon Feb 20 11:50:30 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/patchprocess/precommit/personality/provided.sh git revision trunk / ff9f7fc maven version: Apache Maven 3.3.9 Default Java 1.8.0_151 findbugs v3.1.0-RC1 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/18448/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt whitespace https://builds.apache.org/job/PreCommit-YARN-Build/18448/artifact/out/whitespace-eol.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/18448/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/18448/testReport/ Max. process+thread count 842 (vs. ulimit of 5000) modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/18448/console Powered by Apache Yetus 0.7.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.

            Fixed a couple of other checkstyle and whitespace issues

            suma.shivaprasad Suma Shivaprasad added a comment - Fixed a couple of other checkstyle and whitespace issues
            hadoopqa Hadoop QA added a comment -
            -1 overall



            Vote Subsystem Runtime Comment
            0 reexec 0m 0s Docker mode activated.
            -1 patch 0m 5s YARN-7419 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help.



            This message was automatically generated.

            hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. -1 patch 0m 5s YARN-7419 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. Subsystem Report/Notes JIRA Issue YARN-7419 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12897260/YARN-7419.8.patch Console output https://builds.apache.org/job/PreCommit-YARN-Build/18457/console Powered by Apache Yetus 0.7.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.

            Rebased with runk

            suma.shivaprasad Suma Shivaprasad added a comment - Rebased with runk
            hadoopqa Hadoop QA added a comment -
            -1 overall



            Vote Subsystem Runtime Comment
            0 reexec 0m 15s Docker mode activated.
                  Prechecks
            +1 @author 0m 0s The patch does not contain any @author tags.
            +1 test4tests 0m 0s The patch appears to include 5 new or modified test files.
                  trunk Compile Tests
            +1 mvninstall 16m 8s trunk passed
            +1 compile 0m 37s trunk passed
            +1 checkstyle 0m 33s trunk passed
            +1 mvnsite 0m 39s trunk passed
            +1 shadedclient 10m 57s branch has no errors when building and testing our client artifacts.
            +1 findbugs 1m 2s trunk passed
            +1 javadoc 0m 24s trunk passed
                  Patch Compile Tests
            +1 mvninstall 0m 38s the patch passed
            +1 compile 0m 33s the patch passed
            +1 javac 0m 33s the patch passed
            -0 checkstyle 0m 31s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 41 new + 608 unchanged - 6 fixed = 649 total (was 614)
            +1 mvnsite 0m 36s the patch passed
            +1 whitespace 0m 0s The patch has no whitespace issues.
            +1 shadedclient 11m 4s patch has no errors when building and testing our client artifacts.
            +1 findbugs 1m 8s the patch passed
            +1 javadoc 0m 22s the patch passed
                  Other Tests
            -1 unit 58m 31s hadoop-yarn-server-resourcemanager in the patch failed.
            +1 asflicense 0m 19s The patch does not generate ASF License warnings.
            104m 1s



            Reason Tests
            Failed junit tests hadoop.yarn.server.resourcemanager.scheduler.capacity.TestIncreaseAllocationExpirer
            Timed out junit tests org.apache.hadoop.yarn.server.resourcemanager.TestRMStoreCommands
              org.apache.hadoop.yarn.server.resourcemanager.TestSubmitApplicationWithRMHA
              org.apache.hadoop.yarn.server.resourcemanager.TestKillApplicationWithRMHA



            Subsystem Report/Notes
            Docker Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639
            JIRA Issue YARN-7419
            JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12897264/YARN-7419.9.patch
            Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
            uname Linux 0a985acca980 3.13.0-117-generic #164-Ubuntu SMP Fri Apr 7 11:05:26 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
            Build tool maven
            Personality /testptch/patchprocess/precommit/personality/provided.sh
            git revision trunk / 3e26077
            maven version: Apache Maven 3.3.9
            Default Java 1.8.0_151
            findbugs v3.1.0-RC1
            checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/18458/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
            unit https://builds.apache.org/job/PreCommit-YARN-Build/18458/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
            Test Results https://builds.apache.org/job/PreCommit-YARN-Build/18458/testReport/
            Max. process+thread count 794 (vs. ulimit of 5000)
            modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
            Console output https://builds.apache.org/job/PreCommit-YARN-Build/18458/console
            Powered by Apache Yetus 0.7.0-SNAPSHOT http://yetus.apache.org

            This message was automatically generated.

            hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 15s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 5 new or modified test files.       trunk Compile Tests +1 mvninstall 16m 8s trunk passed +1 compile 0m 37s trunk passed +1 checkstyle 0m 33s trunk passed +1 mvnsite 0m 39s trunk passed +1 shadedclient 10m 57s branch has no errors when building and testing our client artifacts. +1 findbugs 1m 2s trunk passed +1 javadoc 0m 24s trunk passed       Patch Compile Tests +1 mvninstall 0m 38s the patch passed +1 compile 0m 33s the patch passed +1 javac 0m 33s the patch passed -0 checkstyle 0m 31s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 41 new + 608 unchanged - 6 fixed = 649 total (was 614) +1 mvnsite 0m 36s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 shadedclient 11m 4s patch has no errors when building and testing our client artifacts. +1 findbugs 1m 8s the patch passed +1 javadoc 0m 22s the patch passed       Other Tests -1 unit 58m 31s hadoop-yarn-server-resourcemanager in the patch failed. +1 asflicense 0m 19s The patch does not generate ASF License warnings. 104m 1s Reason Tests Failed junit tests hadoop.yarn.server.resourcemanager.scheduler.capacity.TestIncreaseAllocationExpirer Timed out junit tests org.apache.hadoop.yarn.server.resourcemanager.TestRMStoreCommands   org.apache.hadoop.yarn.server.resourcemanager.TestSubmitApplicationWithRMHA   org.apache.hadoop.yarn.server.resourcemanager.TestKillApplicationWithRMHA Subsystem Report/Notes Docker Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 JIRA Issue YARN-7419 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12897264/YARN-7419.9.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle uname Linux 0a985acca980 3.13.0-117-generic #164-Ubuntu SMP Fri Apr 7 11:05:26 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/patchprocess/precommit/personality/provided.sh git revision trunk / 3e26077 maven version: Apache Maven 3.3.9 Default Java 1.8.0_151 findbugs v3.1.0-RC1 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/18458/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/18458/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/18458/testReport/ Max. process+thread count 794 (vs. ulimit of 5000) modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/18458/console Powered by Apache Yetus 0.7.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
            leftnoteasy Wangda Tan added a comment -

            Latest patch looks good, +1. Will commit tomorrow if no objections. Thanks suma.shivaprasad!

            leftnoteasy Wangda Tan added a comment - Latest patch looks good, +1. Will commit tomorrow if no objections. Thanks suma.shivaprasad !
            leftnoteasy Wangda Tan added a comment -

            Committed to trunk, thanks suma.shivaprasad!

            leftnoteasy Wangda Tan added a comment - Committed to trunk, thanks suma.shivaprasad !
            hudson Hudson added a comment -

            SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13249 (See https://builds.apache.org/job/Hadoop-trunk-Commit/13249/)
            YARN-7419. CapacityScheduler: Allow auto leaf queue creation after queue (wangda: rev 0987a7b8cbbbb2c1e4c2095821d98a7db19644df)

            • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestAppManager.java
            • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java
            • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/placement/TestUserGroupMappingPlacementRule.java
            • (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestCapacitySchedulerAutoQueueCreation.java
            • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/placement/PlacementRule.java
            • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/RMAppImpl.java
            • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.java
            • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAppManager.java
            • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/ParentQueue.java
            • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/AutoCreatedLeafQueue.java
            • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/PlanQueue.java
            • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/TestSchedulerUtils.java
            • (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/ManagedParentQueue.java
            • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/placement/PlacementManager.java
            • (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/placement/ApplicationPlacementContext.java
            • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/AbstractCSQueue.java
            • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/placement/UserGroupMappingPlacementRule.java
            • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/event/AppAddedSchedulerEvent.java
            • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/AbstractManagedParentQueue.java
            • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestCapacityScheduler.java
            • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerQueueManager.java
            hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13249 (See https://builds.apache.org/job/Hadoop-trunk-Commit/13249/ ) YARN-7419 . CapacityScheduler: Allow auto leaf queue creation after queue (wangda: rev 0987a7b8cbbbb2c1e4c2095821d98a7db19644df) (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestAppManager.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/placement/TestUserGroupMappingPlacementRule.java (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestCapacitySchedulerAutoQueueCreation.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/placement/PlacementRule.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/RMAppImpl.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAppManager.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/ParentQueue.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/AutoCreatedLeafQueue.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/PlanQueue.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/TestSchedulerUtils.java (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/ManagedParentQueue.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/placement/PlacementManager.java (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/placement/ApplicationPlacementContext.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/AbstractCSQueue.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/placement/UserGroupMappingPlacementRule.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/event/AppAddedSchedulerEvent.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/AbstractManagedParentQueue.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestCapacityScheduler.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerQueueManager.java

            People

              suma.shivaprasad Suma Shivaprasad
              suma.shivaprasad Suma Shivaprasad
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: