Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.9.0, 3.0.0-alpha2
    • Component/s: capacityscheduler
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Support will be added for User-level access permission to use different application-priorities. This is to avoid situations where all users try running max priority in the cluster and thus degrading the value of priorities.

      Access Control Lists can be set per priority level within each queue. Below is an example configuration that can be added in capacity scheduler configuration
      file for each Queue level.
      yarn.scheduler.capacity.root.<queue_name>.<priority>.acl=user1,user2

      1. ApplicationPriority-ACL.pdf
        51 kB
        Sunil G
      2. YARN-3955.wip1.patch
        26 kB
        Sunil G
      3. ApplicationPriority-ACLs-v2.pdf
        51 kB
        Sunil G
      4. YARN-3955.v0.patch
        38 kB
        Sunil G
      5. YARN-3955.v1.patch
        52 kB
        Sunil G
      6. YARN-3955.0001.patch
        49 kB
        Sunil G
      7. YARN-3955.0002.patch
        56 kB
        Sunil G
      8. YARN-3955.0003.patch
        67 kB
        Sunil G
      9. YARN-3955.0004.patch
        70 kB
        Sunil G
      10. YARN-3955.0005.patch
        76 kB
        Sunil G
      11. YARN-3955.0006.patch
        154 kB
        Sunil G
      12. YARN-3955.0007.patch
        77 kB
        Sunil G
      13. YARN-3955.0008.patch
        81 kB
        Sunil G
      14. YARN-3955.0009.patch
        81 kB
        Sunil G
      15. YARN-3955.0010.patch
        81 kB
        Sunil G

        Activity

        Hide
        sunilg Sunil G added a comment -

        Attaching an initial doc. I will share a POC in a short while.

        Show
        sunilg Sunil G added a comment - Attaching an initial doc. I will share a POC in a short while.
        Hide
        sunilg Sunil G added a comment -

        Attaching an initial wip patch as a poc.

        Show
        sunilg Sunil G added a comment - Attaching an initial wip patch as a poc.
        Hide
        sunilg Sunil G added a comment -

        Attaching correct v0 patch. Wangda Tan Jian He pls share your thoughts.

        Show
        sunilg Sunil G added a comment - Attaching correct v0 patch. Wangda Tan Jian He pls share your thoughts.
        Hide
        sunilg Sunil G added a comment -

        Upgrading to a v1 patch with basic test cases.

        Show
        sunilg Sunil G added a comment - Upgrading to a v1 patch with basic test cases.
        Hide
        jianhe Jian He added a comment -

        thanks Sunil, some comments for the patch:

        • PriorityACLConfiguration#getPrioirityAcl typo
        • I think we may not need a public API record PriorityACL as it's not supposed to be used by external application. May use AccessType#SUBMIT_APP_PRIORITY directly
        • AccessType#SUBMIT_APP_PRIORITY, maybe call it ACCESS_PRIORITY ?
        • make the log message clear that "user does not have permission to change priority" ?
                RMAuditLogger.logFailure(callerUGI.getShortUserName(),
                    AuditConstants.UPDATE_APP_PRIORITY,
                    "User doesn't have permissions to "
                        + ApplicationAccessType.MODIFY_APP.toString(),
                    "ClientRMService", AuditConstants.UNAUTHORIZED_USER, applicationId);
                throw RPCUtil.getRemoteException(new AccessControlException("User "
                    + callerUGI.getShortUserName() + " cannot perform operation "
                    + ApplicationAccessType.MODIFY_APP.name() + " on " + applicationId));
          
        • I think readLock is not needed, the field itself is not changing.
              try {
                readLock.lock();
                return priorityAcls;
              } finally {
                readLock.unlock()
          
        • could you add sample configurations in capacity-scheduler.xml, and add comments about what the syntax is for the config ?
        • the priority comes from submissionContext can be an arbitrary value, it's not guarated that "csqueue.getPriorityPrivilegedEntity(priority)" will always return the object?
            Priority priority = submissionContext.getPriority();
                if (null != csqueue) {
                  if ((!authorizer.checkPermission(
                          new AccessRequest(csqueue.getPrivilegedEntity(), userUgi,
                          SchedulerUtils.toAccessType(QueueACL.SUBMIT_APPLICATIONS),
                          applicationId.toString(), appName,
                          Server.getRemoteAddress(), null))
                      && !authorizer.checkPermission(
                          new AccessRequest(csqueue.getPrivilegedEntity(), userUgi,
                              SchedulerUtils.toAccessType(QueueACL.ADMINISTER_QUEUE),
                              applicationId.toString(), appName,
                              Server.getRemoteAddress(), null)))
                      || !authorizer.checkPermission(new AccessRequest(
                          csqueue.getPriorityPrivilegedEntity(priority), userUgi,
          
        • I feel, adding every possible priority at every integer for a queue into the ConfiguredYarnAuthorizer is cumbersome. One solution in my mind is to create a new QueueEntity which overridds PrivilegedEntity, and we add a max-priority field in the new object. On checkPermission, we check whether the accessType is accessPriority, if it is, check if the priority is less than the max-priority. AccessRequest also needs to add a new priority field.
        Show
        jianhe Jian He added a comment - thanks Sunil, some comments for the patch: PriorityACLConfiguration#getPrioirityAcl typo I think we may not need a public API record PriorityACL as it's not supposed to be used by external application. May use AccessType#SUBMIT_APP_PRIORITY directly AccessType#SUBMIT_APP_PRIORITY, maybe call it ACCESS_PRIORITY ? make the log message clear that "user does not have permission to change priority" ? RMAuditLogger.logFailure(callerUGI.getShortUserName(), AuditConstants.UPDATE_APP_PRIORITY, "User doesn't have permissions to " + ApplicationAccessType.MODIFY_APP.toString(), "ClientRMService" , AuditConstants.UNAUTHORIZED_USER, applicationId); throw RPCUtil.getRemoteException( new AccessControlException( "User " + callerUGI.getShortUserName() + " cannot perform operation " + ApplicationAccessType.MODIFY_APP.name() + " on " + applicationId)); I think readLock is not needed, the field itself is not changing. try { readLock.lock(); return priorityAcls; } finally { readLock.unlock() could you add sample configurations in capacity-scheduler.xml, and add comments about what the syntax is for the config ? the priority comes from submissionContext can be an arbitrary value, it's not guarated that "csqueue.getPriorityPrivilegedEntity(priority)" will always return the object? Priority priority = submissionContext.getPriority(); if ( null != csqueue) { if ((!authorizer.checkPermission( new AccessRequest(csqueue.getPrivilegedEntity(), userUgi, SchedulerUtils.toAccessType(QueueACL.SUBMIT_APPLICATIONS), applicationId.toString(), appName, Server.getRemoteAddress(), null )) && !authorizer.checkPermission( new AccessRequest(csqueue.getPrivilegedEntity(), userUgi, SchedulerUtils.toAccessType(QueueACL.ADMINISTER_QUEUE), applicationId.toString(), appName, Server.getRemoteAddress(), null ))) || !authorizer.checkPermission( new AccessRequest( csqueue.getPriorityPrivilegedEntity(priority), userUgi, I feel, adding every possible priority at every integer for a queue into the ConfiguredYarnAuthorizer is cumbersome. One solution in my mind is to create a new QueueEntity which overridds PrivilegedEntity, and we add a max-priority field in the new object. On checkPermission, we check whether the accessType is accessPriority, if it is, check if the priority is less than the max-priority. AccessRequest also needs to add a new priority field.
        Hide
        sunilg Sunil G added a comment -

        Thanks Jian He for the comments.

        I think readLock is not needed, the field itself is not changing

        priorityACLs could be changed during reinitialize. Do we need to consider this point. Also we may add REST based support to add ACLs during runtime. If reinitialize is fine, i could remove lock and add it when REST is in progress. Thoughts?

        Show
        sunilg Sunil G added a comment - Thanks Jian He for the comments. I think readLock is not needed, the field itself is not changing priorityACLs could be changed during reinitialize. Do we need to consider this point. Also we may add REST based support to add ACLs during runtime. If reinitialize is fine, i could remove lock and add it when REST is in progress. Thoughts?
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 18s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
        0 mvndep 0m 9s Maven dependency ordering for branch
        +1 mvninstall 7m 38s trunk passed
        -1 compile 6m 0s hadoop-yarn in trunk failed.
        +1 checkstyle 0m 52s trunk passed
        +1 mvnsite 1m 56s trunk passed
        +1 mvneclipse 1m 9s trunk passed
        +1 findbugs 2m 49s trunk passed
        +1 javadoc 1m 18s trunk passed
        0 mvndep 0m 10s Maven dependency ordering for patch
        +1 mvninstall 1m 11s the patch passed
        -1 compile 4m 44s hadoop-yarn in the patch failed.
        -1 javac 4m 44s hadoop-yarn in the patch failed.
        -0 checkstyle 0m 52s hadoop-yarn-project/hadoop-yarn: The patch generated 26 new + 365 unchanged - 1 fixed = 391 total (was 366)
        +1 mvnsite 1m 36s the patch passed
        +1 mvneclipse 0m 49s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 xml 0m 2s The patch has no ill-formed XML file.
        -1 findbugs 1m 35s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0)
        +1 javadoc 1m 15s the patch passed
        +1 unit 2m 56s hadoop-yarn-common in the patch passed.
        -1 unit 39m 18s hadoop-yarn-server-resourcemanager in the patch failed.
        +1 asflicense 0m 38s The patch does not generate ASF License warnings.
        87m 17s



        Reason Tests
        FindBugs module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
          org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.PriorityACLConfiguration.createACLStringPerPriority(HashMap, Map) invokes inefficient new String() constructor At PriorityACLConfiguration.java:String() constructor At PriorityACLConfiguration.java:[line 137]
          Call to StringBuilder.equals(String) in org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.PriorityACLConfiguration.createACLStringForPriority(Map, Priority, String, PriorityACLConfiguration$PriorityACLConfig) At PriorityACLConfiguration.java:Priority, String, PriorityACLConfiguration$PriorityACLConfig) At PriorityACLConfiguration.java:[line 223]
        Failed junit tests hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesAppsModification
          hadoop.yarn.server.resourcemanager.scheduler.capacity.TestLeafQueue
          hadoop.yarn.server.resourcemanager.scheduler.capacity.TestReservationQueue
          hadoop.yarn.server.resourcemanager.scheduler.capacity.TestReservations
          hadoop.yarn.server.resourcemanager.scheduler.capacity.TestParentQueue
          hadoop.yarn.server.resourcemanager.scheduler.capacity.TestChildQueueOrder
          hadoop.yarn.server.resourcemanager.scheduler.capacity.TestApplicationLimits
          hadoop.yarn.server.resourcemanager.scheduler.capacity.TestApplicationLimitsByPartition



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:e809691
        JIRA Issue YARN-3955
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12838403/YARN-3955.0001.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
        uname Linux a96b30bbd0ff 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28 20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 89354f0
        Default Java 1.8.0_101
        compile https://builds.apache.org/job/PreCommit-YARN-Build/13861/artifact/patchprocess/branch-compile-hadoop-yarn-project_hadoop-yarn.txt
        findbugs v3.0.0
        compile https://builds.apache.org/job/PreCommit-YARN-Build/13861/artifact/patchprocess/patch-compile-hadoop-yarn-project_hadoop-yarn.txt
        javac https://builds.apache.org/job/PreCommit-YARN-Build/13861/artifact/patchprocess/patch-compile-hadoop-yarn-project_hadoop-yarn.txt
        checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/13861/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt
        findbugs https://builds.apache.org/job/PreCommit-YARN-Build/13861/artifact/patchprocess/new-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.html
        unit https://builds.apache.org/job/PreCommit-YARN-Build/13861/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13861/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/13861/console
        Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 18s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. 0 mvndep 0m 9s Maven dependency ordering for branch +1 mvninstall 7m 38s trunk passed -1 compile 6m 0s hadoop-yarn in trunk failed. +1 checkstyle 0m 52s trunk passed +1 mvnsite 1m 56s trunk passed +1 mvneclipse 1m 9s trunk passed +1 findbugs 2m 49s trunk passed +1 javadoc 1m 18s trunk passed 0 mvndep 0m 10s Maven dependency ordering for patch +1 mvninstall 1m 11s the patch passed -1 compile 4m 44s hadoop-yarn in the patch failed. -1 javac 4m 44s hadoop-yarn in the patch failed. -0 checkstyle 0m 52s hadoop-yarn-project/hadoop-yarn: The patch generated 26 new + 365 unchanged - 1 fixed = 391 total (was 366) +1 mvnsite 1m 36s the patch passed +1 mvneclipse 0m 49s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 2s The patch has no ill-formed XML file. -1 findbugs 1m 35s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0) +1 javadoc 1m 15s the patch passed +1 unit 2m 56s hadoop-yarn-common in the patch passed. -1 unit 39m 18s hadoop-yarn-server-resourcemanager in the patch failed. +1 asflicense 0m 38s The patch does not generate ASF License warnings. 87m 17s Reason Tests FindBugs module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager   org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.PriorityACLConfiguration.createACLStringPerPriority(HashMap, Map) invokes inefficient new String() constructor At PriorityACLConfiguration.java:String() constructor At PriorityACLConfiguration.java: [line 137]   Call to StringBuilder.equals(String) in org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.PriorityACLConfiguration.createACLStringForPriority(Map, Priority, String, PriorityACLConfiguration$PriorityACLConfig) At PriorityACLConfiguration.java:Priority, String, PriorityACLConfiguration$PriorityACLConfig) At PriorityACLConfiguration.java: [line 223] Failed junit tests hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesAppsModification   hadoop.yarn.server.resourcemanager.scheduler.capacity.TestLeafQueue   hadoop.yarn.server.resourcemanager.scheduler.capacity.TestReservationQueue   hadoop.yarn.server.resourcemanager.scheduler.capacity.TestReservations   hadoop.yarn.server.resourcemanager.scheduler.capacity.TestParentQueue   hadoop.yarn.server.resourcemanager.scheduler.capacity.TestChildQueueOrder   hadoop.yarn.server.resourcemanager.scheduler.capacity.TestApplicationLimits   hadoop.yarn.server.resourcemanager.scheduler.capacity.TestApplicationLimitsByPartition Subsystem Report/Notes Docker Image:yetus/hadoop:e809691 JIRA Issue YARN-3955 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12838403/YARN-3955.0001.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux a96b30bbd0ff 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28 20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 89354f0 Default Java 1.8.0_101 compile https://builds.apache.org/job/PreCommit-YARN-Build/13861/artifact/patchprocess/branch-compile-hadoop-yarn-project_hadoop-yarn.txt findbugs v3.0.0 compile https://builds.apache.org/job/PreCommit-YARN-Build/13861/artifact/patchprocess/patch-compile-hadoop-yarn-project_hadoop-yarn.txt javac https://builds.apache.org/job/PreCommit-YARN-Build/13861/artifact/patchprocess/patch-compile-hadoop-yarn-project_hadoop-yarn.txt checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/13861/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt findbugs https://builds.apache.org/job/PreCommit-YARN-Build/13861/artifact/patchprocess/new-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.html unit https://builds.apache.org/job/PreCommit-YARN-Build/13861/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13861/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn Console output https://builds.apache.org/job/PreCommit-YARN-Build/13861/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        sunilg Sunil G added a comment -

        Attaching new patch.

        This patch doesnt use YarnAuthorizationProvider. Please help to check
        cc/Jian He Wangda Tan and Rohith Sharma K S

        Show
        sunilg Sunil G added a comment - Attaching new patch. This patch doesnt use YarnAuthorizationProvider. Please help to check cc/ Jian He Wangda Tan and Rohith Sharma K S
        Hide
        sunilg Sunil G added a comment -

        Attaching latest patch to cover the runtime update scenario also.

        Show
        sunilg Sunil G added a comment - Attaching latest patch to cover the runtime update scenario also.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 17s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 3 new or modified test files.
        0 mvndep 0m 15s Maven dependency ordering for branch
        +1 mvninstall 7m 9s trunk passed
        +1 compile 10m 15s trunk passed
        +1 checkstyle 1m 40s trunk passed
        +1 mvnsite 1m 55s trunk passed
        +1 mvneclipse 1m 7s trunk passed
        +1 findbugs 3m 17s trunk passed
        +1 javadoc 1m 42s trunk passed
        0 mvndep 0m 24s Maven dependency ordering for patch
        +1 mvninstall 1m 50s the patch passed
        +1 compile 12m 33s the patch passed
        +1 javac 12m 33s the patch passed
        -0 checkstyle 2m 0s root: The patch generated 29 new + 424 unchanged - 2 fixed = 453 total (was 426)
        +1 mvnsite 2m 32s the patch passed
        +1 mvneclipse 1m 26s the patch passed
        -1 whitespace 0m 0s The patch has 3 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
        +1 xml 0m 2s The patch has no ill-formed XML file.
        -1 findbugs 1m 40s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager generated 3 new + 0 unchanged - 0 fixed = 3 total (was 0)
        -1 javadoc 0m 41s hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager generated 1 new + 913 unchanged - 0 fixed = 914 total (was 913)
        +1 unit 2m 58s hadoop-yarn-common in the patch passed.
        -1 unit 46m 1s hadoop-yarn-server-resourcemanager in the patch failed.
        +1 unit 1m 8s hadoop-sls in the patch passed.
        +1 asflicense 0m 39s The patch does not generate ASF License warnings.
        129m 53s



        Reason Tests
        FindBugs module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
          org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.PriorityACLConfiguration.createACLStringForPriority(List) invokes inefficient new String() constructor At PriorityACLConfiguration.java:constructor At PriorityACLConfiguration.java:[line 159]
          org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.PriorityACLGroup defines compareTo(PriorityACLGroup) and uses Object.equals() At PriorityACLGroup.java:Object.equals() At PriorityACLGroup.java:[line 43]
          Should org.apache.hadoop.yarn.server.resourcemanager.security.AppPriorityACLsManager$PriorityACL be a static inner class? At AppPriorityACLsManager.java:inner class? At AppPriorityACLsManager.java:[lines 51-70]
        Failed junit tests hadoop.yarn.server.resourcemanager.scheduler.capacity.TestQueueState
          hadoop.yarn.server.resourcemanager.scheduler.capacity.TestQueueParsing
          hadoop.yarn.server.resourcemanager.TestRMRestart
          hadoop.yarn.server.resourcemanager.webapp.TestRedirectionErrorPage
          hadoop.yarn.server.resourcemanager.scheduler.capacity.TestQueueMappings
          hadoop.yarn.server.resourcemanager.webapp.TestRMWebApp
          hadoop.yarn.server.resourcemanager.scheduler.capacity.TestLeafQueue
          hadoop.yarn.server.resourcemanager.resourcetracker.TestNMReconnect
          hadoop.yarn.server.resourcemanager.scheduler.capacity.TestReservations
          hadoop.yarn.server.resourcemanager.reservation.TestCapacitySchedulerPlanFollower
          hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer
          hadoop.yarn.server.resourcemanager.reservation.TestReservationSystem
          hadoop.yarn.server.resourcemanager.webapp.TestAppPage
          hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacityScheduler
          hadoop.yarn.server.resourcemanager.webapp.TestNodesPage



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:a9ad5d6
        JIRA Issue YARN-3955
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12842343/YARN-3955.0003.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
        uname Linux 335055fcdfa3 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 0ef7961
        Default Java 1.8.0_111
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/14223/artifact/patchprocess/diff-checkstyle-root.txt
        whitespace https://builds.apache.org/job/PreCommit-YARN-Build/14223/artifact/patchprocess/whitespace-eol.txt
        findbugs https://builds.apache.org/job/PreCommit-YARN-Build/14223/artifact/patchprocess/new-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.html
        javadoc https://builds.apache.org/job/PreCommit-YARN-Build/14223/artifact/patchprocess/diff-javadoc-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
        unit https://builds.apache.org/job/PreCommit-YARN-Build/14223/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/14223/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager hadoop-tools/hadoop-sls U: .
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/14223/console
        Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 17s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 3 new or modified test files. 0 mvndep 0m 15s Maven dependency ordering for branch +1 mvninstall 7m 9s trunk passed +1 compile 10m 15s trunk passed +1 checkstyle 1m 40s trunk passed +1 mvnsite 1m 55s trunk passed +1 mvneclipse 1m 7s trunk passed +1 findbugs 3m 17s trunk passed +1 javadoc 1m 42s trunk passed 0 mvndep 0m 24s Maven dependency ordering for patch +1 mvninstall 1m 50s the patch passed +1 compile 12m 33s the patch passed +1 javac 12m 33s the patch passed -0 checkstyle 2m 0s root: The patch generated 29 new + 424 unchanged - 2 fixed = 453 total (was 426) +1 mvnsite 2m 32s the patch passed +1 mvneclipse 1m 26s the patch passed -1 whitespace 0m 0s The patch has 3 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply +1 xml 0m 2s The patch has no ill-formed XML file. -1 findbugs 1m 40s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager generated 3 new + 0 unchanged - 0 fixed = 3 total (was 0) -1 javadoc 0m 41s hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager generated 1 new + 913 unchanged - 0 fixed = 914 total (was 913) +1 unit 2m 58s hadoop-yarn-common in the patch passed. -1 unit 46m 1s hadoop-yarn-server-resourcemanager in the patch failed. +1 unit 1m 8s hadoop-sls in the patch passed. +1 asflicense 0m 39s The patch does not generate ASF License warnings. 129m 53s Reason Tests FindBugs module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager   org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.PriorityACLConfiguration.createACLStringForPriority(List) invokes inefficient new String() constructor At PriorityACLConfiguration.java:constructor At PriorityACLConfiguration.java: [line 159]   org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.PriorityACLGroup defines compareTo(PriorityACLGroup) and uses Object.equals() At PriorityACLGroup.java:Object.equals() At PriorityACLGroup.java: [line 43]   Should org.apache.hadoop.yarn.server.resourcemanager.security.AppPriorityACLsManager$PriorityACL be a static inner class? At AppPriorityACLsManager.java:inner class? At AppPriorityACLsManager.java: [lines 51-70] Failed junit tests hadoop.yarn.server.resourcemanager.scheduler.capacity.TestQueueState   hadoop.yarn.server.resourcemanager.scheduler.capacity.TestQueueParsing   hadoop.yarn.server.resourcemanager.TestRMRestart   hadoop.yarn.server.resourcemanager.webapp.TestRedirectionErrorPage   hadoop.yarn.server.resourcemanager.scheduler.capacity.TestQueueMappings   hadoop.yarn.server.resourcemanager.webapp.TestRMWebApp   hadoop.yarn.server.resourcemanager.scheduler.capacity.TestLeafQueue   hadoop.yarn.server.resourcemanager.resourcetracker.TestNMReconnect   hadoop.yarn.server.resourcemanager.scheduler.capacity.TestReservations   hadoop.yarn.server.resourcemanager.reservation.TestCapacitySchedulerPlanFollower   hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer   hadoop.yarn.server.resourcemanager.reservation.TestReservationSystem   hadoop.yarn.server.resourcemanager.webapp.TestAppPage   hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacityScheduler   hadoop.yarn.server.resourcemanager.webapp.TestNodesPage Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue YARN-3955 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12842343/YARN-3955.0003.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux 335055fcdfa3 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 0ef7961 Default Java 1.8.0_111 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/14223/artifact/patchprocess/diff-checkstyle-root.txt whitespace https://builds.apache.org/job/PreCommit-YARN-Build/14223/artifact/patchprocess/whitespace-eol.txt findbugs https://builds.apache.org/job/PreCommit-YARN-Build/14223/artifact/patchprocess/new-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.html javadoc https://builds.apache.org/job/PreCommit-YARN-Build/14223/artifact/patchprocess/diff-javadoc-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/14223/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/14223/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager hadoop-tools/hadoop-sls U: . Console output https://builds.apache.org/job/PreCommit-YARN-Build/14223/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        leftnoteasy Wangda Tan added a comment -

        Thanks Sunil G.

        Besides UT failures, findbugs, etc. Some comments:

        1) AppPriorityACLsManager

        • private class PriorityACL add static
        • Why put it to RM(Active)Context? We have checkAndGetApplicationPriority exposed by scheduler already. Is it possible to remove it from contexts?
        • setPrioirityACLsToStore, remove the store? It doesn't have logic of state store? And where is the logic to update state store?
        • Common logic of checkAccess / getDefaultPriority can be merged.
        • getDefaultPriority, I found PriorityAclGroup#getDefaultPriority is not accessed, so this looks like using max priority instead of default. Adding some UTs for this?

        2) Changes to capacity-scheduler.xml

        • Related to this JIRA?

        3) CapacityScheduler

        • checkAndGetApplicationPriority: priorityFromContext -> something like priorityRequestedByApp
        • not reuse "priorityRequestedByApp", and change the appPriority
        • this.appPriorityACLManager.getDefaultPriority can be called before calling queueManager.getDefaultPriorityForQueue. And getDefaultPriorityForQueue can be called only if previous one is null.

        4) CSQueueManager:

        • move setPriorityAcls to setQueueAcls?

        5) ClientRMService:
        Instead of call methods of appPriorityAclManager (I prefer to remove it, see #1). Can we call scheduler#checkAndGetApplicationPriority to achieve the same goal?

        6) LeafQueue:

        • getPriorityAcls, readLock is not necessary. Add a final to priorityAcls should be enough.
        • remove unncessary import

        7) Add comprehensive tests to PriorityAclConfiguration should be better.

        8) RMAppManager:

        • Changes of the if (isRecovery && YarnConfiguration.isAclEnabled(conf) && scheduler instanceof CapacityScheduler) may not be necessary?
        Show
        leftnoteasy Wangda Tan added a comment - Thanks Sunil G . Besides UT failures, findbugs, etc. Some comments: 1) AppPriorityACLsManager private class PriorityACL add static Why put it to RM(Active)Context? We have checkAndGetApplicationPriority exposed by scheduler already. Is it possible to remove it from contexts? setPrioirityACLsToStore, remove the store? It doesn't have logic of state store? And where is the logic to update state store? Common logic of checkAccess / getDefaultPriority can be merged. getDefaultPriority, I found PriorityAclGroup#getDefaultPriority is not accessed, so this looks like using max priority instead of default. Adding some UTs for this? 2) Changes to capacity-scheduler.xml Related to this JIRA? 3) CapacityScheduler checkAndGetApplicationPriority: priorityFromContext -> something like priorityRequestedByApp not reuse "priorityRequestedByApp", and change the appPriority this.appPriorityACLManager.getDefaultPriority can be called before calling queueManager.getDefaultPriorityForQueue. And getDefaultPriorityForQueue can be called only if previous one is null. 4) CSQueueManager: move setPriorityAcls to setQueueAcls? 5) ClientRMService: Instead of call methods of appPriorityAclManager (I prefer to remove it, see #1). Can we call scheduler#checkAndGetApplicationPriority to achieve the same goal? 6) LeafQueue: getPriorityAcls, readLock is not necessary. Add a final to priorityAcls should be enough. remove unncessary import 7) Add comprehensive tests to PriorityAclConfiguration should be better. 8) RMAppManager: Changes of the if (isRecovery && YarnConfiguration.isAclEnabled(conf) && scheduler instanceof CapacityScheduler) may not be necessary?
        Hide
        sunilg Sunil G added a comment -

        HI Wangda Tan

        Thanks for the comments. I will work on tests/findbugs/ and comments.

        In general I have one doubt below point.

        Instead of call methods of appPriorityAclManager (I prefer to remove it, see #1). Can we call scheduler#checkAndGetApplicationPriority to achieve the same goal?

        I think with QueueACLsManager, we took ACLs out of CS and managed externally. Yes, it was to plugin external ACLs policies, however I thought it was good. I think that was the reason I went with a PriorityACLManager model. Do you see adding PrioirtyACLs back to scheduler is better? If so, as suggested we can leverage from scheduler apis. Thoughts?

        Show
        sunilg Sunil G added a comment - HI Wangda Tan Thanks for the comments. I will work on tests/findbugs/ and comments. In general I have one doubt below point. Instead of call methods of appPriorityAclManager (I prefer to remove it, see #1). Can we call scheduler#checkAndGetApplicationPriority to achieve the same goal? I think with QueueACLsManager , we took ACLs out of CS and managed externally. Yes, it was to plugin external ACLs policies, however I thought it was good. I think that was the reason I went with a PriorityACLManager model. Do you see adding PrioirtyACLs back to scheduler is better? If so, as suggested we can leverage from scheduler apis. Thoughts?
        Hide
        sunilg Sunil G added a comment -

        Attaching an interim version of patch to check jenkins.

        Show
        sunilg Sunil G added a comment - Attaching an interim version of patch to check jenkins.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 14s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 6 new or modified test files.
        0 mvndep 0m 16s Maven dependency ordering for branch
        +1 mvninstall 13m 4s trunk passed
        +1 compile 10m 47s trunk passed
        +1 checkstyle 1m 45s trunk passed
        +1 mvnsite 1m 44s trunk passed
        +1 mvneclipse 1m 0s trunk passed
        +1 findbugs 3m 0s trunk passed
        +1 javadoc 1m 24s trunk passed
        0 mvndep 0m 18s Maven dependency ordering for patch
        +1 mvninstall 1m 27s the patch passed
        +1 compile 10m 12s the patch passed
        +1 javac 10m 12s the patch passed
        -0 checkstyle 1m 47s root: The patch generated 41 new + 494 unchanged - 2 fixed = 535 total (was 496)
        +1 mvnsite 1m 51s the patch passed
        +1 mvneclipse 1m 10s the patch passed
        -1 whitespace 0m 0s The patch has 2 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
        +1 xml 0m 2s The patch has no ill-formed XML file.
        -1 findbugs 1m 24s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager generated 3 new + 0 unchanged - 0 fixed = 3 total (was 0)
        -1 javadoc 0m 31s hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager generated 1 new + 913 unchanged - 0 fixed = 914 total (was 913)
        +1 unit 2m 35s hadoop-yarn-common in the patch passed.
        -1 unit 15m 7s hadoop-yarn-server-resourcemanager in the patch failed.
        +1 unit 1m 7s hadoop-sls in the patch passed.
        -1 asflicense 0m 37s The patch generated 1 ASF License warnings.
        98m 30s



        Reason Tests
        FindBugs module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
          Possible null pointer dereference of priorityRequestedByApp in org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.checkAndGetApplicationPriority(Priority, UserGroupInformation, String, ApplicationId) Dereferenced at CapacityScheduler.java:priorityRequestedByApp in org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.checkAndGetApplicationPriority(Priority, UserGroupInformation, String, ApplicationId) Dereferenced at CapacityScheduler.java:[line 2214]
          org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.PriorityACLConfiguration.createACLStringForPriority(List) invokes inefficient new String() constructor At PriorityACLConfiguration.java:constructor At PriorityACLConfiguration.java:[line 159]
          org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.PriorityACLGroup defines compareTo(PriorityACLGroup) and uses Object.equals() At PriorityACLGroup.java:Object.equals() At PriorityACLGroup.java:[line 43]
        Failed junit tests hadoop.yarn.server.resourcemanager.TestRMProxyUsersConf
          hadoop.yarn.server.resourcemanager.webapp.TestNodesPage
          hadoop.yarn.server.resourcemanager.scheduler.capacity.TestLeafQueue
          hadoop.yarn.server.resourcemanager.scheduler.capacity.TestQueueState
          hadoop.yarn.server.resourcemanager.monitor.TestSchedulingMonitor
          hadoop.yarn.server.resourcemanager.webapp.TestRMWebApp
          hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerResizing
          hadoop.yarn.server.resourcemanager.webapp.TestAppPage
          hadoop.yarn.server.resourcemanager.scheduler.capacity.TestApplicationPriority
          hadoop.yarn.server.resourcemanager.scheduler.capacity.TestIncreaseAllocationExpirer
          hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerAllocation
          hadoop.yarn.server.resourcemanager.scheduler.capacity.TestReservations
          hadoop.yarn.server.resourcemanager.TestClientRMService
          hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacitySchedulerLazyPreemption
          hadoop.yarn.server.resourcemanager.TestAMAuthorization
          hadoop.yarn.server.resourcemanager.TestSubmitApplicationWithRMHA
          hadoop.yarn.server.resourcemanager.scheduler.policy.TestFairOrderingPolicy
          hadoop.yarn.server.resourcemanager.TestSignalContainer
          hadoop.yarn.server.resourcemanager.TestReservationSystemWithRMHA
          hadoop.yarn.server.resourcemanager.TestLeaderElectorService
          hadoop.yarn.server.resourcemanager.rmapp.TestNodesListManager
          hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacitySchedulerNodeLabelUpdate
          hadoop.yarn.server.resourcemanager.TestRMHA
          hadoop.yarn.server.resourcemanager.nodelabels.TestRMNodeLabelsManager
          hadoop.yarn.server.resourcemanager.TestRMHAForNodeLabels
          hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesCapacitySched
          hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesAppsModification
          hadoop.yarn.server.resourcemanager.scheduler.TestSchedulerHealth
          hadoop.yarn.server.resourcemanager.applicationsmanager.TestAMRMRPCNodeUpdates
          hadoop.yarn.server.resourcemanager.nodelabels.TestRMDelegatedNodeLabelsUpdater
          hadoop.yarn.server.resourcemanager.TestRMHAMetrics
          hadoop.yarn.server.resourcemanager.TestRMRestart
          hadoop.yarn.server.resourcemanager.TestRMEmbeddedElector
          hadoop.yarn.server.resourcemanager.TestApplicationCleanup
          hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer
          hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesSchedulerActivities
          hadoop.yarn.server.resourcemanager.TestContainerResourceUsage
          hadoop.yarn.server.resourcemanager.rmcontainer.TestRMContainerImpl
          hadoop.yarn.server.resourcemanager.applicationsmanager.TestAMRMRPCResponseId
          hadoop.yarn.server.resourcemanager.recovery.TestZKRMStateStore
          hadoop.yarn.server.resourcemanager.scheduler.capacity.TestWorkPreservingRMRestartForNodeLabel
          hadoop.yarn.server.resourcemanager.security.TestClientToAMTokens
          hadoop.yarn.server.resourcemanager.TestNodeBlacklistingOnAMFailures
          hadoop.yarn.server.resourcemanager.scheduler.capacity.TestQueueMappings
          hadoop.yarn.server.resourcemanager.reservation.TestReservationSystem
          hadoop.yarn.server.resourcemanager.TestAppManager
          hadoop.yarn.server.resourcemanager.scheduler.capacity.TestQueueParsing
          hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesNodeLabels
          hadoop.yarn.server.resourcemanager.ahs.TestRMApplicationHistoryWriter
          hadoop.yarn.server.resourcemanager.resourcetracker.TestNMReconnect
          hadoop.yarn.server.resourcemanager.TestApplicationACLs
          hadoop.yarn.server.resourcemanager.TestResourceTrackerService
          hadoop.yarn.server.resourcemanager.TestKillApplicationWithRMHA
          hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesNodes
          hadoop.yarn.server.resourcemanager.scheduler.capacity.TestApplicationLimits
          hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacityScheduler
          hadoop.yarn.server.resourcemanager.TestRM
          hadoop.yarn.server.resourcemanager.webapp.TestRMWebServiceAppsNodelabel
          hadoop.yarn.server.resourcemanager.scheduler.TestAbstractYarnScheduler
          hadoop.yarn.server.resourcemanager.scheduler.capacity.TestApplicationLimitsByPartition
          hadoop.yarn.server.resourcemanager.rmapp.attempt.TestAMLivelinessMonitor
          hadoop.yarn.server.resourcemanager.TestRMStoreCommands
          hadoop.yarn.server.resourcemanager.security.TestRMDelegationTokens
          hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacitySchedulerDynamicBehavior
          hadoop.yarn.server.resourcemanager.TestApplicationMasterService
          hadoop.yarn.server.resourcemanager.webapp.TestRedirectionErrorPage
          hadoop.yarn.server.resourcemanager.reservation.TestCapacitySchedulerPlanFollower
          hadoop.yarn.server.resourcemanager.TestDecommissioningNodesWatcher
          hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesReservation
          hadoop.yarn.server.resourcemanager.security.TestAMRMTokens
          hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacitySchedulerSurgicalPreemption
          hadoop.yarn.server.resourcemanager.TestOpportunisticContainerAllocatorAMService
          hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacitySchedulerAsyncScheduling
          hadoop.yarn.server.resourcemanager.rmapp.TestApplicationLifetimeMonitor
          hadoop.yarn.server.resourcemanager.scheduler.TestSchedulingWithAllocationRequestId
          hadoop.yarn.server.resourcemanager.applicationsmanager.TestAMRestart
          hadoop.yarn.server.resourcemanager.TestApplicationMasterLauncher
          hadoop.yarn.server.resourcemanager.scheduler.capacity.TestApplicationPriorityACLs
          hadoop.yarn.server.resourcemanager.monitor.capacity.TestProportionalCapacityPreemptionPolicy
          hadoop.yarn.server.resourcemanager.TestWorkPreservingRMRestart
          hadoop.yarn.server.resourcemanager.TestResourceManager
          hadoop.yarn.server.resourcemanager.scheduler.TestSchedulerUtils
          hadoop.yarn.server.resourcemanager.scheduler.capacity.TestNodeLabelContainerAllocation
          hadoop.yarn.server.resourcemanager.TestRMAdminService
          hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacitySchedulerQueueACLs
          hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesForCSWithPartitions



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:a9ad5d6
        JIRA Issue YARN-3955
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12844511/YARN-3955.0004.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
        uname Linux 1d31859b6e0d 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 22befbd
        Default Java 1.8.0_111
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/14449/artifact/patchprocess/diff-checkstyle-root.txt
        whitespace https://builds.apache.org/job/PreCommit-YARN-Build/14449/artifact/patchprocess/whitespace-eol.txt
        findbugs https://builds.apache.org/job/PreCommit-YARN-Build/14449/artifact/patchprocess/new-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.html
        javadoc https://builds.apache.org/job/PreCommit-YARN-Build/14449/artifact/patchprocess/diff-javadoc-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
        unit https://builds.apache.org/job/PreCommit-YARN-Build/14449/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/14449/testReport/
        asflicense https://builds.apache.org/job/PreCommit-YARN-Build/14449/artifact/patchprocess/patch-asflicense-problems.txt
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager hadoop-tools/hadoop-sls U: .
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/14449/console
        Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 14s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 6 new or modified test files. 0 mvndep 0m 16s Maven dependency ordering for branch +1 mvninstall 13m 4s trunk passed +1 compile 10m 47s trunk passed +1 checkstyle 1m 45s trunk passed +1 mvnsite 1m 44s trunk passed +1 mvneclipse 1m 0s trunk passed +1 findbugs 3m 0s trunk passed +1 javadoc 1m 24s trunk passed 0 mvndep 0m 18s Maven dependency ordering for patch +1 mvninstall 1m 27s the patch passed +1 compile 10m 12s the patch passed +1 javac 10m 12s the patch passed -0 checkstyle 1m 47s root: The patch generated 41 new + 494 unchanged - 2 fixed = 535 total (was 496) +1 mvnsite 1m 51s the patch passed +1 mvneclipse 1m 10s the patch passed -1 whitespace 0m 0s The patch has 2 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply +1 xml 0m 2s The patch has no ill-formed XML file. -1 findbugs 1m 24s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager generated 3 new + 0 unchanged - 0 fixed = 3 total (was 0) -1 javadoc 0m 31s hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager generated 1 new + 913 unchanged - 0 fixed = 914 total (was 913) +1 unit 2m 35s hadoop-yarn-common in the patch passed. -1 unit 15m 7s hadoop-yarn-server-resourcemanager in the patch failed. +1 unit 1m 7s hadoop-sls in the patch passed. -1 asflicense 0m 37s The patch generated 1 ASF License warnings. 98m 30s Reason Tests FindBugs module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager   Possible null pointer dereference of priorityRequestedByApp in org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.checkAndGetApplicationPriority(Priority, UserGroupInformation, String, ApplicationId) Dereferenced at CapacityScheduler.java:priorityRequestedByApp in org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.checkAndGetApplicationPriority(Priority, UserGroupInformation, String, ApplicationId) Dereferenced at CapacityScheduler.java: [line 2214]   org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.PriorityACLConfiguration.createACLStringForPriority(List) invokes inefficient new String() constructor At PriorityACLConfiguration.java:constructor At PriorityACLConfiguration.java: [line 159]   org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.PriorityACLGroup defines compareTo(PriorityACLGroup) and uses Object.equals() At PriorityACLGroup.java:Object.equals() At PriorityACLGroup.java: [line 43] Failed junit tests hadoop.yarn.server.resourcemanager.TestRMProxyUsersConf   hadoop.yarn.server.resourcemanager.webapp.TestNodesPage   hadoop.yarn.server.resourcemanager.scheduler.capacity.TestLeafQueue   hadoop.yarn.server.resourcemanager.scheduler.capacity.TestQueueState   hadoop.yarn.server.resourcemanager.monitor.TestSchedulingMonitor   hadoop.yarn.server.resourcemanager.webapp.TestRMWebApp   hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerResizing   hadoop.yarn.server.resourcemanager.webapp.TestAppPage   hadoop.yarn.server.resourcemanager.scheduler.capacity.TestApplicationPriority   hadoop.yarn.server.resourcemanager.scheduler.capacity.TestIncreaseAllocationExpirer   hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerAllocation   hadoop.yarn.server.resourcemanager.scheduler.capacity.TestReservations   hadoop.yarn.server.resourcemanager.TestClientRMService   hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacitySchedulerLazyPreemption   hadoop.yarn.server.resourcemanager.TestAMAuthorization   hadoop.yarn.server.resourcemanager.TestSubmitApplicationWithRMHA   hadoop.yarn.server.resourcemanager.scheduler.policy.TestFairOrderingPolicy   hadoop.yarn.server.resourcemanager.TestSignalContainer   hadoop.yarn.server.resourcemanager.TestReservationSystemWithRMHA   hadoop.yarn.server.resourcemanager.TestLeaderElectorService   hadoop.yarn.server.resourcemanager.rmapp.TestNodesListManager   hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacitySchedulerNodeLabelUpdate   hadoop.yarn.server.resourcemanager.TestRMHA   hadoop.yarn.server.resourcemanager.nodelabels.TestRMNodeLabelsManager   hadoop.yarn.server.resourcemanager.TestRMHAForNodeLabels   hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesCapacitySched   hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesAppsModification   hadoop.yarn.server.resourcemanager.scheduler.TestSchedulerHealth   hadoop.yarn.server.resourcemanager.applicationsmanager.TestAMRMRPCNodeUpdates   hadoop.yarn.server.resourcemanager.nodelabels.TestRMDelegatedNodeLabelsUpdater   hadoop.yarn.server.resourcemanager.TestRMHAMetrics   hadoop.yarn.server.resourcemanager.TestRMRestart   hadoop.yarn.server.resourcemanager.TestRMEmbeddedElector   hadoop.yarn.server.resourcemanager.TestApplicationCleanup   hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer   hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesSchedulerActivities   hadoop.yarn.server.resourcemanager.TestContainerResourceUsage   hadoop.yarn.server.resourcemanager.rmcontainer.TestRMContainerImpl   hadoop.yarn.server.resourcemanager.applicationsmanager.TestAMRMRPCResponseId   hadoop.yarn.server.resourcemanager.recovery.TestZKRMStateStore   hadoop.yarn.server.resourcemanager.scheduler.capacity.TestWorkPreservingRMRestartForNodeLabel   hadoop.yarn.server.resourcemanager.security.TestClientToAMTokens   hadoop.yarn.server.resourcemanager.TestNodeBlacklistingOnAMFailures   hadoop.yarn.server.resourcemanager.scheduler.capacity.TestQueueMappings   hadoop.yarn.server.resourcemanager.reservation.TestReservationSystem   hadoop.yarn.server.resourcemanager.TestAppManager   hadoop.yarn.server.resourcemanager.scheduler.capacity.TestQueueParsing   hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesNodeLabels   hadoop.yarn.server.resourcemanager.ahs.TestRMApplicationHistoryWriter   hadoop.yarn.server.resourcemanager.resourcetracker.TestNMReconnect   hadoop.yarn.server.resourcemanager.TestApplicationACLs   hadoop.yarn.server.resourcemanager.TestResourceTrackerService   hadoop.yarn.server.resourcemanager.TestKillApplicationWithRMHA   hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesNodes   hadoop.yarn.server.resourcemanager.scheduler.capacity.TestApplicationLimits   hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacityScheduler   hadoop.yarn.server.resourcemanager.TestRM   hadoop.yarn.server.resourcemanager.webapp.TestRMWebServiceAppsNodelabel   hadoop.yarn.server.resourcemanager.scheduler.TestAbstractYarnScheduler   hadoop.yarn.server.resourcemanager.scheduler.capacity.TestApplicationLimitsByPartition   hadoop.yarn.server.resourcemanager.rmapp.attempt.TestAMLivelinessMonitor   hadoop.yarn.server.resourcemanager.TestRMStoreCommands   hadoop.yarn.server.resourcemanager.security.TestRMDelegationTokens   hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacitySchedulerDynamicBehavior   hadoop.yarn.server.resourcemanager.TestApplicationMasterService   hadoop.yarn.server.resourcemanager.webapp.TestRedirectionErrorPage   hadoop.yarn.server.resourcemanager.reservation.TestCapacitySchedulerPlanFollower   hadoop.yarn.server.resourcemanager.TestDecommissioningNodesWatcher   hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesReservation   hadoop.yarn.server.resourcemanager.security.TestAMRMTokens   hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacitySchedulerSurgicalPreemption   hadoop.yarn.server.resourcemanager.TestOpportunisticContainerAllocatorAMService   hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacitySchedulerAsyncScheduling   hadoop.yarn.server.resourcemanager.rmapp.TestApplicationLifetimeMonitor   hadoop.yarn.server.resourcemanager.scheduler.TestSchedulingWithAllocationRequestId   hadoop.yarn.server.resourcemanager.applicationsmanager.TestAMRestart   hadoop.yarn.server.resourcemanager.TestApplicationMasterLauncher   hadoop.yarn.server.resourcemanager.scheduler.capacity.TestApplicationPriorityACLs   hadoop.yarn.server.resourcemanager.monitor.capacity.TestProportionalCapacityPreemptionPolicy   hadoop.yarn.server.resourcemanager.TestWorkPreservingRMRestart   hadoop.yarn.server.resourcemanager.TestResourceManager   hadoop.yarn.server.resourcemanager.scheduler.TestSchedulerUtils   hadoop.yarn.server.resourcemanager.scheduler.capacity.TestNodeLabelContainerAllocation   hadoop.yarn.server.resourcemanager.TestRMAdminService   hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacitySchedulerQueueACLs   hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesForCSWithPartitions Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue YARN-3955 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12844511/YARN-3955.0004.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux 1d31859b6e0d 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 22befbd Default Java 1.8.0_111 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/14449/artifact/patchprocess/diff-checkstyle-root.txt whitespace https://builds.apache.org/job/PreCommit-YARN-Build/14449/artifact/patchprocess/whitespace-eol.txt findbugs https://builds.apache.org/job/PreCommit-YARN-Build/14449/artifact/patchprocess/new-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.html javadoc https://builds.apache.org/job/PreCommit-YARN-Build/14449/artifact/patchprocess/diff-javadoc-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/14449/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/14449/testReport/ asflicense https://builds.apache.org/job/PreCommit-YARN-Build/14449/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager hadoop-tools/hadoop-sls U: . Console output https://builds.apache.org/job/PreCommit-YARN-Build/14449/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        sunilg Sunil G added a comment -

        Thanks Wangda Tan.

        Fixed almost all comments. Some comments were on common code, which also is taken care. However I think comment in RMAppManager may not needed to be fixed.

        IN this patch, i added more test case for config and -ve cases. Also essentially used checkAndGetApplicationPriority to do acl check as well. This will also handle default priority for any given user. Tested in a real cluster as well.

        thoughts?

        Show
        sunilg Sunil G added a comment - Thanks Wangda Tan . Fixed almost all comments. Some comments were on common code, which also is taken care. However I think comment in RMAppManager may not needed to be fixed. IN this patch, i added more test case for config and -ve cases. Also essentially used checkAndGetApplicationPriority to do acl check as well. This will also handle default priority for any given user. Tested in a real cluster as well. thoughts?
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 14s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 7 new or modified test files.
        0 mvndep 1m 45s Maven dependency ordering for branch
        +1 mvninstall 14m 40s trunk passed
        +1 compile 10m 35s trunk passed
        +1 checkstyle 1m 39s trunk passed
        +1 mvnsite 1m 43s trunk passed
        +1 mvneclipse 1m 2s trunk passed
        +1 findbugs 2m 56s trunk passed
        +1 javadoc 1m 24s trunk passed
        0 mvndep 0m 17s Maven dependency ordering for patch
        +1 mvninstall 1m 30s the patch passed
        +1 compile 10m 20s the patch passed
        +1 javac 10m 20s the patch passed
        -0 checkstyle 1m 46s root: The patch generated 24 new + 483 unchanged - 2 fixed = 507 total (was 485)
        +1 mvnsite 1m 54s the patch passed
        +1 mvneclipse 1m 12s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 xml 0m 1s The patch has no ill-formed XML file.
        -1 findbugs 1m 24s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
        +1 javadoc 1m 38s the patch passed
        +1 unit 2m 46s hadoop-yarn-common in the patch passed.
        -1 unit 40m 6s hadoop-yarn-server-resourcemanager in the patch failed.
        +1 unit 1m 7s hadoop-sls in the patch passed.
        +1 asflicense 0m 39s The patch does not generate ASF License warnings.
        127m 11s



        Reason Tests
        FindBugs module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
          org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.PriorityACLGroup defines equals and uses Object.hashCode() At PriorityACLGroup.java:Object.hashCode() At PriorityACLGroup.java:[lines 52-63]
        Failed junit tests hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesAppsModification
          hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacitySchedulerQueueACLs
          hadoop.yarn.server.resourcemanager.TestRMHA
          hadoop.yarn.server.resourcemanager.TestRMRestart
          hadoop.yarn.server.resourcemanager.TestWorkPreservingRMRestart
          hadoop.yarn.server.resourcemanager.TestApplicationACLs



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:a9ad5d6
        JIRA Issue YARN-3955
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12844668/YARN-3955.0005.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
        uname Linux 668933af45a8 3.13.0-105-generic #152-Ubuntu SMP Fri Dec 2 15:37:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / ea54752
        Default Java 1.8.0_111
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/14464/artifact/patchprocess/diff-checkstyle-root.txt
        findbugs https://builds.apache.org/job/PreCommit-YARN-Build/14464/artifact/patchprocess/new-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.html
        unit https://builds.apache.org/job/PreCommit-YARN-Build/14464/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/14464/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager hadoop-tools/hadoop-sls U: .
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/14464/console
        Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 14s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 7 new or modified test files. 0 mvndep 1m 45s Maven dependency ordering for branch +1 mvninstall 14m 40s trunk passed +1 compile 10m 35s trunk passed +1 checkstyle 1m 39s trunk passed +1 mvnsite 1m 43s trunk passed +1 mvneclipse 1m 2s trunk passed +1 findbugs 2m 56s trunk passed +1 javadoc 1m 24s trunk passed 0 mvndep 0m 17s Maven dependency ordering for patch +1 mvninstall 1m 30s the patch passed +1 compile 10m 20s the patch passed +1 javac 10m 20s the patch passed -0 checkstyle 1m 46s root: The patch generated 24 new + 483 unchanged - 2 fixed = 507 total (was 485) +1 mvnsite 1m 54s the patch passed +1 mvneclipse 1m 12s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. -1 findbugs 1m 24s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) +1 javadoc 1m 38s the patch passed +1 unit 2m 46s hadoop-yarn-common in the patch passed. -1 unit 40m 6s hadoop-yarn-server-resourcemanager in the patch failed. +1 unit 1m 7s hadoop-sls in the patch passed. +1 asflicense 0m 39s The patch does not generate ASF License warnings. 127m 11s Reason Tests FindBugs module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager   org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.PriorityACLGroup defines equals and uses Object.hashCode() At PriorityACLGroup.java:Object.hashCode() At PriorityACLGroup.java: [lines 52-63] Failed junit tests hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesAppsModification   hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacitySchedulerQueueACLs   hadoop.yarn.server.resourcemanager.TestRMHA   hadoop.yarn.server.resourcemanager.TestRMRestart   hadoop.yarn.server.resourcemanager.TestWorkPreservingRMRestart   hadoop.yarn.server.resourcemanager.TestApplicationACLs Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue YARN-3955 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12844668/YARN-3955.0005.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux 668933af45a8 3.13.0-105-generic #152-Ubuntu SMP Fri Dec 2 15:37:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / ea54752 Default Java 1.8.0_111 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/14464/artifact/patchprocess/diff-checkstyle-root.txt findbugs https://builds.apache.org/job/PreCommit-YARN-Build/14464/artifact/patchprocess/new-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.html unit https://builds.apache.org/job/PreCommit-YARN-Build/14464/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/14464/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager hadoop-tools/hadoop-sls U: . Console output https://builds.apache.org/job/PreCommit-YARN-Build/14464/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        sunilg Sunil G added a comment -

        Fixing test failure and findbugs warning.

        Show
        sunilg Sunil G added a comment - Fixing test failure and findbugs warning.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 10s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 7 new or modified test files.
        0 mvndep 1m 52s Maven dependency ordering for branch
        +1 mvninstall 12m 40s trunk passed
        +1 compile 9m 40s trunk passed
        +1 checkstyle 1m 39s trunk passed
        +1 mvnsite 8m 13s trunk passed
        +1 mvneclipse 1m 14s trunk passed
        0 findbugs 0m 0s Skipped patched modules with no Java source: .
        +1 findbugs 2m 32s trunk passed
        -1 javadoc 4m 13s root in trunk failed.
        0 mvndep 0m 16s Maven dependency ordering for patch
        +1 mvninstall 13m 44s the patch passed
        +1 compile 9m 19s the patch passed
        +1 javac 9m 19s the patch passed
        -0 checkstyle 1m 42s root: The patch generated 24 new + 484 unchanged - 2 fixed = 508 total (was 486)
        +1 mvnsite 8m 12s the patch passed
        +1 mvneclipse 1m 10s the patch passed
        -1 whitespace 0m 0s The patch has 63 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
        +1 xml 0m 1s The patch has no ill-formed XML file.
        0 findbugs 0m 0s Skipped patched modules with no Java source: .
        +1 findbugs 3m 2s the patch passed
        -1 javadoc 4m 19s root in the patch failed.
        -1 unit 13m 43s root in the patch failed.
        -1 asflicense 0m 37s The patch generated 2 ASF License warnings.
        121m 24s



        Reason Tests
        Failed junit tests hadoop.conf.TestCommonConfigurationFields



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:a9ad5d6
        JIRA Issue YARN-3955
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12844689/YARN-3955.0006.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
        uname Linux 8f93b87c022c 3.13.0-103-generic #150-Ubuntu SMP Thu Nov 24 10:34:17 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / ea54752
        Default Java 1.8.0_111
        findbugs v3.0.0
        javadoc https://builds.apache.org/job/PreCommit-YARN-Build/14466/artifact/patchprocess/branch-javadoc-root.txt
        checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/14466/artifact/patchprocess/diff-checkstyle-root.txt
        whitespace https://builds.apache.org/job/PreCommit-YARN-Build/14466/artifact/patchprocess/whitespace-eol.txt
        javadoc https://builds.apache.org/job/PreCommit-YARN-Build/14466/artifact/patchprocess/patch-javadoc-root.txt
        unit https://builds.apache.org/job/PreCommit-YARN-Build/14466/artifact/patchprocess/patch-unit-root.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/14466/testReport/
        asflicense https://builds.apache.org/job/PreCommit-YARN-Build/14466/artifact/patchprocess/patch-asflicense-problems.txt
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager hadoop-tools/hadoop-sls . U: .
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/14466/console
        Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 10s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 7 new or modified test files. 0 mvndep 1m 52s Maven dependency ordering for branch +1 mvninstall 12m 40s trunk passed +1 compile 9m 40s trunk passed +1 checkstyle 1m 39s trunk passed +1 mvnsite 8m 13s trunk passed +1 mvneclipse 1m 14s trunk passed 0 findbugs 0m 0s Skipped patched modules with no Java source: . +1 findbugs 2m 32s trunk passed -1 javadoc 4m 13s root in trunk failed. 0 mvndep 0m 16s Maven dependency ordering for patch +1 mvninstall 13m 44s the patch passed +1 compile 9m 19s the patch passed +1 javac 9m 19s the patch passed -0 checkstyle 1m 42s root: The patch generated 24 new + 484 unchanged - 2 fixed = 508 total (was 486) +1 mvnsite 8m 12s the patch passed +1 mvneclipse 1m 10s the patch passed -1 whitespace 0m 0s The patch has 63 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply +1 xml 0m 1s The patch has no ill-formed XML file. 0 findbugs 0m 0s Skipped patched modules with no Java source: . +1 findbugs 3m 2s the patch passed -1 javadoc 4m 19s root in the patch failed. -1 unit 13m 43s root in the patch failed. -1 asflicense 0m 37s The patch generated 2 ASF License warnings. 121m 24s Reason Tests Failed junit tests hadoop.conf.TestCommonConfigurationFields Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue YARN-3955 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12844689/YARN-3955.0006.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux 8f93b87c022c 3.13.0-103-generic #150-Ubuntu SMP Thu Nov 24 10:34:17 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / ea54752 Default Java 1.8.0_111 findbugs v3.0.0 javadoc https://builds.apache.org/job/PreCommit-YARN-Build/14466/artifact/patchprocess/branch-javadoc-root.txt checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/14466/artifact/patchprocess/diff-checkstyle-root.txt whitespace https://builds.apache.org/job/PreCommit-YARN-Build/14466/artifact/patchprocess/whitespace-eol.txt javadoc https://builds.apache.org/job/PreCommit-YARN-Build/14466/artifact/patchprocess/patch-javadoc-root.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/14466/artifact/patchprocess/patch-unit-root.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/14466/testReport/ asflicense https://builds.apache.org/job/PreCommit-YARN-Build/14466/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager hadoop-tools/hadoop-sls . U: . Console output https://builds.apache.org/job/PreCommit-YARN-Build/14466/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        sunilg Sunil G added a comment -

        java doc error is not related. Its a known failure from hadoop-azure project in trunk.
        ASF warnings are shown in hadoop-tools, and am not so sure why its showing this jenkins run.

        Test case failure s related due to missing config items. I will upload a patch for that.

        Show
        sunilg Sunil G added a comment - java doc error is not related. Its a known failure from hadoop-azure project in trunk. ASF warnings are shown in hadoop-tools, and am not so sure why its showing this jenkins run. Test case failure s related due to missing config items. I will upload a patch for that.
        Hide
        sunilg Sunil G added a comment -

        rebasing patch to latest trunk. Also fixed checkstyle/javadoc errors.

        This patch has almost all major rework suggested by Wangda Tan in below comment

        Show
        sunilg Sunil G added a comment - rebasing patch to latest trunk. Also fixed checkstyle/javadoc errors. This patch has almost all major rework suggested by Wangda Tan in below comment
        Hide
        leftnoteasy Wangda Tan added a comment -

        Thanks Sunil G, some additional comments:

        1) Common logic of checkAccess / getDefaultPriority can be merged further: both can get approvedPriority first.
        2) As I commented above, do changes of capacity-scheduler.xml related to the patch? I cannot find which module uses acl_access_priority in configuration. If not, could you add correct default value?

        3) CapacityScheduler:

        • updateApplicationPriority should hold writeLock?
        • similiarily, checkAndGetApplicationPriority should hold readlock?
        • checkAndGetApplicationPriority: when an app's priority set to negative, I think we should use 0 instead of max. Thoughts?

        4) AppPriorityACLsMgr:

        • addPrioirityACLs, should we do "replace" instead of "add" to acl groups? If it is not intentional, could you add a test to make sure update of acls works? (like change from [1,2,3] to [1,3,4])
        • getPriorityPerUserACL -> getMappedPriorityAclForUGI.

        5) As I mentioned before, remove readlock of LQ#getPriorityAcls, final should be enough.

        6) YarnScheduler: why the new added method has SettableFuture in parameters? It doesn't look very clean ...

        Show
        leftnoteasy Wangda Tan added a comment - Thanks Sunil G , some additional comments: 1) Common logic of checkAccess / getDefaultPriority can be merged further: both can get approvedPriority first. 2) As I commented above, do changes of capacity-scheduler.xml related to the patch? I cannot find which module uses acl_access_priority in configuration. If not, could you add correct default value? 3) CapacityScheduler: updateApplicationPriority should hold writeLock? similiarily, checkAndGetApplicationPriority should hold readlock? checkAndGetApplicationPriority: when an app's priority set to negative, I think we should use 0 instead of max. Thoughts? 4) AppPriorityACLsMgr: addPrioirityACLs, should we do "replace" instead of "add" to acl groups? If it is not intentional, could you add a test to make sure update of acls works? (like change from [1,2,3] to [1,3,4]) getPriorityPerUserACL -> getMappedPriorityAclForUGI. 5) As I mentioned before, remove readlock of LQ#getPriorityAcls, final should be enough. 6) YarnScheduler: why the new added method has SettableFuture in parameters? It doesn't look very clean ...
        Hide
        sunilg Sunil G added a comment -

        Thanks Wangda Tan for the detailed comments. I have some more doubts here.

        1) Common logic of checkAccess / getDefaultPriority can be merged further: both can get approvedPriority first.
        >> priority acls are stored in ascending order. So for checkAccess, we need to see whether ACL match or not and then submitted priority is lesser than configure priority. However in case there are no configurations for priority ACLs or ACLs are disabled, we still need to say access check is passed. Now for default priority, we will loop through all configured priority acls and if any ACLs are matching, we try to get max priority all group from which default could be taken.
        Do you mean that below methods also can be made common.

            if (!isACLsEnable) {
              return true;
            }
            List<PriorityACL> acls = allAcls.get(queueName);
            if (acls == null || acls.isEmpty()) {
              return true;
            }
        

        There is one issue here. If approvedPriorityACL comes are null, for checkAccess it means false. If we put above code also inside getPriorityPerUserACL, then we expect to return true if that returns null. Since there is conflict of interest, i pulled it out. May be you could explain a bit further if I missed some.
        
2) As I commented above, do changes of capacity-scheduler.xml related to the patch? I cannot find which module uses acl_access_priority in configuration. If not, could you add correct default value?
        >> in CapacitySchedulerConfiguration.getAclKey(AccessType acl), we try to get priority ACL config from acl_access_priority . And that is used to parse and then populate internal structures. By default I kept it *, but I have given an example as below.

        The ACL of who can submit applications with configured priority.
        For e.g, [user={name} group={name} max_priority={priority} default_priority={priority}]
        

        3) CapacityScheduler:

        • updateApplicationPriority should hold writeLock?
        • similiarily, checkAndGetApplicationPriority should hold readlock?
          >> Done. Updated in patch
        • checkAndGetApplicationPriority: when an app's priority set to negative, I think we should use 0 instead of max. Thoughts?
                if (appPriority.compareTo(getMaxClusterLevelAppPriority()) < 0) {
                  appPriority = Priority
                      .newInstance(getMaxClusterLevelAppPriority().getPriority());
                }
          

          This code will reset to cluster-max priority only if submitted priority is more than cluster max. Since I used compareTo, it not looks very readable.
          Now to your point, we never worry much of -ve priority as such since we use priority as integer. Do you feel we need to make 0 as lowest priority ?

        4) AppPriorityACLsMgr:

        • addPrioirityACLs, should we do "replace" instead of "add" to acl groups? If it is not intentional, could you add a test to make sure update of acls works? (like change from [1,2,3] to [1,3,4])
          >> Could I add a clear model. It may be more easy. Thoughts? Updated patch as per this.
        • getPriorityPerUserACL -> getMappedPriorityAclForUGI.
          >> Done.
          5) As I mentioned before, remove readlock of LQ#getPriorityAcls, final should be enough.
          >> One doubt here. Since priorityAcls could also be updated in reinitialize, we can’t make it as final rt. refreshQueue’s call flow for eg.
          6) YarnScheduler: why the new added method has SettableFuture in parameters? It doesn't look very clean ...
          >> I agree with you. But we are doing statestore update within scheduler. Hence we need to pass future to see exception is thrown immediately. Hence we had to add this while doing move to queue.
        Show
        sunilg Sunil G added a comment - Thanks Wangda Tan for the detailed comments. I have some more doubts here. 1) Common logic of checkAccess / getDefaultPriority can be merged further: both can get approvedPriority first. >> priority acls are stored in ascending order. So for checkAccess, we need to see whether ACL match or not and then submitted priority is lesser than configure priority. However in case there are no configurations for priority ACLs or ACLs are disabled, we still need to say access check is passed. Now for default priority, we will loop through all configured priority acls and if any ACLs are matching, we try to get max priority all group from which default could be taken. Do you mean that below methods also can be made common. if (!isACLsEnable) { return true; } List<PriorityACL> acls = allAcls.get(queueName); if (acls == null || acls.isEmpty()) { return true; } There is one issue here. If approvedPriorityACL comes are null, for checkAccess it means false. If we put above code also inside getPriorityPerUserACL , then we expect to return true if that returns null. Since there is conflict of interest, i pulled it out. May be you could explain a bit further if I missed some. 
2) As I commented above, do changes of capacity-scheduler.xml related to the patch? I cannot find which module uses acl_access_priority in configuration. If not, could you add correct default value? >> in CapacitySchedulerConfiguration.getAclKey(AccessType acl) , we try to get priority ACL config from acl_access_priority . And that is used to parse and then populate internal structures. By default I kept it *, but I have given an example as below. The ACL of who can submit applications with configured priority. For e.g, [user={name} group={name} max_priority={priority} default_priority={priority}] 3) CapacityScheduler: updateApplicationPriority should hold writeLock? similiarily, checkAndGetApplicationPriority should hold readlock? >> Done. Updated in patch checkAndGetApplicationPriority: when an app's priority set to negative, I think we should use 0 instead of max. Thoughts? if (appPriority.compareTo(getMaxClusterLevelAppPriority()) < 0) { appPriority = Priority .newInstance(getMaxClusterLevelAppPriority().getPriority()); } This code will reset to cluster-max priority only if submitted priority is more than cluster max. Since I used compareTo , it not looks very readable. Now to your point, we never worry much of -ve priority as such since we use priority as integer. Do you feel we need to make 0 as lowest priority ? 4) AppPriorityACLsMgr: addPrioirityACLs, should we do "replace" instead of "add" to acl groups? If it is not intentional, could you add a test to make sure update of acls works? (like change from [1,2,3] to [1,3,4] ) >> Could I add a clear model. It may be more easy. Thoughts? Updated patch as per this. getPriorityPerUserACL -> getMappedPriorityAclForUGI. >> Done. 5) As I mentioned before, remove readlock of LQ#getPriorityAcls, final should be enough. >> One doubt here. Since priorityAcls could also be updated in reinitialize, we can’t make it as final rt. refreshQueue’s call flow for eg. 6) YarnScheduler: why the new added method has SettableFuture in parameters? It doesn't look very clean ... >> I agree with you. But we are doing statestore update within scheduler. Hence we need to pass future to see exception is thrown immediately. Hence we had to add this while doing move to queue.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 17s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 7 new or modified test files.
        0 mvndep 1m 57s Maven dependency ordering for branch
        +1 mvninstall 13m 37s trunk passed
        +1 compile 10m 59s trunk passed
        +1 checkstyle 1m 40s trunk passed
        +1 mvnsite 1m 45s trunk passed
        +1 mvneclipse 1m 4s trunk passed
        +1 findbugs 3m 0s trunk passed
        +1 javadoc 1m 26s trunk passed
        0 mvndep 0m 17s Maven dependency ordering for patch
        +1 mvninstall 1m 27s the patch passed
        +1 compile 9m 37s the patch passed
        +1 javac 9m 37s the patch passed
        -0 checkstyle 1m 42s root: The patch generated 13 new + 484 unchanged - 2 fixed = 497 total (was 486)
        +1 mvnsite 1m 48s the patch passed
        +1 mvneclipse 1m 11s the patch passed
        -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
        +1 xml 0m 2s The patch has no ill-formed XML file.
        +1 findbugs 3m 15s the patch passed
        -1 javadoc 0m 31s hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager generated 1 new + 913 unchanged - 0 fixed = 914 total (was 913)
        +1 unit 2m 30s hadoop-yarn-common in the patch passed.
        +1 unit 39m 37s hadoop-yarn-server-resourcemanager in the patch passed.
        +1 unit 1m 5s hadoop-sls in the patch passed.
        +1 asflicense 0m 38s The patch does not generate ASF License warnings.
        124m 58s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:a9ad5d6
        JIRA Issue YARN-3955
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12845827/YARN-3955.0008.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
        uname Linux 25481fef2165 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / a605ff3
        Default Java 1.8.0_111
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/14566/artifact/patchprocess/diff-checkstyle-root.txt
        whitespace https://builds.apache.org/job/PreCommit-YARN-Build/14566/artifact/patchprocess/whitespace-eol.txt
        javadoc https://builds.apache.org/job/PreCommit-YARN-Build/14566/artifact/patchprocess/diff-javadoc-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/14566/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager hadoop-tools/hadoop-sls U: .
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/14566/console
        Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 17s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 7 new or modified test files. 0 mvndep 1m 57s Maven dependency ordering for branch +1 mvninstall 13m 37s trunk passed +1 compile 10m 59s trunk passed +1 checkstyle 1m 40s trunk passed +1 mvnsite 1m 45s trunk passed +1 mvneclipse 1m 4s trunk passed +1 findbugs 3m 0s trunk passed +1 javadoc 1m 26s trunk passed 0 mvndep 0m 17s Maven dependency ordering for patch +1 mvninstall 1m 27s the patch passed +1 compile 9m 37s the patch passed +1 javac 9m 37s the patch passed -0 checkstyle 1m 42s root: The patch generated 13 new + 484 unchanged - 2 fixed = 497 total (was 486) +1 mvnsite 1m 48s the patch passed +1 mvneclipse 1m 11s the patch passed -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply +1 xml 0m 2s The patch has no ill-formed XML file. +1 findbugs 3m 15s the patch passed -1 javadoc 0m 31s hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager generated 1 new + 913 unchanged - 0 fixed = 914 total (was 913) +1 unit 2m 30s hadoop-yarn-common in the patch passed. +1 unit 39m 37s hadoop-yarn-server-resourcemanager in the patch passed. +1 unit 1m 5s hadoop-sls in the patch passed. +1 asflicense 0m 38s The patch does not generate ASF License warnings. 124m 58s Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue YARN-3955 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12845827/YARN-3955.0008.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux 25481fef2165 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / a605ff3 Default Java 1.8.0_111 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/14566/artifact/patchprocess/diff-checkstyle-root.txt whitespace https://builds.apache.org/job/PreCommit-YARN-Build/14566/artifact/patchprocess/whitespace-eol.txt javadoc https://builds.apache.org/job/PreCommit-YARN-Build/14566/artifact/patchprocess/diff-javadoc-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/14566/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager hadoop-tools/hadoop-sls U: . Console output https://builds.apache.org/job/PreCommit-YARN-Build/14566/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        leftnoteasy Wangda Tan added a comment - - edited

        1)

        There is one issue here. If approvedPriorityACL comes are null, for checkAccess it means false.

        Ok gotcha, my bad, we cannot merge the two.

        2)
        Got it, not related to your patch. The previous design of "acl-key" is bad, it will be hard to find which code path uses it...
        And in addition, I didn't see test case that parses raw priority acls (string) to List of PriorityACLGroup. Could you point me if there's any test cases exist?

        Few renaming suggestions:

        • PriorityACLConfiguration -> AppPriorityACLConfigurationParser (I was trying to find where's the parser code, and since we're adding queue priority YARN-5864, so it will be better to add an App- to distinguish that)
        • Similiarily, PriorityAclConfig -> AppPriorityACLOwnerType (or any better name?)
        • PriorityACLGroup -> AppPriorityACLGroup
        • Do you think is it better to rename acl_access_priority to acl_app_max_priority?

        3)

        This code will reset to cluster-max priority only if submitted priority is more than cluster max. Since I used compareTo, it not looks very readable.

        Yeah, since we're using Priority in different ways, sometimes lower is more important and sometimes higher is more important. Could you use ">" to do the comparision?

        checkAndGetApplicationPriority: when an app's priority set to negative, I think we should use 0 instead of max. Thoughts?

        Negative value looks fine, since app can set lower priority if needed.

        4)

        Could I add a clear model. It may be more easy. Thoughts? Updated patch as per this.

        Not quite sure what did you mean. From my understanding, existing logic read acls from configs while refreshQueues, and what we need to do is to replace all ACLs instead of append to previous acl list, correct?

        One doubt here. Since priorityAcls could also be updated in reinitialize, we can’t make it as final rt. refreshQueue’s call flow for eg.

        Since the returned list can be modified by another thread, so the readLock cannot provide enough protection. The better way might be readLock + copyList.

        But we are doing statestore update within scheduler. Hence we need to pass future to see exception is thrown immediately. Hence we had to add this while doing move to queue.

        Make sense.

        Show
        leftnoteasy Wangda Tan added a comment - - edited 1) There is one issue here. If approvedPriorityACL comes are null, for checkAccess it means false. Ok gotcha, my bad, we cannot merge the two. 2) Got it, not related to your patch. The previous design of "acl-key" is bad, it will be hard to find which code path uses it... And in addition, I didn't see test case that parses raw priority acls (string) to List of PriorityACLGroup. Could you point me if there's any test cases exist? Few renaming suggestions: PriorityACLConfiguration -> AppPriorityACLConfigurationParser (I was trying to find where's the parser code, and since we're adding queue priority YARN-5864 , so it will be better to add an App- to distinguish that) Similiarily, PriorityAclConfig -> AppPriorityACLOwnerType (or any better name?) PriorityACLGroup -> AppPriorityACLGroup Do you think is it better to rename acl_access_priority to acl_app_max_priority? 3) This code will reset to cluster-max priority only if submitted priority is more than cluster max. Since I used compareTo, it not looks very readable. Yeah, since we're using Priority in different ways, sometimes lower is more important and sometimes higher is more important. Could you use ">" to do the comparision? checkAndGetApplicationPriority: when an app's priority set to negative, I think we should use 0 instead of max. Thoughts? Negative value looks fine, since app can set lower priority if needed. 4) Could I add a clear model. It may be more easy. Thoughts? Updated patch as per this. Not quite sure what did you mean. From my understanding, existing logic read acls from configs while refreshQueues, and what we need to do is to replace all ACLs instead of append to previous acl list, correct? One doubt here. Since priorityAcls could also be updated in reinitialize, we can’t make it as final rt. refreshQueue’s call flow for eg. Since the returned list can be modified by another thread, so the readLock cannot provide enough protection. The better way might be readLock + copyList. But we are doing statestore update within scheduler. Hence we need to pass future to see exception is thrown immediately. Hence we had to add this while doing move to queue. Make sense.
        Hide
        sunilg Sunil G added a comment -

        Thanks Wangda Tan

        Mostly addressed all comments.

        And in addition, I didn't see test case that parses raw priority acls (string) to List of PriorityACLGroup. Could you point me if there's any test cases exist?

        TestApplicationPriorityACLConfiguration handles this case.

        Not quite sure what did you mean. From my understanding, existing logic read acls from configs while refreshQueues, and what we need to do is to replace all ACLs instead of append to previous acl list, correct?

        I was planning like below. clear the any queue acl before updating new.

                // Clear Priority ACLs first since reinitialize also call same.
                appPriorityACLManager.clearPriorityACLs(lQueue.getQueueName());
                appPriorityACLManager.addPrioirityACLs(lQueue.getPriorityACLs(),lQueue.getQueueName());
        
        Show
        sunilg Sunil G added a comment - Thanks Wangda Tan Mostly addressed all comments. And in addition, I didn't see test case that parses raw priority acls (string) to List of PriorityACLGroup. Could you point me if there's any test cases exist? TestApplicationPriorityACLConfiguration handles this case. Not quite sure what did you mean. From my understanding, existing logic read acls from configs while refreshQueues, and what we need to do is to replace all ACLs instead of append to previous acl list, correct? I was planning like below. clear the any queue acl before updating new. // Clear Priority ACLs first since reinitialize also call same. appPriorityACLManager.clearPriorityACLs(lQueue.getQueueName()); appPriorityACLManager.addPrioirityACLs(lQueue.getPriorityACLs(),lQueue.getQueueName());
        Hide
        leftnoteasy Wangda Tan added a comment -

        Looks good in general, and could you update capacity-scheduler.xml for the configuration name change?

        Beyond that, +1 to the final patch.

        Show
        leftnoteasy Wangda Tan added a comment - Looks good in general, and could you update capacity-scheduler.xml for the configuration name change? Beyond that, +1 to the final patch.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 15s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 7 new or modified test files.
        0 mvndep 0m 17s Maven dependency ordering for branch
        +1 mvninstall 14m 14s trunk passed
        +1 compile 10m 25s trunk passed
        +1 checkstyle 1m 46s trunk passed
        +1 mvnsite 1m 51s trunk passed
        +1 mvneclipse 1m 7s trunk passed
        +1 findbugs 2m 47s trunk passed
        +1 javadoc 1m 26s trunk passed
        0 mvndep 0m 16s Maven dependency ordering for patch
        +1 mvninstall 1m 21s the patch passed
        +1 compile 9m 35s the patch passed
        +1 javac 9m 35s the patch passed
        -0 checkstyle 1m 40s root: The patch generated 15 new + 485 unchanged - 2 fixed = 500 total (was 487)
        +1 mvnsite 1m 49s the patch passed
        +1 mvneclipse 1m 10s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 xml 0m 2s The patch has no ill-formed XML file.
        +1 findbugs 3m 21s the patch passed
        +1 javadoc 1m 33s the patch passed
        +1 unit 2m 31s hadoop-yarn-common in the patch passed.
        -1 unit 38m 59s hadoop-yarn-server-resourcemanager in the patch failed.
        +1 unit 1m 4s hadoop-sls in the patch passed.
        +1 asflicense 0m 37s The patch does not generate ASF License warnings.
        122m 24s



        Reason Tests
        Failed junit tests hadoop.yarn.server.resourcemanager.TestRMRestart
          hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:a9ad5d6
        JIRA Issue YARN-3955
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12845938/YARN-3955.0009.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
        uname Linux 763d33bbb2bd 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 4a659ff
        Default Java 1.8.0_111
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/14583/artifact/patchprocess/diff-checkstyle-root.txt
        unit https://builds.apache.org/job/PreCommit-YARN-Build/14583/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/14583/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager hadoop-tools/hadoop-sls U: .
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/14583/console
        Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 15s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 7 new or modified test files. 0 mvndep 0m 17s Maven dependency ordering for branch +1 mvninstall 14m 14s trunk passed +1 compile 10m 25s trunk passed +1 checkstyle 1m 46s trunk passed +1 mvnsite 1m 51s trunk passed +1 mvneclipse 1m 7s trunk passed +1 findbugs 2m 47s trunk passed +1 javadoc 1m 26s trunk passed 0 mvndep 0m 16s Maven dependency ordering for patch +1 mvninstall 1m 21s the patch passed +1 compile 9m 35s the patch passed +1 javac 9m 35s the patch passed -0 checkstyle 1m 40s root: The patch generated 15 new + 485 unchanged - 2 fixed = 500 total (was 487) +1 mvnsite 1m 49s the patch passed +1 mvneclipse 1m 10s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 2s The patch has no ill-formed XML file. +1 findbugs 3m 21s the patch passed +1 javadoc 1m 33s the patch passed +1 unit 2m 31s hadoop-yarn-common in the patch passed. -1 unit 38m 59s hadoop-yarn-server-resourcemanager in the patch failed. +1 unit 1m 4s hadoop-sls in the patch passed. +1 asflicense 0m 37s The patch does not generate ASF License warnings. 122m 24s Reason Tests Failed junit tests hadoop.yarn.server.resourcemanager.TestRMRestart   hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue YARN-3955 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12845938/YARN-3955.0009.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux 763d33bbb2bd 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 4a659ff Default Java 1.8.0_111 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/14583/artifact/patchprocess/diff-checkstyle-root.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/14583/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/14583/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager hadoop-tools/hadoop-sls U: . Console output https://builds.apache.org/job/PreCommit-YARN-Build/14583/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        sunilg Sunil G added a comment -

        Thanks Wangda Tan for pointing out. Missed it.

        Attaching a new patch addressing the same.

        Show
        sunilg Sunil G added a comment - Thanks Wangda Tan for pointing out. Missed it. Attaching a new patch addressing the same.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 14s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 7 new or modified test files.
        0 mvndep 0m 14s Maven dependency ordering for branch
        +1 mvninstall 12m 33s trunk passed
        +1 compile 10m 36s trunk passed
        +1 checkstyle 1m 42s trunk passed
        +1 mvnsite 1m 48s trunk passed
        +1 mvneclipse 0m 59s trunk passed
        +1 findbugs 3m 8s trunk passed
        +1 javadoc 1m 25s trunk passed
        0 mvndep 0m 18s Maven dependency ordering for patch
        +1 mvninstall 1m 38s the patch passed
        +1 compile 9m 42s the patch passed
        +1 javac 9m 42s the patch passed
        -0 checkstyle 1m 43s root: The patch generated 15 new + 485 unchanged - 2 fixed = 500 total (was 487)
        +1 mvnsite 1m 50s the patch passed
        +1 mvneclipse 1m 7s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 xml 0m 1s The patch has no ill-formed XML file.
        +1 findbugs 3m 40s the patch passed
        +1 javadoc 1m 35s the patch passed
        +1 unit 2m 40s hadoop-yarn-common in the patch passed.
        -1 unit 39m 18s hadoop-yarn-server-resourcemanager in the patch failed.
        +1 unit 1m 6s hadoop-sls in the patch passed.
        +1 asflicense 0m 38s The patch does not generate ASF License warnings.
        121m 42s



        Reason Tests
        Failed junit tests hadoop.yarn.server.resourcemanager.TestRMRestart



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:a9ad5d6
        JIRA Issue YARN-3955
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12845961/YARN-3955.0010.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
        uname Linux db72ac333e8f 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 4a659ff
        Default Java 1.8.0_111
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/14584/artifact/patchprocess/diff-checkstyle-root.txt
        unit https://builds.apache.org/job/PreCommit-YARN-Build/14584/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/14584/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager hadoop-tools/hadoop-sls U: .
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/14584/console
        Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 14s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 7 new or modified test files. 0 mvndep 0m 14s Maven dependency ordering for branch +1 mvninstall 12m 33s trunk passed +1 compile 10m 36s trunk passed +1 checkstyle 1m 42s trunk passed +1 mvnsite 1m 48s trunk passed +1 mvneclipse 0m 59s trunk passed +1 findbugs 3m 8s trunk passed +1 javadoc 1m 25s trunk passed 0 mvndep 0m 18s Maven dependency ordering for patch +1 mvninstall 1m 38s the patch passed +1 compile 9m 42s the patch passed +1 javac 9m 42s the patch passed -0 checkstyle 1m 43s root: The patch generated 15 new + 485 unchanged - 2 fixed = 500 total (was 487) +1 mvnsite 1m 50s the patch passed +1 mvneclipse 1m 7s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. +1 findbugs 3m 40s the patch passed +1 javadoc 1m 35s the patch passed +1 unit 2m 40s hadoop-yarn-common in the patch passed. -1 unit 39m 18s hadoop-yarn-server-resourcemanager in the patch failed. +1 unit 1m 6s hadoop-sls in the patch passed. +1 asflicense 0m 38s The patch does not generate ASF License warnings. 121m 42s Reason Tests Failed junit tests hadoop.yarn.server.resourcemanager.TestRMRestart Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue YARN-3955 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12845961/YARN-3955.0010.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux db72ac333e8f 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 4a659ff Default Java 1.8.0_111 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/14584/artifact/patchprocess/diff-checkstyle-root.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/14584/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/14584/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager hadoop-tools/hadoop-sls U: . Console output https://builds.apache.org/job/PreCommit-YARN-Build/14584/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        sunilg Sunil G added a comment -

        Test case failures are unrelated.

        Show
        sunilg Sunil G added a comment - Test case failures are unrelated.
        Hide
        leftnoteasy Wangda Tan added a comment -

        Committed to trunk, branch-2, thanks Sunil G for working on the patch and thanks Jian He for review comments.

        Show
        leftnoteasy Wangda Tan added a comment - Committed to trunk, branch-2, thanks Sunil G for working on the patch and thanks Jian He for review comments.
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Jenkins build Hadoop-trunk-Commit #11090 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11090/)
        YARN-3955. Support for application priority ACLs in queues of (wangda: rev 287d3d6804a869723ae36605a3c2d2b3eae3941e)

        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ClientRMService.java
        • (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/security/AppPriorityACLsManager.java
        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestParentQueue.java
        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.java
        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/security/AccessType.java
        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestApplicationMasterService.java
        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/AbstractYarnScheduler.java
        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java
        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/ACLsTestBase.java
        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java
        • (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/AppPriorityACLGroup.java
        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerQueueManager.java
        • (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestApplicationPriorityACLs.java
        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAppManager.java
        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestApplicationPriority.java
        • (edit) hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/scheduler/ResourceSchedulerWrapper.java
        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/conf/capacity-scheduler.xml
        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestClientRMService.java
        • (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/AppPriorityACLConfigurationParser.java
        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/YarnScheduler.java
        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerContext.java
        • (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestApplicationPriorityACLConfiguration.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Jenkins build Hadoop-trunk-Commit #11090 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11090/ ) YARN-3955 . Support for application priority ACLs in queues of (wangda: rev 287d3d6804a869723ae36605a3c2d2b3eae3941e) (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ClientRMService.java (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/security/AppPriorityACLsManager.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestParentQueue.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/security/AccessType.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestApplicationMasterService.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/AbstractYarnScheduler.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/ACLsTestBase.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/AppPriorityACLGroup.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerQueueManager.java (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestApplicationPriorityACLs.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAppManager.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestApplicationPriority.java (edit) hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/scheduler/ResourceSchedulerWrapper.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/conf/capacity-scheduler.xml (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestClientRMService.java (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/AppPriorityACLConfigurationParser.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/YarnScheduler.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerContext.java (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestApplicationPriorityACLConfiguration.java
        Hide
        sunilg Sunil G added a comment -

        Thank you Wangda Tan for the review and commit. Thanks Jian He for additional reviews.

        Show
        sunilg Sunil G added a comment - Thank you Wangda Tan for the review and commit. Thanks Jian He for additional reviews.

          People

          • Assignee:
            sunilg Sunil G
            Reporter:
            sunilg Sunil G
          • Votes:
            0 Vote for this issue
            Watchers:
            12 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development