Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-1963 Support priorities across applications within the same queue
  3. YARN-5802

updateApplicationPriority api in scheduler should ensure to re-insert app to correct ordering policy

    Details

    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      LeafQueue#updateApplicationPriority

       getOrderingPolicy().removeSchedulableEntity(attempt);
      
            // Update new priority in SchedulerApplication
            attempt.setPriority(newAppPriority);
      
            getOrderingPolicy().addSchedulableEntity(attempt);
      

      We should add again to ordering policy only when attempt available in first case.Else during application attempt removal will try to iterate on killed application still available in pending Ordering policy.Which can cause RM to crash.

      1. YARN-5802.branch-2.8.0001.patch
        9 kB
        Bibin A Chundatt
      2. YARN-5802.0006.patch
        9 kB
        Bibin A Chundatt
      3. YARN-5802.0005.patch
        9 kB
        Bibin A Chundatt
      4. YARN-5802.0004.patch
        8 kB
        Bibin A Chundatt
      5. YARN-5802.0003.patch
        9 kB
        Bibin A Chundatt
      6. YARN-5802.0002.patch
        8 kB
        Bibin A Chundatt
      7. YARN-5802.0001.patch
        1 kB
        Bibin A Chundatt

        Activity

        Hide
        sunilg Sunil G added a comment - - edited

        Test case failures are known which is shown now for branch-2.8.

        Thanks Bibin A Chundatt for the contribution. Thanks Varun Saxena for the additional review. Committed to trunk, branch-2, and branch-2.8

        Show
        sunilg Sunil G added a comment - - edited Test case failures are known which is shown now for branch-2.8. Thanks Bibin A Chundatt for the contribution. Thanks Varun Saxena for the additional review. Committed to trunk, branch-2, and branch-2.8
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 19s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
        +1 mvninstall 6m 29s branch-2.8 passed
        +1 compile 0m 29s branch-2.8 passed with JDK v1.8.0_111
        +1 compile 0m 31s branch-2.8 passed with JDK v1.7.0_111
        +1 checkstyle 0m 19s branch-2.8 passed
        +1 mvnsite 0m 35s branch-2.8 passed
        +1 mvneclipse 0m 17s branch-2.8 passed
        +1 findbugs 1m 8s branch-2.8 passed
        +1 javadoc 0m 20s branch-2.8 passed with JDK v1.8.0_111
        +1 javadoc 0m 24s branch-2.8 passed with JDK v1.7.0_111
        +1 mvninstall 0m 29s the patch passed
        +1 compile 0m 25s the patch passed with JDK v1.8.0_111
        +1 javac 0m 25s the patch passed
        +1 compile 0m 29s the patch passed with JDK v1.7.0_111
        +1 javac 0m 29s the patch passed
        -0 checkstyle 0m 17s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 1 new + 109 unchanged - 1 fixed = 110 total (was 110)
        +1 mvnsite 0m 33s the patch passed
        +1 mvneclipse 0m 14s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 1m 17s the patch passed
        +1 javadoc 0m 17s the patch passed with JDK v1.8.0_111
        +1 javadoc 0m 21s the patch passed with JDK v1.7.0_111
        -1 unit 75m 54s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.7.0_111.
        +1 asflicense 0m 17s The patch does not generate ASF License warnings.
        167m 21s



        Reason Tests
        JDK v1.8.0_111 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens
          hadoop.yarn.server.resourcemanager.TestAMAuthorization
        JDK v1.7.0_111 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens
          hadoop.yarn.server.resourcemanager.TestAMAuthorization



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:5af2af1
        JIRA Issue YARN-5802
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12837345/YARN-5802.branch-2.8.0001.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux eeb9c50e969d 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision branch-2.8 / 5073289
        Default Java 1.7.0_111
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_111 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_111
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/13791/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
        unit https://builds.apache.org/job/PreCommit-YARN-Build/13791/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_111.txt
        JDK v1.7.0_111 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13791/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/13791/console
        Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 19s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. +1 mvninstall 6m 29s branch-2.8 passed +1 compile 0m 29s branch-2.8 passed with JDK v1.8.0_111 +1 compile 0m 31s branch-2.8 passed with JDK v1.7.0_111 +1 checkstyle 0m 19s branch-2.8 passed +1 mvnsite 0m 35s branch-2.8 passed +1 mvneclipse 0m 17s branch-2.8 passed +1 findbugs 1m 8s branch-2.8 passed +1 javadoc 0m 20s branch-2.8 passed with JDK v1.8.0_111 +1 javadoc 0m 24s branch-2.8 passed with JDK v1.7.0_111 +1 mvninstall 0m 29s the patch passed +1 compile 0m 25s the patch passed with JDK v1.8.0_111 +1 javac 0m 25s the patch passed +1 compile 0m 29s the patch passed with JDK v1.7.0_111 +1 javac 0m 29s the patch passed -0 checkstyle 0m 17s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 1 new + 109 unchanged - 1 fixed = 110 total (was 110) +1 mvnsite 0m 33s the patch passed +1 mvneclipse 0m 14s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 17s the patch passed +1 javadoc 0m 17s the patch passed with JDK v1.8.0_111 +1 javadoc 0m 21s the patch passed with JDK v1.7.0_111 -1 unit 75m 54s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.7.0_111. +1 asflicense 0m 17s The patch does not generate ASF License warnings. 167m 21s Reason Tests JDK v1.8.0_111 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens   hadoop.yarn.server.resourcemanager.TestAMAuthorization JDK v1.7.0_111 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens   hadoop.yarn.server.resourcemanager.TestAMAuthorization Subsystem Report/Notes Docker Image:yetus/hadoop:5af2af1 JIRA Issue YARN-5802 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12837345/YARN-5802.branch-2.8.0001.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux eeb9c50e969d 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision branch-2.8 / 5073289 Default Java 1.7.0_111 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_111 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_111 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/13791/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/13791/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_111.txt JDK v1.7.0_111 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13791/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/13791/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        bibinchundatt Bibin A Chundatt added a comment -

        Thank you Sunil G for commit to trunk.

        1. Attaching patch for branch-2.8.LeafQueue and TestClientRMService mismatch resolved.
        2. Verified locally new testcases added in branch 2.8
        Show
        bibinchundatt Bibin A Chundatt added a comment - Thank you Sunil G for commit to trunk. Attaching patch for branch-2.8 . LeafQueue and TestClientRMService mismatch resolved. Verified locally new testcases added in branch 2.8
        Hide
        sunilg Sunil G added a comment - - edited

        Bibin A Chundatt
        I have committed this patch to trunk and branch-2.

        Current patch does not apply cleanly to branch-2.8. Please share branch-2.8 patch.

        Show
        sunilg Sunil G added a comment - - edited Bibin A Chundatt I have committed this patch to trunk and branch-2. Current patch does not apply cleanly to branch-2.8. Please share branch-2.8 patch.
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10770 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10770/)
        YARN-5802. updateApplicationPriority api in scheduler should ensure to (Sunil: rev 19b3779ae7455230ed89971141b2667eae624aab)

        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java
        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestApplicationPriority.java
        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestClientRMService.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10770 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10770/ ) YARN-5802 . updateApplicationPriority api in scheduler should ensure to (Sunil: rev 19b3779ae7455230ed89971141b2667eae624aab) (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestApplicationPriority.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestClientRMService.java
        Hide
        sunilg Sunil G added a comment -

        Test case failure is tracked via YARN-5815. Latest patch looks fine for me. +1
        I will commit the same tomorrow if there are no objections.

        Show
        sunilg Sunil G added a comment - Test case failure is tracked via YARN-5815 . Latest patch looks fine for me. +1 I will commit the same tomorrow if there are no objections.
        Hide
        bibinchundatt Bibin A Chundatt added a comment -

        Test case failure looks random due to Nodemanager event not yet received to scheduler after registration. Not related to patch attached.

        Show
        bibinchundatt Bibin A Chundatt added a comment - Test case failure looks random due to Nodemanager event not yet received to scheduler after registration. Not related to patch attached.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 14s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
        +1 mvninstall 7m 7s trunk passed
        +1 compile 0m 32s trunk passed
        +1 checkstyle 0m 22s trunk passed
        +1 mvnsite 0m 40s trunk passed
        +1 mvneclipse 0m 18s trunk passed
        +1 findbugs 0m 59s trunk passed
        +1 javadoc 0m 22s trunk passed
        +1 mvninstall 0m 33s the patch passed
        +1 compile 0m 30s the patch passed
        +1 javac 0m 30s the patch passed
        -0 checkstyle 0m 19s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 1 new + 125 unchanged - 0 fixed = 126 total (was 125)
        +1 mvnsite 0m 36s the patch passed
        +1 mvneclipse 0m 14s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 1m 2s the patch passed
        +1 javadoc 0m 17s the patch passed
        -1 unit 37m 0s hadoop-yarn-server-resourcemanager in the patch failed.
        +1 asflicense 0m 25s The patch does not generate ASF License warnings.
        52m 45s



        Reason Tests
        Failed junit tests hadoop.yarn.server.resourcemanager.scheduler.capacity.TestApplicationPriority



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:9560f25
        JIRA Issue YARN-5802
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12836537/YARN-5802.0006.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 85f9c2b378b3 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / cb5cc0d
        Default Java 1.8.0_101
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/13750/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
        unit https://builds.apache.org/job/PreCommit-YARN-Build/13750/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13750/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/13750/console
        Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 14s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. +1 mvninstall 7m 7s trunk passed +1 compile 0m 32s trunk passed +1 checkstyle 0m 22s trunk passed +1 mvnsite 0m 40s trunk passed +1 mvneclipse 0m 18s trunk passed +1 findbugs 0m 59s trunk passed +1 javadoc 0m 22s trunk passed +1 mvninstall 0m 33s the patch passed +1 compile 0m 30s the patch passed +1 javac 0m 30s the patch passed -0 checkstyle 0m 19s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 1 new + 125 unchanged - 0 fixed = 126 total (was 125) +1 mvnsite 0m 36s the patch passed +1 mvneclipse 0m 14s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 2s the patch passed +1 javadoc 0m 17s the patch passed -1 unit 37m 0s hadoop-yarn-server-resourcemanager in the patch failed. +1 asflicense 0m 25s The patch does not generate ASF License warnings. 52m 45s Reason Tests Failed junit tests hadoop.yarn.server.resourcemanager.scheduler.capacity.TestApplicationPriority Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Issue YARN-5802 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12836537/YARN-5802.0006.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 85f9c2b378b3 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / cb5cc0d Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/13750/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/13750/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13750/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/13750/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        bibinchundatt Bibin A Chundatt added a comment -

        Thank you Sunil G for review comment.
        Update patch handling all review comments..

        Show
        bibinchundatt Bibin A Chundatt added a comment - Thank you Sunil G for review comment. Update patch handling all review comments..
        Hide
        sunilg Sunil G added a comment -

        Thanks Bibin A Chundatt. One last pass over the patch. Few minor nits
        TestClientRMService.java:
        1. testUpdatePriorityAndKillWithEmptyResource -> testUpdatePriorityAndKillAppWithZeroNodes (or ZeroClusterResource)

        TestApplicationPriority.java:
        2. testUpdatePriorityAndRemoveAttempt -> testUpdatePriorityOnPendingAppAndKillAttempt
        3. This code CSQueue defaultQueue = findQueue(rootQueue, "root.default”); could be replaced with LeafQueue q = (LeafQueue) cs.getQueue("default"); given that the scheduler is CS. If so we could remove findQueue
        4. In killAppAndVerifyOrderingPolicy, appsPending and activeApps are calculated before calling updateApplicationPriority. Pls retrieve these values after update and kill event.
        5. In killAppAndVerifyOrderingPolicy, activeApps is directly getting an object of schedulableEntities. pls use getApplications from LeafQueue like pending.

        Show
        sunilg Sunil G added a comment - Thanks Bibin A Chundatt . One last pass over the patch. Few minor nits TestClientRMService.java: 1. testUpdatePriorityAndKillWithEmptyResource -> testUpdatePriorityAndKillAppWithZeroNodes (or ZeroClusterResource) TestApplicationPriority.java: 2. testUpdatePriorityAndRemoveAttempt -> testUpdatePriorityOnPendingAppAndKillAttempt 3. This code CSQueue defaultQueue = findQueue(rootQueue, "root.default”); could be replaced with LeafQueue q = (LeafQueue) cs.getQueue("default"); given that the scheduler is CS. If so we could remove findQueue 4. In killAppAndVerifyOrderingPolicy , appsPending and activeApps are calculated before calling updateApplicationPriority . Pls retrieve these values after update and kill event. 5. In killAppAndVerifyOrderingPolicy , activeApps is directly getting an object of schedulableEntities . pls use getApplications from LeafQueue like pending.
        Hide
        hadoopqa Hadoop QA added a comment -
        +1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 23s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
        +1 mvninstall 8m 22s trunk passed
        +1 compile 0m 38s trunk passed
        +1 checkstyle 0m 25s trunk passed
        +1 mvnsite 0m 44s trunk passed
        +1 mvneclipse 0m 19s trunk passed
        +1 findbugs 1m 9s trunk passed
        +1 javadoc 0m 24s trunk passed
        +1 mvninstall 0m 37s the patch passed
        +1 compile 0m 36s the patch passed
        +1 javac 0m 36s the patch passed
        -0 checkstyle 0m 19s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 1 new + 126 unchanged - 0 fixed = 127 total (was 126)
        +1 mvnsite 0m 45s the patch passed
        +1 mvneclipse 0m 16s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 1m 11s the patch passed
        +1 javadoc 0m 19s the patch passed
        +1 unit 35m 5s hadoop-yarn-server-resourcemanager in the patch passed.
        +1 asflicense 0m 16s The patch does not generate ASF License warnings.
        53m 6s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:9560f25
        JIRA Issue YARN-5802
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12836097/YARN-5802.0005.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 66d1fb3e3da9 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / aa3cab1
        Default Java 1.8.0_101
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/13685/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13685/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/13685/console
        Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 23s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. +1 mvninstall 8m 22s trunk passed +1 compile 0m 38s trunk passed +1 checkstyle 0m 25s trunk passed +1 mvnsite 0m 44s trunk passed +1 mvneclipse 0m 19s trunk passed +1 findbugs 1m 9s trunk passed +1 javadoc 0m 24s trunk passed +1 mvninstall 0m 37s the patch passed +1 compile 0m 36s the patch passed +1 javac 0m 36s the patch passed -0 checkstyle 0m 19s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 1 new + 126 unchanged - 0 fixed = 127 total (was 126) +1 mvnsite 0m 45s the patch passed +1 mvneclipse 0m 16s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 11s the patch passed +1 javadoc 0m 19s the patch passed +1 unit 35m 5s hadoop-yarn-server-resourcemanager in the patch passed. +1 asflicense 0m 16s The patch does not generate ASF License warnings. 53m 6s Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Issue YARN-5802 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12836097/YARN-5802.0005.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 66d1fb3e3da9 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / aa3cab1 Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/13685/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13685/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/13685/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        bibinchundatt Bibin A Chundatt added a comment -

        Thank you Sunil G for review comments

        Extremely sorry for confusing earlier with the comment of introducing contains interface

        Nop.Doesnt create too much performance impact since the call is not frequent.

        It will really great if we can have such a case in TestApplicationPriority

        updated scenarios to cover all cases

        Show
        bibinchundatt Bibin A Chundatt added a comment - Thank you Sunil G for review comments Extremely sorry for confusing earlier with the comment of introducing contains interface Nop.Doesnt create too much performance impact since the call is not frequent. It will really great if we can have such a case in TestApplicationPriority updated scenarios to cover all cases
        Hide
        sunilg Sunil G added a comment -

        HI Bibin A Chundatt for the patch.

        Extremely sorry for confusing earlier with the comment of introducing contains interface. I have looked into LQ#removeApplicationAttempt. And it uses like below. And it is similar to the approach you have told first offline.

              boolean wasActive = orderingPolicy.removeSchedulableEntity(application);
              if (!wasActive) {
                pendingOrderingPolicy.removeSchedulableEntity(application);
              }
        

        So could we do something like below which i think matches your initial impression.

              boolean wasActive = orderingPolicy.removeSchedulableEntity(application);
              if (!wasActive) {
                pendingOrderingPolicy.removeSchedulableEntity(application);
              }
              // Update new priority in SchedulerApplication
              attempt.setPriority(newAppPriority);
        
              if (wasActive) {
                getOrderingPolicy().addSchedulableEntity(attempt);
              } else {
                getPendingAppsOrderingPolicy().addSchedulableEntity(attempt);
              }
        

        TestCase:
        1. Could we move testUpdatePriorityAndRemoveAttempt to TestApplicationPriority
        2. I think currently in testUpdatePriorityAndRemoveAttempt (as per patch), second priority update is not happening.
        So could we remove the kill part, and call cs.updateApplicationPriority direct so that update priority will be done for pending app (assuming no node is registered). Then register a new node, and ensure that this app is moved to oderingPolicy which is the active list. With this, we can now complete full test case. It will really great if we can have such a case in TestApplicationPriority

        Thanks.

        Show
        sunilg Sunil G added a comment - HI Bibin A Chundatt for the patch. Extremely sorry for confusing earlier with the comment of introducing contains interface. I have looked into LQ#removeApplicationAttempt . And it uses like below. And it is similar to the approach you have told first offline. boolean wasActive = orderingPolicy.removeSchedulableEntity(application); if (!wasActive) { pendingOrderingPolicy.removeSchedulableEntity(application); } So could we do something like below which i think matches your initial impression. boolean wasActive = orderingPolicy.removeSchedulableEntity(application); if (!wasActive) { pendingOrderingPolicy.removeSchedulableEntity(application); } // Update new priority in SchedulerApplication attempt.setPriority(newAppPriority); if (wasActive) { getOrderingPolicy().addSchedulableEntity(attempt); } else { getPendingAppsOrderingPolicy().addSchedulableEntity(attempt); } TestCase: 1. Could we move testUpdatePriorityAndRemoveAttempt to TestApplicationPriority 2. I think currently in testUpdatePriorityAndRemoveAttempt (as per patch), second priority update is not happening. So could we remove the kill part, and call cs.updateApplicationPriority direct so that update priority will be done for pending app (assuming no node is registered). Then register a new node, and ensure that this app is moved to oderingPolicy which is the active list. With this, we can now complete full test case. It will really great if we can have such a case in TestApplicationPriority Thanks.
        Hide
        bibinchundatt Bibin A Chundatt added a comment -

        Failure looks random.YARN-5803 raised to track the same

        Show
        bibinchundatt Bibin A Chundatt added a comment - Failure looks random. YARN-5803 raised to track the same
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 17s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
        +1 mvninstall 6m 43s trunk passed
        +1 compile 0m 32s trunk passed
        +1 checkstyle 0m 24s trunk passed
        +1 mvnsite 0m 37s trunk passed
        +1 mvneclipse 0m 16s trunk passed
        +1 findbugs 0m 57s trunk passed
        +1 javadoc 0m 21s trunk passed
        +1 mvninstall 0m 31s the patch passed
        +1 compile 0m 29s the patch passed
        +1 javac 0m 29s the patch passed
        +1 checkstyle 0m 21s the patch passed
        +1 mvnsite 0m 35s the patch passed
        +1 mvneclipse 0m 13s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 1m 1s the patch passed
        +1 javadoc 0m 19s the patch passed
        -1 unit 38m 30s hadoop-yarn-server-resourcemanager in the patch failed.
        +1 asflicense 0m 15s The patch does not generate ASF License warnings.
        53m 36s



        Reason Tests
        Failed junit tests hadoop.yarn.server.resourcemanager.applicationsmanager.TestAMRestart



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:9560f25
        JIRA Issue YARN-5802
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12836049/YARN-5802.0004.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 794f8bdfd0e5 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / aa3cab1
        Default Java 1.8.0_101
        findbugs v3.0.0
        unit https://builds.apache.org/job/PreCommit-YARN-Build/13681/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13681/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/13681/console
        Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 17s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. +1 mvninstall 6m 43s trunk passed +1 compile 0m 32s trunk passed +1 checkstyle 0m 24s trunk passed +1 mvnsite 0m 37s trunk passed +1 mvneclipse 0m 16s trunk passed +1 findbugs 0m 57s trunk passed +1 javadoc 0m 21s trunk passed +1 mvninstall 0m 31s the patch passed +1 compile 0m 29s the patch passed +1 javac 0m 29s the patch passed +1 checkstyle 0m 21s the patch passed +1 mvnsite 0m 35s the patch passed +1 mvneclipse 0m 13s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 1s the patch passed +1 javadoc 0m 19s the patch passed -1 unit 38m 30s hadoop-yarn-server-resourcemanager in the patch failed. +1 asflicense 0m 15s The patch does not generate ASF License warnings. 53m 36s Reason Tests Failed junit tests hadoop.yarn.server.resourcemanager.applicationsmanager.TestAMRestart Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Issue YARN-5802 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12836049/YARN-5802.0004.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 794f8bdfd0e5 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / aa3cab1 Default Java 1.8.0_101 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-YARN-Build/13681/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13681/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/13681/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        bibinchundatt Bibin A Chundatt added a comment -

        hadoop.yarn.server.resourcemanager.TestRMRestart- 5548 is available for tracking

        Show
        bibinchundatt Bibin A Chundatt added a comment - hadoop.yarn.server.resourcemanager.TestRMRestart- 5548 is available for tracking
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 16s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
        +1 mvninstall 6m 46s trunk passed
        +1 compile 0m 32s trunk passed
        +1 checkstyle 0m 23s trunk passed
        +1 mvnsite 0m 37s trunk passed
        +1 mvneclipse 0m 16s trunk passed
        +1 findbugs 0m 56s trunk passed
        +1 javadoc 0m 21s trunk passed
        +1 mvninstall 0m 32s the patch passed
        +1 compile 0m 30s the patch passed
        +1 javac 0m 30s the patch passed
        -0 checkstyle 0m 20s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 1 new + 309 unchanged - 0 fixed = 310 total (was 309)
        +1 mvnsite 0m 35s the patch passed
        +1 mvneclipse 0m 14s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 1m 2s the patch passed
        -1 javadoc 0m 18s hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager generated 2 new + 938 unchanged - 0 fixed = 940 total (was 938)
        -1 unit 38m 26s hadoop-yarn-server-resourcemanager in the patch failed.
        +1 asflicense 0m 15s The patch does not generate ASF License warnings.
        53m 34s



        Reason Tests
        Failed junit tests hadoop.yarn.server.resourcemanager.TestRMRestart



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:9560f25
        JIRA Issue YARN-5802
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12836046/YARN-5802.0003.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 72da115595ac 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / aa3cab1
        Default Java 1.8.0_101
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/13680/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
        javadoc https://builds.apache.org/job/PreCommit-YARN-Build/13680/artifact/patchprocess/diff-javadoc-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
        unit https://builds.apache.org/job/PreCommit-YARN-Build/13680/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13680/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/13680/console
        Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 16s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. +1 mvninstall 6m 46s trunk passed +1 compile 0m 32s trunk passed +1 checkstyle 0m 23s trunk passed +1 mvnsite 0m 37s trunk passed +1 mvneclipse 0m 16s trunk passed +1 findbugs 0m 56s trunk passed +1 javadoc 0m 21s trunk passed +1 mvninstall 0m 32s the patch passed +1 compile 0m 30s the patch passed +1 javac 0m 30s the patch passed -0 checkstyle 0m 20s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 1 new + 309 unchanged - 0 fixed = 310 total (was 309) +1 mvnsite 0m 35s the patch passed +1 mvneclipse 0m 14s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 2s the patch passed -1 javadoc 0m 18s hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager generated 2 new + 938 unchanged - 0 fixed = 940 total (was 938) -1 unit 38m 26s hadoop-yarn-server-resourcemanager in the patch failed. +1 asflicense 0m 15s The patch does not generate ASF License warnings. 53m 34s Reason Tests Failed junit tests hadoop.yarn.server.resourcemanager.TestRMRestart Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Issue YARN-5802 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12836046/YARN-5802.0003.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 72da115595ac 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / aa3cab1 Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/13680/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt javadoc https://builds.apache.org/job/PreCommit-YARN-Build/13680/artifact/patchprocess/diff-javadoc-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/13680/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13680/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/13680/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        bibinchundatt Bibin A Chundatt added a comment -

        Thank you Varun Saxena for review

        Checkstyle and javadoc looks related. Can you fix them ?

        done

        this test be added in TestCapacityScheduler instead fetching queues and ordering policy in TestClientRMService

        Moved ordering policy assertion to TestCapacityScheduler.One testcase to verify kill apps with empty resource is added to TestClientRMService.Hope that should be fine since its one additional scenarios test.

        Show
        bibinchundatt Bibin A Chundatt added a comment - Thank you Varun Saxena for review Checkstyle and javadoc looks related. Can you fix them ? done this test be added in TestCapacityScheduler instead fetching queues and ordering policy in TestClientRMService Moved ordering policy assertion to TestCapacityScheduler .One testcase to verify kill apps with empty resource is added to TestClientRMService .Hope that should be fine since its one additional scenarios test.
        Hide
        varun_saxena Varun Saxena added a comment -

        Thanks Bibin A Chundatt for the patch. Changes overall LGTM.

        Checkstyle and javadoc looks related. Can you fix them ?
        Also in the test case added, we can explicitly set scheduler to capacity in config as test makes sense only for capacity scheduler. instanceof check should not be required.
        Moreover, shouldn't this test be added in TestCapacityScheduler instead fetching queues and ordering policy in TestClientRMService ? Because although we found the issue via kill but the real issue is during attempt removal from capacity scheduler.
        However, I do not have a strong opinion on this. If its easy to add, probably add this test in TestCapacityScheduler.

        Show
        varun_saxena Varun Saxena added a comment - Thanks Bibin A Chundatt for the patch. Changes overall LGTM. Checkstyle and javadoc looks related. Can you fix them ? Also in the test case added, we can explicitly set scheduler to capacity in config as test makes sense only for capacity scheduler. instanceof check should not be required. Moreover, shouldn't this test be added in TestCapacityScheduler instead fetching queues and ordering policy in TestClientRMService ? Because although we found the issue via kill but the real issue is during attempt removal from capacity scheduler. However, I do not have a strong opinion on this. If its easy to add, probably add this test in TestCapacityScheduler.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 20s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        +1 mvninstall 6m 44s trunk passed
        +1 compile 0m 32s trunk passed
        +1 checkstyle 0m 23s trunk passed
        +1 mvnsite 0m 38s trunk passed
        +1 mvneclipse 0m 17s trunk passed
        +1 findbugs 0m 56s trunk passed
        +1 javadoc 0m 21s trunk passed
        +1 mvninstall 0m 32s the patch passed
        +1 compile 0m 30s the patch passed
        +1 javac 0m 30s the patch passed
        -0 checkstyle 0m 20s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 2 new + 129 unchanged - 0 fixed = 131 total (was 129)
        +1 mvnsite 0m 37s the patch passed
        +1 mvneclipse 0m 15s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 1m 4s the patch passed
        -1 javadoc 0m 18s hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager generated 1 new + 938 unchanged - 0 fixed = 939 total (was 938)
        +1 unit 35m 6s hadoop-yarn-server-resourcemanager in the patch passed.
        +1 asflicense 0m 15s The patch does not generate ASF License warnings.
        50m 23s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:9560f25
        JIRA Issue YARN-5802
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12836024/YARN-5802.0002.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux acbec2863550 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / aa3cab1
        Default Java 1.8.0_101
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/13677/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
        javadoc https://builds.apache.org/job/PreCommit-YARN-Build/13677/artifact/patchprocess/diff-javadoc-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13677/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/13677/console
        Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 20s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 6m 44s trunk passed +1 compile 0m 32s trunk passed +1 checkstyle 0m 23s trunk passed +1 mvnsite 0m 38s trunk passed +1 mvneclipse 0m 17s trunk passed +1 findbugs 0m 56s trunk passed +1 javadoc 0m 21s trunk passed +1 mvninstall 0m 32s the patch passed +1 compile 0m 30s the patch passed +1 javac 0m 30s the patch passed -0 checkstyle 0m 20s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 2 new + 129 unchanged - 0 fixed = 131 total (was 129) +1 mvnsite 0m 37s the patch passed +1 mvneclipse 0m 15s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 4s the patch passed -1 javadoc 0m 18s hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager generated 1 new + 938 unchanged - 0 fixed = 939 total (was 938) +1 unit 35m 6s hadoop-yarn-server-resourcemanager in the patch passed. +1 asflicense 0m 15s The patch does not generate ASF License warnings. 50m 23s Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Issue YARN-5802 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12836024/YARN-5802.0002.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux acbec2863550 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / aa3cab1 Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/13677/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt javadoc https://builds.apache.org/job/PreCommit-YARN-Build/13677/artifact/patchprocess/diff-javadoc-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13677/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/13677/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        bibinchundatt Bibin A Chundatt added a comment -

        Thank you Sunil G looking into the issue.

        Current approach in the patch is not handling runtime update priority for accepted/pending apps.

        Had added a query for the same. Thank you for confirming that we require update of pendingOrderingPolicy too.

        We might need to add contains method to OrderingPolicy interface for this.

        Thank you for input. Attaching patch after handling comment and adding testcases

        Show
        bibinchundatt Bibin A Chundatt added a comment - Thank you Sunil G looking into the issue. Current approach in the patch is not handling runtime update priority for accepted/pending apps. Had added a query for the same. Thank you for confirming that we require update of pendingOrderingPolicy too. We might need to add contains method to OrderingPolicy interface for this. Thank you for input. Attaching patch after handling comment and adding testcases
        Hide
        sunilg Sunil G added a comment -

        Marking Target Version as 2.8 because priority feature is introduced in 2.8

        Show
        sunilg Sunil G added a comment - Marking Target Version as 2.8 because priority feature is introduced in 2.8
        Hide
        sunilg Sunil G added a comment - - edited

        Good catch. After app priority, there were few optimizations done to orderingPolicies. And hence two policies are now used. Priority update should consider both.

        App priority can be updated when app is in ACCEPTED or RUNNING state. Hence in the scheduler, app can be either in pendingOrderingPolicy or in normal orderingPolicy. So please check whether app is present in pendingOrderingPolicy or orderingPolicy, then update back to same object. Current approach in the patch is not handling runtime update priority for accepted/pending apps.

        We might need to add contains method to OrderingPolicy interface for this.

        Show
        sunilg Sunil G added a comment - - edited Good catch. After app priority, there were few optimizations done to orderingPolicies. And hence two policies are now used. Priority update should consider both. App priority can be updated when app is in ACCEPTED or RUNNING state. Hence in the scheduler, app can be either in pendingOrderingPolicy or in normal orderingPolicy . So please check whether app is present in pendingOrderingPolicy or orderingPolicy , then update back to same object. Current approach in the patch is not handling runtime update priority for accepted/pending apps. We might need to add contains method to OrderingPolicy interface for this.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 16s Docker mode activated.
        +1 @author 0m 1s The patch does not contain any @author tags.
        -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        +1 mvninstall 6m 44s trunk passed
        +1 compile 0m 33s trunk passed
        +1 checkstyle 0m 21s trunk passed
        +1 mvnsite 0m 36s trunk passed
        +1 mvneclipse 0m 17s trunk passed
        +1 findbugs 0m 56s trunk passed
        +1 javadoc 0m 21s trunk passed
        +1 mvninstall 0m 30s the patch passed
        +1 compile 0m 29s the patch passed
        +1 javac 0m 29s the patch passed
        +1 checkstyle 0m 18s the patch passed
        +1 mvnsite 0m 35s the patch passed
        +1 mvneclipse 0m 14s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 1m 1s the patch passed
        +1 javadoc 0m 18s the patch passed
        +1 unit 38m 40s hadoop-yarn-server-resourcemanager in the patch passed.
        +1 asflicense 0m 15s The patch does not generate ASF License warnings.
        53m 39s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:9560f25
        JIRA Issue YARN-5802
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12835999/YARN-5802.0001.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux fb1441c4a7ee 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / aa3cab1
        Default Java 1.8.0_101
        findbugs v3.0.0
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13672/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/13672/console
        Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 16s Docker mode activated. +1 @author 0m 1s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 6m 44s trunk passed +1 compile 0m 33s trunk passed +1 checkstyle 0m 21s trunk passed +1 mvnsite 0m 36s trunk passed +1 mvneclipse 0m 17s trunk passed +1 findbugs 0m 56s trunk passed +1 javadoc 0m 21s trunk passed +1 mvninstall 0m 30s the patch passed +1 compile 0m 29s the patch passed +1 javac 0m 29s the patch passed +1 checkstyle 0m 18s the patch passed +1 mvnsite 0m 35s the patch passed +1 mvneclipse 0m 14s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 1s the patch passed +1 javadoc 0m 18s the patch passed +1 unit 38m 40s hadoop-yarn-server-resourcemanager in the patch passed. +1 asflicense 0m 15s The patch does not generate ASF License warnings. 53m 39s Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Issue YARN-5802 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12835999/YARN-5802.0001.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux fb1441c4a7ee 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / aa3cab1 Default Java 1.8.0_101 findbugs v3.0.0 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13672/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/13672/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        bibinchundatt Bibin A Chundatt added a comment -

        Attaching patch for the same

        Show
        bibinchundatt Bibin A Chundatt added a comment - Attaching patch for the same
        Hide
        bibinchundatt Bibin A Chundatt added a comment - - edited

        Sunil G
        Currently only the for orderingPolicy update is done during priority update. For pendingOrderingPolicy also IIUC should remove and add. To update ordering.

        Show
        bibinchundatt Bibin A Chundatt added a comment - - edited Sunil G Currently only the for orderingPolicy update is done during priority update. For pendingOrderingPolicy also IIUC should remove and add. To update ordering.

          People

          • Assignee:
            bibinchundatt Bibin A Chundatt
            Reporter:
            bibinchundatt Bibin A Chundatt
          • Votes:
            0 Vote for this issue
            Watchers:
            8 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development