Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-5932

Retrospect moveApplicationToQueue in align with YARN-5611

    Details

    • Hadoop Flags:
      Reviewed

      Description

      All dynamic api's of an application's state change could follow a general design approach. Currently priority and app timeouts are following this approach all corner cases.
      Steps

      • Do a pre-validate check to ensure that changes are fine.
      • Update this information to state-store
      • Perform real move operation and update in-memory data structures.
      1. YARN-5932.0001.patch
        46 kB
        Sunil G
      2. YARN-5932.0002.patch
        41 kB
        Sunil G
      3. YARN-5932.0003.patch
        41 kB
        Sunil G
      4. YARN-5932.v0.patch
        19 kB
        Sunil G
      5. YARN-5932.v1.patch
        46 kB
        Sunil G
      6. YARN-5932-branch-2.0001.patch
        41 kB
        Sunil G

        Issue Links

          Activity

          Hide
          sunilg Sunil G added a comment -

          Updating a version 0 patch to capture the flow.

          Changes done:

          • Currently this patch handles changes only for Capacity Scheduler
          • A new pre-validate check is added for moveApplication in CapacityScheduler and Leaf/ParentQueue. This helps to ensure that the target queue can accept this application or not. Any failure could be handled at this stage itself.
          • StateStore also updated with the queue name to which this app is moved to. We could update submission context for same.
          • Finally perform real move operation. Ideally there should not be any exception thrown out. However with some corner cases, there is very less chance to happen this. In such cases, we can revert from StateStore.

          cc/Jian HeRohith Sharma K S

          Show
          sunilg Sunil G added a comment - Updating a version 0 patch to capture the flow. Changes done: Currently this patch handles changes only for Capacity Scheduler A new pre-validate check is added for moveApplication in CapacityScheduler and Leaf/ParentQueue. This helps to ensure that the target queue can accept this application or not. Any failure could be handled at this stage itself. StateStore also updated with the queue name to which this app is moved to. We could update submission context for same. Finally perform real move operation. Ideally there should not be any exception thrown out. However with some corner cases, there is very less chance to happen this. In such cases, we can revert from StateStore. cc/ Jian He Rohith Sharma K S
          Hide
          sunilg Sunil G added a comment - - edited

          Attaching v1 patch.

          Changes:

          • Introduced new command for move : yarn application -appId <app Id> -move queueB. Old one is deprecated, but will still work.
          • Removed RMAppMoveEvent handling. RMApp need not have to handle MOVE event as we introduced new api in RMAppManager.
          • Changed FairScheduler in align with new design approach.

          cc/ Rohith Sharma K S and Jian He

          Show
          sunilg Sunil G added a comment - - edited Attaching v1 patch. Changes: Introduced new command for move : yarn application -appId <app Id> -move queueB . Old one is deprecated, but will still work. Removed RMAppMoveEvent handling. RMApp need not have to handle MOVE event as we introduced new api in RMAppManager. Changed FairScheduler in align with new design approach. cc/ Rohith Sharma K S and Jian He
          Hide
          rohithsharma Rohith Sharma K S added a comment -

          Approach looks fine to me,I will have detailed look at the patch.

          Show
          rohithsharma Rohith Sharma K S added a comment - Approach looks fine to me,I will have detailed look at the patch.
          Hide
          rohithsharma Rohith Sharma K S added a comment -

          overall patch looks good to me.

          Show
          rohithsharma Rohith Sharma K S added a comment - overall patch looks good to me.
          Hide
          sunilg Sunil G added a comment -

          Thanks Rohith Sharma K S.

          As discussed, there are some similar changes done in FairScheduler too.

          A quick summary of changes.
          1. Introduced preValidateMoveApplication api to validate whether move operation can be successful or not. Few code is taken out from FS#moveApplication for same.
          2. By using above api, we are trying to achieve a similar behavior for all app update changes from cli/rest etc.

          Karthik Kambatla, could you please help to take a look.

          Rohith Sharma K S, please help to add any points if I missed some. Thank you.

          Show
          sunilg Sunil G added a comment - Thanks Rohith Sharma K S . As discussed, there are some similar changes done in FairScheduler too. A quick summary of changes. 1. Introduced preValidateMoveApplication api to validate whether move operation can be successful or not. Few code is taken out from FS#moveApplication for same. 2. By using above api, we are trying to achieve a similar behavior for all app update changes from cli/rest etc. Karthik Kambatla , could you please help to take a look. Rohith Sharma K S , please help to add any points if I missed some. Thank you.
          Hide
          jianhe Jian He added a comment -

          one question about move, what if the target queue goes over its capacity limit, will the move continue ?

          • change this to if (! (Running && accepted) ) ?
                if (EnumSet.of(RMAppState.NEW, RMAppState.NEW_SAVING, RMAppState.SUBMITTED,
                    RMAppState.FINAL_SAVING, RMAppState.FINISHING, RMAppState.FINISHED,
                    RMAppState.KILLED, RMAppState.KILLING, RMAppState.FAILED)
            
          • the type cast is not needed
             ((RMAppImpl) app).setQueue(queue);
              FSAppAttempt attempt = (FSAppAttempt) app.getCurrentAppAttempt();
             
          Show
          jianhe Jian He added a comment - one question about move, what if the target queue goes over its capacity limit, will the move continue ? change this to if (! (Running && accepted) ) ? if (EnumSet.of(RMAppState.NEW, RMAppState.NEW_SAVING, RMAppState.SUBMITTED, RMAppState.FINAL_SAVING, RMAppState.FINISHING, RMAppState.FINISHED, RMAppState.KILLED, RMAppState.KILLING, RMAppState.FAILED) the type cast is not needed ((RMAppImpl) app).setQueue(queue); FSAppAttempt attempt = (FSAppAttempt) app.getCurrentAppAttempt();
          Hide
          sunilg Sunil G added a comment -

          Thanks Jian He

          Yes. We are not checking this condition as of now in CS. Ideally we are moving resource from one queue to another. hence post move, there could be some imbalances. Could such cases be handled in next allocation / preemption cycle itself.

          I had few thoughts here for some more potential validations (app priority etc), but it may make move more stricter. Could that be a pblm to end user as its an old api to customer.

          Addressed other comments and uploading a new patch.

          Show
          sunilg Sunil G added a comment - Thanks Jian He Yes. We are not checking this condition as of now in CS. Ideally we are moving resource from one queue to another. hence post move, there could be some imbalances. Could such cases be handled in next allocation / preemption cycle itself. I had few thoughts here for some more potential validations (app priority etc), but it may make move more stricter. Could that be a pblm to end user as its an old api to customer. Addressed other comments and uploading a new patch.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 13s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
          0 mvndep 0m 9s Maven dependency ordering for branch
          +1 mvninstall 7m 33s trunk passed
          +1 compile 6m 8s trunk passed
          +1 checkstyle 0m 59s trunk passed
          +1 mvnsite 1m 30s trunk passed
          +1 mvneclipse 0m 49s trunk passed
          +1 findbugs 2m 9s trunk passed
          +1 javadoc 0m 55s trunk passed
          0 mvndep 0m 11s Maven dependency ordering for patch
          +1 mvninstall 1m 10s the patch passed
          +1 compile 5m 29s the patch passed
          +1 javac 5m 28s the patch passed
          -0 checkstyle 0m 56s hadoop-yarn-project/hadoop-yarn: The patch generated 28 new + 653 unchanged - 13 fixed = 681 total (was 666)
          +1 mvnsite 1m 28s the patch passed
          +1 mvneclipse 0m 46s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 2m 28s the patch passed
          -1 javadoc 0m 30s hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager generated 1 new + 913 unchanged - 0 fixed = 914 total (was 913)
          -1 unit 42m 37s hadoop-yarn-server-resourcemanager in the patch failed.
          +1 unit 16m 7s hadoop-yarn-client in the patch passed.
          +1 asflicense 0m 29s The patch does not generate ASF License warnings.
          101m 11s



          Reason Tests
          Failed junit tests hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacitySchedulerNodeLabelUpdate
            hadoop.yarn.server.resourcemanager.scheduler.fair.TestContinuousScheduling



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue YARN-5932
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12841259/YARN-5932.0001.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 85d6dedb6673 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 1f7613b
          Default Java 1.8.0_111
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/14143/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt
          javadoc https://builds.apache.org/job/PreCommit-YARN-Build/14143/artifact/patchprocess/diff-javadoc-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/14143/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/14143/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client U: hadoop-yarn-project/hadoop-yarn
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/14143/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 13s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. 0 mvndep 0m 9s Maven dependency ordering for branch +1 mvninstall 7m 33s trunk passed +1 compile 6m 8s trunk passed +1 checkstyle 0m 59s trunk passed +1 mvnsite 1m 30s trunk passed +1 mvneclipse 0m 49s trunk passed +1 findbugs 2m 9s trunk passed +1 javadoc 0m 55s trunk passed 0 mvndep 0m 11s Maven dependency ordering for patch +1 mvninstall 1m 10s the patch passed +1 compile 5m 29s the patch passed +1 javac 5m 28s the patch passed -0 checkstyle 0m 56s hadoop-yarn-project/hadoop-yarn: The patch generated 28 new + 653 unchanged - 13 fixed = 681 total (was 666) +1 mvnsite 1m 28s the patch passed +1 mvneclipse 0m 46s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 2m 28s the patch passed -1 javadoc 0m 30s hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager generated 1 new + 913 unchanged - 0 fixed = 914 total (was 913) -1 unit 42m 37s hadoop-yarn-server-resourcemanager in the patch failed. +1 unit 16m 7s hadoop-yarn-client in the patch passed. +1 asflicense 0m 29s The patch does not generate ASF License warnings. 101m 11s Reason Tests Failed junit tests hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacitySchedulerNodeLabelUpdate   hadoop.yarn.server.resourcemanager.scheduler.fair.TestContinuousScheduling Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue YARN-5932 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12841259/YARN-5932.0001.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 85d6dedb6673 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 1f7613b Default Java 1.8.0_111 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/14143/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt javadoc https://builds.apache.org/job/PreCommit-YARN-Build/14143/artifact/patchprocess/diff-javadoc-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/14143/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/14143/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client U: hadoop-yarn-project/hadoop-yarn Console output https://builds.apache.org/job/PreCommit-YARN-Build/14143/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          sunilg Sunil G added a comment -

          Thanks Jian He

          Fixed test cases failures ande checkstyle.
          Also removed the CLI change from this patch as this patch is handling many different aspects as per thoughts from Vinod offline. I will attach CLI patch once this is committed.

          Jian He and Rohith Sharma K S pls review.

          Show
          sunilg Sunil G added a comment - Thanks Jian He Fixed test cases failures ande checkstyle. Also removed the CLI change from this patch as this patch is handling many different aspects as per thoughts from Vinod offline. I will attach CLI patch once this is committed. Jian He and Rohith Sharma K S pls review.
          Hide
          hadoopqa Hadoop QA added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 11s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
          +1 mvninstall 7m 11s trunk passed
          +1 compile 0m 35s trunk passed
          +1 checkstyle 0m 27s trunk passed
          +1 mvnsite 0m 40s trunk passed
          +1 mvneclipse 0m 17s trunk passed
          +1 findbugs 2m 10s trunk passed
          +1 javadoc 1m 11s trunk passed
          +1 mvninstall 1m 33s the patch passed
          +1 compile 1m 30s the patch passed
          +1 javac 1m 30s the patch passed
          -0 checkstyle 1m 20s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 4 new + 524 unchanged - 15 fixed = 528 total (was 539)
          +1 mvnsite 1m 32s the patch passed
          +1 mvneclipse 0m 43s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 3m 4s the patch passed
          +1 javadoc 0m 51s the patch passed
          +1 unit 38m 44s hadoop-yarn-server-resourcemanager in the patch passed.
          +1 asflicense 0m 20s The patch does not generate ASF License warnings.
          65m 30s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue YARN-5932
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12841444/YARN-5932.0002.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 0f167e7782f3 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / c87b3a4
          Default Java 1.8.0_111
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/14158/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/14158/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/14158/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 11s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. +1 mvninstall 7m 11s trunk passed +1 compile 0m 35s trunk passed +1 checkstyle 0m 27s trunk passed +1 mvnsite 0m 40s trunk passed +1 mvneclipse 0m 17s trunk passed +1 findbugs 2m 10s trunk passed +1 javadoc 1m 11s trunk passed +1 mvninstall 1m 33s the patch passed +1 compile 1m 30s the patch passed +1 javac 1m 30s the patch passed -0 checkstyle 1m 20s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 4 new + 524 unchanged - 15 fixed = 528 total (was 539) +1 mvnsite 1m 32s the patch passed +1 mvneclipse 0m 43s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 3m 4s the patch passed +1 javadoc 0m 51s the patch passed +1 unit 38m 44s hadoop-yarn-server-resourcemanager in the patch passed. +1 asflicense 0m 20s The patch does not generate ASF License warnings. 65m 30s Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue YARN-5932 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12841444/YARN-5932.0002.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 0f167e7782f3 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / c87b3a4 Default Java 1.8.0_111 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/14158/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/14158/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/14158/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          sunilg Sunil G added a comment -

          Updating patch after fixing a minor bug in ParentQueue previalidation check.

          Show
          sunilg Sunil G added a comment - Updating patch after fixing a minor bug in ParentQueue previalidation check.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 16s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
          +1 mvninstall 8m 14s trunk passed
          +1 compile 0m 39s trunk passed
          +1 checkstyle 0m 28s trunk passed
          +1 mvnsite 0m 43s trunk passed
          +1 mvneclipse 0m 19s trunk passed
          +1 findbugs 1m 5s trunk passed
          +1 javadoc 0m 24s trunk passed
          +1 mvninstall 0m 37s the patch passed
          +1 compile 0m 35s the patch passed
          +1 javac 0m 35s the patch passed
          -0 checkstyle 0m 28s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 4 new + 523 unchanged - 15 fixed = 527 total (was 538)
          +1 mvnsite 0m 42s the patch passed
          +1 mvneclipse 0m 17s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 1m 20s the patch passed
          +1 javadoc 0m 22s the patch passed
          -1 unit 45m 32s hadoop-yarn-server-resourcemanager in the patch failed.
          +1 asflicense 0m 18s The patch does not generate ASF License warnings.
          63m 42s



          Reason Tests
          Failed junit tests hadoop.yarn.server.resourcemanager.TestRMRestart



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue YARN-5932
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12841466/YARN-5932.0003.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 211ab3724ef5 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / c87b3a4
          Default Java 1.8.0_111
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/14162/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/14162/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/14162/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/14162/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 16s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. +1 mvninstall 8m 14s trunk passed +1 compile 0m 39s trunk passed +1 checkstyle 0m 28s trunk passed +1 mvnsite 0m 43s trunk passed +1 mvneclipse 0m 19s trunk passed +1 findbugs 1m 5s trunk passed +1 javadoc 0m 24s trunk passed +1 mvninstall 0m 37s the patch passed +1 compile 0m 35s the patch passed +1 javac 0m 35s the patch passed -0 checkstyle 0m 28s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 4 new + 523 unchanged - 15 fixed = 527 total (was 538) +1 mvnsite 0m 42s the patch passed +1 mvneclipse 0m 17s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 20s the patch passed +1 javadoc 0m 22s the patch passed -1 unit 45m 32s hadoop-yarn-server-resourcemanager in the patch failed. +1 asflicense 0m 18s The patch does not generate ASF License warnings. 63m 42s Reason Tests Failed junit tests hadoop.yarn.server.resourcemanager.TestRMRestart Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue YARN-5932 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12841466/YARN-5932.0003.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 211ab3724ef5 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / c87b3a4 Default Java 1.8.0_111 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/14162/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/14162/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/14162/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/14162/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          sunilg Sunil G added a comment -

          Test case failures are unrelated.

          Show
          sunilg Sunil G added a comment - Test case failures are unrelated.
          Hide
          rohithsharma Rohith Sharma K S added a comment -

          +1 LGTM, I will commit it later of the day if no more objections

          Show
          rohithsharma Rohith Sharma K S added a comment - +1 LGTM, I will commit it later of the day if no more objections
          Hide
          jianhe Jian He added a comment -

          Could such cases be handled in next allocation / preemption cycle itself.

          The capacity could even go over max-capacity, which violates capacity guarantees. Anyway, this is a different question.

          The patch itself looks good to me.

          Show
          jianhe Jian He added a comment - Could such cases be handled in next allocation / preemption cycle itself. The capacity could even go over max-capacity, which violates capacity guarantees. Anyway, this is a different question. The patch itself looks good to me.
          Hide
          rohithsharma Rohith Sharma K S added a comment -

          committed to trunk.. the patch do not apply to branch-2. Sunil G could you give patch to branch-2?

          Show
          rohithsharma Rohith Sharma K S added a comment - committed to trunk.. the patch do not apply to branch-2. Sunil G could you give patch to branch-2?
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10958 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10958/)
          YARN-5932. Retrospect moveApplicationToQueue in align with YARN-5611. (rohithsharmaks: rev 563480dccd0136d82730f4228f1df44449ed5822)

          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java
          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAppManager.java
          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairScheduler.java
          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAppManagerEventType.java
          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/ParentQueue.java
          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CSQueue.java
          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAppManagerEvent.java
          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestMoveApplication.java
          • (delete) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/RMAppMoveEvent.java
          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/YarnScheduler.java
          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ClientRMService.java
          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/RMAppEventType.java
          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestCapacitySchedulerNodeLabelUpdate.java
          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/AbstractYarnScheduler.java
          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/RMAppImpl.java
          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/AbstractCSQueue.java
          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10958 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10958/ ) YARN-5932 . Retrospect moveApplicationToQueue in align with YARN-5611 . (rohithsharmaks: rev 563480dccd0136d82730f4228f1df44449ed5822) (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAppManager.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairScheduler.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAppManagerEventType.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/ParentQueue.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CSQueue.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAppManagerEvent.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestMoveApplication.java (delete) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/RMAppMoveEvent.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/YarnScheduler.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ClientRMService.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/RMAppEventType.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestCapacitySchedulerNodeLabelUpdate.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/AbstractYarnScheduler.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/RMAppImpl.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/AbstractCSQueue.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.java
          Hide
          sunilg Sunil G added a comment -

          Thanks Rohith Sharma K S.
          Attaching patch for branch-2.

          Show
          sunilg Sunil G added a comment - Thanks Rohith Sharma K S . Attaching patch for branch-2.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 21s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
          +1 mvninstall 8m 56s branch-2 passed
          +1 compile 0m 29s branch-2 passed with JDK v1.8.0_111
          +1 compile 0m 32s branch-2 passed with JDK v1.7.0_121
          +1 checkstyle 0m 30s branch-2 passed
          +1 mvnsite 0m 40s branch-2 passed
          +1 mvneclipse 0m 19s branch-2 passed
          +1 findbugs 1m 12s branch-2 passed
          +1 javadoc 0m 21s branch-2 passed with JDK v1.8.0_111
          +1 javadoc 0m 24s branch-2 passed with JDK v1.7.0_121
          +1 mvninstall 0m 31s the patch passed
          +1 compile 0m 27s the patch passed with JDK v1.8.0_111
          +1 javac 0m 27s the patch passed
          +1 compile 0m 30s the patch passed with JDK v1.7.0_121
          +1 javac 0m 30s the patch passed
          -0 checkstyle 0m 26s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 3 new + 525 unchanged - 15 fixed = 528 total (was 540)
          +1 mvnsite 0m 36s the patch passed
          +1 mvneclipse 0m 14s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 1m 18s the patch passed
          +1 javadoc 0m 18s the patch passed with JDK v1.8.0_111
          +1 javadoc 0m 22s the patch passed with JDK v1.7.0_121
          -1 unit 43m 43s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.7.0_121.
          +1 asflicense 0m 17s The patch does not generate ASF License warnings.
          106m 21s



          Reason Tests
          JDK v1.7.0_121 Failed junit tests hadoop.yarn.server.resourcemanager.TestRMRestart
            hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:b59b8b7
          JIRA Issue YARN-5932
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12842119/YARN-5932-branch-2.0001.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 6a3d2990642a 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision branch-2 / 3350d0c
          Default Java 1.7.0_121
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_111 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_121
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/14205/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/14205/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_121.txt
          JDK v1.7.0_121 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/14205/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/14205/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 21s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. +1 mvninstall 8m 56s branch-2 passed +1 compile 0m 29s branch-2 passed with JDK v1.8.0_111 +1 compile 0m 32s branch-2 passed with JDK v1.7.0_121 +1 checkstyle 0m 30s branch-2 passed +1 mvnsite 0m 40s branch-2 passed +1 mvneclipse 0m 19s branch-2 passed +1 findbugs 1m 12s branch-2 passed +1 javadoc 0m 21s branch-2 passed with JDK v1.8.0_111 +1 javadoc 0m 24s branch-2 passed with JDK v1.7.0_121 +1 mvninstall 0m 31s the patch passed +1 compile 0m 27s the patch passed with JDK v1.8.0_111 +1 javac 0m 27s the patch passed +1 compile 0m 30s the patch passed with JDK v1.7.0_121 +1 javac 0m 30s the patch passed -0 checkstyle 0m 26s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 3 new + 525 unchanged - 15 fixed = 528 total (was 540) +1 mvnsite 0m 36s the patch passed +1 mvneclipse 0m 14s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 18s the patch passed +1 javadoc 0m 18s the patch passed with JDK v1.8.0_111 +1 javadoc 0m 22s the patch passed with JDK v1.7.0_121 -1 unit 43m 43s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.7.0_121. +1 asflicense 0m 17s The patch does not generate ASF License warnings. 106m 21s Reason Tests JDK v1.7.0_121 Failed junit tests hadoop.yarn.server.resourcemanager.TestRMRestart   hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer Subsystem Report/Notes Docker Image:yetus/hadoop:b59b8b7 JIRA Issue YARN-5932 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12842119/YARN-5932-branch-2.0001.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 6a3d2990642a 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision branch-2 / 3350d0c Default Java 1.7.0_121 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_111 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_121 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/14205/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/14205/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_121.txt JDK v1.7.0_121 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/14205/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/14205/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          sunilg Sunil G added a comment -

          Test case failures are unrelated and tracked via separate tickets.

          Show
          sunilg Sunil G added a comment - Test case failures are unrelated and tracked via separate tickets.
          Hide
          rohithsharma Rohith Sharma K S added a comment -

          committed to trunk/banch-2.. thanks Sunil for the patch and Jian for the review.

          Show
          rohithsharma Rohith Sharma K S added a comment - committed to trunk/banch-2.. thanks Sunil for the patch and Jian for the review.

            People

            • Assignee:
              sunilg Sunil G
              Reporter:
              sunilg Sunil G
            • Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development