Details

    • Hadoop Flags:
      Reviewed

      Description

      Today, we could add or modify queues without restarting the RM, via a CS refresh. But for deleting queue, we have to restart the ResourceManager. We could support for deleting queues without requiring a RM restart

      1. YARN-5556.v1.001.patch
        12 kB
        Naganarasimha G R
      2. YARN-5556.v1.002.patch
        24 kB
        Naganarasimha G R
      3. YARN-5556.v1.003.patch
        24 kB
        Naganarasimha G R
      4. YARN-5556.v1.004.patch
        24 kB
        Naganarasimha G R
      5. YARN-5556.v2.005.patch
        20 kB
        Naganarasimha G R
      6. YARN-5556.v2.006.patch
        22 kB
        Naganarasimha G R

        Issue Links

          Activity

          Hide
          Naganarasimha Naganarasimha G R added a comment -

          Hi Xuan Gong,
          We were already working on this, is it ok we take this work up ?

          Show
          Naganarasimha Naganarasimha G R added a comment - Hi Xuan Gong , We were already working on this, is it ok we take this work up ?
          Hide
          Naganarasimha Naganarasimha G R added a comment -

          Basically we wanted to support as part of refreshQueues, if there are no apps running in a queue we should be able to delete the queue and by default if some apps (running or pending) we intend to fail the refresh,
          Further we planned to give a option to forceKill to kill the apps of the queue, which by default is false. Thoughts?

          Show
          Naganarasimha Naganarasimha G R added a comment - Basically we wanted to support as part of refreshQueues , if there are no apps running in a queue we should be able to delete the queue and by default if some apps (running or pending) we intend to fail the refresh, Further we planned to give a option to forceKill to kill the apps of the queue, which by default is false. Thoughts?
          Hide
          Naganarasimha Naganarasimha G R added a comment - - edited

          Hi Xuan Gong, as discussed offline i am taking over this jira and here i have give a basic implementation that leaf queue's can be deleted if there are no applications associated with it. If further options are required as mentioned in previous comment then i can work on it, else will add some test cases to complete the patch. Have tested the patch in local pseudo cluster.

          Show
          Naganarasimha Naganarasimha G R added a comment - - edited Hi Xuan Gong , as discussed offline i am taking over this jira and here i have give a basic implementation that leaf queue's can be deleted if there are no applications associated with it. If further options are required as mentioned in previous comment then i can work on it, else will add some test cases to complete the patch. Have tested the patch in local pseudo cluster.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 14m 11s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 mvninstall 8m 56s trunk passed
          +1 compile 0m 43s trunk passed
          +1 checkstyle 0m 27s trunk passed
          +1 mvnsite 0m 47s trunk passed
          +1 mvneclipse 0m 20s trunk passed
          +1 findbugs 1m 12s trunk passed
          +1 javadoc 0m 27s trunk passed
          +1 mvninstall 0m 38s the patch passed
          +1 compile 0m 40s the patch passed
          +1 javac 0m 40s the patch passed
          -1 checkstyle 0m 26s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 4 new + 239 unchanged - 0 fixed = 243 total (was 239)
          +1 mvnsite 0m 48s the patch passed
          +1 mvneclipse 0m 18s the patch passed
          -1 whitespace 0m 0s The patch has 4 line(s) that end in whitespace. Use git apply --whitespace=fix.
          +1 findbugs 1m 24s the patch passed
          +1 javadoc 0m 22s the patch passed
          -1 unit 36m 57s hadoop-yarn-server-resourcemanager in the patch failed.
          +1 asflicense 0m 19s The patch does not generate ASF License warnings.
          69m 45s



          Reason Tests
          Failed junit tests hadoop.yarn.server.resourcemanager.scheduler.capacity.TestNodeLabelContainerAllocation
            hadoop.yarn.server.resourcemanager.TestRMAdminService
            hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacitySchedulerDynamicBehavior
            hadoop.yarn.server.resourcemanager.TestRM



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12830409/YARN-5556.v1.001.patch
          JIRA Issue YARN-5556
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 3fe190dbc274 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 8e06d86
          Default Java 1.8.0_101
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/13222/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
          whitespace https://builds.apache.org/job/PreCommit-YARN-Build/13222/artifact/patchprocess/whitespace-eol.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/13222/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
          unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/13222/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13222/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/13222/console
          Powered by Apache Yetus 0.3.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 14m 11s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 8m 56s trunk passed +1 compile 0m 43s trunk passed +1 checkstyle 0m 27s trunk passed +1 mvnsite 0m 47s trunk passed +1 mvneclipse 0m 20s trunk passed +1 findbugs 1m 12s trunk passed +1 javadoc 0m 27s trunk passed +1 mvninstall 0m 38s the patch passed +1 compile 0m 40s the patch passed +1 javac 0m 40s the patch passed -1 checkstyle 0m 26s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 4 new + 239 unchanged - 0 fixed = 243 total (was 239) +1 mvnsite 0m 48s the patch passed +1 mvneclipse 0m 18s the patch passed -1 whitespace 0m 0s The patch has 4 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 findbugs 1m 24s the patch passed +1 javadoc 0m 22s the patch passed -1 unit 36m 57s hadoop-yarn-server-resourcemanager in the patch failed. +1 asflicense 0m 19s The patch does not generate ASF License warnings. 69m 45s Reason Tests Failed junit tests hadoop.yarn.server.resourcemanager.scheduler.capacity.TestNodeLabelContainerAllocation   hadoop.yarn.server.resourcemanager.TestRMAdminService   hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacitySchedulerDynamicBehavior   hadoop.yarn.server.resourcemanager.TestRM Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12830409/YARN-5556.v1.001.patch JIRA Issue YARN-5556 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 3fe190dbc274 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 8e06d86 Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/13222/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt whitespace https://builds.apache.org/job/PreCommit-YARN-Build/13222/artifact/patchprocess/whitespace-eol.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/13222/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/13222/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13222/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/13222/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
          Hide
          Naganarasimha Naganarasimha G R added a comment -

          Xuan Gong, i think you wanted to update target version as 2.9 right ? seems like you have updated the wrong field and also may be you can review the patch most of the test case doesnt seem to relate to patch.

          Show
          Naganarasimha Naganarasimha G R added a comment - Xuan Gong , i think you wanted to update target version as 2.9 right ? seems like you have updated the wrong field and also may be you can review the patch most of the test case doesnt seem to relate to patch.
          Hide
          xgong Xuan Gong added a comment -

          Naganarasimha G R
          Thanks for the patch. We could use it as the initial step. Will create the following tickets, such as kill the running apps or drain the apps.

          a basic implementation that leaf queue's can be deleted if there are no applications associated with it.

          Let us also support deleting the parent queue here.

          Show
          xgong Xuan Gong added a comment - Naganarasimha G R Thanks for the patch. We could use it as the initial step. Will create the following tickets, such as kill the running apps or drain the apps. a basic implementation that leaf queue's can be deleted if there are no applications associated with it. Let us also support deleting the parent queue here.
          Hide
          Naganarasimha Naganarasimha G R added a comment -

          Thanks for the comment Xuan Gong, i am assuming we can delete the parent queue when none of its children(or its children) have running or pending apps with them. right ?

          Show
          Naganarasimha Naganarasimha G R added a comment - Thanks for the comment Xuan Gong , i am assuming we can delete the parent queue when none of its children(or its children) have running or pending apps with them. right ?
          Hide
          xgong Xuan Gong added a comment -

          Naganarasimha G R

          we can delete the parent queue when none of its children(or its children) have running or pending apps with them. right ?

          Yes.

          Show
          xgong Xuan Gong added a comment - Naganarasimha G R we can delete the parent queue when none of its children(or its children) have running or pending apps with them. right ? Yes.
          Hide
          xgong Xuan Gong added a comment -

          Naganarasimha G R

          Any updates ?

          Show
          xgong Xuan Gong added a comment - Naganarasimha G R Any updates ?
          Hide
          naganarasimha_gr@apache.org Naganarasimha Garla added a comment -

          Sure working on the testcase will get it out today

          Show
          naganarasimha_gr@apache.org Naganarasimha Garla added a comment - Sure working on the testcase will get it out today
          Hide
          Naganarasimha Naganarasimha G R added a comment -

          Hi Xuan Gong, i have updated the patch with delete for parent queue as discussed along with test cases covering the scenarios.

          Show
          Naganarasimha Naganarasimha G R added a comment - Hi Xuan Gong , i have updated the patch with delete for parent queue as discussed along with test cases covering the scenarios.
          Hide
          Naganarasimha Naganarasimha G R added a comment -

          Triggered the build manually not sure why it dint trigger automatically, may be Xuan Gong you too could take a look if you require it ?
          Also shall i raise jira's and start working for other issues which we discussed ?

          Show
          Naganarasimha Naganarasimha G R added a comment - Triggered the build manually not sure why it dint trigger automatically, may be Xuan Gong you too could take a look if you require it ? Also shall i raise jira's and start working for other issues which we discussed ?
          Hide
          Naganarasimha Naganarasimha G R added a comment -

          reattaching the patch not sure why jenkins is not picking even after manually triggering

          Show
          Naganarasimha Naganarasimha G R added a comment - reattaching the patch not sure why jenkins is not picking even after manually triggering
          Hide
          Naganarasimha Naganarasimha G R added a comment -

          Xuan Gong, any cycles for reviewing the patch ?

          Show
          Naganarasimha Naganarasimha G R added a comment - Xuan Gong , any cycles for reviewing the patch ?
          Hide
          templedf Daniel Templeton added a comment -

          Thanks for working on this, Naganarasimha G R. Some comments:

          • In CSQueue, the javadoc comment should end with a period:
               * remove the specified child queue
          • In your message; " will be deleted as it as not present in the latest xml", I'd make a couple changes. Instead of future tense ("will"), I'd make it present tense ("deleting queue because..."). Also, I'd be more specific than "latest xml"; give the file name.
          • In your message " cannot be deleted as one of its children" + queueHavingAppsRunning + " has apps.", I'd say "at least one", and I'd put the queue name in parentheses or put it at the end after a colon. Depending on the queue name, the sentence could otherwise be hard to parse. Finally, I'd say "has active applications" or "has running applications".
          • There's a typo in "Queue " + queueName + " cannot be deleted as it as " – as/has
          • In TestCapacityScheduler, after deleting the parent queue, you should also check to see that the child queue has been deleted.
          Show
          templedf Daniel Templeton added a comment - Thanks for working on this, Naganarasimha G R . Some comments: In CSQueue , the javadoc comment should end with a period: * remove the specified child queue In your message; " will be deleted as it as not present in the latest xml" , I'd make a couple changes. Instead of future tense ("will"), I'd make it present tense ("deleting queue because..."). Also, I'd be more specific than "latest xml"; give the file name. In your message " cannot be deleted as one of its children" + queueHavingAppsRunning + " has apps." , I'd say "at least one", and I'd put the queue name in parentheses or put it at the end after a colon. Depending on the queue name, the sentence could otherwise be hard to parse. Finally, I'd say "has active applications" or "has running applications". There's a typo in "Queue " + queueName + " cannot be deleted as it as " – as/has In TestCapacityScheduler , after deleting the parent queue, you should also check to see that the child queue has been deleted.
          Hide
          Naganarasimha Naganarasimha G R added a comment -

          Thanks for the review Daniel Templeton, have attached a patch after fixing your review comments

          Show
          Naganarasimha Naganarasimha G R added a comment - Thanks for the review Daniel Templeton , have attached a patch after fixing your review comments
          Hide
          templedf Daniel Templeton added a comment -

          That'll work for me, except for one tiny typo:

                            + " cannot be deleted as atleast one of its children has"

          atleast/at least

          Show
          templedf Daniel Templeton added a comment - That'll work for me, except for one tiny typo: + " cannot be deleted as atleast one of its children has" atleast/at least
          Hide
          Naganarasimha Naganarasimha G R added a comment -

          Thanks Daniel Templeton, attaching the patch for the nit which you mentioned earlier

          Show
          Naganarasimha Naganarasimha G R added a comment - Thanks Daniel Templeton , attaching the patch for the nit which you mentioned earlier
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 18s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 6m 44s trunk passed
          +1 compile 0m 32s trunk passed
          +1 checkstyle 0m 25s trunk passed
          +1 mvnsite 0m 37s trunk passed
          +1 mvneclipse 0m 17s trunk passed
          +1 findbugs 0m 56s trunk passed
          +1 javadoc 0m 20s trunk passed
          +1 mvninstall 0m 31s the patch passed
          +1 compile 0m 29s the patch passed
          +1 javac 0m 29s the patch passed
          -0 checkstyle 0m 22s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 6 new + 390 unchanged - 0 fixed = 396 total (was 390)
          +1 mvnsite 0m 36s the patch passed
          +1 mvneclipse 0m 14s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 1m 1s the patch passed
          +1 javadoc 0m 19s the patch passed
          -1 unit 38m 26s hadoop-yarn-server-resourcemanager in the patch failed.
          +1 asflicense 0m 15s The patch does not generate ASF License warnings.
          53m 37s



          Reason Tests
          Failed junit tests hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacitySchedulerDynamicBehavior



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Issue YARN-5556
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12836091/YARN-5556.v1.004.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 7defa0b60da8 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / aa3cab1
          Default Java 1.8.0_101
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/13683/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/13683/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13683/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/13683/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 18s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 6m 44s trunk passed +1 compile 0m 32s trunk passed +1 checkstyle 0m 25s trunk passed +1 mvnsite 0m 37s trunk passed +1 mvneclipse 0m 17s trunk passed +1 findbugs 0m 56s trunk passed +1 javadoc 0m 20s trunk passed +1 mvninstall 0m 31s the patch passed +1 compile 0m 29s the patch passed +1 javac 0m 29s the patch passed -0 checkstyle 0m 22s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 6 new + 390 unchanged - 0 fixed = 396 total (was 390) +1 mvnsite 0m 36s the patch passed +1 mvneclipse 0m 14s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 1s the patch passed +1 javadoc 0m 19s the patch passed -1 unit 38m 26s hadoop-yarn-server-resourcemanager in the patch failed. +1 asflicense 0m 15s The patch does not generate ASF License warnings. 53m 37s Reason Tests Failed junit tests hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacitySchedulerDynamicBehavior Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Issue YARN-5556 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12836091/YARN-5556.v1.004.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 7defa0b60da8 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / aa3cab1 Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/13683/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/13683/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13683/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/13683/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 22s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 6m 50s trunk passed
          +1 compile 0m 32s trunk passed
          +1 checkstyle 0m 25s trunk passed
          +1 mvnsite 0m 38s trunk passed
          +1 mvneclipse 0m 17s trunk passed
          +1 findbugs 0m 59s trunk passed
          +1 javadoc 0m 21s trunk passed
          +1 mvninstall 0m 31s the patch passed
          +1 compile 0m 30s the patch passed
          +1 javac 0m 30s the patch passed
          -0 checkstyle 0m 23s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 6 new + 390 unchanged - 0 fixed = 396 total (was 390)
          +1 mvnsite 0m 35s the patch passed
          +1 mvneclipse 0m 14s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 1m 3s the patch passed
          +1 javadoc 0m 18s the patch passed
          -1 unit 35m 10s hadoop-yarn-server-resourcemanager in the patch failed.
          +1 asflicense 0m 15s The patch does not generate ASF License warnings.
          50m 44s



          Reason Tests
          Failed junit tests hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacitySchedulerDynamicBehavior



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Issue YARN-5556
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12836091/YARN-5556.v1.004.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 6c1dabdb04c3 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / e0bebbb
          Default Java 1.8.0_101
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/13691/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/13691/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13691/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/13691/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 22s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 6m 50s trunk passed +1 compile 0m 32s trunk passed +1 checkstyle 0m 25s trunk passed +1 mvnsite 0m 38s trunk passed +1 mvneclipse 0m 17s trunk passed +1 findbugs 0m 59s trunk passed +1 javadoc 0m 21s trunk passed +1 mvninstall 0m 31s the patch passed +1 compile 0m 30s the patch passed +1 javac 0m 30s the patch passed -0 checkstyle 0m 23s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 6 new + 390 unchanged - 0 fixed = 396 total (was 390) +1 mvnsite 0m 35s the patch passed +1 mvneclipse 0m 14s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 3s the patch passed +1 javadoc 0m 18s the patch passed -1 unit 35m 10s hadoop-yarn-server-resourcemanager in the patch failed. +1 asflicense 0m 15s The patch does not generate ASF License warnings. 50m 44s Reason Tests Failed junit tests hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacitySchedulerDynamicBehavior Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Issue YARN-5556 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12836091/YARN-5556.v1.004.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 6c1dabdb04c3 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / e0bebbb Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/13691/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/13691/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13691/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/13691/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          templedf Daniel Templeton added a comment -

          Thanks, Naganarasimha G R. The latest patch looks good to me. +1 I'll commit tomorrow if you haven't already.

          Show
          templedf Daniel Templeton added a comment - Thanks, Naganarasimha G R . The latest patch looks good to me. +1 I'll commit tomorrow if you haven't already.
          Hide
          Naganarasimha Naganarasimha G R added a comment -

          Thanks Daniel Templeton, please go ahead and commit, was anyway waiting for one of you guys to give +1 on it.

          Show
          Naganarasimha Naganarasimha G R added a comment - Thanks Daniel Templeton , please go ahead and commit, was anyway waiting for one of you guys to give +1 on it.
          Hide
          leftnoteasy Wangda Tan added a comment -

          Sorry for jumping in late, not sure if deleting queues should use YARN-5746. According to design doc of YARN-5746, queue could be deleted only if queue's state becomes STOPPED.

          Show
          leftnoteasy Wangda Tan added a comment - Sorry for jumping in late, not sure if deleting queues should use YARN-5746 . According to design doc of YARN-5746 , queue could be deleted only if queue's state becomes STOPPED.
          Hide
          Naganarasimha Naganarasimha G R added a comment -

          Thanks Tan, Wangda,
          IIUC its design is in YARN-5724, earlier was wondering whether this will take considerable amount of time hence let the basic version of it go in to 2.8 version and based on the new design achieve it in a more better way for further versions...
          Xuan Gong, Which version are we targeting this modification to go in ?

          Show
          Naganarasimha Naganarasimha G R added a comment - Thanks Tan, Wangda , IIUC its design is in YARN-5724 , earlier was wondering whether this will take considerable amount of time hence let the basic version of it go in to 2.8 version and based on the new design achieve it in a more better way for further versions... Xuan Gong , Which version are we targeting this modification to go in ?
          Hide
          Naganarasimha Naganarasimha G R added a comment -

          Hi Xuan Gong & Tan, Wangda, please advice whether to have similar implementation as captured in design or get some work around as captured in the patch for 2.8 release would be helpfull ?
          Based on this i can cancel the patch !

          Show
          Naganarasimha Naganarasimha G R added a comment - Hi Xuan Gong & Tan, Wangda , please advice whether to have similar implementation as captured in design or get some work around as captured in the patch for 2.8 release would be helpfull ? Based on this i can cancel the patch !
          Hide
          leftnoteasy Wangda Tan added a comment -

          Thanks Naganarasimha G R,

          I'm OK with pulling YARN-5761 into branch-2.8 (assume this patch will build on top of YARN-5761).

          Some comments:
          1) Verify no app running is not enough – we need also check queue is in stopped state. so I think YARN-5746 is required for this patch.
          2) removeChildQueue should be method of ParentQueue only.

          So I think it's better to wait YARN-5746 / YARN-5761 committed before working on this patch. Which should happen soon.

          Thoughts?

          Show
          leftnoteasy Wangda Tan added a comment - Thanks Naganarasimha G R , I'm OK with pulling YARN-5761 into branch-2.8 (assume this patch will build on top of YARN-5761 ). Some comments: 1) Verify no app running is not enough – we need also check queue is in stopped state. so I think YARN-5746 is required for this patch. 2) removeChildQueue should be method of ParentQueue only. So I think it's better to wait YARN-5746 / YARN-5761 committed before working on this patch. Which should happen soon. Thoughts?
          Hide
          Naganarasimha Naganarasimha G R added a comment -

          Thanks Tan, Wangda,
          In that case i will wait for YARN-5746 / YARN-5761 and also pitch in the reviews to speed it up to 2.8. Cancelling the patch

          Show
          Naganarasimha Naganarasimha G R added a comment - Thanks Tan, Wangda , In that case i will wait for YARN-5746 / YARN-5761 and also pitch in the reviews to speed it up to 2.8. Cancelling the patch
          Hide
          xgong Xuan Gong added a comment -

          Naganarasimha G R
          Given all the dependent patches have been committed, could you rebase the patch, please ?

          Show
          xgong Xuan Gong added a comment - Naganarasimha G R Given all the dependent patches have been committed, could you rebase the patch, please ?
          Hide
          naganarasimha_gr@apache.org Naganarasimha Garla added a comment -

          Thanks Xuan, will start working on it and update the patch at the
          earliest...

          Show
          naganarasimha_gr@apache.org Naganarasimha Garla added a comment - Thanks Xuan, will start working on it and update the patch at the earliest...
          Hide
          Naganarasimha Naganarasimha G R added a comment -

          Hi Xuan Gong and Tan, Wangda,
          I have few queries based on the design document attached in YARN-5724,

          1. So user needs to delete a queue(say a2) then he needs to remove the queue from its parent's "yarn.scheduler.capacity.<parent queue>.queues" config and also mention its state(yarn.scheduler.capacity.<root...a2>.state) as DELETED right ?
          2. How to delete intermediate queues? i presume we need NOT configure state for each of its children right ? or do we plan to support delete of only leaf queue?
          3. Do we need to consider the moving of queues(along with its apps) from one queue hiearchy to another ? IMO it complicates but not sure about the real world usecases.
          4. In case of HA, i think it further complicates as if both the RM's are initialiased with old queue settings and then if new queue is updated then CS is aware of deleted queue else if the RM starts of with updated xml(with deleted queue) then deleted queue information is not available and if failover happens to this RM then apps running on the deleted queue cannot be recovered as the queue doesnt exist. so do we need to start maintaining the deleted queue in statestore or need handling of creating queue objects for the queues whose state has been marked as deleted (then we need to consider 2nd point) ?
          5. More of a test scenario, i have a queue with apps running, now i delete the queue which will make it go into drain state (or new state as Deleted but queue is not deleted until all apps under it are finished) but apps take some time to finish and now xml is again updated with new queue which has same name and path as of one which was deleted earlier, so do we need to support addition of this new queue or dont allow if earlier queue is in process of deletion ?
          6. do we need to consider showing of the deleted queues in the webui ? may be in another jira but the code needs to be updated.
          7. for the below comment in the doc :

            "For the resources of the deleted/stopped queue, users should explicitly distribute them away to its siblings."

          • While we allow running apps to complete do we allow pending container requests to be catered for these apps ? if so deleted queue's capacity is considered to be 0% and max cap as per its config ?
          • if deleted queue's capacity is not explicitly redistributed to its siblings i presume we need to throw exception ?
          • Do we need to consider preemption of resources from these deleted queues when there is shortage of resources?
          • What should happen to the pending(not activated) apps in the queue, kill or give a chance to complere like running apps ?

          I can port/rebase based on my original patch (which just goes ahead and deletes the queue if apps are not present), but i presume that scope has got changed hence once these points are clarified we can decide the scope of this jira and then will work upon it.

          Show
          Naganarasimha Naganarasimha G R added a comment - Hi Xuan Gong and Tan, Wangda , I have few queries based on the design document attached in YARN-5724 , So user needs to delete a queue(say a2) then he needs to remove the queue from its parent's "yarn.scheduler.capacity.<parent queue>.queues" config and also mention its state(yarn.scheduler.capacity.<root...a2>.state) as DELETED right ? How to delete intermediate queues? i presume we need NOT configure state for each of its children right ? or do we plan to support delete of only leaf queue? Do we need to consider the moving of queues(along with its apps) from one queue hiearchy to another ? IMO it complicates but not sure about the real world usecases. In case of HA, i think it further complicates as if both the RM's are initialiased with old queue settings and then if new queue is updated then CS is aware of deleted queue else if the RM starts of with updated xml(with deleted queue) then deleted queue information is not available and if failover happens to this RM then apps running on the deleted queue cannot be recovered as the queue doesnt exist. so do we need to start maintaining the deleted queue in statestore or need handling of creating queue objects for the queues whose state has been marked as deleted (then we need to consider 2nd point) ? More of a test scenario, i have a queue with apps running, now i delete the queue which will make it go into drain state (or new state as Deleted but queue is not deleted until all apps under it are finished) but apps take some time to finish and now xml is again updated with new queue which has same name and path as of one which was deleted earlier, so do we need to support addition of this new queue or dont allow if earlier queue is in process of deletion ? do we need to consider showing of the deleted queues in the webui ? may be in another jira but the code needs to be updated. for the below comment in the doc : "For the resources of the deleted/stopped queue, users should explicitly distribute them away to its siblings." While we allow running apps to complete do we allow pending container requests to be catered for these apps ? if so deleted queue's capacity is considered to be 0% and max cap as per its config ? if deleted queue's capacity is not explicitly redistributed to its siblings i presume we need to throw exception ? Do we need to consider preemption of resources from these deleted queues when there is shortage of resources? What should happen to the pending(not activated) apps in the queue, kill or give a chance to complere like running apps ? I can port/rebase based on my original patch (which just goes ahead and deletes the queue if apps are not present), but i presume that scope has got changed hence once these points are clarified we can decide the scope of this jira and then will work upon it.
          Hide
          xgong Xuan Gong added a comment -

          Naganarasimha G R Thanks for the comments

          Wangda Tan Please comment if you have any further suggestions.

          So user needs to delete a queue(say a2) then he needs to remove the queue from its parent's "yarn.scheduler.capacity.<parent queue>.queues" config and also mention its state(yarn.scheduler.capacity.<root...a2>.state) as DELETED right ?

          Do not need to remove the queue from its parent's "yarn.scheduler.capacity.<parent queue>.queues" config, just mention its state(yarn.scheduler.capacity.<root...a2>.state) as DELETED.

          How to delete intermediate queues? i presume we need NOT configure state for each of its children right ? or do we plan to support delete of only leaf queue?

          We need not configure the state for each of its children. Just mark delete for the queue itself.

          Do we need to consider the moving of queues(along with its apps) from one queue hiearchy to another ? IMO it complicates but not sure about the real world usecases.

          we can consider this scenario later.

          In case of HA, i think it further complicates as if both the RM's are initialiased with old queue settings and then if new queue is updated then CS is aware of deleted queue else if the RM starts of with updated xml(with deleted queue) then deleted queue information is not available and if failover happens to this RM then apps running on the deleted queue cannot be recovered as the queue doesnt exist. so do we need to start maintaining the deleted queue in statestore or need handling of creating queue objects for the queues whose state has been marked as deleted (then we need to consider 2nd point) ?

          Yes, this is the fundamental issue with the "configuration-based" approach. This api-based approach would solve this issue: https://issues.apache.org/jira/browse/YARN-5734. But for "configuration-based" approach, in RM HA case, we have to make sure the configuration file for every RM nodes is updated.

          do we need to consider showing of the deleted queues in the webui ? may be in another jira but the code needs to be updated.

          Yes, we could file a separate jira, and do it later.

          The basic workflow could be: before we can actually delete the queue, we should make sure the queue in STOPPED state which means this queue can not accept any new applications, and all apps (including pending request) have been finished (for now, we could simply wait. or add a command/flag to force kill later). Then, we could delete the queue and split capacity.

          Thanks

          Xuan Gong

          Show
          xgong Xuan Gong added a comment - Naganarasimha G R Thanks for the comments Wangda Tan Please comment if you have any further suggestions. So user needs to delete a queue(say a2) then he needs to remove the queue from its parent's "yarn.scheduler.capacity.<parent queue>.queues" config and also mention its state(yarn.scheduler.capacity.<root...a2>.state) as DELETED right ? Do not need to remove the queue from its parent's "yarn.scheduler.capacity.<parent queue>.queues" config, just mention its state(yarn.scheduler.capacity.<root...a2>.state) as DELETED. How to delete intermediate queues? i presume we need NOT configure state for each of its children right ? or do we plan to support delete of only leaf queue? We need not configure the state for each of its children. Just mark delete for the queue itself. Do we need to consider the moving of queues(along with its apps) from one queue hiearchy to another ? IMO it complicates but not sure about the real world usecases. we can consider this scenario later. In case of HA, i think it further complicates as if both the RM's are initialiased with old queue settings and then if new queue is updated then CS is aware of deleted queue else if the RM starts of with updated xml(with deleted queue) then deleted queue information is not available and if failover happens to this RM then apps running on the deleted queue cannot be recovered as the queue doesnt exist. so do we need to start maintaining the deleted queue in statestore or need handling of creating queue objects for the queues whose state has been marked as deleted (then we need to consider 2nd point) ? Yes, this is the fundamental issue with the "configuration-based" approach. This api-based approach would solve this issue: https://issues.apache.org/jira/browse/YARN-5734 . But for "configuration-based" approach, in RM HA case, we have to make sure the configuration file for every RM nodes is updated. do we need to consider showing of the deleted queues in the webui ? may be in another jira but the code needs to be updated. Yes, we could file a separate jira, and do it later. The basic workflow could be: before we can actually delete the queue, we should make sure the queue in STOPPED state which means this queue can not accept any new applications, and all apps (including pending request) have been finished (for now, we could simply wait. or add a command/flag to force kill later). Then, we could delete the queue and split capacity. Thanks Xuan Gong
          Hide
          leftnoteasy Wangda Tan added a comment -

          Just offline discussed with Xuan Gong.

          I think we don't need the additional DELETED state, first it generate some maintenance overheads, for example we need to maintain state transition to/from of the DELETED state. And since by design a queue can be deleted only if queue is stopped and no app running, so the impact of typo should be minimum. Our preference is simply remove queue from config.

          And for re-distribution of stopped/deleted queue. For delete queue it should be obvious, since the queue is gone, sum of its siblings should be 100. For stopped queue, our expectation is, it will be reactivated at some time. So it will be better to keep the capacity as-is, and admin can update max-capacity of its siblings to make sure queue capacity can be utilized.

          I think we need to update design doc to make it up-to-date.

          Thoughts?

          Show
          leftnoteasy Wangda Tan added a comment - Just offline discussed with Xuan Gong . I think we don't need the additional DELETED state, first it generate some maintenance overheads, for example we need to maintain state transition to/from of the DELETED state. And since by design a queue can be deleted only if queue is stopped and no app running, so the impact of typo should be minimum. Our preference is simply remove queue from config. And for re-distribution of stopped/deleted queue. For delete queue it should be obvious, since the queue is gone, sum of its siblings should be 100. For stopped queue, our expectation is, it will be reactivated at some time. So it will be better to keep the capacity as-is, and admin can update max-capacity of its siblings to make sure queue capacity can be utilized. I think we need to update design doc to make it up-to-date. Thoughts?
          Hide
          xgong Xuan Gong added a comment -

          sounds good. Will update the design doc

          Show
          xgong Xuan Gong added a comment - sounds good. Will update the design doc
          Hide
          Naganarasimha Naganarasimha G R added a comment -

          Thanks Tan, Wangda & Xuan Gong,

          I think we don't need the additional DELETED state, first it generate some maintenance overheads, for example we need to maintain state transition to/from of the DELETED state. And since by design a queue can be deleted only if queue is stopped and no app running, so the impact of typo should be minimum. Our preference is simply remove queue from config.

          This makes the things pretty clear and straight forward for users/admin, and clears all my queries. Will modify and upload the patch at the earliest.

          Show
          Naganarasimha Naganarasimha G R added a comment - Thanks Tan, Wangda & Xuan Gong , I think we don't need the additional DELETED state, first it generate some maintenance overheads, for example we need to maintain state transition to/from of the DELETED state. And since by design a queue can be deleted only if queue is stopped and no app running, so the impact of typo should be minimum. Our preference is simply remove queue from config. This makes the things pretty clear and straight forward for users/admin, and clears all my queries. Will modify and upload the patch at the earliest.
          Hide
          Naganarasimha Naganarasimha G R added a comment -

          Attached the patch based on new approach !

          Show
          Naganarasimha Naganarasimha G R added a comment - Attached the patch based on new approach !
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 18s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 14m 20s trunk passed
          +1 compile 0m 38s trunk passed
          +1 checkstyle 0m 26s trunk passed
          +1 mvnsite 0m 42s trunk passed
          +1 mvneclipse 0m 19s trunk passed
          +1 findbugs 1m 9s trunk passed
          +1 javadoc 0m 24s trunk passed
          +1 mvninstall 0m 43s the patch passed
          +1 compile 0m 33s the patch passed
          +1 javac 0m 33s the patch passed
          -0 checkstyle 0m 22s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 6 new + 207 unchanged - 2 fixed = 213 total (was 209)
          +1 mvnsite 0m 34s the patch passed
          +1 mvneclipse 0m 14s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 1m 12s the patch passed
          +1 javadoc 0m 20s the patch passed
          -1 unit 42m 21s hadoop-yarn-server-resourcemanager in the patch failed.
          +1 asflicense 0m 21s The patch does not generate ASF License warnings.
          66m 18s



          Reason Tests
          Failed junit tests hadoop.yarn.server.resourcemanager.TestRMRestart



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue YARN-5556
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12846311/YARN-5556.v2.005.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 452e00e08ed0 3.13.0-105-generic #152-Ubuntu SMP Fri Dec 2 15:37:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 287d3d6
          Default Java 1.8.0_111
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/14608/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/14608/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/14608/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/14608/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 18s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 14m 20s trunk passed +1 compile 0m 38s trunk passed +1 checkstyle 0m 26s trunk passed +1 mvnsite 0m 42s trunk passed +1 mvneclipse 0m 19s trunk passed +1 findbugs 1m 9s trunk passed +1 javadoc 0m 24s trunk passed +1 mvninstall 0m 43s the patch passed +1 compile 0m 33s the patch passed +1 javac 0m 33s the patch passed -0 checkstyle 0m 22s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 6 new + 207 unchanged - 2 fixed = 213 total (was 209) +1 mvnsite 0m 34s the patch passed +1 mvneclipse 0m 14s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 12s the patch passed +1 javadoc 0m 20s the patch passed -1 unit 42m 21s hadoop-yarn-server-resourcemanager in the patch failed. +1 asflicense 0m 21s The patch does not generate ASF License warnings. 66m 18s Reason Tests Failed junit tests hadoop.yarn.server.resourcemanager.TestRMRestart Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue YARN-5556 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12846311/YARN-5556.v2.005.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 452e00e08ed0 3.13.0-105-generic #152-Ubuntu SMP Fri Dec 2 15:37:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 287d3d6 Default Java 1.8.0_111 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/14608/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/14608/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/14608/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/14608/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          Naganarasimha Naganarasimha G R added a comment -

          Had tested the changes in the local setup, seems to be fine. Few nits which i was able to think of were

          • "root.Q1.Q1b seems to be deleted but the queue is not yet in stopped state" error message can be told as removed in the updated xml than deleted.
          • "refreshQueues: java.io.IOException: Failed to re-init queues" is the message shown in console though down in the stacktrace exact message is captured. Better to append the exception message to "Failed to re-init queues" at CapacityScheduler.reinitialize on exception
          • documentation needs to be updated, can either raise a new jira or add as part of this jira for the steps to be delete the Queue in CS

          Can work on this along with other comments .

          Show
          Naganarasimha Naganarasimha G R added a comment - Had tested the changes in the local setup, seems to be fine. Few nits which i was able to think of were "root.Q1.Q1b seems to be deleted but the queue is not yet in stopped state" error message can be told as removed in the updated xml than deleted. "refreshQueues: java.io.IOException: Failed to re-init queues" is the message shown in console though down in the stacktrace exact message is captured. Better to append the exception message to "Failed to re-init queues" at CapacityScheduler.reinitialize on exception documentation needs to be updated, can either raise a new jira or add as part of this jira for the steps to be delete the Queue in CS Can work on this along with other comments .
          Hide
          leftnoteasy Wangda Tan added a comment -

          Thanks Naganarasimha G R,

          Generally looks good, few minor suggestions:

          1) CSQueueManager:

          • Better to update warn to info:
            LOG.warn("Deleting Queue " + queueName + ", as it is not"
                            + " present in the modified capacity configuration xml");
            

            Since delete queue is a normal behavior

          • "seems to be deleted" -> "is deleted from the new configuration"
          • In updateQueues, oldQueues -> existingQueues

          2) ParentQueue:

          • In the "// remove the deleted queue in the refreshed xml", oldQueueName -> queueName

          And for your comment above:

          Better to append the exception message ...

          Agree

          documentation needs to be updated, can either raise a new jira or add as part of this jira for the steps to be delete the Queue in CS

          It's better to have a separate JIRA to add document.

          Show
          leftnoteasy Wangda Tan added a comment - Thanks Naganarasimha G R , Generally looks good, few minor suggestions: 1) CSQueueManager: Better to update warn to info: LOG.warn( "Deleting Queue " + queueName + ", as it is not" + " present in the modified capacity configuration xml" ); Since delete queue is a normal behavior "seems to be deleted" -> "is deleted from the new configuration" In updateQueues , oldQueues -> existingQueues 2) ParentQueue: In the "// remove the deleted queue in the refreshed xml", oldQueueName -> queueName And for your comment above: Better to append the exception message ... Agree documentation needs to be updated, can either raise a new jira or add as part of this jira for the steps to be delete the Queue in CS It's better to have a separate JIRA to add document.
          Hide
          Naganarasimha Naganarasimha G R added a comment -

          attaching a patch for addressing Tan, Wangda's comments

          Show
          Naganarasimha Naganarasimha G R added a comment - attaching a patch for addressing Tan, Wangda 's comments
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 16s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 12m 45s trunk passed
          +1 compile 0m 32s trunk passed
          +1 checkstyle 0m 25s trunk passed
          +1 mvnsite 0m 34s trunk passed
          +1 mvneclipse 0m 17s trunk passed
          +1 findbugs 0m 58s trunk passed
          +1 javadoc 0m 20s trunk passed
          +1 mvninstall 0m 30s the patch passed
          +1 compile 0m 30s the patch passed
          +1 javac 0m 30s the patch passed
          -0 checkstyle 0m 22s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 6 new + 280 unchanged - 2 fixed = 286 total (was 282)
          +1 mvnsite 0m 32s the patch passed
          +1 mvneclipse 0m 14s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 1m 3s the patch passed
          +1 javadoc 0m 18s the patch passed
          -1 unit 39m 58s hadoop-yarn-server-resourcemanager in the patch failed.
          +1 asflicense 0m 18s The patch does not generate ASF License warnings.
          61m 11s



          Reason Tests
          Failed junit tests hadoop.yarn.server.resourcemanager.TestRMRestart



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue YARN-5556
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12847065/YARN-5556.v2.006.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 0e50c55d491e 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / e648b6e
          Default Java 1.8.0_111
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/14640/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/14640/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/14640/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/14640/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 16s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 12m 45s trunk passed +1 compile 0m 32s trunk passed +1 checkstyle 0m 25s trunk passed +1 mvnsite 0m 34s trunk passed +1 mvneclipse 0m 17s trunk passed +1 findbugs 0m 58s trunk passed +1 javadoc 0m 20s trunk passed +1 mvninstall 0m 30s the patch passed +1 compile 0m 30s the patch passed +1 javac 0m 30s the patch passed -0 checkstyle 0m 22s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 6 new + 280 unchanged - 2 fixed = 286 total (was 282) +1 mvnsite 0m 32s the patch passed +1 mvneclipse 0m 14s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 3s the patch passed +1 javadoc 0m 18s the patch passed -1 unit 39m 58s hadoop-yarn-server-resourcemanager in the patch failed. +1 asflicense 0m 18s The patch does not generate ASF License warnings. 61m 11s Reason Tests Failed junit tests hadoop.yarn.server.resourcemanager.TestRMRestart Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue YARN-5556 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12847065/YARN-5556.v2.006.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 0e50c55d491e 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / e648b6e Default Java 1.8.0_111 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/14640/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/14640/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/14640/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/14640/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          leftnoteasy Wangda Tan added a comment -

          Thanks Naganarasimha G R, +1 to latest patch, will commit next Monday if no opposite opinions.

          Show
          leftnoteasy Wangda Tan added a comment - Thanks Naganarasimha G R , +1 to latest patch, will commit next Monday if no opposite opinions.
          Hide
          leftnoteasy Wangda Tan added a comment -

          Committed to trunk/branch-2.

          Thanks Naganarasimha G R and thanks reviews from Xuan Gong/Daniel Templeton!

          Show
          leftnoteasy Wangda Tan added a comment - Committed to trunk/branch-2. Thanks Naganarasimha G R and thanks reviews from Xuan Gong / Daniel Templeton !
          Hide
          hudson Hudson added a comment -

          ABORTED: Integrated in Jenkins build Hadoop-trunk-Commit #11136 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11136/)
          YARN-5556. CapacityScheduler: Support deleting queues without requiring (wangda: rev 72054a817dfb43e93916d7036eba19cf2f49cea2)

          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestCapacityScheduler.java
          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/ParentQueue.java
          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.java
          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerQueueManager.java
          Show
          hudson Hudson added a comment - ABORTED: Integrated in Jenkins build Hadoop-trunk-Commit #11136 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11136/ ) YARN-5556 . CapacityScheduler: Support deleting queues without requiring (wangda: rev 72054a817dfb43e93916d7036eba19cf2f49cea2) (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestCapacityScheduler.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/ParentQueue.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerQueueManager.java
          Hide
          leftnoteasy Wangda Tan added a comment -

          Hi Naganarasimha Garla,

          When I working on other tickets, I found a test failure:

          java.lang.AssertionError: Expected to NOT throw exception when refresh queue tries to delete a queue WITHOUT running apps
          	at org.junit.Assert.fail(Assert.java:88)
          	at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacityScheduler.testRefreshQueuesWithQueueDelete(TestCapacityScheduler.java:3870)
          

          Is it an intermittent failure or it is caused by other changes?

          See: https://builds.apache.org/job/PreCommit-YARN-Build/14735/testReport/org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity/TestCapacityScheduler/testRefreshQueuesWithQueueDelete/

          Thanks.

          Show
          leftnoteasy Wangda Tan added a comment - Hi Naganarasimha Garla , When I working on other tickets, I found a test failure: java.lang.AssertionError: Expected to NOT throw exception when refresh queue tries to delete a queue WITHOUT running apps at org.junit.Assert.fail(Assert.java:88) at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacityScheduler.testRefreshQueuesWithQueueDelete(TestCapacityScheduler.java:3870) Is it an intermittent failure or it is caused by other changes? See: https://builds.apache.org/job/PreCommit-YARN-Build/14735/testReport/org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity/TestCapacityScheduler/testRefreshQueuesWithQueueDelete/ Thanks.
          Hide
          Naganarasimha Naganarasimha G R added a comment -

          Oops missed to see this ! Thanks for informing Tan, Wangda but tried running in local build multiple times (specific method and all) but seems like its not failing, may be we can wait for one more time and if required raise a jira, so that i can add more logs to capture the failure.

          Show
          Naganarasimha Naganarasimha G R added a comment - Oops missed to see this ! Thanks for informing Tan, Wangda but tried running in local build multiple times (specific method and all) but seems like its not failing, may be we can wait for one more time and if required raise a jira, so that i can add more logs to capture the failure.
          Hide
          leftnoteasy Wangda Tan added a comment -

          Naganarasimha G R, thanks for confirming this!

          Show
          leftnoteasy Wangda Tan added a comment - Naganarasimha G R , thanks for confirming this!
          Hide
          sunilg Sunil G added a comment -

          I got this test failure in a recent run here

          I think i ll raise a ticket for same.

          Show
          sunilg Sunil G added a comment - I got this test failure in a recent run here I think i ll raise a ticket for same.
          Hide
          Naganarasimha Naganarasimha G R added a comment -

          Thanks Sunil G for raising the issue will look into it !

          Show
          Naganarasimha Naganarasimha G R added a comment - Thanks Sunil G for raising the issue will look into it !

            People

            • Assignee:
              Naganarasimha Naganarasimha G R
              Reporter:
              xgong Xuan Gong
            • Votes:
              0 Vote for this issue
              Watchers:
              18 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development