Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-4492

Add documentation for preemption supported in Capacity scheduler

    Details

    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      As part of YARN-2056, Support has been added to disable preemption for a specific queue. This is a useful feature in a multiload cluster but currently missing documentation.
      Complete preemption is not documented hence update all configurations for capacity scheduler preemption

      1. CapacityScheduler.html
        48 kB
        Naganarasimha G R
      2. YARN-4492.v1.001.patch
        2 kB
        Naganarasimha G R
      3. YARN-4492.v1.002.patch
        2 kB
        Naganarasimha G R
      4. YARN-4492.v1.003.patch
        2 kB
        Naganarasimha G R
      5. YARN-4492.v2.001.patch
        11 kB
        Naganarasimha G R
      6. YARN-4492.v2.002.patch
        11 kB
        Naganarasimha G R
      7. YARN-4492.v2.003.patch
        11 kB
        Naganarasimha G R
      8. YARN-4492-branch-2.7.001.patch
        6 kB
        Naganarasimha G R

        Issue Links

          Activity

          Hide
          vinodkv Vinod Kumar Vavilapalli added a comment -

          Closing the JIRA as part of 2.7.3 release.

          Show
          vinodkv Vinod Kumar Vavilapalli added a comment - Closing the JIRA as part of 2.7.3 release.
          Hide
          jlowe Jason Lowe added a comment -

          but Jira is not resolved, any thing pending/missed ?

          Sorry, I got interrupted during the commit process, should be good now. Busy, busy!

          Show
          jlowe Jason Lowe added a comment - but Jira is not resolved, any thing pending/missed ? Sorry, I got interrupted during the commit process, should be good now. Busy, busy!
          Hide
          jlowe Jason Lowe added a comment -

          Thanks to Naganarasimha G R for the contribution and to Daniel Templeton and Eric Payne for additional review! I committed this to trunk, branch-2, branch-2.8, and branch-2.7.

          Show
          jlowe Jason Lowe added a comment - Thanks to Naganarasimha G R for the contribution and to Daniel Templeton and Eric Payne for additional review! I committed this to trunk, branch-2, branch-2.8, and branch-2.7.
          Hide
          Naganarasimha Naganarasimha G R added a comment -

          Thanks for the review and commit Jason Lowe, Daniel Templeton & Eric Payne.
          Jason Lowe, I was able to see that the patch was committed to trunk and 2.7 branches but Jira is not resolved, any thing pending/missed ?

          Show
          Naganarasimha Naganarasimha G R added a comment - Thanks for the review and commit Jason Lowe , Daniel Templeton & Eric Payne . Jason Lowe , I was able to see that the patch was committed to trunk and 2.7 branches but Jira is not resolved, any thing pending/missed ?
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #9149 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9149/)
          YARN-4492. Add documentation for preemption supported in Capacity (jlowe: rev f858ac24269c101bdee1f7b7b2566bdbda2b4ecd)

          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/CapacityScheduler.md
          • hadoop-yarn-project/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #9149 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9149/ ) YARN-4492 . Add documentation for preemption supported in Capacity (jlowe: rev f858ac24269c101bdee1f7b7b2566bdbda2b4ecd) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/CapacityScheduler.md hadoop-yarn-project/CHANGES.txt
          Hide
          jlowe Jason Lowe added a comment -

          Sorry for the delay.

          +1 for the branch-2.7 patch. Committing this.

          Show
          jlowe Jason Lowe added a comment - Sorry for the delay. +1 for the branch-2.7 patch. Committing this.
          Hide
          Naganarasimha Naganarasimha G R added a comment -

          Hi Jason Lowe If you have some cycles can you take a look at this patch ? As its almost done may be one last review for version 2.7 we can get this in !

          Show
          Naganarasimha Naganarasimha G R added a comment - Hi Jason Lowe If you have some cycles can you take a look at this patch ? As its almost done may be one last review for version 2.7 we can get this in !
          Hide
          Naganarasimha Naganarasimha G R added a comment -

          Hi Jason Lowe, Any other updates required for the patch ?

          Show
          Naganarasimha Naganarasimha G R added a comment - Hi Jason Lowe , Any other updates required for the patch ?
          Hide
          Naganarasimha Naganarasimha G R added a comment -

          hi Jason Lowe,
          I triggered the jenkins again but seems like issue still exists with the build process, seems like report for whitespace and asf warnings are not related to the modifications in the patch !

          Show
          Naganarasimha Naganarasimha G R added a comment - hi Jason Lowe , I triggered the jenkins again but seems like issue still exists with the build process, seems like report for whitespace and asf warnings are not related to the modifications in the patch !
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 mvnsite 0m 42s branch-2.7 passed
          +1 mvnsite 0m 13s the patch passed
          -1 whitespace 0m 1s The patch has 1430 line(s) that end in whitespace. Use git apply --whitespace=fix.
          -1 whitespace 0m 35s The patch has 59 line(s) with tabs.
          -1 asflicense 42m 40s Patch generated 34 ASF License warnings.
          44m 50s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:date2016-01-16
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12782661/YARN-4492-branch-2.7.001.patch
          JIRA Issue YARN-4492
          Optional Tests asflicense mvnsite
          uname Linux 45416f42d60e 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision branch-2.7 / 296ecbd
          whitespace https://builds.apache.org/job/PreCommit-YARN-Build/10313/artifact/patchprocess/whitespace-eol.txt
          whitespace https://builds.apache.org/job/PreCommit-YARN-Build/10313/artifact/patchprocess/whitespace-tabs.txt
          asflicense https://builds.apache.org/job/PreCommit-YARN-Build/10313/artifact/patchprocess/patch-asflicense-problems.txt
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site
          Max memory used 29MB
          Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/10313/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 mvnsite 0m 42s branch-2.7 passed +1 mvnsite 0m 13s the patch passed -1 whitespace 0m 1s The patch has 1430 line(s) that end in whitespace. Use git apply --whitespace=fix. -1 whitespace 0m 35s The patch has 59 line(s) with tabs. -1 asflicense 42m 40s Patch generated 34 ASF License warnings. 44m 50s Subsystem Report/Notes Docker Image:yetus/hadoop:date2016-01-16 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12782661/YARN-4492-branch-2.7.001.patch JIRA Issue YARN-4492 Optional Tests asflicense mvnsite uname Linux 45416f42d60e 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision branch-2.7 / 296ecbd whitespace https://builds.apache.org/job/PreCommit-YARN-Build/10313/artifact/patchprocess/whitespace-eol.txt whitespace https://builds.apache.org/job/PreCommit-YARN-Build/10313/artifact/patchprocess/whitespace-tabs.txt asflicense https://builds.apache.org/job/PreCommit-YARN-Build/10313/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site Max memory used 29MB Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-YARN-Build/10313/console This message was automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 mvnsite 0m 48s branch-2.7 passed
          +1 mvnsite 0m 13s the patch passed
          -1 whitespace 0m 1s The patch has 1287 line(s) that end in whitespace. Use git apply --whitespace=fix.
          -1 whitespace 0m 32s The patch has 59 line(s) with tabs.
          -1 asflicense 43m 19s Patch generated 34 ASF License warnings.
          60m 52s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:date2016-01-16
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12782661/YARN-4492-branch-2.7.001.patch
          JIRA Issue YARN-4492
          Optional Tests asflicense mvnsite
          uname Linux 11fc8b9f3bca 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision branch-2.7 / 296ecbd
          whitespace https://builds.apache.org/job/PreCommit-YARN-Build/10312/artifact/patchprocess/whitespace-eol.txt
          whitespace https://builds.apache.org/job/PreCommit-YARN-Build/10312/artifact/patchprocess/whitespace-tabs.txt
          asflicense https://builds.apache.org/job/PreCommit-YARN-Build/10312/artifact/patchprocess/patch-asflicense-problems.txt
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site
          Max memory used 29MB
          Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/10312/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 mvnsite 0m 48s branch-2.7 passed +1 mvnsite 0m 13s the patch passed -1 whitespace 0m 1s The patch has 1287 line(s) that end in whitespace. Use git apply --whitespace=fix. -1 whitespace 0m 32s The patch has 59 line(s) with tabs. -1 asflicense 43m 19s Patch generated 34 ASF License warnings. 60m 52s Subsystem Report/Notes Docker Image:yetus/hadoop:date2016-01-16 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12782661/YARN-4492-branch-2.7.001.patch JIRA Issue YARN-4492 Optional Tests asflicense mvnsite uname Linux 11fc8b9f3bca 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision branch-2.7 / 296ecbd whitespace https://builds.apache.org/job/PreCommit-YARN-Build/10312/artifact/patchprocess/whitespace-eol.txt whitespace https://builds.apache.org/job/PreCommit-YARN-Build/10312/artifact/patchprocess/whitespace-tabs.txt asflicense https://builds.apache.org/job/PreCommit-YARN-Build/10312/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site Max memory used 29MB Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-YARN-Build/10312/console This message was automatically generated.
          Hide
          Naganarasimha Naganarasimha G R added a comment -

          Jason Lowe, I have attached a patch for branch-2.7, can you please check whether its fine ?
          From my side i have verified the configurations mentioned in the doc is also present in the 2.7 branch.

          Show
          Naganarasimha Naganarasimha G R added a comment - Jason Lowe , I have attached a patch for branch-2.7, can you please check whether its fine ? From my side i have verified the configurations mentioned in the doc is also present in the 2.7 branch.
          Hide
          jlowe Jason Lowe added a comment -

          Turns out this no longer applies cleanly to branch-2.7 since it doesn't have any of the application priority stuff. Could you provide a patch for 2.7 as well?

          Show
          jlowe Jason Lowe added a comment - Turns out this no longer applies cleanly to branch-2.7 since it doesn't have any of the application priority stuff. Could you provide a patch for 2.7 as well?
          Hide
          jlowe Jason Lowe added a comment -

          +1 lgtm. Committing this.

          Show
          jlowe Jason Lowe added a comment - +1 lgtm. Committing this.
          Hide
          Naganarasimha Naganarasimha G R added a comment -

          Hi Jason Lowe,
          If you have cycles, can you take a look @ the patch as it seems to be almost to ready !

          Show
          Naganarasimha Naganarasimha G R added a comment - Hi Jason Lowe , If you have cycles, can you take a look @ the patch as it seems to be almost to ready !
          Hide
          hadoopqa Hadoop QA added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 mvnsite 0m 14s trunk passed
          +1 mvnsite 0m 13s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 asflicense 0m 16s Patch does not generate ASF License warnings.
          1m 1s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12782211/YARN-4492.v2.003.patch
          JIRA Issue YARN-4492
          Optional Tests asflicense mvnsite
          uname Linux 58d16168f636 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 62b0d61
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site
          Max memory used 30MB
          Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/10278/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 mvnsite 0m 14s trunk passed +1 mvnsite 0m 13s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 asflicense 0m 16s Patch does not generate ASF License warnings. 1m 1s Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12782211/YARN-4492.v2.003.patch JIRA Issue YARN-4492 Optional Tests asflicense mvnsite uname Linux 58d16168f636 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 62b0d61 modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site Max memory used 30MB Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-YARN-Build/10278/console This message was automatically generated.
          Hide
          Naganarasimha Naganarasimha G R added a comment -

          Attaching a patch with corrections for the comments, Jason Lowe, can you please review now ?

          Show
          Naganarasimha Naganarasimha G R added a comment - Attaching a patch with corrections for the comments, Jason Lowe , can you please review now ?
          Hide
          jlowe Jason Lowe added a comment -

          Thanks for the patch, Naganarasimha G R! Just a few more phrasing nits:

          "Free resources can be allocated to any queue beyond it's capacity." should be "Free resources can be allocated to any queue beyond its capacity." ("it's" vs. "its") I know this was from before, but since we're in the neighborhood...

          "Any application that is submitted with a priority is greater then cluster max priority then ..." should be "Any application submitted with a priority greater than the cluster-max priority will have its priority reset to the cluster-max priority." (or something similar).

          "If application is not submitted with any priority, then default priority of the queue will be taken in account." would be clearer as "An application submitted without a specified priority will have the priority set to the queue's default priority." or "The queue's default priority will be used for any application submitted without a specified priority."

          "Following configuration parameters ..." should be "The following configuration parameters ..."

          Show
          jlowe Jason Lowe added a comment - Thanks for the patch, Naganarasimha G R ! Just a few more phrasing nits: "Free resources can be allocated to any queue beyond it's capacity." should be "Free resources can be allocated to any queue beyond its capacity." ("it's" vs. "its") I know this was from before, but since we're in the neighborhood... "Any application that is submitted with a priority is greater then cluster max priority then ..." should be "Any application submitted with a priority greater than the cluster-max priority will have its priority reset to the cluster-max priority." (or something similar). "If application is not submitted with any priority, then default priority of the queue will be taken in account." would be clearer as "An application submitted without a specified priority will have the priority set to the queue's default priority." or "The queue's default priority will be used for any application submitted without a specified priority." "Following configuration parameters ..." should be "The following configuration parameters ..."
          Hide
          Naganarasimha Naganarasimha G R added a comment -

          Hi Jian He, Jason Lowe, Tan, Wangda & Rohith Sharma K S,
          Can one of you take a look at this documentation for pre-emption in capacity scheduler ?

          Show
          Naganarasimha Naganarasimha G R added a comment - Hi Jian He , Jason Lowe , Tan, Wangda & Rohith Sharma K S , Can one of you take a look at this documentation for pre-emption in capacity scheduler ?
          Hide
          eepayne Eric Payne added a comment -

          Thanks Naganarasimha G R.

          +1 (non-binding)

          Show
          eepayne Eric Payne added a comment - Thanks Naganarasimha G R . +1 (non-binding)
          Hide
          hadoopqa Hadoop QA added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 mvnsite 0m 20s trunk passed
          +1 mvnsite 0m 14s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 asflicense 0m 21s Patch does not generate ASF License warnings.
          1m 18s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12781908/YARN-4492.v2.002.patch
          JIRA Issue YARN-4492
          Optional Tests asflicense mvnsite
          uname Linux 528274ef8375 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 126705f
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site
          Max memory used 29MB
          Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/10251/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 mvnsite 0m 20s trunk passed +1 mvnsite 0m 14s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 asflicense 0m 21s Patch does not generate ASF License warnings. 1m 18s Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12781908/YARN-4492.v2.002.patch JIRA Issue YARN-4492 Optional Tests asflicense mvnsite uname Linux 528274ef8375 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 126705f modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site Max memory used 29MB Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-YARN-Build/10251/console This message was automatically generated.
          Hide
          Naganarasimha Naganarasimha G R added a comment -

          Thanks for reviewing so fast and sharing the comments Eric Payne , attaching updated patch for fixing the review comments

          Show
          Naganarasimha Naganarasimha G R added a comment - Thanks for reviewing so fast and sharing the comments Eric Payne , attaching updated patch for fixing the review comments
          Hide
          eepayne Eric Payne added a comment -

          Thanks for updating the patch so quickly, Naganarasimha G R. I think it looks good overall. I have a couple of minor grammatical nits:

          The CapacityScheduler supports preemption of container(s) from the queues whose resource usage is more than their guaranteed capacity. Following configuration parameters needs to be enabled in yarn-site.xml, for supporting preemption of application containers.

          • I would say container rather than container(s)
          • The following configuration parameters need to ...
          • Take out comma after yarn-site.xml

          yarn.resourcemanager.scheduler.monitor.policies

          • Configured policies need to be ...

          Following configuration parameters can be configured in yarn-site.xml, to control the preemption of containers when ProportionalCapacityPreemptionPolicy class is configured for yarn.resourcemanager.scheduler.monitor.policies

          • I would remove the comma after yarn-site.xml

          Other than that, it looks good.

          Show
          eepayne Eric Payne added a comment - Thanks for updating the patch so quickly, Naganarasimha G R . I think it looks good overall. I have a couple of minor grammatical nits: The CapacityScheduler supports preemption of container(s) from the queues whose resource usage is more than their guaranteed capacity. Following configuration parameters needs to be enabled in yarn-site.xml, for supporting preemption of application containers. I would say container rather than container(s) The following configuration parameters need to ... Take out comma after yarn-site.xml yarn.resourcemanager.scheduler.monitor.policies Configured policies need to be ... Following configuration parameters can be configured in yarn-site.xml, to control the preemption of containers when ProportionalCapacityPreemptionPolicy class is configured for yarn.resourcemanager.scheduler.monitor.policies I would remove the comma after yarn-site.xml Other than that, it looks good.
          Hide
          hadoopqa Hadoop QA added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 mvnsite 0m 15s trunk passed
          +1 mvnsite 0m 15s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 asflicense 0m 22s Patch does not generate ASF License warnings.
          1m 11s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12781847/YARN-4492.v2.001.patch
          JIRA Issue YARN-4492
          Optional Tests asflicense mvnsite
          uname Linux 68878194c5bd 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 13de835
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site
          Max memory used 29MB
          Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/10243/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 mvnsite 0m 15s trunk passed +1 mvnsite 0m 15s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 asflicense 0m 22s Patch does not generate ASF License warnings. 1m 11s Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12781847/YARN-4492.v2.001.patch JIRA Issue YARN-4492 Optional Tests asflicense mvnsite uname Linux 68878194c5bd 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 13de835 modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site Max memory used 29MB Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-YARN-Build/10243/console This message was automatically generated.
          Hide
          Naganarasimha Naganarasimha G R added a comment -

          Attaching generated html file for reference

          Show
          Naganarasimha Naganarasimha G R added a comment - Attaching generated html file for reference
          Hide
          Naganarasimha Naganarasimha G R added a comment -

          Thanks for the review Eric Payne,

          The new section added to CapacityScheduler.md is titled * Queue preemption support. Should these (and other preemption-related properties) should be documented here.

          I felt it was required and i have re done the patch with following modifications

          • It was wrongly documented in Elasticity Feature that pre-emption is not supported, have corrected it.
          • Captured all preemption configurations for capacity scheduler
          • Created a new heading under configuration and have updated the contents
          • Re organised the contents under configurations as Priority configuration was wrongly placed in yarn-4098
          Show
          Naganarasimha Naganarasimha G R added a comment - Thanks for the review Eric Payne , The new section added to CapacityScheduler.md is titled * Queue preemption support. Should these (and other preemption-related properties) should be documented here. I felt it was required and i have re done the patch with following modifications It was wrongly documented in Elasticity Feature that pre-emption is not supported, have corrected it. Captured all preemption configurations for capacity scheduler Created a new heading under configuration and have updated the contents Re organised the contents under configurations as Priority configuration was wrongly placed in yarn-4098
          Hide
          eepayne Eric Payne added a comment -

          Thanks, Naganarasimha G R, for providing the documentation for this feature. I just have one minor grammatical nit:

          This configuration can be set true ...

          I would say This configuration can be set to true ... (added the word to)

          Also, one question: this documentation references the yarn.resourcemanager.scheduler.monitor.enable and the yarn.resourcemanager.scheduler.monitor.policies properties, but I don't see those documented anywhere, either in the CapacityScheduler.md file or elsewhere in the source base. The new section added to CapacityScheduler.md is titled * Queue preemption support. Should these (and other preemption-related properties) should be documented here?

          Show
          eepayne Eric Payne added a comment - Thanks, Naganarasimha G R , for providing the documentation for this feature. I just have one minor grammatical nit: This configuration can be set true ... I would say This configuration can be set to true ... (added the word to ) Also, one question: this documentation references the yarn.resourcemanager.scheduler.monitor.enable and the yarn.resourcemanager.scheduler.monitor.policies properties, but I don't see those documented anywhere, either in the CapacityScheduler.md file or elsewhere in the source base. The new section added to CapacityScheduler.md is titled * Queue preemption support . Should these (and other preemption-related properties) should be documented here?
          Hide
          Naganarasimha Naganarasimha G R added a comment -

          Thanks Daniel Templeton, for the review
          Hope Jason Lowe / Tan, Wangda, (reviewers of YARN-2056) can take a look at this ?

          Show
          Naganarasimha Naganarasimha G R added a comment - Thanks Daniel Templeton , for the review Hope Jason Lowe / Tan, Wangda , (reviewers of YARN-2056 ) can take a look at this ?
          Hide
          templedf Daniel Templeton added a comment -

          +1 (non-binding). Thanks, Naganarasimha G R!

          Show
          templedf Daniel Templeton added a comment - +1 (non-binding). Thanks, Naganarasimha G R !
          Hide
          hadoopqa Hadoop QA added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 mvnsite 0m 17s trunk passed
          +1 mvnsite 0m 16s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 asflicense 0m 20s Patch does not generate ASF License warnings.
          1m 6s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12778943/YARN-4492.v1.003.patch
          JIRA Issue YARN-4492
          Optional Tests asflicense mvnsite
          uname Linux 0be32395cbd3 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / a0de702
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site
          Max memory used 29MB
          Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/10066/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 mvnsite 0m 17s trunk passed +1 mvnsite 0m 16s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 asflicense 0m 20s Patch does not generate ASF License warnings. 1m 6s Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12778943/YARN-4492.v1.003.patch JIRA Issue YARN-4492 Optional Tests asflicense mvnsite uname Linux 0be32395cbd3 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / a0de702 modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site Max memory used 29MB Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-YARN-Build/10066/console This message was automatically generated.
          Hide
          Naganarasimha Naganarasimha G R added a comment -

          updated with your last comment !

          Show
          Naganarasimha Naganarasimha G R added a comment - updated with your last comment !
          Hide
          hadoopqa Hadoop QA added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 mvnsite 0m 14s trunk passed
          +1 mvnsite 0m 14s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 asflicense 0m 18s Patch does not generate ASF License warnings.
          0m 56s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12778872/YARN-4492.v1.002.patch
          JIRA Issue YARN-4492
          Optional Tests asflicense mvnsite
          uname Linux a443b8cfe42f 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 2cb5aff
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site
          Max memory used 29MB
          Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/10060/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 mvnsite 0m 14s trunk passed +1 mvnsite 0m 14s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 asflicense 0m 18s Patch does not generate ASF License warnings. 0m 56s Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12778872/YARN-4492.v1.002.patch JIRA Issue YARN-4492 Optional Tests asflicense mvnsite uname Linux a443b8cfe42f 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 2cb5aff modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site Max memory used 29MB Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-YARN-Build/10060/console This message was automatically generated.
          Hide
          templedf Daniel Templeton added a comment -

          Excellent. There was one more change in there that I think was missed. This will take impact only when system wide preemption is enabled should be This property applies only when system wide preemption is enabled.

          Show
          templedf Daniel Templeton added a comment - Excellent. There was one more change in there that I think was missed. This will take impact only when system wide preemption is enabled should be This property applies only when system wide preemption is enabled .
          Hide
          Naganarasimha Naganarasimha G R added a comment -

          Thanks Daniel Templeton,
          Yes i am fine your last suggestion, updating the patch with your review comments fixed

          Show
          Naganarasimha Naganarasimha G R added a comment - Thanks Daniel Templeton , Yes i am fine your last suggestion, updating the patch with your review comments fixed
          Hide
          templedf Daniel Templeton added a comment -

          Too many "it"s with unclear antecedents. How about: If this property is not set for a queue, then the property value is inherited from the queue's parent ?

          Show
          templedf Daniel Templeton added a comment - Too many "it"s with unclear antecedents. How about: If this property is not set for a queue, then the property value is inherited from the queue's parent ?
          Hide
          Naganarasimha Naganarasimha G R added a comment -

          Well i think its almost there but just that property is used twice ! how about If this property is not set for a queue, then it inherits from its parent

          Show
          Naganarasimha Naganarasimha G R added a comment - Well i think its almost there but just that property is used twice ! how about If this property is not set for a queue, then it inherits from its parent
          Hide
          templedf Daniel Templeton added a comment -

          I like the third one, too. Let's maybe say: If this property is not set for a queue, that queue inherits the value for this property from its parent. Does that work?

          Show
          templedf Daniel Templeton added a comment - I like the third one, too. Let's maybe say: If this property is not set for a queue, that queue inherits the value for this property from its parent. Does that work?
          Hide
          Naganarasimha Naganarasimha G R added a comment -

          Thanks for the comments Daniel Templeton,

          A queue's ability to be preempted is inherited from its parent unless explicitly overridden for that queue.

          will this sound like queue is preemptable? can this be A queue's ability for its apps resources to be preempted is inherited from its parent unless explicitly overridden for that queue.
          or Preempt-ability of Queue's application resources will be inherited from its parent unless explicitly overridden
          or This property of a Queue is inherited from its parent unless explicitly overridden thoughts?
          my choice is 3rd!

          Show
          Naganarasimha Naganarasimha G R added a comment - Thanks for the comments Daniel Templeton , A queue's ability to be preempted is inherited from its parent unless explicitly overridden for that queue. will this sound like queue is preemptable? can this be A queue's ability for its apps resources to be preempted is inherited from its parent unless explicitly overridden for that queue. or Preempt-ability of Queue's application resources will be inherited from its parent unless explicitly overridden or This property of a Queue is inherited from its parent unless explicitly overridden thoughts? my choice is 3rd!
          Hide
          templedf Daniel Templeton added a comment -

          Minor language edits:

          The `CapacityScheduler` supports the following parameters to control the preemption of containers of applications submitted to a queue:
          

          should be:

           The `CapacityScheduler` supports the following parameters to control the preemption of application containers submitted to a queue:
          

          and:

          | `yarn.scheduler.capacity.<queue-path>.disable_preemption` | This configuration can be set true, to selectively disable preemption of application containers submitted to a given queue. This will take impact only when system wide preemption is enabled by configuring `yarn.resourcemanager.scheduler.monitor.enable` to *true* and `yarn.resourcemanager.scheduler.monitor.policies` to *ProportionalCapacityPreemptionPolicy*. Preemptability will be inherited from the parent's hierarchy unless explicitly overridden by a queue. Default value is false.
          

          should maybe be:

          | `yarn.scheduler.capacity.<queue-path>.disable_preemption` | This configuration can be set true to selectively disable preemption of application containers submitted to a given queue. This will take effect only when system wide preemption is enabled by configuring `yarn.resourcemanager.scheduler.monitor.enable` to *true* and `yarn.resourcemanager.scheduler.monitor.policies` to *ProportionalCapacityPreemptionPolicy*. A queue's ability to be preempted is inherited from its parent unless explicitly overridden for that queue. The default value is false.
          
          Show
          templedf Daniel Templeton added a comment - Minor language edits: The `CapacityScheduler` supports the following parameters to control the preemption of containers of applications submitted to a queue: should be: The `CapacityScheduler` supports the following parameters to control the preemption of application containers submitted to a queue: and: | `yarn.scheduler.capacity.<queue-path>.disable_preemption` | This configuration can be set true, to selectively disable preemption of application containers submitted to a given queue. This will take impact only when system wide preemption is enabled by configuring `yarn.resourcemanager.scheduler.monitor.enable` to *true* and `yarn.resourcemanager.scheduler.monitor.policies` to *ProportionalCapacityPreemptionPolicy*. Preemptability will be inherited from the parent's hierarchy unless explicitly overridden by a queue. Default value is false. should maybe be: | `yarn.scheduler.capacity.<queue-path>.disable_preemption` | This configuration can be set true to selectively disable preemption of application containers submitted to a given queue. This will take effect only when system wide preemption is enabled by configuring `yarn.resourcemanager.scheduler.monitor.enable` to *true* and `yarn.resourcemanager.scheduler.monitor.policies` to *ProportionalCapacityPreemptionPolicy*. A queue's ability to be preempted is inherited from its parent unless explicitly overridden for that queue. The default value is false.
          Hide
          Naganarasimha Naganarasimha G R added a comment -

          ASF warning is due to HDFS-9173 and already it has been reopened !

          Show
          Naganarasimha Naganarasimha G R added a comment - ASF warning is due to HDFS-9173 and already it has been reopened !
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 mvnsite 0m 16s trunk passed
          +1 mvnsite 0m 16s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          -1 asflicense 0m 18s Patch generated 1 ASF License warnings.
          1m 0s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12778745/YARN-4492.v1.001.patch
          JIRA Issue YARN-4492
          Optional Tests asflicense mvnsite
          uname Linux 1fd0202e740c 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 7995a6e
          asflicense https://builds.apache.org/job/PreCommit-YARN-Build/10052/artifact/patchprocess/patch-asflicense-problems.txt
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site
          Max memory used 29MB
          Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/10052/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 mvnsite 0m 16s trunk passed +1 mvnsite 0m 16s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. -1 asflicense 0m 18s Patch generated 1 ASF License warnings. 1m 0s Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12778745/YARN-4492.v1.001.patch JIRA Issue YARN-4492 Optional Tests asflicense mvnsite uname Linux 1fd0202e740c 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 7995a6e asflicense https://builds.apache.org/job/PreCommit-YARN-Build/10052/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site Max memory used 29MB Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-YARN-Build/10052/console This message was automatically generated.
          Hide
          Naganarasimha Naganarasimha G R added a comment -

          uploading with initial patch

          Show
          Naganarasimha Naganarasimha G R added a comment - uploading with initial patch

            People

            • Assignee:
              Naganarasimha Naganarasimha G R
              Reporter:
              Naganarasimha Naganarasimha G R
            • Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development