Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-2394

FairScheduler: Configure fairSharePreemptionThreshold per queue

Details

    • New Feature
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 2.6.0
    • fairscheduler
    • None
    • Reviewed

    Description

      Preemption based on fair share starvation happens when usage of a queue is less than 50% of its fair share. This 50% is hardcoded. We'd like to make this configurable on a per queue basis, so that we can choose the threshold at which we want to preempt. Calling this config "fairSharePreemptionThreshold".

      Attachments

        1. YARN-2394-1.patch
          23 kB
          Wei Yan
        2. YARN-2394-2.patch
          34 kB
          Wei Yan
        3. YARN-2394-3.patch
          39 kB
          Wei Yan
        4. YARN-2394-4.patch
          46 kB
          Wei Yan
        5. YARN-2394-5.patch
          55 kB
          Wei Yan
        6. YARN-2394-6.patch
          56 kB
          Wei Yan
        7. YARN-2394-7.patch
          56 kB
          Wei Yan

        Activity

          ywskycn Wei Yan added a comment -

          I'll look into this.

          ywskycn Wei Yan added a comment - I'll look into this.
          ywskycn Wei Yan added a comment -

          Update a patch following the same way as YARN-2395. Each queue inherits fairSharePreemptionThreshold from its parent queue if it doesn't configure in the allocation file. Will rebase the patch once YARN-2395 is in.

          ywskycn Wei Yan added a comment - Update a patch following the same way as YARN-2395 . Each queue inherits fairSharePreemptionThreshold from its parent queue if it doesn't configure in the allocation file. Will rebase the patch once YARN-2395 is in.
          ashwinshankar77 Ashwin Shankar added a comment -

          ywskycn, kasha
          Inheriting fairSharePreemptionThreshold wouldn't work for us. Do you guys have a use case for inheriting this config ?
          We have the following requirement :
          root.parentUserQueue : fair share = 80% fairSharePreemptionThreshold=1
          root.parentUserQueue.user(1-100) : defaultFairSharePreemptionThreshold=0.5

          Basically when parentUserQueue is starved, we want to preempt more aggressively(threshold=1) from its siblings under root.
          While when a user queue is starved, we want to preempt less aggressively from other user queues(assuming other user queues are above fair share).
          We can do this by setting fairSharePreemptionThreshold=1 at parentUserQueue and defaultFairSharePreemptionThreshold=0.5 which would be used by user queues.
          You might say that we can override fairSharePreemptionThreshold to 0.5 in the user queues, but these queues are created on the fly when a new user
          submits an app and NOT through the alloc xml and there are hundreds of them, so its not very feasible to override each one of them in the config.

          Would it be okay if we stick to version-1 of the patch which doesn't inherit threshold ?

          ashwinshankar77 Ashwin Shankar added a comment - ywskycn , kasha Inheriting fairSharePreemptionThreshold wouldn't work for us. Do you guys have a use case for inheriting this config ? We have the following requirement : root.parentUserQueue : fair share = 80% fairSharePreemptionThreshold=1 root.parentUserQueue.user(1-100) : defaultFairSharePreemptionThreshold=0.5 Basically when parentUserQueue is starved, we want to preempt more aggressively(threshold=1) from its siblings under root. While when a user queue is starved, we want to preempt less aggressively from other user queues(assuming other user queues are above fair share). We can do this by setting fairSharePreemptionThreshold=1 at parentUserQueue and defaultFairSharePreemptionThreshold=0.5 which would be used by user queues. You might say that we can override fairSharePreemptionThreshold to 0.5 in the user queues, but these queues are created on the fly when a new user submits an app and NOT through the alloc xml and there are hundreds of them, so its not very feasible to override each one of them in the config. Would it be okay if we stick to version-1 of the patch which doesn't inherit threshold ?
          hadoopqa Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12663259/YARN-2394-2.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 2 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-YARN-Build/4749//testReport/
          Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4749//console

          This message is automatically generated.

          hadoopqa Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12663259/YARN-2394-2.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 2 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/4749//testReport/ Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4749//console This message is automatically generated.

          If the user queues inherit from the parent and have a threshold of 1, the scheduler would never preempt unless the queue has over its fairshare. If we are preempting resources that are over the fairshare, I think it is fair game. We can improve choosing the victim, but we can work on it in another JIRA.

          OTOH, choosing default-threshold of 0.5 for user queues can lead to:

          1. one of the user queues starve (stay under fairshare) while other user queues are higher than threshold
          2. depending on preemption logic (being revisited in YARN-2154), it might take a while to reach the threshold for parent queues
          kkambatl Karthik Kambatla (Inactive) added a comment - If the user queues inherit from the parent and have a threshold of 1, the scheduler would never preempt unless the queue has over its fairshare. If we are preempting resources that are over the fairshare, I think it is fair game. We can improve choosing the victim, but we can work on it in another JIRA. OTOH, choosing default-threshold of 0.5 for user queues can lead to: one of the user queues starve (stay under fairshare) while other user queues are higher than threshold depending on preemption logic (being revisited in YARN-2154 ), it might take a while to reach the threshold for parent queues
          ashwinshankar77 Ashwin Shankar added a comment -

          I have different opinion on this, but I spoke to some guys here, I think we can work around by setting higher timeouts or slightly lesser threshold(0.8/0.9) at the parent to prevent aggressive preemption at its children.
          So I'm fine with inheriting.

          ashwinshankar77 Ashwin Shankar added a comment - I have different opinion on this, but I spoke to some guys here, I think we can work around by setting higher timeouts or slightly lesser threshold(0.8/0.9) at the parent to prevent aggressive preemption at its children. So I'm fine with inheriting.

          Canceling patch to wait until YARN-2395 gets in.

          ywskycn - can you post a patch on top of YARN-2395?

          kkambatl Karthik Kambatla (Inactive) added a comment - Canceling patch to wait until YARN-2395 gets in. ywskycn - can you post a patch on top of YARN-2395 ?
          ywskycn Wei Yan added a comment -

          Update a patch based on YARN-2395.

          ywskycn Wei Yan added a comment - Update a patch based on YARN-2395 .
          ywskycn Wei Yan added a comment -

          re-submit patch as YARN-2395 is in.

          ywskycn Wei Yan added a comment - re-submit patch as YARN-2395 is in.
          hadoopqa Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12665552/YARN-2394-3.patch
          against trunk revision 0f34e6f.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 2 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-YARN-Build/4787//testReport/
          Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4787//console

          This message is automatically generated.

          hadoopqa Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12665552/YARN-2394-3.patch against trunk revision 0f34e6f. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 2 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/4787//testReport/ Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4787//console This message is automatically generated.

          Just noticed that there is duplication between FSQueue#updatePreemptionParameters and QueueManager#setPreemptionParameters? Can we get rid of one?

          Other than that, I see a couple of lines over 80 chars in AllocationFileLoaderService.

          kkambatl Karthik Kambatla (Inactive) added a comment - Just noticed that there is duplication between FSQueue#updatePreemptionParameters and QueueManager#setPreemptionParameters? Can we get rid of one? Other than that, I see a couple of lines over 80 chars in AllocationFileLoaderService.
          ywskycn Wei Yan added a comment -

          Thanks, kasha, updated a patch to address the two comments. And, move the two isStarved functions in FairScheduler to FSLeafQueue and combine to one function; add one testcase to verify the online created queue's preemption configurations.

          ywskycn Wei Yan added a comment - Thanks, kasha , updated a patch to address the two comments. And, move the two isStarved functions in FairScheduler to FSLeafQueue and combine to one function; add one testcase to verify the online created queue's preemption configurations.
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12665585/YARN-2394-4.patch
          against trunk revision 258c7d0.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 2 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:

          org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesAppsModification
          org.apache.hadoop.yarn.server.resourcemanager.TestApplicationACLs
          org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesNodes

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-YARN-Build/4789//testReport/
          Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4789//console

          This message is automatically generated.

          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12665585/YARN-2394-4.patch against trunk revision 258c7d0. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 2 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesAppsModification org.apache.hadoop.yarn.server.resourcemanager.TestApplicationACLs org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesNodes +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/4789//testReport/ Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4789//console This message is automatically generated.
          ywskycn Wei Yan added a comment -

          The test errors are not related to this patch. Tests passed locally.

          ywskycn Wei Yan added a comment - The test errors are not related to this patch. Tests passed locally.

          Thanks for cleaning up the FairScheduler code, Wei. It looks much better now. While at it, can we do a couple of other things:

          1. FSLeafQueue#isStarved need not take the ResourceCalculator and resource - we can fetch this from the scheduler.
          2. How about renaming FairScheduler#updatePreemptionVariables to FairScheduler#updateStarvationStats?
          3. Add a new method FSLeafQueue#updateStarvationStats that handles the stats for a leaf queue. FairScheduler#updateStarvationStats can call this. Subsequently, we can make FSLeafQueue#setLastTimeAt and FSLeafQueue#isStarved methods private (or package-private with VisibleForTesting annotation if we need these for tests).
          4. Now that we are moving some of these methods to FSLeafQueue and FSQueue, can we move the corresponding method unit tests to respective TestFS*Queue files where possible? These tests should extend FairSchedulerTestBase and the setup etc. should be modeled after TestFairSchedulerFairShare - fetch the scheduler from the MockRM instead of creating a separate scheduler and using the RMContext from an RM?
          kkambatl Karthik Kambatla (Inactive) added a comment - Thanks for cleaning up the FairScheduler code, Wei. It looks much better now. While at it, can we do a couple of other things: FSLeafQueue#isStarved need not take the ResourceCalculator and resource - we can fetch this from the scheduler. How about renaming FairScheduler#updatePreemptionVariables to FairScheduler#updateStarvationStats? Add a new method FSLeafQueue#updateStarvationStats that handles the stats for a leaf queue. FairScheduler#updateStarvationStats can call this. Subsequently, we can make FSLeafQueue#setLastTimeAt and FSLeafQueue#isStarved methods private (or package-private with VisibleForTesting annotation if we need these for tests). Now that we are moving some of these methods to FSLeafQueue and FSQueue, can we move the corresponding method unit tests to respective TestFS*Queue files where possible? These tests should extend FairSchedulerTestBase and the setup etc. should be modeled after TestFairSchedulerFairShare - fetch the scheduler from the MockRM instead of creating a separate scheduler and using the RMContext from an RM?
          ywskycn Wei Yan added a comment -

          Update a patch addressing the latest comments.

          ywskycn Wei Yan added a comment - Update a patch addressing the latest comments.
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12665858/YARN-2394-5.patch
          against trunk revision 258c7d0.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:

          org.apache.hadoop.yarn.server.resourcemanager.TestKillApplicationWithRMHA
          org.apache.hadoop.yarn.server.resourcemanager.recovery.TestZKRMStateStore

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-YARN-Build/4796//testReport/
          Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4796//console

          This message is automatically generated.

          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12665858/YARN-2394-5.patch against trunk revision 258c7d0. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 3 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: org.apache.hadoop.yarn.server.resourcemanager.TestKillApplicationWithRMHA org.apache.hadoop.yarn.server.resourcemanager.recovery.TestZKRMStateStore +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/4796//testReport/ Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4796//console This message is automatically generated.

          Latest patch looks good to me.

          kkambatl Karthik Kambatla (Inactive) added a comment - Latest patch looks good to me.
          ashwinshankar77 Ashwin Shankar added a comment -

          ywskycn, patch looks almost good. Couple of comments :
          1.

          FSLeafQueue.java
          Resource desiredShare = Resources.min(scheduler.getResourceCalculator(),
                  scheduler.getClusterResource(), share, getDemand());
          return Resources.lessThan(scheduler.getResourceCalculator(),
                  scheduler.getClusterResource(), getResourceUsage(), desiredShare);
          

          getResourceCalculator() is a static method, can you please access it through the class and not the object.

          2. Nit

          AllocationConfiguration.java
          // The fair share preemption threshold for each queue. It a queue waits
          

          "It" should be "If"

          ashwinshankar77 Ashwin Shankar added a comment - ywskycn , patch looks almost good. Couple of comments : 1. FSLeafQueue.java Resource desiredShare = Resources.min(scheduler.getResourceCalculator(), scheduler.getClusterResource(), share, getDemand()); return Resources.lessThan(scheduler.getResourceCalculator(), scheduler.getClusterResource(), getResourceUsage(), desiredShare); getResourceCalculator() is a static method, can you please access it through the class and not the object. 2. Nit AllocationConfiguration.java // The fair share preemption threshold for each queue. It a queue waits "It" should be "If"
          ashwinshankar77 Ashwin Shankar added a comment -

          Also, a subjective opinion(feel free to ignore the comment) - In the code below, instead of passing a boolean, can we have the original
          isStarvedFor(Fair|Min)Share which seems more readable to me and put the last two lines of isStarved(boolean) into another method if you
          think that would be redundant between isStarvedFor(Fair|Min)Share ?

          FSLeafQueue
          boolean isStarved(boolean checkMinShare) {
              Resource share = checkMinShare ? getMinShare() :
                  Resources.multiply(getFairShare(), getFairSharePreemptionThreshold());
          
              Resource desiredShare = Resources.min(scheduler.getResourceCalculator(),
                  scheduler.getClusterResource(), share, getDemand());
              return Resources.lessThan(scheduler.getResourceCalculator(),
                  scheduler.getClusterResource(), getResourceUsage(), desiredShare);
            }
          
          ashwinshankar77 Ashwin Shankar added a comment - Also, a subjective opinion(feel free to ignore the comment) - In the code below, instead of passing a boolean, can we have the original isStarvedFor(Fair|Min)Share which seems more readable to me and put the last two lines of isStarved(boolean) into another method if you think that would be redundant between isStarvedFor(Fair|Min)Share ? FSLeafQueue boolean isStarved( boolean checkMinShare) { Resource share = checkMinShare ? getMinShare() : Resources.multiply(getFairShare(), getFairSharePreemptionThreshold()); Resource desiredShare = Resources.min(scheduler.getResourceCalculator(), scheduler.getClusterResource(), share, getDemand()); return Resources.lessThan(scheduler.getResourceCalculator(), scheduler.getClusterResource(), getResourceUsage(), desiredShare); }
          ywskycn Wei Yan added a comment -

          Thanks, ashwinshankar77. Update a new patch to fix your comments.

          ywskycn Wei Yan added a comment - Thanks, ashwinshankar77 . Update a new patch to fix your comments.
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12666241/YARN-2394-6.patch
          against trunk revision 3425ae5.

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4813//console

          This message is automatically generated.

          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12666241/YARN-2394-6.patch against trunk revision 3425ae5. -1 patch . The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4813//console This message is automatically generated.
          ywskycn Wei Yan added a comment -

          rebase the patch

          ywskycn Wei Yan added a comment - rebase the patch
          hadoopqa Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12666250/YARN-2394-7.patch
          against trunk revision 3425ae5.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-YARN-Build/4814//testReport/
          Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4814//console

          This message is automatically generated.

          hadoopqa Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12666250/YARN-2394-7.patch against trunk revision 3425ae5. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 3 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/4814//testReport/ Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4814//console This message is automatically generated.

          +1. Checking this in..

          kkambatl Karthik Kambatla (Inactive) added a comment - +1. Checking this in..

          Thanks Wei for working on this and Ashwin for the reviews. Just committed this to trunk and branch-2.

          kkambatl Karthik Kambatla (Inactive) added a comment - Thanks Wei for working on this and Ashwin for the reviews. Just committed this to trunk and branch-2.
          ywskycn Wei Yan added a comment -

          Thanks for the reviews, kasha, ashwinshankar77.

          ywskycn Wei Yan added a comment - Thanks for the reviews, kasha , ashwinshankar77 .
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk #670 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/670/)
          YARN-2394. FairScheduler: Configure fairSharePreemptionThreshold per queue. (Wei Yan via kasha) (kasha: rev 1dcaba9a7aa27f7ca4ba693e3abb56ab3c59c8a7)

          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/QueueManager.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSParentQueue.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairScheduler.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/AllocationConfiguration.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/AllocationFileLoaderService.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSQueue.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestAllocationFileLoaderService.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSLeafQueue.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/FairScheduler.apt.vm
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairScheduler.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFSLeafQueue.java
          • hadoop-yarn-project/CHANGES.txt
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #670 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/670/ ) YARN-2394 . FairScheduler: Configure fairSharePreemptionThreshold per queue. (Wei Yan via kasha) (kasha: rev 1dcaba9a7aa27f7ca4ba693e3abb56ab3c59c8a7) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/QueueManager.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSParentQueue.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairScheduler.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/AllocationConfiguration.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/AllocationFileLoaderService.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSQueue.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestAllocationFileLoaderService.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSLeafQueue.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/FairScheduler.apt.vm hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairScheduler.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFSLeafQueue.java hadoop-yarn-project/CHANGES.txt
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk #1861 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1861/)
          YARN-2394. FairScheduler: Configure fairSharePreemptionThreshold per queue. (Wei Yan via kasha) (kasha: rev 1dcaba9a7aa27f7ca4ba693e3abb56ab3c59c8a7)

          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSParentQueue.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairScheduler.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/QueueManager.java
          • hadoop-yarn-project/CHANGES.txt
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestAllocationFileLoaderService.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSQueue.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSLeafQueue.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/AllocationFileLoaderService.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/FairScheduler.apt.vm
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairScheduler.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFSLeafQueue.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/AllocationConfiguration.java
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #1861 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1861/ ) YARN-2394 . FairScheduler: Configure fairSharePreemptionThreshold per queue. (Wei Yan via kasha) (kasha: rev 1dcaba9a7aa27f7ca4ba693e3abb56ab3c59c8a7) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSParentQueue.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairScheduler.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/QueueManager.java hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestAllocationFileLoaderService.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSQueue.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSLeafQueue.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/AllocationFileLoaderService.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/FairScheduler.apt.vm hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairScheduler.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFSLeafQueue.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/AllocationConfiguration.java
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk #1886 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1886/)
          YARN-2394. FairScheduler: Configure fairSharePreemptionThreshold per queue. (Wei Yan via kasha) (kasha: rev 1dcaba9a7aa27f7ca4ba693e3abb56ab3c59c8a7)

          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestAllocationFileLoaderService.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairScheduler.java
          • hadoop-yarn-project/CHANGES.txt
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairScheduler.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/QueueManager.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/FairScheduler.apt.vm
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSLeafQueue.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSParentQueue.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/AllocationConfiguration.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSQueue.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/AllocationFileLoaderService.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFSLeafQueue.java
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #1886 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1886/ ) YARN-2394 . FairScheduler: Configure fairSharePreemptionThreshold per queue. (Wei Yan via kasha) (kasha: rev 1dcaba9a7aa27f7ca4ba693e3abb56ab3c59c8a7) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestAllocationFileLoaderService.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairScheduler.java hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairScheduler.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/QueueManager.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/FairScheduler.apt.vm hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSLeafQueue.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSParentQueue.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/AllocationConfiguration.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSQueue.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/AllocationFileLoaderService.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFSLeafQueue.java

          People

            ywskycn Wei Yan
            ashwinshankar77 Ashwin Shankar
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: