Details
-
New Feature
-
Status: Closed
-
Major
-
Resolution: Fixed
-
None
-
None
-
Reviewed
Description
Preemption based on fair share starvation happens when usage of a queue is less than 50% of its fair share. This 50% is hardcoded. We'd like to make this configurable on a per queue basis, so that we can choose the threshold at which we want to preempt. Calling this config "fairSharePreemptionThreshold".
Attachments
Attachments
- YARN-2394-1.patch
- 23 kB
- Wei Yan
- YARN-2394-2.patch
- 34 kB
- Wei Yan
- YARN-2394-3.patch
- 39 kB
- Wei Yan
- YARN-2394-4.patch
- 46 kB
- Wei Yan
- YARN-2394-5.patch
- 55 kB
- Wei Yan
- YARN-2394-6.patch
- 56 kB
- Wei Yan
- YARN-2394-7.patch
- 56 kB
- Wei Yan
Activity
Update a patch following the same way as YARN-2395. Each queue inherits fairSharePreemptionThreshold from its parent queue if it doesn't configure in the allocation file. Will rebase the patch once YARN-2395 is in.
ywskycn, kasha
Inheriting fairSharePreemptionThreshold wouldn't work for us. Do you guys have a use case for inheriting this config ?
We have the following requirement :
root.parentUserQueue : fair share = 80% fairSharePreemptionThreshold=1
root.parentUserQueue.user(1-100) : defaultFairSharePreemptionThreshold=0.5
Basically when parentUserQueue is starved, we want to preempt more aggressively(threshold=1) from its siblings under root.
While when a user queue is starved, we want to preempt less aggressively from other user queues(assuming other user queues are above fair share).
We can do this by setting fairSharePreemptionThreshold=1 at parentUserQueue and defaultFairSharePreemptionThreshold=0.5 which would be used by user queues.
You might say that we can override fairSharePreemptionThreshold to 0.5 in the user queues, but these queues are created on the fly when a new user
submits an app and NOT through the alloc xml and there are hundreds of them, so its not very feasible to override each one of them in the config.
Would it be okay if we stick to version-1 of the patch which doesn't inherit threshold ?
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12663259/YARN-2394-2.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 2 new or modified test files.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 javadoc. There were no new javadoc warning messages.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-YARN-Build/4749//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4749//console
This message is automatically generated.
If the user queues inherit from the parent and have a threshold of 1, the scheduler would never preempt unless the queue has over its fairshare. If we are preempting resources that are over the fairshare, I think it is fair game. We can improve choosing the victim, but we can work on it in another JIRA.
OTOH, choosing default-threshold of 0.5 for user queues can lead to:
- one of the user queues starve (stay under fairshare) while other user queues are higher than threshold
- depending on preemption logic (being revisited in
YARN-2154), it might take a while to reach the threshold for parent queues
I have different opinion on this, but I spoke to some guys here, I think we can work around by setting higher timeouts or slightly lesser threshold(0.8/0.9) at the parent to prevent aggressive preemption at its children.
So I'm fine with inheriting.
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12665552/YARN-2394-3.patch
against trunk revision 0f34e6f.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 2 new or modified test files.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 javadoc. There were no new javadoc warning messages.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-YARN-Build/4787//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4787//console
This message is automatically generated.
Just noticed that there is duplication between FSQueue#updatePreemptionParameters and QueueManager#setPreemptionParameters? Can we get rid of one?
Other than that, I see a couple of lines over 80 chars in AllocationFileLoaderService.
Thanks, kasha, updated a patch to address the two comments. And, move the two isStarved functions in FairScheduler to FSLeafQueue and combine to one function; add one testcase to verify the online created queue's preemption configurations.
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12665585/YARN-2394-4.patch
against trunk revision 258c7d0.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 2 new or modified test files.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 javadoc. There were no new javadoc warning messages.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed these unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesAppsModification
org.apache.hadoop.yarn.server.resourcemanager.TestApplicationACLs
org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesNodes
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-YARN-Build/4789//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4789//console
This message is automatically generated.
Thanks for cleaning up the FairScheduler code, Wei. It looks much better now. While at it, can we do a couple of other things:
- FSLeafQueue#isStarved need not take the ResourceCalculator and resource - we can fetch this from the scheduler.
- How about renaming FairScheduler#updatePreemptionVariables to FairScheduler#updateStarvationStats?
- Add a new method FSLeafQueue#updateStarvationStats that handles the stats for a leaf queue. FairScheduler#updateStarvationStats can call this. Subsequently, we can make FSLeafQueue#setLastTimeAt and FSLeafQueue#isStarved methods private (or package-private with VisibleForTesting annotation if we need these for tests).
- Now that we are moving some of these methods to FSLeafQueue and FSQueue, can we move the corresponding method unit tests to respective TestFS*Queue files where possible? These tests should extend FairSchedulerTestBase and the setup etc. should be modeled after TestFairSchedulerFairShare - fetch the scheduler from the MockRM instead of creating a separate scheduler and using the RMContext from an RM?
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12665858/YARN-2394-5.patch
against trunk revision 258c7d0.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified test files.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 javadoc. There were no new javadoc warning messages.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed these unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
org.apache.hadoop.yarn.server.resourcemanager.TestKillApplicationWithRMHA
org.apache.hadoop.yarn.server.resourcemanager.recovery.TestZKRMStateStore
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-YARN-Build/4796//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4796//console
This message is automatically generated.
ywskycn, patch looks almost good. Couple of comments :
1.
Resource desiredShare = Resources.min(scheduler.getResourceCalculator(),
scheduler.getClusterResource(), share, getDemand());
return Resources.lessThan(scheduler.getResourceCalculator(),
scheduler.getClusterResource(), getResourceUsage(), desiredShare);
getResourceCalculator() is a static method, can you please access it through the class and not the object.
2. Nit
// The fair share preemption threshold for each queue. It a queue waits
"It" should be "If"
Also, a subjective opinion(feel free to ignore the comment) - In the code below, instead of passing a boolean, can we have the original
isStarvedFor(Fair|Min)Share which seems more readable to me and put the last two lines of isStarved(boolean) into another method if you
think that would be redundant between isStarvedFor(Fair|Min)Share ?
boolean isStarved(boolean checkMinShare) { Resource share = checkMinShare ? getMinShare() : Resources.multiply(getFairShare(), getFairSharePreemptionThreshold()); Resource desiredShare = Resources.min(scheduler.getResourceCalculator(), scheduler.getClusterResource(), share, getDemand()); return Resources.lessThan(scheduler.getResourceCalculator(), scheduler.getClusterResource(), getResourceUsage(), desiredShare); }
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12666241/YARN-2394-6.patch
against trunk revision 3425ae5.
-1 patch. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4813//console
This message is automatically generated.
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12666250/YARN-2394-7.patch
against trunk revision 3425ae5.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified test files.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 javadoc. There were no new javadoc warning messages.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-YARN-Build/4814//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4814//console
This message is automatically generated.
Thanks Wei for working on this and Ashwin for the reviews. Just committed this to trunk and branch-2.
FAILURE: Integrated in Hadoop-Yarn-trunk #670 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/670/)
YARN-2394. FairScheduler: Configure fairSharePreemptionThreshold per queue. (Wei Yan via kasha) (kasha: rev 1dcaba9a7aa27f7ca4ba693e3abb56ab3c59c8a7)
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/QueueManager.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSParentQueue.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairScheduler.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/AllocationConfiguration.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/AllocationFileLoaderService.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSQueue.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestAllocationFileLoaderService.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSLeafQueue.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/FairScheduler.apt.vm
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairScheduler.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFSLeafQueue.java
- hadoop-yarn-project/CHANGES.txt
FAILURE: Integrated in Hadoop-Hdfs-trunk #1861 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1861/)
YARN-2394. FairScheduler: Configure fairSharePreemptionThreshold per queue. (Wei Yan via kasha) (kasha: rev 1dcaba9a7aa27f7ca4ba693e3abb56ab3c59c8a7)
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSParentQueue.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairScheduler.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/QueueManager.java
- hadoop-yarn-project/CHANGES.txt
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestAllocationFileLoaderService.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSQueue.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSLeafQueue.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/AllocationFileLoaderService.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/FairScheduler.apt.vm
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairScheduler.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFSLeafQueue.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/AllocationConfiguration.java
FAILURE: Integrated in Hadoop-Mapreduce-trunk #1886 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1886/)
YARN-2394. FairScheduler: Configure fairSharePreemptionThreshold per queue. (Wei Yan via kasha) (kasha: rev 1dcaba9a7aa27f7ca4ba693e3abb56ab3c59c8a7)
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestAllocationFileLoaderService.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairScheduler.java
- hadoop-yarn-project/CHANGES.txt
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairScheduler.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/QueueManager.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/FairScheduler.apt.vm
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSLeafQueue.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSParentQueue.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/AllocationConfiguration.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSQueue.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/AllocationFileLoaderService.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFSLeafQueue.java
I'll look into this.