Details
-
Type:
Bug
-
Status: Resolved
-
Priority:
Major
-
Resolution: Fixed
-
Affects Version/s: None
-
Fix Version/s: 2.8.0, 3.0.0-alpha1
-
Component/s: fairscheduler
-
Labels:None
Description
I has been noticed that for off-switch requests, the FairScheduler reserves resources on all nodes. This could lead to the entire cluster being unavailable for all other applications.
Ideally, the reservations should be on a configurable number of nodes, default 1.
Issue Links
Activity
- All
- Comments
- Work Log
- History
- Activity
- Transitions
 -1 overall |
| Vote | Subsystem | Runtime | Comment |
|---|---|---|---|
| 0 | pre-patch | 18m 51s | Pre-patch trunk compilation is healthy. |
| +1 | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 | tests included | 0m 0s | The patch appears to include 1 new or modified test files. |
| +1 | javac | 7m 50s | There were no new javac warning messages. |
| +1 | javadoc | 10m 18s | There were no new javadoc warning messages. |
| +1 | release audit | 0m 24s | The applied patch does not increase the total number of release audit warnings. |
| -1 | checkstyle | 1m 32s | The applied patch generated 3 new checkstyle issues (total was 211, now 213). |
| -1 | whitespace | 0m 2s | The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix. |
| +1 | install | 1m 32s | mvn install still works. |
| +1 | eclipse:eclipse | 0m 34s | The patch built with eclipse:eclipse. |
| +1 | findbugs | 3m 5s | The patch does not introduce any new Findbugs (version 3.0.0) warnings. |
| -1 | yarn tests | 0m 22s | Tests failed in hadoop-yarn-api. |
| -1 | yarn tests | 61m 33s | Tests failed in hadoop-yarn-server-resourcemanager. |
| 106m 23s |
| Reason | Tests |
|---|---|
| Failed unit tests | hadoop.yarn.conf.TestYarnConfigurationFields |
| hadoop.yarn.server.resourcemanager.security.TestRMDelegationTokens |
This message was automatically generated.
Updating patch :
- Restricted changes to only the FairScheduler
- Some changes to testcase
| -1 overall |
| Vote | Subsystem | Runtime | Comment |
|---|---|---|---|
| 0 | pre-patch | 16m 57s | Pre-patch trunk compilation is healthy. |
| +1 | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 | tests included | 0m 0s | The patch appears to include 1 new or modified test files. |
| +1 | javac | 7m 50s | There were no new javac warning messages. |
| +1 | javadoc | 10m 24s | There were no new javadoc warning messages. |
| +1 | release audit | 0m 24s | The applied patch does not increase the total number of release audit warnings. |
| -1 | checkstyle | 0m 49s | The applied patch generated 3 new checkstyle issues (total was 128, now 128). |
| +1 | whitespace | 0m 2s | The patch has no lines that end in whitespace. |
| +1 | install | 1m 29s | mvn install still works. |
| +1 | eclipse:eclipse | 0m 34s | The patch built with eclipse:eclipse. |
| +1 | findbugs | 1m 28s | The patch does not introduce any new Findbugs (version 3.0.0) warnings. |
| -1 | yarn tests | 61m 57s | Tests failed in hadoop-yarn-server-resourcemanager. |
| 101m 58s |
| Reason | Tests |
|---|---|
| Failed unit tests | hadoop.yarn.server.resourcemanager.TestSubmitApplicationWithRMHA |
| Subsystem | Report/Notes |
|---|---|
| Patch URL | http://issues.apache.org/jira/secure/attachment/12767243/YARN-4270.2.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 58590fe |
| checkstyle | https://builds.apache.org/job/PreCommit-YARN-Build/9471/artifact/patchprocess/diffcheckstylehadoop-yarn-server-resourcemanager.txt |
| hadoop-yarn-server-resourcemanager test log | https://builds.apache.org/job/PreCommit-YARN-Build/9471/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt |
| Test Results | https://builds.apache.org/job/PreCommit-YARN-Build/9471/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output | https://builds.apache.org/job/PreCommit-YARN-Build/9471/console |
This message was automatically generated.
Previous test case failure was spurious.
Updating Patch anyway:
- Added one more testcase to verify behavior for rack local reservations
- Fixed checkstyle warnings
Karthik Kambatla, do you want to give it a quick rev ?
| -1 overall |
| Vote | Subsystem | Runtime | Comment |
|---|---|---|---|
| 0 | pre-patch | 17m 45s | Pre-patch trunk compilation is healthy. |
| +1 | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 | tests included | 0m 0s | The patch appears to include 1 new or modified test files. |
| +1 | javac | 8m 15s | There were no new javac warning messages. |
| +1 | javadoc | 10m 37s | There were no new javadoc warning messages. |
| +1 | release audit | 0m 23s | The applied patch does not increase the total number of release audit warnings. |
| -1 | checkstyle | 0m 52s | The applied patch generated 1 new checkstyle issues (total was 128, now 129). |
| +1 | whitespace | 0m 3s | The patch has no lines that end in whitespace. |
| +1 | install | 1m 31s | mvn install still works. |
| +1 | eclipse:eclipse | 0m 35s | The patch built with eclipse:eclipse. |
| +1 | findbugs | 1m 30s | The patch does not introduce any new Findbugs (version 3.0.0) warnings. |
| +1 | yarn tests | 62m 15s | Tests passed in hadoop-yarn-server-resourcemanager. |
| 103m 51s |
| Subsystem | Report/Notes |
|---|---|
| Patch URL | http://issues.apache.org/jira/secure/attachment/12767259/YARN-4270.3.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 0ab3f9d |
| checkstyle | https://builds.apache.org/job/PreCommit-YARN-Build/9473/artifact/patchprocess/diffcheckstylehadoop-yarn-server-resourcemanager.txt |
| hadoop-yarn-server-resourcemanager test log | https://builds.apache.org/job/PreCommit-YARN-Build/9473/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt |
| Test Results | https://builds.apache.org/job/PreCommit-YARN-Build/9473/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output | https://builds.apache.org/job/PreCommit-YARN-Build/9473/console |
This message was automatically generated.
Addressing checkstyle warning..
 +1 overall |
| Vote | Subsystem | Runtime | Comment |
|---|---|---|---|
| 0 | pre-patch | 17m 33s | Pre-patch trunk compilation is healthy. |
| +1 | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 | tests included | 0m 0s | The patch appears to include 1 new or modified test files. |
| +1 | javac | 8m 11s | There were no new javac warning messages. |
| +1 | javadoc | 10m 47s | There were no new javadoc warning messages. |
| +1 | release audit | 0m 24s | The applied patch does not increase the total number of release audit warnings. |
| +1 | checkstyle | 0m 50s | There were no new checkstyle issues. |
| +1 | whitespace | 0m 3s | The patch has no lines that end in whitespace. |
| +1 | install | 1m 35s | mvn install still works. |
| +1 | eclipse:eclipse | 0m 34s | The patch built with eclipse:eclipse. |
| +1 | findbugs | 1m 32s | The patch does not introduce any new Findbugs (version 3.0.0) warnings. |
| +1 | yarn tests | 63m 16s | Tests passed in hadoop-yarn-server-resourcemanager. |
| 104m 48s |
| Subsystem | Report/Notes |
|---|---|
| Patch URL | http://issues.apache.org/jira/secure/attachment/12767262/YARN-4270.4.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 0ab3f9d |
| hadoop-yarn-server-resourcemanager test log | https://builds.apache.org/job/PreCommit-YARN-Build/9474/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt |
| Test Results | https://builds.apache.org/job/PreCommit-YARN-Build/9474/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output | https://builds.apache.org/job/PreCommit-YARN-Build/9474/console |
This message was automatically generated.
Thanks for reporting and working on this, Arun. The approach looks good to me. Few minor comments on the patch:
- Can we call the config and variables reservableNodes? Calling it appReservationThreshold increases the ambiguity with SLA-reservations.
- In FairScheduler#removeNode, can we log an error in the else block. i.e., when a rack is not found in the app or has a node-count <=0?
- FSAppAttempt: can we add a comment to describe what the map holds? i.e., what String corresponds to?
private Map<String, Set<String>> reservations = new HashMap<>();
- Should we use a smaller value for the default? Consider a cluster with 2k nodes: with this default, we end up reserving on 1000 nodes. How about using 0.05?
- I believe we should be reserving nodes for an app only after the locality of the corresponding RR is fully relaxed. That is likely for another JIRA.
- Nit: Spurious changes in FairSchedulerConfiguration
Added comment too soon. Couple of comments:
- FSAppAttempt#reservationExceedsThreshold: Instead of doing a ratio, would it be better to check existingReservations > Math.ceil(reservableNodes() * totalAvailableNodes)? We avoid a division and the check to see if there is at least one reservation is not necessary.
- Nice tests.
Thanks for the review Karthik Kambatla !!
Updating patch with your suggestions. Will file a follow up for delaying reservations until locality relaxation has maxed out.
| +1 overall |
| Vote | Subsystem | Runtime | Comment |
|---|---|---|---|
| 0 | pre-patch | 20m 8s | Pre-patch trunk compilation is healthy. |
| +1 | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 | tests included | 0m 0s | The patch appears to include 1 new or modified test files. |
| +1 | javac | 8m 59s | There were no new javac warning messages. |
| +1 | javadoc | 11m 35s | There were no new javadoc warning messages. |
| +1 | release audit | 0m 24s | The applied patch does not increase the total number of release audit warnings. |
| +1 | checkstyle | 0m 55s | There were no new checkstyle issues. |
| +1 | whitespace | 0m 4s | The patch has no lines that end in whitespace. |
| +1 | install | 1m 41s | mvn install still works. |
| +1 | eclipse:eclipse | 0m 38s | The patch built with eclipse:eclipse. |
| +1 | findbugs | 1m 44s | The patch does not introduce any new Findbugs (version 3.0.0) warnings. |
| +1 | yarn tests | 58m 40s | Tests passed in hadoop-yarn-server-resourcemanager. |
| 104m 54s |
| Subsystem | Report/Notes |
|---|---|
| Patch URL | http://issues.apache.org/jira/secure/attachment/12767467/YARN-4270.5.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 6144e01 |
| hadoop-yarn-server-resourcemanager test log | https://builds.apache.org/job/PreCommit-YARN-Build/9484/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt |
| Test Results | https://builds.apache.org/job/PreCommit-YARN-Build/9484/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output | https://builds.apache.org/job/PreCommit-YARN-Build/9484/console |
This message was automatically generated.
+1
Committed to trunk and branch-2. Thanks again for the review Karthik Kambatla
FAILURE: Integrated in Hadoop-trunk-Commit #8668 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8668/)
YARN-4270. Limit application resource reservation on nodes for (arun suresh: rev 7e2837f830382835838c82398db6fc9823d612a7)
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairScheduler.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairScheduler.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSAppAttempt.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairSchedulerConfiguration.java
- hadoop-yarn-project/CHANGES.txt
FAILURE: Integrated in Hadoop-Yarn-trunk #1291 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1291/)
YARN-4270. Limit application resource reservation on nodes for (arun suresh: rev 7e2837f830382835838c82398db6fc9823d612a7)
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSAppAttempt.java
- hadoop-yarn-project/CHANGES.txt
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairScheduler.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairSchedulerConfiguration.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairScheduler.java
FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #555 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/555/)
YARN-4270. Limit application resource reservation on nodes for (arun suresh: rev 7e2837f830382835838c82398db6fc9823d612a7)
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairScheduler.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSAppAttempt.java
- hadoop-yarn-project/CHANGES.txt
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairSchedulerConfiguration.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairScheduler.java
FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #571 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/571/)
YARN-4270. Limit application resource reservation on nodes for (arun suresh: rev 7e2837f830382835838c82398db6fc9823d612a7)
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairScheduler.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairScheduler.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSAppAttempt.java
- hadoop-yarn-project/CHANGES.txt
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairSchedulerConfiguration.java
FAILURE: Integrated in Hadoop-Mapreduce-trunk #2504 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2504/)
YARN-4270. Limit application resource reservation on nodes for (arun suresh: rev 7e2837f830382835838c82398db6fc9823d612a7)
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairScheduler.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairScheduler.java
- hadoop-yarn-project/CHANGES.txt
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSAppAttempt.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairSchedulerConfiguration.java
FAILURE: Integrated in Hadoop-Hdfs-trunk #2454 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2454/)
YARN-4270. Limit application resource reservation on nodes for (arun suresh: rev 7e2837f830382835838c82398db6fc9823d612a7)
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSAppAttempt.java
- hadoop-yarn-project/CHANGES.txt
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairScheduler.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairScheduler.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairSchedulerConfiguration.java
FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #517 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/517/)
YARN-4270. Limit application resource reservation on nodes for (arun suresh: rev 7e2837f830382835838c82398db6fc9823d612a7)
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSAppAttempt.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairScheduler.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairSchedulerConfiguration.java
- hadoop-yarn-project/CHANGES.txt
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairScheduler.java
Attaching initial patch