Details
-
Bug
-
Status: Resolved
-
Minor
-
Resolution: Fixed
-
2.7.1
-
None
-
Reviewed
Description
When the queue with vcores is in decimal value, the value after the decimal point is taken as vcores by FairScheduler.
For the below queue,
<minResources>20000 mb,20 vcores,20.25 disks</minResources>
<maxResources>30000 mb,40.2 vcores,30.25 disks</maxResources>
When many applications submitted parallely into queue, all were in PENDING state as the vcores is taken as 2 skipping the value 40.
The code FairSchedulerConfiguration.java to Pattern match the vcores has to be improved in such a way either throw AllocationConfigurationException("Missing resource") or consider the value before decimal.
Attachments
Attachments
- YARN-4256.001.patch
- 3 kB
- Jun Gong
- YARN-4256.002.patch
- 3 kB
- Jun Gong
Activity
+1 overall |
Vote | Subsystem | Runtime | Comment |
---|---|---|---|
0 | pre-patch | 19m 4s | Pre-patch trunk compilation is healthy. |
+1 | @author | 0m 0s | The patch does not contain any @author tags. |
+1 | tests included | 0m 0s | The patch appears to include 1 new or modified test files. |
+1 | javac | 8m 48s | There were no new javac warning messages. |
+1 | javadoc | 11m 47s | There were no new javadoc warning messages. |
+1 | release audit | 0m 25s | The applied patch does not increase the total number of release audit warnings. |
+1 | checkstyle | 0m 55s | There were no new checkstyle issues. |
+1 | whitespace | 0m 0s | The patch has no lines that end in whitespace. |
+1 | install | 1m 41s | mvn install still works. |
+1 | eclipse:eclipse | 0m 37s | The patch built with eclipse:eclipse. |
+1 | findbugs | 1m 40s | The patch does not introduce any new Findbugs (version 3.0.0) warnings. |
+1 | yarn tests | 63m 3s | Tests passed in hadoop-yarn-server-resourcemanager. |
108m 4s |
Subsystem | Report/Notes |
---|---|
Patch URL | http://issues.apache.org/jira/secure/attachment/12766517/YARN-4256.001.patch |
Optional Tests | javadoc javac unit findbugs checkstyle |
git revision | trunk / d6c8bad |
hadoop-yarn-server-resourcemanager test log | https://builds.apache.org/job/PreCommit-YARN-Build/9440/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt |
Test Results | https://builds.apache.org/job/PreCommit-YARN-Build/9440/testReport/ |
Java | 1.7.0_55 |
uname | Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
Console output | https://builds.apache.org/job/PreCommit-YARN-Build/9440/console |
This message was automatically generated.
Thanks for reporting this issue prabhujoseph! Thanks for the patch hex108! The patch looks most good. Can we change '+' to '*'
(\\.
d+)? => (\\.
d*)? So we can relax the condition to support 1024. mb.
Thanks zxu for the review and comments.
Attach a new patch to address it and add a new test for the case '1024. mb'.
+1 overall |
Vote | Subsystem | Runtime | Comment |
---|---|---|---|
0 | pre-patch | 19m 26s | Pre-patch trunk compilation is healthy. |
+1 | @author | 0m 0s | The patch does not contain any @author tags. |
+1 | tests included | 0m 0s | The patch appears to include 1 new or modified test files. |
+1 | javac | 9m 32s | There were no new javac warning messages. |
+1 | javadoc | 11m 47s | There were no new javadoc warning messages. |
+1 | release audit | 0m 26s | The applied patch does not increase the total number of release audit warnings. |
+1 | checkstyle | 0m 57s | There were no new checkstyle issues. |
+1 | whitespace | 0m 0s | The patch has no lines that end in whitespace. |
+1 | install | 1m 48s | mvn install still works. |
+1 | eclipse:eclipse | 0m 38s | The patch built with eclipse:eclipse. |
+1 | findbugs | 1m 36s | The patch does not introduce any new Findbugs (version 3.0.0) warnings. |
+1 | yarn tests | 64m 35s | Tests passed in hadoop-yarn-server-resourcemanager. |
110m 49s |
Subsystem | Report/Notes |
---|---|
Patch URL | http://issues.apache.org/jira/secure/attachment/12767706/YARN-4256.002.patch |
Optional Tests | javadoc javac unit findbugs checkstyle |
git revision | trunk / 0c4af0f |
hadoop-yarn-server-resourcemanager test log | https://builds.apache.org/job/PreCommit-YARN-Build/9504/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt |
Test Results | https://builds.apache.org/job/PreCommit-YARN-Build/9504/testReport/ |
Java | 1.7.0_55 |
uname | Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
Console output | https://builds.apache.org/job/PreCommit-YARN-Build/9504/console |
This message was automatically generated.
committed it to trunk and branch-2, Thanks prabhujoseph for reporting this issue, thanks hex108 for the patch and thanks brahmareddy for additional review!
FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #570 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/570/)
YARN-4256. YARN fair scheduler vcores with decimal values. Contributed (zxu: rev 960201b79b9f2ca40f8eadb21a2f9fe37dde2b5d)
- hadoop-yarn-project/CHANGES.txt
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairSchedulerConfiguration.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairSchedulerConfiguration.java
FAILURE: Integrated in Hadoop-trunk-Commit #8691 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8691/)
YARN-4256. YARN fair scheduler vcores with decimal values. Contributed (zxu: rev 960201b79b9f2ca40f8eadb21a2f9fe37dde2b5d)
- hadoop-yarn-project/CHANGES.txt
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairSchedulerConfiguration.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairSchedulerConfiguration.java
SUCCESS: Integrated in Hadoop-Yarn-trunk #1306 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1306/)
YARN-4256. YARN fair scheduler vcores with decimal values. Contributed (zxu: rev 960201b79b9f2ca40f8eadb21a2f9fe37dde2b5d)
- hadoop-yarn-project/CHANGES.txt
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairSchedulerConfiguration.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairSchedulerConfiguration.java
FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #585 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/585/)
YARN-4256. YARN fair scheduler vcores with decimal values. Contributed (zxu: rev 960201b79b9f2ca40f8eadb21a2f9fe37dde2b5d)
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairSchedulerConfiguration.java
- hadoop-yarn-project/CHANGES.txt
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairSchedulerConfiguration.java
FAILURE: Integrated in Hadoop-Mapreduce-trunk #2517 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2517/)
YARN-4256. YARN fair scheduler vcores with decimal values. Contributed (zxu: rev 960201b79b9f2ca40f8eadb21a2f9fe37dde2b5d)
- hadoop-yarn-project/CHANGES.txt
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairSchedulerConfiguration.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairSchedulerConfiguration.java
FAILURE: Integrated in Hadoop-Hdfs-trunk #2465 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2465/)
YARN-4256. YARN fair scheduler vcores with decimal values. Contributed (zxu: rev 960201b79b9f2ca40f8eadb21a2f9fe37dde2b5d)
- hadoop-yarn-project/CHANGES.txt
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairSchedulerConfiguration.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairSchedulerConfiguration.java
FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #528 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/528/)
YARN-4256. YARN fair scheduler vcores with decimal values. Contributed (zxu: rev 960201b79b9f2ca40f8eadb21a2f9fe37dde2b5d)
- hadoop-yarn-project/CHANGES.txt
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairSchedulerConfiguration.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairSchedulerConfiguration.java
Thanks prabhujoseph for reporting. How about considering the value before decimal? If OK, I will attach a patch.