Details
-
Sub-task
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
None
-
None
-
Reviewed
Description
YARN-5881 introduced the notion of configuring queues with Absolute resource specifications instead of percentage. As part of that , each leaf queue has an effective min/max capacity that needs to be set when queue is created and whenever queue capacity is changed
Attachments
Attachments
- YARN-7632.3.patch
- 23 kB
- Suma Shivaprasad
- YARN-7632.2.patch
- 23 kB
- Suma Shivaprasad
- YARN-7632.1.patch
- 15 kB
- Suma Shivaprasad
Activity
-1 overall |
Vote | Subsystem | Runtime | Comment |
---|---|---|---|
0 | reexec | 2m 27s | Docker mode activated. |
Prechecks | |||
+1 | @author | 0m 0s | The patch does not contain any @author tags. |
+1 | test4tests | 0m 0s | The patch appears to include 2 new or modified test files. |
trunk Compile Tests | |||
+1 | mvninstall | 18m 0s | trunk passed |
+1 | compile | 0m 41s | trunk passed |
+1 | checkstyle | 0m 29s | trunk passed |
+1 | mvnsite | 0m 45s | trunk passed |
+1 | shadedclient | 11m 6s | branch has no errors when building and testing our client artifacts. |
+1 | findbugs | 1m 12s | trunk passed |
+1 | javadoc | 0m 27s | trunk passed |
Patch Compile Tests | |||
+1 | mvninstall | 0m 43s | the patch passed |
+1 | compile | 0m 42s | the patch passed |
+1 | javac | 0m 42s | the patch passed |
-0 | checkstyle | 0m 28s | hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 7 new + 17 unchanged - 0 fixed = 24 total (was 17) |
+1 | mvnsite | 0m 43s | the patch passed |
+1 | whitespace | 0m 0s | The patch has no whitespace issues. |
+1 | shadedclient | 11m 4s | patch has no errors when building and testing our client artifacts. |
+1 | findbugs | 1m 23s | the patch passed |
+1 | javadoc | 0m 26s | the patch passed |
Other Tests | |||
-1 | unit | 61m 29s | hadoop-yarn-server-resourcemanager in the patch failed. |
+1 | asflicense | 0m 20s | The patch does not generate ASF License warnings. |
112m 14s |
Reason | Tests |
---|---|
Failed junit tests | hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesSchedulerActivities |
hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesCapacitySched | |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestNodeLabelContainerAllocation |
Subsystem | Report/Notes |
---|---|
Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
JIRA Issue | |
JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12901384/YARN-7632.1.patch |
Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle |
uname | Linux 6f12c8ccf803 3.13.0-129-generic #178-Ubuntu SMP Fri Aug 11 12:48:20 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
Build tool | maven |
Personality | /testptch/patchprocess/precommit/personality/provided.sh |
git revision | trunk / 670e8d4 |
maven | version: Apache Maven 3.3.9 |
Default Java | 1.8.0_151 |
findbugs | v3.1.0-RC1 |
checkstyle | https://builds.apache.org/job/PreCommit-YARN-Build/18853/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt |
unit | https://builds.apache.org/job/PreCommit-YARN-Build/18853/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt |
Test Results | https://builds.apache.org/job/PreCommit-YARN-Build/18853/testReport/ |
Max. process+thread count | 806 (vs. ulimit of 5000) |
modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager |
Console output | https://builds.apache.org/job/PreCommit-YARN-Build/18853/console |
Powered by | Apache Yetus 0.7.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
Thanks suma.shivaprasad.
Could you please help to add a case where nodes are added/removed and cross validate it with getEffectiveMinResource/getEffectiveMaxResource of new queues.
Thanks sunilg Added testcase where nodes are added/reduced and effective capacity increase/decreased accordingly
-1 overall |
Vote | Subsystem | Runtime | Comment |
---|---|---|---|
0 | reexec | 15m 12s | Docker mode activated. |
Prechecks | |||
+1 | @author | 0m 0s | The patch does not contain any @author tags. |
+1 | test4tests | 0m 0s | The patch appears to include 3 new or modified test files. |
trunk Compile Tests | |||
+1 | mvninstall | 15m 10s | trunk passed |
+1 | compile | 0m 37s | trunk passed |
+1 | checkstyle | 0m 25s | trunk passed |
+1 | mvnsite | 0m 39s | trunk passed |
+1 | shadedclient | 10m 21s | branch has no errors when building and testing our client artifacts. |
+1 | findbugs | 0m 59s | trunk passed |
+1 | javadoc | 0m 24s | trunk passed |
Patch Compile Tests | |||
+1 | mvninstall | 0m 38s | the patch passed |
+1 | compile | 0m 33s | the patch passed |
+1 | javac | 0m 33s | the patch passed |
-0 | checkstyle | 0m 22s | hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 16 new + 19 unchanged - 2 fixed = 35 total (was 21) |
+1 | mvnsite | 0m 36s | the patch passed |
+1 | whitespace | 0m 0s | The patch has no whitespace issues. |
+1 | shadedclient | 10m 23s | patch has no errors when building and testing our client artifacts. |
+1 | findbugs | 1m 7s | the patch passed |
+1 | javadoc | 0m 22s | the patch passed |
Other Tests | |||
-1 | unit | 61m 56s | hadoop-yarn-server-resourcemanager in the patch failed. |
+1 | asflicense | 0m 17s | The patch does not generate ASF License warnings. |
120m 38s |
Reason | Tests |
---|---|
Failed junit tests | hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesSchedulerActivities |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestNodeLabelContainerAllocation | |
hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesCapacitySched |
Subsystem | Report/Notes |
---|---|
Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
JIRA Issue | |
JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12901458/YARN-7632.2.patch |
Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle |
uname | Linux de623631352e 4.4.0-64-generic #85-Ubuntu SMP Mon Feb 20 11:50:30 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
Build tool | maven |
Personality | /testptch/patchprocess/precommit/personality/provided.sh |
git revision | trunk / a2edc4c |
maven | version: Apache Maven 3.3.9 |
Default Java | 1.8.0_151 |
findbugs | v3.1.0-RC1 |
checkstyle | https://builds.apache.org/job/PreCommit-YARN-Build/18860/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt |
unit | https://builds.apache.org/job/PreCommit-YARN-Build/18860/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt |
Test Results | https://builds.apache.org/job/PreCommit-YARN-Build/18860/testReport/ |
Max. process+thread count | 868 (vs. ulimit of 5000) |
modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager |
Console output | https://builds.apache.org/job/PreCommit-YARN-Build/18860/console |
Powered by | Apache Yetus 0.7.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
Patch seems fine. But checkstyle issue could be fixable as many are indentation issues.
Sorry. I missed couple of more things and we have to fix checkstyle, hence sharing same.
- Could we move toSet to a util. Many class copy this
- NODE1_MEMORY is 16 and its same for all nodes. We could have only one variable.
-1 overall |
Vote | Subsystem | Runtime | Comment |
---|---|---|---|
0 | reexec | 0m 15s | Docker mode activated. |
Prechecks | |||
+1 | @author | 0m 0s | The patch does not contain any @author tags. |
+1 | test4tests | 0m 0s | The patch appears to include 4 new or modified test files. |
trunk Compile Tests | |||
+1 | mvninstall | 16m 45s | trunk passed |
+1 | compile | 0m 37s | trunk passed |
+1 | checkstyle | 0m 27s | trunk passed |
+1 | mvnsite | 0m 41s | trunk passed |
+1 | shadedclient | 10m 42s | branch has no errors when building and testing our client artifacts. |
+1 | findbugs | 1m 8s | trunk passed |
+1 | javadoc | 0m 25s | trunk passed |
Patch Compile Tests | |||
+1 | mvninstall | 0m 42s | the patch passed |
+1 | compile | 0m 37s | the patch passed |
+1 | javac | 0m 37s | the patch passed |
-0 | checkstyle | 0m 26s | hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 10 new + 62 unchanged - 2 fixed = 72 total (was 64) |
+1 | mvnsite | 0m 38s | the patch passed |
+1 | whitespace | 0m 0s | The patch has no whitespace issues. |
+1 | shadedclient | 11m 3s | patch has no errors when building and testing our client artifacts. |
+1 | findbugs | 1m 10s | the patch passed |
+1 | javadoc | 0m 24s | the patch passed |
Other Tests | |||
-1 | unit | 60m 57s | hadoop-yarn-server-resourcemanager in the patch failed. |
+1 | asflicense | 0m 21s | The patch does not generate ASF License warnings. |
107m 8s |
Reason | Tests |
---|---|
Failed junit tests | hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesSchedulerActivities |
hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesCapacitySched | |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestNodeLabelContainerAllocation |
Subsystem | Report/Notes |
---|---|
Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
JIRA Issue | |
JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12901480/YARN-7632.3.patch |
Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle |
uname | Linux ff4eba2ce599 3.13.0-129-generic #178-Ubuntu SMP Fri Aug 11 12:48:20 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
Build tool | maven |
Personality | /testptch/patchprocess/precommit/personality/provided.sh |
git revision | trunk / a2edc4c |
maven | version: Apache Maven 3.3.9 |
Default Java | 1.8.0_151 |
findbugs | v3.1.0-RC1 |
checkstyle | https://builds.apache.org/job/PreCommit-YARN-Build/18861/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt |
unit | https://builds.apache.org/job/PreCommit-YARN-Build/18861/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt |
Test Results | https://builds.apache.org/job/PreCommit-YARN-Build/18861/testReport/ |
Max. process+thread count | 820 (vs. ulimit of 5000) |
modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager |
Console output | https://builds.apache.org/job/PreCommit-YARN-Build/18861/console |
Powered by | Apache Yetus 0.7.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13353 (See https://builds.apache.org/job/Hadoop-trunk-Commit/13353/)
YARN-7632. Effective min and max resource need to be set for auto (sunilg: rev 312ceebde8ef8881fc43d82a096fb852f833a206)
- (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestCapacitySchedulerAutoQueueCreation.java
- (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/AutoCreatedLeafQueue.java
- (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestUtils.java
- (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestCapacitySchedulerAutoCreatedQueueBase.java
- (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestAbsoluteResourceConfiguration.java
Attached patch for review