Details
-
Bug
-
Status: Resolved
-
Blocker
-
Resolution: Fixed
-
None
-
None
-
None
-
Reviewed
Description
Currently CapacityScheduler fails to refresh maximum allocation for multiple resource types.
Attachments
Attachments
- YARN-7738.001.patch
- 13 kB
- Wangda Tan
- YARN-7738.002.patch
- 23 kB
- Wangda Tan
- YARN-7738.003.patch
- 24 kB
- Sunil G
- YARN-7738.004.patch
- 24 kB
- Sunil G
Issue Links
Activity
cc: daniel@cloudera.com, I'm not sure if FS supports refresh maximum allocation while RM is running, if yes, FS needs this fix as well.
Attached ver.1 patch and verified it works on a cluster with GPU enabled.
Will add test cases in the next update.
-1 overall |
Vote | Subsystem | Runtime | Comment |
---|---|---|---|
0 | reexec | 9m 41s | Docker mode activated. |
Prechecks | |||
+1 | @author | 0m 0s | The patch does not contain any @author tags. |
+1 | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. |
trunk Compile Tests | |||
0 | mvndep | 0m 59s | Maven dependency ordering for branch |
+1 | mvninstall | 18m 1s | trunk passed |
+1 | compile | 8m 7s | trunk passed |
+1 | checkstyle | 1m 6s | trunk passed |
+1 | mvnsite | 1m 26s | trunk passed |
+1 | shadedclient | 12m 42s | branch has no errors when building and testing our client artifacts. |
-1 | findbugs | 1m 17s | hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api in trunk has 1 extant Findbugs warnings. |
+1 | javadoc | 0m 58s | trunk passed |
Patch Compile Tests | |||
0 | mvndep | 0m 10s | Maven dependency ordering for patch |
+1 | mvninstall | 1m 9s | the patch passed |
+1 | compile | 6m 31s | the patch passed |
+1 | javac | 6m 31s | the patch passed |
-0 | checkstyle | 1m 16s | hadoop-yarn-project/hadoop-yarn: The patch generated 2 new + 367 unchanged - 0 fixed = 369 total (was 367) |
+1 | mvnsite | 1m 25s | the patch passed |
+1 | whitespace | 0m 0s | The patch has no whitespace issues. |
+1 | shadedclient | 10m 56s | patch has no errors when building and testing our client artifacts. |
+1 | findbugs | 2m 37s | the patch passed |
+1 | javadoc | 1m 0s | the patch passed |
Other Tests | |||
+1 | unit | 0m 39s | hadoop-yarn-api in the patch passed. |
-1 | unit | 59m 50s | hadoop-yarn-server-resourcemanager in the patch failed. |
+1 | asflicense | 0m 32s | The patch does not generate ASF License warnings. |
140m 50s |
Reason | Tests |
---|---|
Failed junit tests | hadoop.yarn.server.resourcemanager.scheduler.capacity.TestIncreaseAllocationExpirer |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacityScheduler |
Subsystem | Report/Notes |
---|---|
Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
JIRA Issue | |
JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12905765/YARN-7738.001.patch |
Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle |
uname | Linux 9106d45b396a 3.13.0-129-generic #178-Ubuntu SMP Fri Aug 11 12:48:20 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
Build tool | maven |
Personality | /testptch/patchprocess/precommit/personality/provided.sh |
git revision | trunk / bc285da |
maven | version: Apache Maven 3.3.9 |
Default Java | 1.8.0_151 |
findbugs | v3.1.0-RC1 |
findbugs | https://builds.apache.org/job/PreCommit-YARN-Build/19208/artifact/out/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-api-warnings.html |
checkstyle | https://builds.apache.org/job/PreCommit-YARN-Build/19208/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt |
unit | https://builds.apache.org/job/PreCommit-YARN-Build/19208/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt |
Test Results | https://builds.apache.org/job/PreCommit-YARN-Build/19208/testReport/ |
Max. process+thread count | 799 (vs. ulimit of 5000) |
modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn |
Console output | https://builds.apache.org/job/PreCommit-YARN-Build/19208/console |
Powered by | Apache Yetus 0.7.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
-1 overall |
Vote | Subsystem | Runtime | Comment |
---|---|---|---|
0 | reexec | 0m 15s | Docker mode activated. |
Prechecks | |||
+1 | @author | 0m 0s | The patch does not contain any @author tags. |
+1 | test4tests | 0m 0s | The patch appears to include 2 new or modified test files. |
trunk Compile Tests | |||
0 | mvndep | 0m 10s | Maven dependency ordering for branch |
+1 | mvninstall | 16m 45s | trunk passed |
+1 | compile | 8m 7s | trunk passed |
+1 | checkstyle | 1m 5s | trunk passed |
+1 | mvnsite | 1m 26s | trunk passed |
+1 | shadedclient | 12m 51s | branch has no errors when building and testing our client artifacts. |
-1 | findbugs | 1m 20s | hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api in trunk has 1 extant Findbugs warnings. |
+1 | javadoc | 1m 0s | trunk passed |
Patch Compile Tests | |||
0 | mvndep | 0m 11s | Maven dependency ordering for patch |
+1 | mvninstall | 1m 18s | the patch passed |
+1 | compile | 7m 28s | the patch passed |
+1 | javac | 7m 28s | the patch passed |
-0 | checkstyle | 1m 4s | hadoop-yarn-project/hadoop-yarn: The patch generated 3 new + 413 unchanged - 0 fixed = 416 total (was 413) |
+1 | mvnsite | 1m 22s | the patch passed |
+1 | whitespace | 0m 0s | The patch has no whitespace issues. |
+1 | shadedclient | 10m 50s | patch has no errors when building and testing our client artifacts. |
+1 | findbugs | 2m 44s | the patch passed |
+1 | javadoc | 1m 0s | the patch passed |
Other Tests | |||
+1 | unit | 0m 42s | hadoop-yarn-api in the patch passed. |
+1 | unit | 66m 50s | hadoop-yarn-server-resourcemanager in the patch passed. |
+1 | asflicense | 0m 37s | The patch does not generate ASF License warnings. |
137m 37s |
Subsystem | Report/Notes |
---|---|
Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
JIRA Issue | |
JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12905812/YARN-7738.002.patch |
Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle |
uname | Linux dfe31b26722f 3.13.0-129-generic #178-Ubuntu SMP Fri Aug 11 12:48:20 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
Build tool | maven |
Personality | /testptch/patchprocess/precommit/personality/provided.sh |
git revision | trunk / addbcd8 |
maven | version: Apache Maven 3.3.9 |
Default Java | 1.8.0_151 |
findbugs | v3.1.0-RC1 |
findbugs | https://builds.apache.org/job/PreCommit-YARN-Build/19215/artifact/out/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-api-warnings.html |
checkstyle | https://builds.apache.org/job/PreCommit-YARN-Build/19215/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt |
Test Results | https://builds.apache.org/job/PreCommit-YARN-Build/19215/testReport/ |
Max. process+thread count | 843 (vs. ulimit of 5000) |
modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn |
Console output | https://builds.apache.org/job/PreCommit-YARN-Build/19215/console |
Powered by | Apache Yetus 0.7.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
Thanks leftnoteasy. Fix seems fine to me
Few comments:
- Resource#setResourceValue will internally take care of checking whether resource name is memory or vcores and accordingly call setMemorySize or setVirtualCores. Hence we could avoid setMaximumAllocationByConfiguredResourceInformation in ResourceUtils.
- if (!Resources.fitsIn(oldMax, newMax)) could we make its like if (Resources.fitsIn(newMax, oldMax)). fitsIn checks smaller as first arg.
TO check test cases, i ll try attach a patch with this changes.
-1 overall |
Vote | Subsystem | Runtime | Comment |
---|---|---|---|
0 | reexec | 0m 30s | Docker mode activated. |
Prechecks | |||
+1 | @author | 0m 0s | The patch does not contain any @author tags. |
+1 | test4tests | 0m 0s | The patch appears to include 2 new or modified test files. |
trunk Compile Tests | |||
0 | mvndep | 0m 48s | Maven dependency ordering for branch |
+1 | mvninstall | 15m 1s | trunk passed |
+1 | compile | 9m 22s | trunk passed |
+1 | checkstyle | 0m 57s | trunk passed |
+1 | mvnsite | 1m 16s | trunk passed |
+1 | shadedclient | 11m 1s | branch has no errors when building and testing our client artifacts. |
-1 | findbugs | 1m 8s | hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api in trunk has 1 extant Findbugs warnings. |
+1 | javadoc | 1m 0s | trunk passed |
Patch Compile Tests | |||
0 | mvndep | 0m 11s | Maven dependency ordering for patch |
+1 | mvninstall | 1m 5s | the patch passed |
+1 | compile | 8m 26s | the patch passed |
+1 | javac | 8m 26s | the patch passed |
-0 | checkstyle | 1m 0s | hadoop-yarn-project/hadoop-yarn: The patch generated 3 new + 415 unchanged - 0 fixed = 418 total (was 415) |
+1 | mvnsite | 1m 13s | the patch passed |
+1 | whitespace | 0m 0s | The patch has no whitespace issues. |
+1 | shadedclient | 9m 33s | patch has no errors when building and testing our client artifacts. |
+1 | findbugs | 2m 26s | the patch passed |
+1 | javadoc | 0m 48s | the patch passed |
Other Tests | |||
+1 | unit | 0m 34s | hadoop-yarn-api in the patch passed. |
-1 | unit | 74m 32s | hadoop-yarn-server-resourcemanager in the patch failed. |
+1 | asflicense | 0m 33s | The patch does not generate ASF License warnings. |
141m 43s |
Reason | Tests |
---|---|
Failed junit tests | hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacityScheduler |
hadoop.yarn.server.resourcemanager.TestKillApplicationWithRMHA | |
hadoop.yarn.server.resourcemanager.monitor.TestSchedulingMonitor | |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestQueueParsing | |
hadoop.yarn.server.resourcemanager.scheduler.capacity.conf.TestZKConfigurationStore | |
hadoop.yarn.server.resourcemanager.TestRMHATimelineCollectors | |
hadoop.yarn.server.resourcemanager.TestRMHAForAsyncScheduler | |
hadoop.yarn.server.resourcemanager.TestLeaderElectorService | |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacitySchedulerAutoQueueCreation | |
hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesConfigurationMutation | |
hadoop.yarn.server.resourcemanager.TestReservationSystemWithRMHA | |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestQueueState | |
hadoop.yarn.server.resourcemanager.TestRMHA | |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerAllocation | |
hadoop.yarn.server.resourcemanager.federation.TestFederationRMStateStoreService | |
hadoop.yarn.server.resourcemanager.TestRMHAForNodeLabels | |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestLeafQueue | |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestReservations | |
hadoop.yarn.server.resourcemanager.TestRMAdminService | |
hadoop.yarn.server.resourcemanager.TestRMStoreCommands | |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestAbsoluteResourceConfiguration | |
hadoop.yarn.server.resourcemanager.TestSubmitApplicationWithRMHA | |
hadoop.yarn.server.resourcemanager.scheduler.capacity.conf.TestLeveldbConfigurationStore | |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestNodeLabelContainerAllocation | |
hadoop.yarn.server.resourcemanager.reservation.TestReservationSystem | |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestQueueMappings | |
hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesAppsModification | |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacitySchedulerDynamicBehavior | |
hadoop.yarn.server.resourcemanager.recovery.TestZKRMStateStore |
Subsystem | Report/Notes |
---|---|
Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
JIRA Issue | |
JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12905840/YARN-7738.003.patch |
Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle |
uname | Linux 1a13ac9a4554 4.4.0-64-generic #85-Ubuntu SMP Mon Feb 20 11:50:30 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
Build tool | maven |
Personality | /testptch/patchprocess/precommit/personality/provided.sh |
git revision | trunk / addbcd8 |
maven | version: Apache Maven 3.3.9 |
Default Java | 1.8.0_151 |
findbugs | v3.1.0-RC1 |
findbugs | https://builds.apache.org/job/PreCommit-YARN-Build/19217/artifact/out/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-api-warnings.html |
checkstyle | https://builds.apache.org/job/PreCommit-YARN-Build/19217/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt |
unit | https://builds.apache.org/job/PreCommit-YARN-Build/19217/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt |
Test Results | https://builds.apache.org/job/PreCommit-YARN-Build/19217/testReport/ |
Max. process+thread count | 869 (vs. ulimit of 5000) |
modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn |
Console output | https://builds.apache.org/job/PreCommit-YARN-Build/19217/console |
Powered by | Apache Yetus 0.7.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
-1 overall |
Vote | Subsystem | Runtime | Comment |
---|---|---|---|
0 | reexec | 0m 0s | Docker mode activated. |
-1 | docker | 6m 59s | Docker failed to build yetus/hadoop:5b98639. |
Subsystem | Report/Notes |
---|---|
JIRA Issue | |
JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12905866/YARN-7738.004.patch |
Console output | https://builds.apache.org/job/PreCommit-YARN-Build/19221/console |
Powered by | Apache Yetus 0.7.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
-1 overall |
Vote | Subsystem | Runtime | Comment |
---|---|---|---|
0 | reexec | 0m 0s | Docker mode activated. |
-1 | docker | 5m 38s | Docker failed to build yetus/hadoop:5b98639. |
Subsystem | Report/Notes |
---|---|
JIRA Issue | |
JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12905866/YARN-7738.004.patch |
Console output | https://builds.apache.org/job/PreCommit-YARN-Build/19222/console |
Powered by | Apache Yetus 0.7.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
-1 overall |
Vote | Subsystem | Runtime | Comment |
---|---|---|---|
0 | reexec | 0m 18s | Docker mode activated. |
Prechecks | |||
+1 | @author | 0m 0s | The patch does not contain any @author tags. |
+1 | test4tests | 0m 0s | The patch appears to include 2 new or modified test files. |
trunk Compile Tests | |||
0 | mvndep | 0m 10s | Maven dependency ordering for branch |
+1 | mvninstall | 14m 57s | trunk passed |
+1 | compile | 7m 21s | trunk passed |
+1 | checkstyle | 0m 54s | trunk passed |
+1 | mvnsite | 1m 13s | trunk passed |
+1 | shadedclient | 10m 57s | branch has no errors when building and testing our client artifacts. |
-1 | findbugs | 1m 8s | hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api in trunk has 1 extant Findbugs warnings. |
+1 | javadoc | 0m 53s | trunk passed |
Patch Compile Tests | |||
0 | mvndep | 0m 10s | Maven dependency ordering for patch |
+1 | mvninstall | 1m 5s | the patch passed |
+1 | compile | 8m 12s | the patch passed |
+1 | javac | 8m 12s | the patch passed |
-0 | checkstyle | 0m 58s | hadoop-yarn-project/hadoop-yarn: The patch generated 3 new + 415 unchanged - 0 fixed = 418 total (was 415) |
+1 | mvnsite | 1m 12s | the patch passed |
+1 | whitespace | 0m 0s | The patch has no whitespace issues. |
+1 | shadedclient | 9m 29s | patch has no errors when building and testing our client artifacts. |
+1 | findbugs | 2m 25s | the patch passed |
+1 | javadoc | 0m 51s | the patch passed |
Other Tests | |||
+1 | unit | 0m 37s | hadoop-yarn-api in the patch passed. |
-1 | unit | 76m 35s | hadoop-yarn-server-resourcemanager in the patch failed. |
+1 | asflicense | 0m 28s | The patch does not generate ASF License warnings. |
140m 29s |
Subsystem | Report/Notes |
---|---|
Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
JIRA Issue | |
JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12905866/YARN-7738.004.patch |
Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle |
uname | Linux 7dd03ab48ebf 4.4.0-64-generic #85-Ubuntu SMP Mon Feb 20 11:50:30 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
Build tool | maven |
Personality | /testptch/patchprocess/precommit/personality/provided.sh |
git revision | trunk / b278f7b |
maven | version: Apache Maven 3.3.9 |
Default Java | 1.8.0_151 |
findbugs | v3.1.0-RC1 |
findbugs | https://builds.apache.org/job/PreCommit-YARN-Build/19224/artifact/out/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-api-warnings.html |
checkstyle | https://builds.apache.org/job/PreCommit-YARN-Build/19224/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt |
unit | https://builds.apache.org/job/PreCommit-YARN-Build/19224/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt |
Test Results | https://builds.apache.org/job/PreCommit-YARN-Build/19224/testReport/ |
Max. process+thread count | 896 (vs. ulimit of 5000) |
modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn |
Console output | https://builds.apache.org/job/PreCommit-YARN-Build/19224/console |
Powered by | Apache Yetus 0.7.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
-1 overall |
Vote | Subsystem | Runtime | Comment |
---|---|---|---|
0 | reexec | 9m 44s | Docker mode activated. |
Prechecks | |||
+1 | @author | 0m 0s | The patch does not contain any @author tags. |
+1 | test4tests | 0m 0s | The patch appears to include 2 new or modified test files. |
trunk Compile Tests | |||
0 | mvndep | 0m 9s | Maven dependency ordering for branch |
+1 | mvninstall | 16m 51s | trunk passed |
+1 | compile | 7m 56s | trunk passed |
+1 | checkstyle | 1m 6s | trunk passed |
+1 | mvnsite | 1m 31s | trunk passed |
+1 | shadedclient | 12m 39s | branch has no errors when building and testing our client artifacts. |
-1 | findbugs | 1m 19s | hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api in trunk has 1 extant Findbugs warnings. |
+1 | javadoc | 0m 58s | trunk passed |
Patch Compile Tests | |||
0 | mvndep | 0m 12s | Maven dependency ordering for patch |
+1 | mvninstall | 1m 19s | the patch passed |
+1 | compile | 7m 0s | the patch passed |
+1 | javac | 7m 0s | the patch passed |
-0 | checkstyle | 1m 2s | hadoop-yarn-project/hadoop-yarn: The patch generated 3 new + 415 unchanged - 0 fixed = 418 total (was 415) |
+1 | mvnsite | 1m 21s | the patch passed |
+1 | whitespace | 0m 0s | The patch has no whitespace issues. |
+1 | shadedclient | 11m 1s | patch has no errors when building and testing our client artifacts. |
+1 | findbugs | 2m 47s | the patch passed |
+1 | javadoc | 0m 56s | the patch passed |
Other Tests | |||
+1 | unit | 0m 40s | hadoop-yarn-api in the patch passed. |
-1 | unit | 62m 30s | hadoop-yarn-server-resourcemanager in the patch failed. |
+1 | asflicense | 0m 34s | The patch does not generate ASF License warnings. |
142m 0s |
Reason | Tests |
---|---|
Failed junit tests | hadoop.yarn.server.resourcemanager.TestLeaderElectorService |
Subsystem | Report/Notes |
---|---|
Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
JIRA Issue | |
JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12905866/YARN-7738.004.patch |
Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle |
uname | Linux e380e3d87536 3.13.0-129-generic #178-Ubuntu SMP Fri Aug 11 12:48:20 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
Build tool | maven |
Personality | /testptch/patchprocess/precommit/personality/provided.sh |
git revision | trunk / 3d65dbe |
maven | version: Apache Maven 3.3.9 |
Default Java | 1.8.0_151 |
findbugs | v3.1.0-RC1 |
findbugs | https://builds.apache.org/job/PreCommit-YARN-Build/19227/artifact/out/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-api-warnings.html |
checkstyle | https://builds.apache.org/job/PreCommit-YARN-Build/19227/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt |
unit | https://builds.apache.org/job/PreCommit-YARN-Build/19227/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt |
Test Results | https://builds.apache.org/job/PreCommit-YARN-Build/19227/testReport/ |
Max. process+thread count | 866 (vs. ulimit of 5000) |
modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn |
Console output | https://builds.apache.org/job/PreCommit-YARN-Build/19227/console |
Powered by | Apache Yetus 0.7.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
sunilg could u help check the latest patch? Test failure should not related.
Committed to trunk. wangda, is it needed to commit to branch-3.0. I find patch is not applying. Could you please help to check this, i found TestCapacitySchedulerWithMultiResourceTypes is not preset in branch-3.0
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13517 (See https://builds.apache.org/job/Hadoop-trunk-Commit/13517/)
YARN-7738. CapacityScheduler: Support refresh maximum allocation for (sunilg: rev 06cceba1cb07340c412c4467439c16ea6812e685)
- (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java
- (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/util/resource/ResourceUtils.java
- (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.java
- (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestCapacityScheduler.java
- (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/Resource.java
- (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestCapacitySchedulerWithMultiResourceTypes.java
- (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/AdminService.java
- (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java
Hi leftnoteasy sunilg, it seems we hits a similar issue on our production system (Hadoop 2.7.3). Could we have a patch for branch-2 (or branch-2.7.3) ?
I am trying to back-port the patch to branch-2.7.3, but the patch is based on resource-types and ResourceUtils, which are not available in branch-2.
leftnoteasy thanks for taking care of this! I see. I will study it and get back here if I could have more.
leftnoteasy, could you tell me a little more about why CS supports refresh maximum allocation while RM is running? It looks kind of weird to me that reload the whole setting in resource-type.xml isn't supported, but only one of these setting is allowed to reload dynamically. Thanks.
yufeigu, CS support refresh maximum allocation while RM is running, and the new_max_alloc must >= old_max_alloc.
The patch fixed an issue that resource-types.xml isn't loaded during the refresh. This causes after refresh, all maximum-allocation for non-memory/vcore resource becomes 0
Thanks leftnoteasy. If that is so, is it reasonable to make sure resource-types.xml to be loaded before any scheduler refresh? So that we don't need to reload maximum allocation any more.
Thanks ssathish@hortonworks.com for reporting this issue, updated title/desc and will put a patch soon.