Details
-
Sub-task
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
None
-
None
-
Reviewed
Description
This involves changes to queue mapping flow to pass along context information for auto queue creation. Auto creation of queues will be part of Capacity Scheduler flow while attempting to resolve queues during application submission. The leaf queues which do not exist are auto created under parent queues which have been explicitly enabled for auto queue creation . In order to determine which parent queue to create the leaf queues under - parent queues need to be specified in queue mapping configuration
Attachments
Attachments
- YARN-7419.1.patch
- 142 kB
- Suma Shivaprasad
- YARN-7419.2.patch
- 115 kB
- Suma Shivaprasad
- YARN-7419.3.patch
- 116 kB
- Suma Shivaprasad
- YARN-7419.4.patch
- 120 kB
- Suma Shivaprasad
- YARN-7419.5.patch
- 122 kB
- Suma Shivaprasad
- YARN-7419.6.patch
- 122 kB
- Suma Shivaprasad
- YARN-7419.7.patch
- 124 kB
- Suma Shivaprasad
- YARN-7419.8.patch
- 119 kB
- Suma Shivaprasad
- YARN-7419.9.patch
- 124 kB
- Suma Shivaprasad
- YARN-7419.patch
- 142 kB
- Suma Shivaprasad
Issue Links
- is blocked by
-
YARN-7394 Merge code paths for Reservation/Plan queues and Auto Created queues
- Resolved
Activity
-1 overall |
Vote | Subsystem | Runtime | Comment |
---|---|---|---|
0 | reexec | 0m 12s | Docker mode activated. |
Prechecks | |||
+1 | @author | 0m 0s | The patch does not contain any @author tags. |
+1 | test4tests | 0m 0s | The patch appears to include 8 new or modified test files. |
trunk Compile Tests | |||
+1 | mvninstall | 16m 19s | trunk passed |
+1 | compile | 0m 36s | trunk passed |
+1 | checkstyle | 0m 31s | trunk passed |
+1 | mvnsite | 0m 38s | trunk passed |
+1 | shadedclient | 10m 13s | branch has no errors when building and testing our client artifacts. |
-1 | findbugs | 1m 5s | hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager in trunk has 2 extant Findbugs warnings. |
+1 | javadoc | 0m 21s | trunk passed |
Patch Compile Tests | |||
+1 | mvninstall | 0m 38s | the patch passed |
+1 | compile | 0m 34s | the patch passed |
+1 | javac | 0m 34s | the patch passed |
-0 | checkstyle | 0m 31s | hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 159 new + 632 unchanged - 16 fixed = 791 total (was 648) |
+1 | mvnsite | 0m 37s | the patch passed |
+1 | whitespace | 0m 0s | The patch has no whitespace issues. |
+1 | shadedclient | 10m 13s | patch has no errors when building and testing our client artifacts. |
-1 | findbugs | 1m 14s | hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager generated 3 new + 2 unchanged - 0 fixed = 5 total (was 2) |
-1 | javadoc | 0m 20s | hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager generated 1 new + 4 unchanged - 0 fixed = 5 total (was 4) |
Other Tests | |||
-1 | unit | 54m 49s | hadoop-yarn-server-resourcemanager in the patch failed. |
+1 | asflicense | 0m 17s | The patch does not generate ASF License warnings. |
98m 48s |
Reason | Tests |
---|---|
FindBugs | module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager |
Self assignment of field ApplicationPlacementContext.parentQueue in new org.apache.hadoop.yarn.server.resourcemanager.placement.ApplicationPlacementContext(String) At ApplicationPlacementContext.java:in new org.apache.hadoop.yarn.server.resourcemanager.placement.ApplicationPlacementContext(String) At ApplicationPlacementContext.java:[line 37] | |
instanceof will always return true for all non-null values in org.apache.hadoop.yarn.server.resourcemanager.placement.UserGroupMappingPlacementRule.validateParentQueue(CSQueue, String, String), since all org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ManagedParentQueue are instances of org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ManagedParentQueue At UserGroupMappingPlacementRule.java:for all non-null values in org.apache.hadoop.yarn.server.resourcemanager.placement.UserGroupMappingPlacementRule.validateParentQueue(CSQueue, String, String), since all org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ManagedParentQueue are instances of org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ManagedParentQueue At UserGroupMappingPlacementRule.java:[line 385] | |
Unread field:field be static? At UserGroupMappingPlacementRule.java:[line 82] | |
Failed junit tests | hadoop.yarn.server.resourcemanager.reservation.TestCapacityOverTimePolicy |
This message was automatically generated.
-1 overall |
Vote | Subsystem | Runtime | Comment |
---|---|---|---|
0 | reexec | 0m 11s | Docker mode activated. |
Prechecks | |||
+1 | @author | 0m 0s | The patch does not contain any @author tags. |
+1 | test4tests | 0m 0s | The patch appears to include 8 new or modified test files. |
trunk Compile Tests | |||
+1 | mvninstall | 17m 11s | trunk passed |
+1 | compile | 0m 40s | trunk passed |
+1 | checkstyle | 0m 37s | trunk passed |
+1 | mvnsite | 0m 44s | trunk passed |
+1 | shadedclient | 11m 17s | branch has no errors when building and testing our client artifacts. |
-1 | findbugs | 1m 3s | hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager in trunk has 2 extant Findbugs warnings. |
+1 | javadoc | 0m 22s | trunk passed |
Patch Compile Tests | |||
+1 | mvninstall | 0m 38s | the patch passed |
+1 | compile | 0m 33s | the patch passed |
+1 | javac | 0m 33s | the patch passed |
-0 | checkstyle | 0m 34s | hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 157 new + 632 unchanged - 16 fixed = 789 total (was 648) |
+1 | mvnsite | 0m 36s | the patch passed |
+1 | whitespace | 0m 0s | The patch has no whitespace issues. |
+1 | shadedclient | 11m 13s | patch has no errors when building and testing our client artifacts. |
+1 | findbugs | 1m 12s | the patch passed |
-1 | javadoc | 0m 21s | hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager generated 1 new + 4 unchanged - 0 fixed = 5 total (was 4) |
Other Tests | |||
-1 | unit | 56m 12s | hadoop-yarn-server-resourcemanager in the patch failed. |
+1 | asflicense | 0m 20s | The patch does not generate ASF License warnings. |
103m 21s |
Reason | Tests |
---|---|
Failed junit tests | hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacitySchedulerNodeLabelUpdate |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacitySchedulerAutoQueueCreation |
This message was automatically generated.
Thanks suma.shivaprasad for working on this.
Some comments:
1) Why changes of getClock are needed, seems nobody is using them?
2) Following methods can reuse leaf queue's methods (just pass <parent-queue-path>.AUTO_CREATED_LEAF_QUEUE_TEMPLATE_PREFIX as queue path).
getAutoCreatedLeafQueueTemplateMaxCapacity(String) getAutoCreatedQueuesMaxChildQueuesLimit(String) setAutoCreatedLeafQueueTemplateCapacity(String, float) setAutoCreatedLeafQueueTemplateMaxApplicationsPerQueue(String, int) setAutoCreatedLeafQueueTemplateMaxCapacity(String, float)
And this config: AUTO_CREATE_QUEUE_MAX_QUEUES should be part of parent queue instead of leaf queue templete.
3) Could you create a AutoCreatedLeafQueueTemplate class which encapsulate all leaf queue's properties and replace all getter/setters of leaf queue properties.
4) sumOfChildCapacities/sumOfChildAbsCapacities can be moved to AbstractManagedParentQueue, correct?
5) ApplicationPlacementContext, why keep PlacementRule as one of the fields?
6) There're two rmApp.getApplicationQueuePlacementContext() calls inside CapacityScheduler, one of them can be removed. (Which we can avoid access RMApps map twice).
Thanks for the review wangda Attached patch with review comments addressed
Had missed removing sumOfChildCapacities from ParentQueue. Fixed in this patch
-1 overall |
Vote | Subsystem | Runtime | Comment |
---|---|---|---|
0 | reexec | 0m 9s | Docker mode activated. |
Prechecks | |||
+1 | @author | 0m 0s | The patch does not contain any @author tags. |
+1 | test4tests | 0m 0s | The patch appears to include 6 new or modified test files. |
trunk Compile Tests | |||
+1 | mvninstall | 16m 43s | trunk passed |
+1 | compile | 0m 47s | trunk passed |
+1 | checkstyle | 0m 36s | trunk passed |
+1 | mvnsite | 0m 45s | trunk passed |
+1 | shadedclient | 11m 38s | branch has no errors when building and testing our client artifacts. |
+1 | findbugs | 1m 17s | trunk passed |
+1 | javadoc | 0m 29s | trunk passed |
Patch Compile Tests | |||
+1 | mvninstall | 0m 50s | the patch passed |
+1 | compile | 0m 55s | the patch passed |
+1 | javac | 0m 55s | the patch passed |
-0 | checkstyle | 0m 37s | hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 167 new + 581 unchanged - 0 fixed = 748 total (was 581) |
+1 | mvnsite | 0m 42s | the patch passed |
+1 | whitespace | 0m 0s | The patch has no whitespace issues. |
+1 | shadedclient | 11m 35s | patch has no errors when building and testing our client artifacts. |
+1 | findbugs | 1m 23s | the patch passed |
+1 | javadoc | 0m 25s | the patch passed |
Other Tests | |||
-1 | unit | 57m 10s | hadoop-yarn-server-resourcemanager in the patch failed. |
+1 | asflicense | 0m 22s | The patch does not generate ASF License warnings. |
106m 6s |
Reason | Tests |
---|---|
Failed junit tests | hadoop.yarn.server.resourcemanager.reservation.TestCapacityOverTimePolicy |
hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer |
Subsystem | Report/Notes |
---|---|
Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
JIRA Issue | |
JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12896522/YARN-7419.3.patch |
Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle |
uname | Linux 3c7f5c99d398 3.13.0-129-generic #178-Ubuntu SMP Fri Aug 11 12:48:20 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
Build tool | maven |
Personality | /testptch/patchprocess/precommit/personality/provided.sh |
git revision | trunk / 13fa2d4 |
maven | version: Apache Maven 3.3.9 |
Default Java | 1.8.0_131 |
findbugs | v3.1.0-RC1 |
checkstyle | https://builds.apache.org/job/PreCommit-YARN-Build/18388/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt |
unit | https://builds.apache.org/job/PreCommit-YARN-Build/18388/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt |
Test Results | https://builds.apache.org/job/PreCommit-YARN-Build/18388/testReport/ |
Max. process+thread count | 872 (vs. ulimit of 5000) |
modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager |
Console output | https://builds.apache.org/job/PreCommit-YARN-Build/18388/console |
Powered by | Apache Yetus 0.7.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
Thanks suma.shivaprasad for updating the patch, more comments:
1) CapacityScheduler:
1.1 Instead of fetching ApplicationPlacementContext from RMApp (to avoid perf/locking issue and code flow becomes more clear), you can add the ApplicationPlacementContext to AppAddedSchedulerEvent. And getPlacementContext API can be removed from RMApp.
1.2 Move following if ... to addApplication:
if (placementContext != null) { // ... }
Like,
if (queue == null && placementContext != null) { //Could be a potential auto-created leaf queue }
Only enter the autoCreateLeafQueue function when necessary.
1.3 Following two catch exception can be merged by using YarnException | IOException e syntax.
catch (YarnException e) { LOG.error("Could not auto-create leaf queue due to : ", e); final String message = "Application " + applicationId + " submission by user " + user + " to queue: " + queueName + " failed : " + e.getMessage(); this.rmContext.getDispatcher().getEventHandler().handle( new RMAppEvent(applicationId, RMAppEventType.APP_REJECTED, message)); } catch (IOException e) { final String message = "Application " + applicationId + " submission by user " + user + " to queue: " + queueName + " failed : " + e.getMessage(); LOG.error("Could not auto-create leaf queue due to : ", e); this.rmContext.getDispatcher().getEventHandler().handle( new RMAppEvent(applicationId, RMAppEventType.APP_REJECTED, message)); }
1.4 Following message is not clear enough:
String message = "Application " + applicationId + " submission by user " + user + " to queue: " + queueName + " failed : " + "Queue mapping does not exist for user";
It should say directly specify a autocreated queue name is prohibited, it has to be automatically mapped, etc.
1.5 I'm not sure if this check is necessary, I think previous logics should be enough to detect this correct?
else if (!queue.getParent().getQueueName().equals( placementContext.getParentQueue())) { String message = "Auto created Leaf queue " + placementContext.getQueue() + " already exists under " + queue .getParent().getQueuePath() + ".But Queue mapping has a different parent queue " + placementContext.getParentQueue() + " for the specified user : " + user; this.rmContext.getDispatcher().getEventHandler().handle( new RMAppEvent(applicationId, RMAppEventType.APP_REJECTED, message)); return; }
1.6 clock is still here, move to a separate patch?
2) CapacitySchedulerConfiguration:
- getQueuePlacementRules is unused.
- Make sure all new added methods/fields are @Private
- FAIL_AUTO_CREATION_ON_EXCEEDING_CAPACITY is this necessary? Should we just fail leaf queue creation when it exceeds parent queue's limit?
Renames:
- AutoCreatedLeafQueueTemplate.Builder#capacity => capacities
Unecesssary changes:
- CapacitySchedulerContext
- AbstractCSQueue
Miscs:
- Did you accidentally included
YARN-6124in this patch? Could you revert that part?
Thanks Suma Shivaprasad for updating the patch, more comments:
1) CapacityScheduler:
1.1 Instead of fetching ApplicationPlacementContext from RMApp (to avoid perf/locking issue and code flow becomes more clear), you can add the ApplicationPlacementContext to AppAddedSchedulerEvent. And getPlacementContext API can be removed from RMApp.
Fixed for both new application submissions and recovery
1.2 Move following if ... to addApplication:
if (placementContext != null)Unknown macro: { // ... }Like,
if (queue == null && placementContext != null)Unknown macro: { //Could be a potential auto-created leaf queue }Only enter the autoCreateLeafQueue function when necessary.
Fixed
1.3 Following two catch exception can be merged by using YarnException | IOException e syntax.
catch (YarnException e)Unknown macro: { LOG.error("Could not auto-create leaf queue due to }catch (IOException e)
Unknown macro: { final String message = "Application " + applicationId + " submission by user " + user + " to queue}
Fixed
1.4 Following message is not clear enough:
String message =
"Application " + applicationId + " submission by user " + user
+ " to queue: " + queueName + " failed : "
+ "Queue mapping does not exist for user";
It should say directly specify a autocreated queue name is prohibited, it has to be automatically mapped, etc.
Fixed
1.5 I'm not sure if this check is necessary, I think previous logics should be enough to detect this correct?
else if (!queue.getParent().getQueueName().equals(
placementContext.getParentQueue()))Unknown macro: { String message = "Auto created Leaf queue " + placementContext.getQueue() + " already exists under " + queue .getParent().getQueuePath() + ".But Queue mapping has a different parent queue " + placementContext.getParentQueue() + " for the specified user }
This is needed in the case where auto created leaf queue already exists but its parent queue has been updated in the queue mapping to a different queue which could lead to inconsistencies
since we do not allow same leaf queue name under different hierarchies
1.6 clock is still here, move to a separate patch?
Fixed
2) CapacitySchedulerConfiguration:
getQueuePlacementRules is unused.
Make sure all new added methods/fields are @Private
Fixed
FAIL_AUTO_CREATION_ON_EXCEEDING_CAPACITY is this necessary? Should we just fail leaf queue creation when it exceeds parent queue's limit?
FixedRenames:
AutoCreatedLeafQueueTemplate.Builder#capacity => capacities
Unecesssary changes:
CapacitySchedulerContext
AbstractCSQueue
Miscs:
Did you accidentally included
-1 overall |
Vote | Subsystem | Runtime | Comment |
---|---|---|---|
0 | reexec | 0m 25s | Docker mode activated. |
Prechecks | |||
+1 | @author | 0m 0s | The patch does not contain any @author tags. |
+1 | test4tests | 0m 0s | The patch appears to include 6 new or modified test files. |
trunk Compile Tests | |||
+1 | mvninstall | 24m 48s | trunk passed |
+1 | compile | 1m 3s | trunk passed |
+1 | checkstyle | 0m 46s | trunk passed |
+1 | mvnsite | 1m 10s | trunk passed |
+1 | shadedclient | 14m 35s | branch has no errors when building and testing our client artifacts. |
+1 | findbugs | 2m 11s | trunk passed |
+1 | javadoc | 0m 55s | trunk passed |
Patch Compile Tests | |||
+1 | mvninstall | 1m 25s | the patch passed |
+1 | compile | 1m 18s | the patch passed |
+1 | javac | 1m 18s | the patch passed |
-0 | checkstyle | 1m 4s | hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 150 new + 625 unchanged - 2 fixed = 775 total (was 627) |
+1 | mvnsite | 1m 16s | the patch passed |
-1 | whitespace | 0m 0s | The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply |
+1 | shadedclient | 15m 17s | patch has no errors when building and testing our client artifacts. |
+1 | findbugs | 2m 42s | the patch passed |
+1 | javadoc | 1m 9s | the patch passed |
Other Tests | |||
-1 | unit | 2m 1s | hadoop-yarn-server-resourcemanager in the patch failed. |
-1 | asflicense | 0m 55s | The patch generated 15 ASF License warnings. |
73m 20s |
Reason | Tests |
---|---|
Failed junit tests | hadoop.yarn.server.resourcemanager.TestRM |
Subsystem | Report/Notes |
---|---|
Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
JIRA Issue | |
JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12896970/YARN-7419.4.patch |
Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle |
uname | Linux 81d4e2a5abc1 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
Build tool | maven |
Personality | /testptch/patchprocess/precommit/personality/provided.sh |
git revision | trunk / 5eb7dbe |
maven | version: Apache Maven 3.3.9 |
Default Java | 1.8.0_131 |
findbugs | v3.1.0-RC1 |
checkstyle | https://builds.apache.org/job/PreCommit-YARN-Build/18423/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt |
whitespace | https://builds.apache.org/job/PreCommit-YARN-Build/18423/artifact/out/whitespace-eol.txt |
unit | https://builds.apache.org/job/PreCommit-YARN-Build/18423/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt |
Test Results | https://builds.apache.org/job/PreCommit-YARN-Build/18423/testReport/ |
asflicense | https://builds.apache.org/job/PreCommit-YARN-Build/18423/artifact/out/patch-asflicense-problems.txt |
Max. process+thread count | 296 (vs. ulimit of 5000) |
modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager |
Console output | https://builds.apache.org/job/PreCommit-YARN-Build/18423/console |
Powered by | Apache Yetus 0.7.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
Fixed some checkstyle issues and added missing return for CapacityScheduler when queue is null
-1 overall |
Vote | Subsystem | Runtime | Comment |
---|---|---|---|
0 | reexec | 0m 12s | Docker mode activated. |
Prechecks | |||
+1 | @author | 0m 0s | The patch does not contain any @author tags. |
+1 | test4tests | 0m 0s | The patch appears to include 6 new or modified test files. |
trunk Compile Tests | |||
+1 | mvninstall | 19m 33s | trunk passed |
+1 | compile | 0m 36s | trunk passed |
+1 | checkstyle | 0m 31s | trunk passed |
+1 | mvnsite | 0m 40s | trunk passed |
+1 | shadedclient | 10m 58s | branch has no errors when building and testing our client artifacts. |
+1 | findbugs | 1m 7s | trunk passed |
+1 | javadoc | 0m 25s | trunk passed |
Patch Compile Tests | |||
+1 | mvninstall | 0m 39s | the patch passed |
+1 | compile | 0m 36s | the patch passed |
+1 | javac | 0m 36s | the patch passed |
-0 | checkstyle | 0m 31s | hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 109 new + 625 unchanged - 3 fixed = 734 total (was 628) |
+1 | mvnsite | 0m 38s | the patch passed |
-1 | whitespace | 0m 0s | The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply |
+1 | shadedclient | 10m 44s | patch has no errors when building and testing our client artifacts. |
+1 | findbugs | 1m 9s | the patch passed |
+1 | javadoc | 0m 22s | the patch passed |
Other Tests | |||
-1 | unit | 55m 5s | hadoop-yarn-server-resourcemanager in the patch failed. |
+1 | asflicense | 0m 20s | The patch does not generate ASF License warnings. |
103m 56s |
Reason | Tests |
---|---|
Failed junit tests | hadoop.yarn.server.resourcemanager.scheduler.capacity.TestIncreaseAllocationExpirer |
Subsystem | Report/Notes |
---|---|
Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
JIRA Issue | |
JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12897001/YARN-7419.5.patch |
Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle |
uname | Linux e7df8630bbb6 3.13.0-133-generic #182-Ubuntu SMP Tue Sep 19 15:49:21 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
Build tool | maven |
Personality | /testptch/patchprocess/precommit/personality/provided.sh |
git revision | trunk / 10a1f55 |
maven | version: Apache Maven 3.3.9 |
Default Java | 1.8.0_151 |
findbugs | v3.1.0-RC1 |
checkstyle | https://builds.apache.org/job/PreCommit-YARN-Build/18426/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt |
whitespace | https://builds.apache.org/job/PreCommit-YARN-Build/18426/artifact/out/whitespace-eol.txt |
unit | https://builds.apache.org/job/PreCommit-YARN-Build/18426/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt |
Test Results | https://builds.apache.org/job/PreCommit-YARN-Build/18426/testReport/ |
Max. process+thread count | 849 (vs. ulimit of 5000) |
modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager |
Console output | https://builds.apache.org/job/PreCommit-YARN-Build/18426/console |
Powered by | Apache Yetus 0.7.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
Thanks suma.shivaprasad for updating the patch, in general patch looks good. Could you help to fix following checkstyle issues if possible?
HiddenField
LineLength
UnusedImports
-1 overall |
Vote | Subsystem | Runtime | Comment |
---|---|---|---|
0 | reexec | 12m 44s | Docker mode activated. |
Prechecks | |||
+1 | @author | 0m 0s | The patch does not contain any @author tags. |
+1 | test4tests | 0m 0s | The patch appears to include 5 new or modified test files. |
trunk Compile Tests | |||
+1 | mvninstall | 33m 18s | trunk passed |
+1 | compile | 1m 29s | trunk passed |
+1 | checkstyle | 1m 5s | trunk passed |
+1 | mvnsite | 1m 33s | trunk passed |
+1 | shadedclient | 18m 6s | branch has no errors when building and testing our client artifacts. |
+1 | findbugs | 2m 23s | trunk passed |
+1 | javadoc | 0m 52s | trunk passed |
Patch Compile Tests | |||
+1 | mvninstall | 1m 35s | the patch passed |
+1 | compile | 1m 29s | the patch passed |
+1 | javac | 1m 29s | the patch passed |
-0 | checkstyle | 1m 3s | hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 53 new + 611 unchanged - 4 fixed = 664 total (was 615) |
+1 | mvnsite | 1m 32s | the patch passed |
-1 | whitespace | 0m 0s | The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply |
+1 | shadedclient | 17m 32s | patch has no errors when building and testing our client artifacts. |
+1 | findbugs | 1m 14s | the patch passed |
+1 | javadoc | 0m 23s | the patch passed |
Other Tests | |||
+1 | unit | 55m 55s | hadoop-yarn-server-resourcemanager in the patch passed. |
+1 | asflicense | 0m 28s | The patch does not generate ASF License warnings. |
151m 27s |
Subsystem | Report/Notes |
---|---|
Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
JIRA Issue | |
JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12897132/YARN-7419.6.patch |
Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle |
uname | Linux 7d1a48856853 3.13.0-133-generic #182-Ubuntu SMP Tue Sep 19 15:49:21 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
Build tool | maven |
Personality | /testptch/patchprocess/precommit/personality/provided.sh |
git revision | trunk / 8a1bd9a |
maven | version: Apache Maven 3.3.9 |
Default Java | 1.8.0_151 |
findbugs | v3.1.0-RC1 |
checkstyle | https://builds.apache.org/job/PreCommit-YARN-Build/18433/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt |
whitespace | https://builds.apache.org/job/PreCommit-YARN-Build/18433/artifact/out/whitespace-eol.txt |
Test Results | https://builds.apache.org/job/PreCommit-YARN-Build/18433/testReport/ |
Max. process+thread count | 812 (vs. ulimit of 5000) |
modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager |
Console output | https://builds.apache.org/job/PreCommit-YARN-Build/18433/console |
Powered by | Apache Yetus 0.7.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
-1 overall |
Vote | Subsystem | Runtime | Comment |
---|---|---|---|
0 | reexec | 0m 18s | Docker mode activated. |
Prechecks | |||
+1 | @author | 0m 0s | The patch does not contain any @author tags. |
+1 | test4tests | 0m 0s | The patch appears to include 5 new or modified test files. |
trunk Compile Tests | |||
+1 | mvninstall | 14m 50s | trunk passed |
+1 | compile | 0m 37s | trunk passed |
+1 | checkstyle | 0m 34s | trunk passed |
+1 | mvnsite | 0m 39s | trunk passed |
+1 | shadedclient | 10m 26s | branch has no errors when building and testing our client artifacts. |
+1 | findbugs | 0m 59s | trunk passed |
+1 | javadoc | 0m 24s | trunk passed |
Patch Compile Tests | |||
+1 | mvninstall | 0m 36s | the patch passed |
+1 | compile | 0m 33s | the patch passed |
+1 | javac | 0m 33s | the patch passed |
-0 | checkstyle | 0m 31s | hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 44 new + 609 unchanged - 6 fixed = 653 total (was 615) |
+1 | mvnsite | 0m 36s | the patch passed |
-1 | whitespace | 0m 0s | The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply |
+1 | shadedclient | 10m 16s | patch has no errors when building and testing our client artifacts. |
+1 | findbugs | 1m 3s | the patch passed |
+1 | javadoc | 0m 23s | the patch passed |
Other Tests | |||
-1 | unit | 60m 42s | hadoop-yarn-server-resourcemanager in the patch failed. |
+1 | asflicense | 0m 20s | The patch does not generate ASF License warnings. |
103m 39s |
Reason | Tests |
---|---|
Timed out junit tests | org.apache.hadoop.yarn.server.resourcemanager.TestRMStoreCommands |
org.apache.hadoop.yarn.server.resourcemanager.TestSubmitApplicationWithRMHA | |
org.apache.hadoop.yarn.server.resourcemanager.TestKillApplicationWithRMHA |
Subsystem | Report/Notes |
---|---|
Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
JIRA Issue | |
JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12897222/YARN-7419.7.patch |
Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle |
uname | Linux e1b98de310a9 4.4.0-64-generic #85-Ubuntu SMP Mon Feb 20 11:50:30 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
Build tool | maven |
Personality | /testptch/patchprocess/precommit/personality/provided.sh |
git revision | trunk / ff9f7fc |
maven | version: Apache Maven 3.3.9 |
Default Java | 1.8.0_151 |
findbugs | v3.1.0-RC1 |
checkstyle | https://builds.apache.org/job/PreCommit-YARN-Build/18448/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt |
whitespace | https://builds.apache.org/job/PreCommit-YARN-Build/18448/artifact/out/whitespace-eol.txt |
unit | https://builds.apache.org/job/PreCommit-YARN-Build/18448/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt |
Test Results | https://builds.apache.org/job/PreCommit-YARN-Build/18448/testReport/ |
Max. process+thread count | 842 (vs. ulimit of 5000) |
modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager |
Console output | https://builds.apache.org/job/PreCommit-YARN-Build/18448/console |
Powered by | Apache Yetus 0.7.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
-1 overall |
Vote | Subsystem | Runtime | Comment |
---|---|---|---|
0 | reexec | 0m 0s | Docker mode activated. |
-1 | patch | 0m 5s | |
Subsystem | Report/Notes |
---|---|
JIRA Issue | |
JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12897260/YARN-7419.8.patch |
Console output | https://builds.apache.org/job/PreCommit-YARN-Build/18457/console |
Powered by | Apache Yetus 0.7.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
-1 overall |
Vote | Subsystem | Runtime | Comment |
---|---|---|---|
0 | reexec | 0m 15s | Docker mode activated. |
Prechecks | |||
+1 | @author | 0m 0s | The patch does not contain any @author tags. |
+1 | test4tests | 0m 0s | The patch appears to include 5 new or modified test files. |
trunk Compile Tests | |||
+1 | mvninstall | 16m 8s | trunk passed |
+1 | compile | 0m 37s | trunk passed |
+1 | checkstyle | 0m 33s | trunk passed |
+1 | mvnsite | 0m 39s | trunk passed |
+1 | shadedclient | 10m 57s | branch has no errors when building and testing our client artifacts. |
+1 | findbugs | 1m 2s | trunk passed |
+1 | javadoc | 0m 24s | trunk passed |
Patch Compile Tests | |||
+1 | mvninstall | 0m 38s | the patch passed |
+1 | compile | 0m 33s | the patch passed |
+1 | javac | 0m 33s | the patch passed |
-0 | checkstyle | 0m 31s | hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 41 new + 608 unchanged - 6 fixed = 649 total (was 614) |
+1 | mvnsite | 0m 36s | the patch passed |
+1 | whitespace | 0m 0s | The patch has no whitespace issues. |
+1 | shadedclient | 11m 4s | patch has no errors when building and testing our client artifacts. |
+1 | findbugs | 1m 8s | the patch passed |
+1 | javadoc | 0m 22s | the patch passed |
Other Tests | |||
-1 | unit | 58m 31s | hadoop-yarn-server-resourcemanager in the patch failed. |
+1 | asflicense | 0m 19s | The patch does not generate ASF License warnings. |
104m 1s |
Reason | Tests |
---|---|
Failed junit tests | hadoop.yarn.server.resourcemanager.scheduler.capacity.TestIncreaseAllocationExpirer |
Timed out junit tests | org.apache.hadoop.yarn.server.resourcemanager.TestRMStoreCommands |
org.apache.hadoop.yarn.server.resourcemanager.TestSubmitApplicationWithRMHA | |
org.apache.hadoop.yarn.server.resourcemanager.TestKillApplicationWithRMHA |
Subsystem | Report/Notes |
---|---|
Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
JIRA Issue | |
JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12897264/YARN-7419.9.patch |
Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle |
uname | Linux 0a985acca980 3.13.0-117-generic #164-Ubuntu SMP Fri Apr 7 11:05:26 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
Build tool | maven |
Personality | /testptch/patchprocess/precommit/personality/provided.sh |
git revision | trunk / 3e26077 |
maven | version: Apache Maven 3.3.9 |
Default Java | 1.8.0_151 |
findbugs | v3.1.0-RC1 |
checkstyle | https://builds.apache.org/job/PreCommit-YARN-Build/18458/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt |
unit | https://builds.apache.org/job/PreCommit-YARN-Build/18458/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt |
Test Results | https://builds.apache.org/job/PreCommit-YARN-Build/18458/testReport/ |
Max. process+thread count | 794 (vs. ulimit of 5000) |
modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager |
Console output | https://builds.apache.org/job/PreCommit-YARN-Build/18458/console |
Powered by | Apache Yetus 0.7.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
Latest patch looks good, +1. Will commit tomorrow if no objections. Thanks suma.shivaprasad!
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13249 (See https://builds.apache.org/job/Hadoop-trunk-Commit/13249/)
YARN-7419. CapacityScheduler: Allow auto leaf queue creation after queue (wangda: rev 0987a7b8cbbbb2c1e4c2095821d98a7db19644df)
- (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestAppManager.java
- (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java
- (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/placement/TestUserGroupMappingPlacementRule.java
- (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestCapacitySchedulerAutoQueueCreation.java
- (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/placement/PlacementRule.java
- (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/RMAppImpl.java
- (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.java
- (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAppManager.java
- (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/ParentQueue.java
- (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/AutoCreatedLeafQueue.java
- (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/PlanQueue.java
- (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/TestSchedulerUtils.java
- (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/ManagedParentQueue.java
- (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/placement/PlacementManager.java
- (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/placement/ApplicationPlacementContext.java
- (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/AbstractCSQueue.java
- (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/placement/UserGroupMappingPlacementRule.java
- (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/event/AppAddedSchedulerEvent.java
- (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/AbstractManagedParentQueue.java
- (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestCapacityScheduler.java
- (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerQueueManager.java
Attaching patch on top of code changes in
YARN-7394