Details
-
Sub-task
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
None
-
None
-
Reviewed
Description
Placement policy exists in the API but is not implemented yet.
I have filed YARN-8074 to move the composite constraints implementation out of this phase-1 implementation of placement policy.
Attachments
Attachments
- YARN-7142.001.patch
- 112 kB
- Gour Saha
- YARN-7142.002.patch
- 94 kB
- Gour Saha
- YARN-7142.003.patch
- 94 kB
- Gour Saha
- YARN-7142.004.patch
- 94 kB
- Gour Saha
- YARN-7142-branch-3.1.004.patch
- 93 kB
- Gour Saha
Issue Links
- depends upon
-
YARN-6592 [Umbrella] Rich placement constraints in YARN
- Resolved
Activity
leftnoteasy, based on our offline discussions and on the documentation provided in YARN-6592 I am uploading a patch for incorporating rich placement policy in YARN Service. Please review when you get a chance.
-1 overall |
Vote | Subsystem | Runtime | Comment |
---|---|---|---|
0 | reexec | 0m 16s | Docker mode activated. |
Prechecks | |||
+1 | @author | 0m 0s | The patch does not contain any @author tags. |
+1 | test4tests | 0m 0s | The patch appears to include 3 new or modified test files. |
trunk Compile Tests | |||
0 | mvndep | 0m 11s | Maven dependency ordering for branch |
+1 | mvninstall | 15m 16s | trunk passed |
+1 | compile | 7m 33s | trunk passed |
+1 | checkstyle | 1m 4s | trunk passed |
+1 | mvnsite | 2m 16s | trunk passed |
+1 | shadedclient | 11m 42s | branch has no errors when building and testing our client artifacts. |
0 | findbugs | 0m 0s | Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site |
-1 | findbugs | 1m 6s | hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api in trunk has 1 extant Findbugs warnings. |
+1 | javadoc | 1m 54s | trunk passed |
Patch Compile Tests | |||
0 | mvndep | 0m 9s | Maven dependency ordering for patch |
+1 | mvninstall | 1m 50s | the patch passed |
+1 | compile | 6m 51s | the patch passed |
+1 | javac | 6m 51s | the patch passed |
-0 | checkstyle | 1m 11s | hadoop-yarn-project/hadoop-yarn: The patch generated 33 new + 148 unchanged - 10 fixed = 181 total (was 158) |
+1 | mvnsite | 2m 32s | the patch passed |
+1 | whitespace | 0m 0s | The patch has no whitespace issues. |
+1 | shadedclient | 9m 40s | patch has no errors when building and testing our client artifacts. |
0 | findbugs | 0m 1s | Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site |
+1 | findbugs | 3m 47s | the patch passed |
+1 | javadoc | 1m 52s | the patch passed |
Other Tests | |||
+1 | unit | 0m 36s | hadoop-yarn-api in the patch passed. |
+1 | unit | 3m 13s | hadoop-yarn-common in the patch passed. |
+1 | unit | 5m 54s | hadoop-yarn-services-core in the patch passed. |
+1 | unit | 0m 31s | hadoop-yarn-services-api in the patch passed. |
+1 | unit | 0m 15s | hadoop-yarn-site in the patch passed. |
+1 | asflicense | 0m 25s | The patch does not generate ASF License warnings. |
80m 54s |
Subsystem | Report/Notes |
---|---|
Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:d4cc50f |
JIRA Issue | |
JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12915435/YARN-7142.001.patch |
Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle |
uname | Linux b79a06d8f83f 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12 13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
Build tool | maven |
Personality | /testptch/patchprocess/precommit/personality/provided.sh |
git revision | trunk / 6c63cc7 |
maven | version: Apache Maven 3.3.9 |
Default Java | 1.8.0_151 |
findbugs | v3.1.0-RC1 |
findbugs | https://builds.apache.org/job/PreCommit-YARN-Build/20028/artifact/out/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-api-warnings.html |
checkstyle | https://builds.apache.org/job/PreCommit-YARN-Build/20028/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt |
Test Results | https://builds.apache.org/job/PreCommit-YARN-Build/20028/testReport/ |
Max. process+thread count | 684 (vs. ulimit of 10000) |
modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site U: hadoop-yarn-project/hadoop-yarn |
Console output | https://builds.apache.org/job/PreCommit-YARN-Build/20028/console |
Powered by | Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
Thanks gsaha, my comments:
1) Placement policy removed from Service? How user to specify common placement policies for components.
2) Compatibility of the placement policy?
3) validatePlacementPolicy:
- Should we enforce target tag name == self component name?
4) Remove expression name from Example.md: - "name": "CA1"
5) Mark unsupported APIs from the documentation/definition.
6) Add node partition to PlacementConstraint. Ref: org.apache.hadoop.yarn.api.resource.PlacementConstraints.PlacementTargets#nodePartition
7) tags => targetTags
8) Use SchedulingRequest when ANY component uses placement policy.
leftnoteasy thank you for the review.
1) Placement policy removed from Service? How user to specify common placement policies for components.
I don't think common/service-level placement policies make sense at least in this first phase of implementation since we support self allocation tag namespace only. We can introduce service level placement policy later once we support other namespaces and if it seems helpful. For now, it will be good to keep it simple.
2) Compatibility of the placement policy?
The existing placement policy was a dummy placeholder and not supported. There was no implementation backing it. Hence there should not be any backward compatibility issue.
I incorporated all your other suggestions in patch 002.
-1 overall |
Vote | Subsystem | Runtime | Comment |
---|---|---|---|
0 | reexec | 0m 22s | Docker mode activated. |
Prechecks | |||
+1 | @author | 0m 0s | The patch does not contain any @author tags. |
+1 | test4tests | 0m 0s | The patch appears to include 3 new or modified test files. |
trunk Compile Tests | |||
0 | mvndep | 0m 25s | Maven dependency ordering for branch |
+1 | mvninstall | 22m 46s | trunk passed |
+1 | compile | 8m 11s | trunk passed |
+1 | checkstyle | 1m 4s | trunk passed |
+1 | mvnsite | 2m 38s | trunk passed |
+1 | shadedclient | 13m 8s | branch has no errors when building and testing our client artifacts. |
0 | findbugs | 0m 0s | Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site |
-1 | findbugs | 1m 12s | hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api in trunk has 1 extant Findbugs warnings. |
+1 | javadoc | 2m 2s | trunk passed |
Patch Compile Tests | |||
0 | mvndep | 0m 12s | Maven dependency ordering for patch |
+1 | mvninstall | 2m 6s | the patch passed |
+1 | compile | 7m 18s | the patch passed |
+1 | javac | 7m 18s | the patch passed |
-0 | checkstyle | 1m 4s | hadoop-yarn-project/hadoop-yarn: The patch generated 12 new + 150 unchanged - 9 fixed = 162 total (was 159) |
+1 | mvnsite | 2m 29s | the patch passed |
+1 | whitespace | 0m 0s | The patch has no whitespace issues. |
+1 | shadedclient | 9m 48s | patch has no errors when building and testing our client artifacts. |
0 | findbugs | 0m 0s | Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site |
-1 | findbugs | 0m 49s | hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) |
+1 | javadoc | 2m 20s | the patch passed |
Other Tests | |||
+1 | unit | 0m 44s | hadoop-yarn-api in the patch passed. |
+1 | unit | 3m 20s | hadoop-yarn-common in the patch passed. |
+1 | unit | 6m 36s | hadoop-yarn-services-core in the patch passed. |
+1 | unit | 0m 34s | hadoop-yarn-services-api in the patch passed. |
+1 | unit | 0m 22s | hadoop-yarn-site in the patch passed. |
+1 | asflicense | 0m 34s | The patch does not generate ASF License warnings. |
94m 21s |
Reason | Tests |
---|---|
FindBugs | module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core |
Useless object stored in variable constraintsMap of method org.apache.hadoop.yarn.service.component.Component.requestContainers(long) At Component.java:constraintsMap of method org.apache.hadoop.yarn.service.component.Component.requestContainers(long) At Component.java:[line 470] |
Subsystem | Report/Notes |
---|---|
Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8620d2b |
JIRA Issue | |
JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12916500/YARN-7142.002.patch |
Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle |
uname | Linux f829d4fe529a 4.4.0-64-generic #85-Ubuntu SMP Mon Feb 20 11:50:30 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
Build tool | maven |
Personality | /testptch/patchprocess/precommit/personality/provided.sh |
git revision | trunk / 3fe41c6 |
maven | version: Apache Maven 3.3.9 |
Default Java | 1.8.0_151 |
findbugs | v3.1.0-RC1 |
findbugs | https://builds.apache.org/job/PreCommit-YARN-Build/20112/artifact/out/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-api-warnings.html |
checkstyle | https://builds.apache.org/job/PreCommit-YARN-Build/20112/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt |
findbugs | https://builds.apache.org/job/PreCommit-YARN-Build/20112/artifact/out/new-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-applications_hadoop-yarn-services_hadoop-yarn-services-core.html |
Test Results | https://builds.apache.org/job/PreCommit-YARN-Build/20112/testReport/ |
Max. process+thread count | 683 (vs. ulimit of 10000) |
modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site U: hadoop-yarn-project/hadoop-yarn |
Console output | https://builds.apache.org/job/PreCommit-YARN-Build/20112/console |
Powered by | Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
Uploaded a 003 patch with the following -
Fixed the findbug issue and 2 of the 12 checkstyle issues reported. 9 of the remaining 10 fall in the bucket of the existing builder pattern followed by other resource objects in the o.a.h.y.service.api.records package. The final one is asking for a default clause for a switch on an enum, which does not make sense/is not a good idea since it suppresses warnings when new enum values are added in the future.
-1 overall |
Vote | Subsystem | Runtime | Comment |
---|---|---|---|
0 | reexec | 0m 26s | Docker mode activated. |
Prechecks | |||
+1 | @author | 0m 0s | The patch does not contain any @author tags. |
+1 | test4tests | 0m 0s | The patch appears to include 3 new or modified test files. |
trunk Compile Tests | |||
0 | mvndep | 0m 53s | Maven dependency ordering for branch |
+1 | mvninstall | 30m 50s | trunk passed |
+1 | compile | 11m 28s | trunk passed |
+1 | checkstyle | 1m 28s | trunk passed |
+1 | mvnsite | 3m 44s | trunk passed |
+1 | shadedclient | 16m 58s | branch has no errors when building and testing our client artifacts. |
0 | findbugs | 0m 0s | Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site |
-1 | findbugs | 1m 31s | hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api in trunk has 1 extant Findbugs warnings. |
+1 | javadoc | 2m 44s | trunk passed |
Patch Compile Tests | |||
0 | mvndep | 0m 12s | Maven dependency ordering for patch |
+1 | mvninstall | 2m 10s | the patch passed |
+1 | compile | 7m 17s | the patch passed |
+1 | javac | 7m 17s | the patch passed |
-0 | checkstyle | 1m 21s | hadoop-yarn-project/hadoop-yarn: The patch generated 10 new + 148 unchanged - 9 fixed = 158 total (was 157) |
+1 | mvnsite | 2m 46s | the patch passed |
+1 | whitespace | 0m 0s | The patch has no whitespace issues. |
+1 | shadedclient | 11m 18s | patch has no errors when building and testing our client artifacts. |
0 | findbugs | 0m 0s | Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site |
+1 | findbugs | 4m 29s | the patch passed |
+1 | javadoc | 2m 21s | the patch passed |
Other Tests | |||
+1 | unit | 0m 48s | hadoop-yarn-api in the patch passed. |
+1 | unit | 3m 24s | hadoop-yarn-common in the patch passed. |
+1 | unit | 6m 38s | hadoop-yarn-services-core in the patch passed. |
+1 | unit | 0m 34s | hadoop-yarn-services-api in the patch passed. |
+1 | unit | 0m 20s | hadoop-yarn-site in the patch passed. |
+1 | asflicense | 0m 33s | The patch does not generate ASF License warnings. |
114m 25s |
Subsystem | Report/Notes |
---|---|
Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8620d2b |
JIRA Issue | |
JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12916515/YARN-7142.003.patch |
Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle |
uname | Linux f7f7434d28cc 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
Build tool | maven |
Personality | /testptch/patchprocess/precommit/personality/provided.sh |
git revision | trunk / 2a2ef15 |
maven | version: Apache Maven 3.3.9 |
Default Java | 1.8.0_151 |
findbugs | v3.1.0-RC1 |
findbugs | https://builds.apache.org/job/PreCommit-YARN-Build/20113/artifact/out/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-api-warnings.html |
checkstyle | https://builds.apache.org/job/PreCommit-YARN-Build/20113/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt |
Test Results | https://builds.apache.org/job/PreCommit-YARN-Build/20113/testReport/ |
Max. process+thread count | 667 (vs. ulimit of 10000) |
modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site U: hadoop-yarn-project/hadoop-yarn |
Console output | https://builds.apache.org/job/PreCommit-YARN-Build/20113/console |
Powered by | Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
leftnoteasy, the -1 on findbugs is an extant issue in hadoop-yarn-api -
org.apache.hadoop.yarn.api.records.Resource.getResources() may expose internal representation by returning Resource.resources
Thanks leftnoteasy for the offline comment. I missed changing tags to target_tags in the YarnServiceAPI.md file. Uploading 004 patch with the fix.
Thanks gsaha, the last patch looks good.
I would prefer to let another set of eyes to look at this patch, sunilg could you help with the patch review? I plan to commit the patch by end of tomorrow if no objections / additional reviews.
Thanks leftnoteasy.
billie.rinaldi if possible, can you please review too.
-1 overall |
Vote | Subsystem | Runtime | Comment |
---|---|---|---|
0 | reexec | 0m 22s | Docker mode activated. |
Prechecks | |||
+1 | @author | 0m 0s | The patch does not contain any @author tags. |
+1 | test4tests | 0m 0s | The patch appears to include 3 new or modified test files. |
trunk Compile Tests | |||
0 | mvndep | 0m 11s | Maven dependency ordering for branch |
+1 | mvninstall | 23m 7s | trunk passed |
+1 | compile | 7m 55s | trunk passed |
+1 | checkstyle | 1m 15s | trunk passed |
+1 | mvnsite | 3m 0s | trunk passed |
+1 | shadedclient | 13m 51s | branch has no errors when building and testing our client artifacts. |
0 | findbugs | 0m 0s | Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site |
-1 | findbugs | 1m 13s | hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api in trunk has 1 extant Findbugs warnings. |
+1 | javadoc | 2m 28s | trunk passed |
Patch Compile Tests | |||
0 | mvndep | 0m 11s | Maven dependency ordering for patch |
+1 | mvninstall | 2m 3s | the patch passed |
+1 | compile | 6m 45s | the patch passed |
+1 | javac | 6m 45s | the patch passed |
-0 | checkstyle | 1m 13s | hadoop-yarn-project/hadoop-yarn: The patch generated 10 new + 150 unchanged - 9 fixed = 160 total (was 159) |
+1 | mvnsite | 2m 22s | the patch passed |
+1 | whitespace | 0m 0s | The patch has no whitespace issues. |
+1 | shadedclient | 9m 11s | patch has no errors when building and testing our client artifacts. |
0 | findbugs | 0m 0s | Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site |
+1 | findbugs | 3m 50s | the patch passed |
+1 | javadoc | 1m 49s | the patch passed |
Other Tests | |||
+1 | unit | 0m 38s | hadoop-yarn-api in the patch passed. |
+1 | unit | 3m 2s | hadoop-yarn-common in the patch passed. |
-1 | unit | 6m 26s | hadoop-yarn-services-core in the patch failed. |
+1 | unit | 0m 29s | hadoop-yarn-services-api in the patch passed. |
+1 | unit | 0m 19s | hadoop-yarn-site in the patch passed. |
+1 | asflicense | 0m 34s | The patch does not generate ASF License warnings. |
92m 32s |
Reason | Tests |
---|---|
Failed junit tests | hadoop.yarn.service.TestYarnNativeServices |
Subsystem | Report/Notes |
---|---|
Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8620d2b |
JIRA Issue | |
JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12916677/YARN-7142.004.patch |
Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle |
uname | Linux c63ec8203b16 4.4.0-64-generic #85-Ubuntu SMP Mon Feb 20 11:50:30 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
Build tool | maven |
Personality | /testptch/patchprocess/precommit/personality/provided.sh |
git revision | trunk / a991e89 |
maven | version: Apache Maven 3.3.9 |
Default Java | 1.8.0_151 |
findbugs | v3.1.0-RC1 |
findbugs | https://builds.apache.org/job/PreCommit-YARN-Build/20129/artifact/out/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-api-warnings.html |
checkstyle | https://builds.apache.org/job/PreCommit-YARN-Build/20129/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt |
unit | https://builds.apache.org/job/PreCommit-YARN-Build/20129/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-applications_hadoop-yarn-services_hadoop-yarn-services-core.txt |
Test Results | https://builds.apache.org/job/PreCommit-YARN-Build/20129/testReport/ |
Max. process+thread count | 683 (vs. ulimit of 10000) |
modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site U: hadoop-yarn-project/hadoop-yarn |
Console output | https://builds.apache.org/job/PreCommit-YARN-Build/20129/console |
Powered by | Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
FAILURE: Integrated in Jenkins build Hadoop-trunk-Commit #13911 (See https://builds.apache.org/job/Hadoop-trunk-Commit/13911/)
YARN-7142. Support placement policy in yarn native services. (Gour Saha (wangda: rev a0bde7d525911680f9e5fb0a939604865eb8e164)
- (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/main/java/org/apache/hadoop/yarn/service/exceptions/RestApiErrorMessages.java
- (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/yarn-service/YarnServiceAPI.md
- (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services-api/src/main/resources/definition/YARN-Simplified-V1-API-Layer-For-Services.yaml
- (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/main/java/org/apache/hadoop/yarn/service/ServiceScheduler.java
- (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/SchedulingRequestPBImpl.java
- (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/main/java/org/apache/hadoop/yarn/service/component/Component.java
- (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/test/java/org/apache/hadoop/yarn/service/TestYarnNativeServices.java
- (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/main/java/org/apache/hadoop/yarn/service/utils/ServiceApiUtil.java
- (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/main/java/org/apache/hadoop/yarn/service/api/records/ConfigFile.java
- (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/main/java/org/apache/hadoop/yarn/service/api/records/PlacementType.java
- (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/test/java/org/apache/hadoop/yarn/service/TestServiceApiUtil.java
- (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/main/java/org/apache/hadoop/yarn/service/api/records/Resource.java
- (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/main/java/org/apache/hadoop/yarn/service/api/records/PlacementScope.java
- (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/main/java/org/apache/hadoop/yarn/service/api/records/Service.java
- (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/main/java/org/apache/hadoop/yarn/service/api/records/ServiceStatus.java
- (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/test/java/org/apache/hadoop/yarn/service/timelineservice/TestServiceTimelinePublisher.java
- (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/main/java/org/apache/hadoop/yarn/service/api/records/PlacementPolicy.java
- (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/main/java/org/apache/hadoop/yarn/service/UpgradeComponentsFinder.java
- (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/main/java/org/apache/hadoop/yarn/service/timelineservice/ServiceTimelinePublisher.java
- (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/main/java/org/apache/hadoop/yarn/service/api/records/Component.java
- (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/resource/PlacementConstraint.java
- (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/main/java/org/apache/hadoop/yarn/service/api/records/PlacementConstraint.java
gsaha, does it make sense to bring this fix to branch-3.1 so we can get this fix in 3.1.1. If you agree, could you help to create a branch-3.1 patch, I saw many conflicts. Maybe caused by the upgrade patch (I will be fine to bring that to branch-3.1 as well if needed).
Upgrade is in the works and not complete yet. It has been broken into multiple tasks to keep the patches from becoming too big. So far only the first patch got committed and it might not be a good idea to merge an incomplete feature into branch-3.1.
Having said that, the placement policy patch totally makes sense. I am attaching a branch-3.1 patch (YARN-7142-branch-3.1.004.patch) for this jira with conflicts resolved. You can use this to commit to branch-3.1.
+1 overall |
Vote | Subsystem | Runtime | Comment |
---|---|---|---|
0 | reexec | 0m 38s | Docker mode activated. |
Prechecks | |||
+1 | @author | 0m 0s | The patch does not contain any @author tags. |
+1 | test4tests | 0m 0s | The patch appears to include 3 new or modified test files. |
branch-3.1 Compile Tests | |||
0 | mvndep | 3m 30s | Maven dependency ordering for branch |
+1 | mvninstall | 20m 26s | branch-3.1 passed |
+1 | compile | 8m 54s | branch-3.1 passed |
+1 | checkstyle | 1m 23s | branch-3.1 passed |
+1 | mvnsite | 3m 1s | branch-3.1 passed |
+1 | shadedclient | 14m 49s | branch has no errors when building and testing our client artifacts. |
0 | findbugs | 0m 0s | Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site |
+1 | findbugs | 4m 5s | branch-3.1 passed |
+1 | javadoc | 2m 25s | branch-3.1 passed |
Patch Compile Tests | |||
0 | mvndep | 0m 12s | Maven dependency ordering for patch |
+1 | mvninstall | 2m 10s | the patch passed |
+1 | compile | 7m 32s | the patch passed |
+1 | javac | 7m 32s | the patch passed |
-0 | checkstyle | 1m 18s | hadoop-yarn-project/hadoop-yarn: The patch generated 10 new + 149 unchanged - 9 fixed = 159 total (was 158) |
+1 | mvnsite | 2m 47s | the patch passed |
+1 | whitespace | 0m 0s | The patch has no whitespace issues. |
+1 | shadedclient | 11m 12s | patch has no errors when building and testing our client artifacts. |
0 | findbugs | 0m 0s | Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site |
+1 | findbugs | 4m 30s | the patch passed |
+1 | javadoc | 2m 19s | the patch passed |
Other Tests | |||
+1 | unit | 0m 46s | hadoop-yarn-api in the patch passed. |
+1 | unit | 3m 10s | hadoop-yarn-common in the patch passed. |
+1 | unit | 6m 2s | hadoop-yarn-services-core in the patch passed. |
+1 | unit | 0m 33s | hadoop-yarn-services-api in the patch passed. |
+1 | unit | 0m 20s | hadoop-yarn-site in the patch passed. |
+1 | asflicense | 0m 35s | The patch does not generate ASF License warnings. |
100m 27s |
Subsystem | Report/Notes |
---|---|
Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:d4cc50f |
JIRA Issue | |
JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12917630/YARN-7142-branch-3.1.004.patch |
Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle |
uname | Linux 3206b4e51b81 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
Build tool | maven |
Personality | /testptch/patchprocess/precommit/personality/provided.sh |
git revision | branch-3.1 / a336561 |
maven | version: Apache Maven 3.3.9 |
Default Java | 1.8.0_162 |
findbugs | v3.1.0-RC1 |
checkstyle | https://builds.apache.org/job/PreCommit-YARN-Build/20250/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt |
Test Results | https://builds.apache.org/job/PreCommit-YARN-Build/20250/testReport/ |
Max. process+thread count | 669 (vs. ulimit of 10000) |
modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site U: hadoop-yarn-project/hadoop-yarn |
Console output | https://builds.apache.org/job/PreCommit-YARN-Build/20250/console |
Powered by | Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
Hi gsaha/leftnoteasy
Thanks for backing port this to branch-3.1. Not related to this task, I have a question about the format of placement policy in yaml file. It looks like it is more like an interpretation of how we specify placement constraints using Java API. I think we should be able to support a simple PC language, by specifying something like:
notin,node,foo
see more in this doc in YARN-7921. I know this is only used distributed shell as a demo, but I think if we find this more easier to write, maybe we can use such expression here too? Just want to know your opinion.
Thanks
Hi leftnoteasy/gsaha
Is the patch for 3.1 branch ready to commit? I see some checkstyle issues but looks like those are already included in trunk so probably doesn't matter? Please take a look, thanks.
I think we should be able to support a simple PC language, by specifying something like: notin,node,foo
cheersyang thank you for the good suggestions. In my opinion, YARN Service should be viewed as a higher level abstraction for layman users who do not understand and don't want to understand YARN internals. They only understand their own application and their app deployment model. If the YARN Service API and spec are not simple and crisp it will immediately cause a barrier to such application-owners from coming to YARN. What do you think?
cheersyang, thanks for reviewing this Jira, I agree with gsaha: unlike DS mostly for dev testing, placement spec of native service should be more clear. The proposed one in this Jira is clearer than DS spec for end user to use.
Currently we're planning to backport several dependencies to branch-3.1 so YARN-7142 can be backported w/o modification and makes native service implementation less divergency between trunk and branch-3.1. Once YARN-8118 backported, we can backport this one.
Thanks gsaha/leftnoteasy, that's OK lets keep it this way. I was just hesitating about which one is more descriptive. Thanks for your feedback.
YARN-8018 has been committed to branch-3.1 (8118 was a typo), so this patch will cherry-pick cleanly now. I will go ahead and do this today.
Just cherry-picked to branch-3.1
Apologize that I missed last comment from billie.rinaldi.
YARN-6592has been merged to trunk. I will work on a patch for this.