Details
-
Sub-task
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
None
-
None
Description
In YARN-3030, to start the per-app aggregator only for a started AM container, we need to determine if the container is an AM container or not from the context in NM (we can do it on RM). This information is missing, such that we worked around to considered the container with ID "xxxx_01" as the AM container. Unfortunately, this is neither necessary or sufficient condition. We need to have a way to determine if a container is an AM container on NM. We can add flag to the container object or create an API to do the judgement. Perhaps the distributed AM information may also be useful to YARN-2877.
Attachments
Attachments
- YARN-3116.patch
- 4 kB
- Giovanni Matteo Fumarola
- YARN-3116.v10.patch
- 36 kB
- Giovanni Matteo Fumarola
- YARN-3116.v2.patch
- 15 kB
- Giovanni Matteo Fumarola
- YARN-3116.v3.patch
- 19 kB
- Giovanni Matteo Fumarola
- YARN-3116.v4.patch
- 21 kB
- Giovanni Matteo Fumarola
- YARN-3116.v5.patch
- 27 kB
- Giovanni Matteo Fumarola
- YARN-3116.v6.patch
- 29 kB
- Zhijie Shen
- YARN-3116.v7.patch
- 30 kB
- Giovanni Matteo Fumarola
- YARN-3116.v8.patch
- 30 kB
- Zhijie Shen
- YARN-3116.v9.patch
- 38 kB
- Giovanni Matteo Fumarola
Issue Links
- blocks
-
YARN-2884 Proxying all AM-RM communications
- Resolved
- breaks
-
YARN-4839 ResourceManager deadlock between RMAppAttemptImpl and SchedulerApplicationAttempt
- Resolved
- duplicates
-
YARN-3828 Add a flag in container to indicate whether it's an AM container or not
- Resolved
- is related to
-
YARN-2882 Add an OPPORTUNISTIC ExecutionType
- Resolved
- relates to
-
YARN-3030 [Collector wireup] Set up timeline collector with basic request serving structure and lifecycle
- Resolved
Activity
Hi junping_du,
+1, Approach seems to be fine except for one concern, we might need to ensure that Only RM sets this in ContainerLaunchContext, as once API is exposed it may get used in many other places also and ContainerLaunchContext can be created by apps too.
Close this jira as the duplicate of YARN-3828, whether the contributor has already started working on the issue.
We usually close the newly filed JIRAs as dups.
I am closing the other one as dup and assigning this to giovanni.fumarola.
giovanni.fumarola, thanks for the patch. Two quick comments about it:
1. IMHO, I prefer to use ContainerTokenIdentifier instead of ContainerLaunchContext to pass the AM flag, as we did for node label and aggregation log context. ContainerLaunchContext is user API, such that user has the freedom to change the flag and says whether the container he starts is AM or not. Theoretically, user can still change the container token too, but it's not that obvious to user. Thoughts?
2. Not sure if AMRMProxy wants the AM flag to be propagated somewhere, but in our use case, once NM receives the flag, we want to further push it to ContainerInitializationContext, such that we can get the flag inside aux service.
-1 overall |
Vote | Subsystem | Runtime | Comment |
---|---|---|---|
0 | pre-patch | 18m 42s | Pre-patch trunk compilation is healthy. |
+1 | @author | 0m 0s | The patch does not contain any @author tags. |
-1 | tests included | 0m 0s | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. |
+1 | javac | 7m 32s | There were no new javac warning messages. |
+1 | javadoc | 9m 38s | There were no new javadoc warning messages. |
+1 | release audit | 0m 23s | The applied patch does not increase the total number of release audit warnings. |
+1 | checkstyle | 2m 18s | There were no new checkstyle issues. |
+1 | whitespace | 0m 0s | The patch has no lines that end in whitespace. |
+1 | install | 1m 34s | mvn install still works. |
+1 | eclipse:eclipse | 0m 32s | The patch built with eclipse:eclipse. |
+1 | findbugs | 4m 30s | The patch does not introduce any new Findbugs (version 3.0.0) warnings. |
+1 | yarn tests | 0m 25s | Tests passed in hadoop-yarn-api. |
+1 | yarn tests | 2m 1s | Tests passed in hadoop-yarn-common. |
-1 | yarn tests | 50m 50s | Tests failed in hadoop-yarn-server-resourcemanager. |
98m 29s |
Reason | Tests |
---|---|
Failed unit tests | hadoop.yarn.server.resourcemanager.TestWorkPreservingRMRestart |
Subsystem | Report/Notes |
---|---|
Patch URL | http://issues.apache.org/jira/secure/attachment/12740712/YARN-3116.patch |
Optional Tests | javadoc javac unit findbugs checkstyle |
git revision | trunk / 20c03c9 |
hadoop-yarn-api test log | https://builds.apache.org/job/PreCommit-YARN-Build/8298/artifact/patchprocess/testrun_hadoop-yarn-api.txt |
hadoop-yarn-common test log | https://builds.apache.org/job/PreCommit-YARN-Build/8298/artifact/patchprocess/testrun_hadoop-yarn-common.txt |
hadoop-yarn-server-resourcemanager test log | https://builds.apache.org/job/PreCommit-YARN-Build/8298/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt |
Test Results | https://builds.apache.org/job/PreCommit-YARN-Build/8298/testReport/ |
Java | 1.7.0_55 |
uname | Linux asf902.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
Console output | https://builds.apache.org/job/PreCommit-YARN-Build/8298/console |
This message was automatically generated.
Thanks zjshen for quickly reviewing the patch & your comments.
1. I agree that ContainerTokenIdentifier would be a better place to do it so that we keep the flag internal but the ContainerTokenIdentifier is created before the state transition in RMAppAttempt that sets the AM flag in RMContainer. I can try to recreate ContainerTokenIdentifier at the AM launch but that looks unwieldy. Do you have any suggestions on how to do it cleaner?
2. Again a good observation, I'll add this in the next iteration of the patch based on your suggestion for (1) above.
Thanks zjshen for the offline discussion with giovanni.fumarola. The conclusion was that:
- we will add a flag in ContainerTokenIdentifier to indicate whether its an AM container or not
- we'll set the flag in SchedulerApplicationAttempt::pullNewlyAllocatedContainersAndNMTokens where the ContainerTokenIdentifier is created based on whether the current container has a master container. The working knowledge is that masterContainer for AM is null as it itself is the master container.
Sounds good to me. Would you mind taking care of the point (2). At least in YARN-2928 use case, we need the am container flag to be propagated to aux service too. Thanks!
Thanks subru and zjshen for the offline discussion and the feedback. As agreed upon, I am uploading an updated patch that:
- Adds a flag to indicate if a container is AM or not in ContainerTokenIdentifier.
- Adds a flag to indicate if a container is AM or not in ContainerContext. We require the flag in both ContainerInitializationContext and ContainerTerminationContext for
YARN-2884.
-1 overall |
Vote | Subsystem | Runtime | Comment |
---|---|---|---|
0 | pre-patch | 20m 36s | Pre-patch trunk compilation is healthy. |
+1 | @author | 0m 0s | The patch does not contain any @author tags. |
+1 | tests included | 0m 0s | The patch appears to include 2 new or modified test files. |
+1 | javac | 7m 51s | There were no new javac warning messages. |
+1 | javadoc | 10m 29s | There were no new javadoc warning messages. |
+1 | release audit | 0m 30s | The applied patch does not increase the total number of release audit warnings. |
-1 | checkstyle | 2m 54s | The applied patch generated 1 new checkstyle issues (total was 9, now 9). |
-1 | whitespace | 0m 1s | The patch has 2 line(s) that end in whitespace. Use git apply --whitespace=fix. |
+1 | install | 2m 10s | mvn install still works. |
+1 | eclipse:eclipse | 0m 42s | The patch built with eclipse:eclipse. |
+1 | findbugs | 7m 22s | The patch does not introduce any new Findbugs (version 3.0.0) warnings. |
+1 | yarn tests | 0m 35s | Tests passed in hadoop-yarn-api. |
+1 | yarn tests | 2m 49s | Tests passed in hadoop-yarn-common. |
-1 | yarn tests | 6m 40s | Tests failed in hadoop-yarn-server-nodemanager. |
-1 | yarn tests | 63m 25s | Tests failed in hadoop-yarn-server-resourcemanager. |
126m 41s |
Reason | Tests |
---|---|
Failed unit tests | hadoop.yarn.server.nodemanager.TestNodeStatusUpdaterForLabels |
hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairScheduler | |
hadoop.yarn.server.resourcemanager.scheduler.TestSchedulerHealth | |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacityScheduler | |
hadoop.yarn.server.resourcemanager.TestResourceManager | |
Timed out tests | org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestNodeLabelContainerAllocation |
This message was automatically generated.
subru and giovanni.fumarola, in yesterdays offline discussion, it seems that we have missed one case, which is unmanaged AM. If the app is using the unmanaged AM, RMAppAttemptImpl#masterContainer should always be null. Therefore, the more precise condition to determine whether the requested container is AM container is that getUnmanagedAM == false && getMasterContainer == null.
And I think there's a bug around the existing code:
// AM container allocation doesn't support non-exclusive allocation to // avoid painful of preempt an AM container if (schedulingMode == SchedulingMode.IGNORE_PARTITION_EXCLUSIVITY) { RMAppAttempt rmAppAttempt = csContext.getRMContext().getRMApps() .get(application.getApplicationId()).getCurrentAppAttempt(); if (null == rmAppAttempt.getMasterContainer()) { if (LOG.isDebugEnabled()) { LOG.debug("Skip allocating AM container to app_attempt=" + application.getApplicationAttemptId() + ", don't allow to allocate AM container in non-exclusive mode"); } break; } }
Some more comments about the patch:
1. For the constructors added in ContainerXXXX, I suggest creating a new method with the additional param for the compatibility purpose, especially for the api classes (unless the modified constructors are also just added to the new release). According to the previous lessons, though some of them marked @Private, users may ignore the annotations.
2. Can we test the new flag in ContainerTokenIdentifier#testContainerTokenIdentifier?
Thanks zjshen for the comments & for the bug identified in LeafQueue.
1. I modified the Classes ContainerXXXX according your suggestion.
2. I added a new test to check the value of the new flag and I edit an existing test to check its value.
3. I edited LeafQueue to avoid the existing bug.
I ran test-patch.sh in my local box and I got some checkstyle issues.
LeafQueue.java:1: File length is 2,088 lines (max allowed is 2,000) - I added only 1 line.
LeafQueue.java:742:3: Method length is 233 lines (max allowed is 150) - I added only 1 line.
LeafQueue.java:833:73: Expression can be simplified - It is only 2 simple Boolean conditions.
SchedulerApplicationAttempt.java:480:65: Expression can be simplified - It is only 2 simple Boolean conditions.
RMContainerTokenSecretManager.java:206: First sentence should end with a period - I did not figure out this. The some comment is in a different methods.
RMContainerTokenSecretManager.java:220:16: More than 7 parameters (found 9) - I added only 1 parameter.
ContainerTokenIdentifier.java:81:10: More than 7 parameters (found 12) - I added only 1 parameter.
In fact jenkins complains much fewer checkstyle issue, and it seems not to be caused by your patch. But can you take care of the test failure?
I have already taken care of the test failures. All of them were raised due the same missing null check.
I included this in the last patch
-1 overall |
Vote | Subsystem | Runtime | Comment |
---|---|---|---|
0 | pre-patch | 19m 56s | Pre-patch trunk compilation is healthy. |
+1 | @author | 0m 0s | The patch does not contain any @author tags. |
+1 | tests included | 0m 0s | The patch appears to include 1 new or modified test files. |
+1 | javac | 7m 38s | There were no new javac warning messages. |
+1 | javadoc | 9m 34s | There were no new javadoc warning messages. |
+1 | release audit | 0m 24s | The applied patch does not increase the total number of release audit warnings. |
-1 | checkstyle | 2m 36s | The applied patch generated 1 new checkstyle issues (total was 9, now 10). |
+1 | whitespace | 0m 1s | The patch has no lines that end in whitespace. |
+1 | install | 1m 36s | mvn install still works. |
+1 | eclipse:eclipse | 0m 33s | The patch built with eclipse:eclipse. |
+1 | findbugs | 5m 44s | The patch does not introduce any new Findbugs (version 3.0.0) warnings. |
+1 | yarn tests | 0m 30s | Tests passed in hadoop-yarn-api. |
+1 | yarn tests | 1m 57s | Tests passed in hadoop-yarn-common. |
+1 | yarn tests | 6m 4s | Tests passed in hadoop-yarn-server-nodemanager. |
-1 | yarn tests | 51m 45s | Tests failed in hadoop-yarn-server-resourcemanager. |
108m 52s |
Reason | Tests |
---|---|
Failed unit tests | hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerAllocation |
This message was automatically generated.
Looks good to me overall. Two nits:
1. I thought it may not be necessary, as they should be not null at the moment to allocate a container.
479 if (appAttempt != null && appAttempt.getSubmissionContext() != null) {
2. Can we avoid duplicating most of the code in "testAMContainerTokenIdentifier"?
1. Few tests that use that function they don't have appAttempt or submissionContext.
This means that in a normal execution these 2 values can be null.
2. Yes, I'm editing the test.
-1 overall |
Vote | Subsystem | Runtime | Comment |
---|---|---|---|
0 | pre-patch | 20m 34s | Pre-patch trunk compilation is healthy. |
+1 | @author | 0m 0s | The patch does not contain any @author tags. |
+1 | tests included | 0m 0s | The patch appears to include 2 new or modified test files. |
+1 | javac | 7m 48s | There were no new javac warning messages. |
+1 | javadoc | 10m 20s | There were no new javadoc warning messages. |
+1 | release audit | 0m 22s | The applied patch does not increase the total number of release audit warnings. |
-1 | checkstyle | 2m 37s | The applied patch generated 1 new checkstyle issues (total was 9, now 10). |
+1 | whitespace | 0m 2s | The patch has no lines that end in whitespace. |
+1 | install | 1m 37s | mvn install still works. |
+1 | eclipse:eclipse | 0m 34s | The patch built with eclipse:eclipse. |
+1 | findbugs | 5m 56s | The patch does not introduce any new Findbugs (version 3.0.0) warnings. |
+1 | yarn tests | 0m 23s | Tests passed in hadoop-yarn-api. |
+1 | yarn tests | 2m 1s | Tests passed in hadoop-yarn-common. |
-1 | yarn tests | 5m 46s | Tests failed in hadoop-yarn-server-nodemanager. |
-1 | yarn tests | 48m 0s | Tests failed in hadoop-yarn-server-resourcemanager. |
106m 26s |
Reason | Tests |
---|---|
Failed unit tests | hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesAppsModification |
hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesApps | |
hadoop.yarn.server.resourcemanager.applicationsmanager.TestAMRestart | |
hadoop.yarn.server.resourcemanager.ahs.TestRMApplicationHistoryWriter | |
hadoop.yarn.server.resourcemanager.webapp.TestNodesPage | |
Timed out tests | org.apache.hadoop.yarn.server.nodemanager.TestNodeStatusUpdater |
This message was automatically generated.
1. I think normal execution won't have null attempt, but the tests have omitted it. You probably want to fix the test code instead, such as mock the currentAttempt and fix Application#submmit to add the attempt to rmapp.
Thanks zjshen for your feedback. I have uploaded a new patch that fixes the test code based on your suggestions.
All the tests pass except for TestFairScheduler::testQueueMaxAMShare.
I tried but could not find an easy fix for that one. Can you help me to fix it?
-1 overall |
Vote | Subsystem | Runtime | Comment |
---|---|---|---|
0 | pre-patch | 20m 28s | Pre-patch trunk compilation is healthy. |
+1 | @author | 0m 0s | The patch does not contain any @author tags. |
+1 | tests included | 0m 0s | The patch appears to include 5 new or modified test files. |
+1 | javac | 9m 4s | There were no new javac warning messages. |
+1 | javadoc | 10m 8s | There were no new javadoc warning messages. |
+1 | release audit | 0m 22s | The applied patch does not increase the total number of release audit warnings. |
-1 | checkstyle | 2m 49s | The applied patch generated 1 new checkstyle issues (total was 9, now 10). |
+1 | whitespace | 0m 3s | The patch has no lines that end in whitespace. |
+1 | install | 1m 36s | mvn install still works. |
+1 | eclipse:eclipse | 0m 34s | The patch built with eclipse:eclipse. |
+1 | findbugs | 6m 2s | The patch does not introduce any new Findbugs (version 3.0.0) warnings. |
+1 | yarn tests | 0m 25s | Tests passed in hadoop-yarn-api. |
+1 | yarn tests | 2m 3s | Tests passed in hadoop-yarn-common. |
-1 | yarn tests | 6m 47s | Tests failed in hadoop-yarn-server-nodemanager. |
-1 | yarn tests | 69m 45s | Tests failed in hadoop-yarn-server-resourcemanager. |
130m 34s |
Reason | Tests |
---|---|
Failed unit tests | hadoop.yarn.server.nodemanager.containermanager.linux.privileged.TestPrivilegedOperationExecutor |
hadoop.yarn.server.nodemanager.TestNodeStatusUpdaterForLabels | |
hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairScheduler | |
hadoop.yarn.server.resourcemanager.ahs.TestRMApplicationHistoryWriter |
This message was automatically generated.
Fixed the test failure in the new patch. Otherwise, the previous patch looks good to me. As I'm touching the patch also, I need a second committer to take a look. jianhe, would you mind doing me a favor?
-1 overall |
Vote | Subsystem | Runtime | Comment |
---|---|---|---|
0 | pre-patch | 20m 26s | Pre-patch trunk compilation is healthy. |
+1 | @author | 0m 0s | The patch does not contain any @author tags. |
+1 | tests included | 0m 0s | The patch appears to include 5 new or modified test files. |
+1 | javac | 7m 47s | There were no new javac warning messages. |
+1 | javadoc | 10m 25s | There were no new javadoc warning messages. |
+1 | release audit | 0m 23s | The applied patch does not increase the total number of release audit warnings. |
-1 | checkstyle | 2m 35s | The applied patch generated 1 new checkstyle issues (total was 9, now 10). |
+1 | whitespace | 0m 3s | The patch has no lines that end in whitespace. |
+1 | install | 1m 39s | mvn install still works. |
+1 | eclipse:eclipse | 0m 33s | The patch built with eclipse:eclipse. |
+1 | findbugs | 5m 54s | The patch does not introduce any new Findbugs (version 3.0.0) warnings. |
+1 | yarn tests | 0m 24s | Tests passed in hadoop-yarn-api. |
+1 | yarn tests | 2m 0s | Tests passed in hadoop-yarn-common. |
-1 | yarn tests | 6m 6s | Tests failed in hadoop-yarn-server-nodemanager. |
-1 | yarn tests | 51m 9s | Tests failed in hadoop-yarn-server-resourcemanager. |
109m 49s |
Reason | Tests |
---|---|
Failed unit tests | hadoop.yarn.server.nodemanager.containermanager.linux.privileged.TestPrivilegedOperationExecutor |
Timed out tests | org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestAppRunnability |
This message was automatically generated.
Thanks zjshen for fixing the test failure.
For the TestPrivilegedOperationExecutor I just applied a new patch with the fix.
I got the same problem with TestAppRunnability when I was working on TestFairScheduler::testQueueMaxAMShare.
Is TestAppRunnability failure related to this patch? The normal practice is to check if the test failure is related to the code change in this jira. If not, you can go ahead to fix a separate jira to tackling it.
Thanks for fixing TestPrivilegedOperationExecutor. It seems to be straightforward. So let's keep it here.
The patch looks fine overall. Only one comment:
Instead of just specify a boolean flag to indicate the AM container, how about adding an enum of the containerType for the future extensibility ? such as https://issues.apache.org/jira/browse/YARN-2261, we will have the post application clean up container.
TestAppRunnability::testNotUserAsDefaultQueue is related to this patch.
The fix to solve the issue on TestFairScheduler::testQueueMaxAMShare brings the previous test to failure.
Xuan, thanks for your comment. I think this is a good point. To be forward compatible, it's better to use the enum here instead of the boolean flag. In this case, we can add more enum, such as SystemContainer and so on in the future without adding new flag and breaking the compatibility. giovanni.fumarola, subru, how do you think?
xgong thanks for the comment, it's an accurate observation.
zjshen, I think it is a good idea. I can start remove the flag and insert a new enum as you suggested.
-1 overall |
Vote | Subsystem | Runtime | Comment |
---|---|---|---|
-1 | pre-patch | 19m 26s | Findbugs (version 3.0.0) appears to be broken on trunk. |
+1 | @author | 0m 0s | The patch does not contain any @author tags. |
+1 | tests included | 0m 0s | The patch appears to include 6 new or modified test files. |
+1 | javac | 7m 53s | There were no new javac warning messages. |
+1 | javadoc | 9m 44s | There were no new javadoc warning messages. |
+1 | release audit | 0m 22s | The applied patch does not increase the total number of release audit warnings. |
-1 | checkstyle | 2m 25s | The applied patch generated 1 new checkstyle issues (total was 9, now 10). |
+1 | whitespace | 0m 4s | The patch has no lines that end in whitespace. |
+1 | install | 1m 22s | mvn install still works. |
+1 | eclipse:eclipse | 0m 32s | The patch built with eclipse:eclipse. |
+1 | findbugs | 5m 47s | The patch does not introduce any new Findbugs (version 3.0.0) warnings. |
+1 | yarn tests | 0m 26s | Tests passed in hadoop-yarn-api. |
+1 | yarn tests | 1m 58s | Tests passed in hadoop-yarn-common. |
+1 | yarn tests | 6m 17s | Tests passed in hadoop-yarn-server-nodemanager. |
-1 | yarn tests | 51m 6s | Tests failed in hadoop-yarn-server-resourcemanager. |
107m 48s |
Reason | Tests |
---|---|
Timed out tests | org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestAppRunnability |
This message was automatically generated.
-1 overall |
Vote | Subsystem | Runtime | Comment |
---|---|---|---|
0 | pre-patch | 21m 9s | Pre-patch trunk compilation is healthy. |
+1 | @author | 0m 0s | The patch does not contain any @author tags. |
+1 | tests included | 0m 0s | The patch appears to include 6 new or modified test files. |
+1 | javac | 7m 54s | There were no new javac warning messages. |
+1 | javadoc | 10m 4s | There were no new javadoc warning messages. |
+1 | release audit | 0m 24s | The applied patch does not increase the total number of release audit warnings. |
-1 | checkstyle | 2m 36s | The applied patch generated 1 new checkstyle issues (total was 9, now 10). |
+1 | whitespace | 0m 3s | The patch has no lines that end in whitespace. |
+1 | install | 1m 31s | mvn install still works. |
+1 | eclipse:eclipse | 0m 35s | The patch built with eclipse:eclipse. |
+1 | findbugs | 6m 13s | The patch does not introduce any new Findbugs (version 3.0.0) warnings. |
+1 | yarn tests | 0m 24s | Tests passed in hadoop-yarn-api. |
+1 | yarn tests | 2m 1s | Tests passed in hadoop-yarn-common. |
+1 | yarn tests | 6m 15s | Tests passed in hadoop-yarn-server-nodemanager. |
-1 | yarn tests | 27m 51s | Tests failed in hadoop-yarn-server-resourcemanager. |
87m 29s |
Reason | Tests |
---|---|
Failed unit tests | hadoop.yarn.server.resourcemanager.security.TestAMRMTokens |
hadoop.yarn.server.resourcemanager.TestResourceTrackerService | |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestWorkPreservingRMRestartForNodeLabel | |
hadoop.yarn.server.resourcemanager.TestContainerResourceUsage | |
hadoop.yarn.server.resourcemanager.applicationsmanager.TestAMRMRPCNodeUpdates | |
hadoop.yarn.server.resourcemanager.TestResourceManager | |
hadoop.yarn.server.resourcemanager.applicationsmanager.TestAMRMRPCResponseId | |
hadoop.yarn.server.resourcemanager.TestRM | |
hadoop.yarn.server.resourcemanager.resourcetracker.TestNMExpiry | |
hadoop.yarn.server.resourcemanager.TestApplicationMasterService | |
hadoop.yarn.server.resourcemanager.resourcetracker.TestNMReconnect | |
hadoop.yarn.server.resourcemanager.TestClientRMService | |
hadoop.yarn.server.resourcemanager.scheduler.TestSchedulerHealth | |
hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairScheduler | |
hadoop.yarn.server.resourcemanager.resourcetracker.TestRMNMRPCResponseId | |
hadoop.yarn.server.resourcemanager.TestApplicationMasterLauncher | |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestNodeLabelContainerAllocation | |
hadoop.yarn.server.resourcemanager.applicationsmanager.TestAMRestart | |
hadoop.yarn.server.resourcemanager.TestRMHA | |
hadoop.yarn.server.resourcemanager.security.TestClientToAMTokens | |
hadoop.yarn.server.resourcemanager.TestRMRestart | |
hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer | |
hadoop.yarn.server.resourcemanager.scheduler.TestSchedulerUtils | |
hadoop.yarn.server.resourcemanager.scheduler.TestAbstractYarnScheduler | |
hadoop.yarn.server.resourcemanager.TestKillApplicationWithRMHA | |
hadoop.yarn.server.resourcemanager.scheduler.fifo.TestFifoScheduler | |
hadoop.yarn.server.resourcemanager.TestWorkPreservingRMRestart | |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacityScheduler | |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerAllocation | |
hadoop.yarn.server.resourcemanager.TestAMAuthorization | |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacitySchedulerNodeLabelUpdate | |
Timed out tests | org.apache.hadoop.yarn.server.resourcemanager.TestSubmitApplicationWithRMHA |
org.apache.hadoop.yarn.server.resourcemanager.logaggregationstatus.TestRMAppLogAggregationStatus |
This message was automatically generated.
giovanni.fumarola, xgong, zjshen:
Given you are thinking to substitute the boolean with an enum that indicates the container type, I think this is becoming very related to YARN-2882, which is part of the more general YARN-2877 (that introduces distributed scheduling in YARN).
In YARN-2882, we introduce container types to differentiate between GUARANTEED containers allocated by the central RM, and QUEUEABLE containers allocated by one of the distributed schedulers.
We already have a patch available for this JIRA.
What would be interesting to see is whether the AM_CONTAINER should become yet another type of container or whether it should be a separate field within the container type.
The former would probably make more sense at the current implementation, as an AM_CONTAINER can only be allocated by the central RM (we cannot have a QUEUEABLE container that is also an AM_CONTAINER).
The latter, however, would probably give more flexibility.
kkaranasos, thanks for notifying us of YARN-2882. I took a quick look at the jira. Our approach seems to be similar, but it seems that we're on parallel tracks. While YARN-2882 defines two container type for container related API so as to differ the container request to RM or NM, what we want to label a container here aims to let NM know if the container hosts AM or not. This is completely internal information, and users are blind to this type and also not able to set/change it. And this is why we propose to pass this information via ContainerTokenIndentifier instead of ContainerLaunchContext. Thoughts?
kkaranasos, thanks for the observation. As zjshen rightly pointed out, YARN-2882 and YARN-3116 are complementary. We are adding the container type enum to notify the NM whether its an AM container or not. This is purely internal and we deliberately don't want to expose it to the Application for security reasons while in YARN-2882, you want to expose the container type to the application.
Thanks for the feedback, giovanni.fumarola and zjshen.
I agree that the two JIRAs are complementary.
One first observation is that probably we should have two different flavours of container types: ContainerRequestType and ContainerType. The former should be exposed to the application (e.g., centralized or distributed scheduling), while the latter should encode information for the NM to properly handle the container request without exposing it to the application (e.g., GUARANTEED or QUEUEABLE, AM_CONTAINER or "non-AM"_CONTAINER). This looks like a more future-proof design.
That said, I am wondering whether the ContainerTokenIdentifier is the proper place to set the container type to accommodate other types of containers (and not just the isAMContainer flag). I am on the go today without access to the code – will get back to you tomorrow regarding that.
kkaranasos, thanks for taking at look at this JIRA. We felt ContainerTokenIdentifier _ is a reasonably secure way to propagate the _containerType to NM from the RM.The containerType is set in the ContainerContext in the NM so that it is available for auxiliary services. kishorch is already integrating this with YARN-2884 so it should be aligned with what you are trying to achieve in YARN-2877. Additionally based on xgong's feedback, we updated containerType to be an enum from the earlier boolean flag so should cover your future requirements of adding additional container types.
kkaranasos, I didn't touch the detail on YARN-2884, but it seems to be the API change that needs to be exposed to the users. In this case, user faced objects, i.e., ContainerLaunchContext, is the better choice for you.
-1 overall |
Vote | Subsystem | Runtime | Comment |
---|---|---|---|
0 | pre-patch | 20m 19s | Pre-patch trunk compilation is healthy. |
+1 | @author | 0m 0s | The patch does not contain any @author tags. |
+1 | tests included | 0m 0s | The patch appears to include 5 new or modified test files. |
+1 | javac | 7m 56s | There were no new javac warning messages. |
+1 | javadoc | 9m 46s | There were no new javadoc warning messages. |
+1 | release audit | 0m 22s | The applied patch does not increase the total number of release audit warnings. |
+1 | checkstyle | 2m 48s | There were no new checkstyle issues. |
+1 | whitespace | 0m 6s | The patch has no lines that end in whitespace. |
+1 | install | 1m 24s | mvn install still works. |
+1 | eclipse:eclipse | 0m 33s | The patch built with eclipse:eclipse. |
+1 | findbugs | 5m 50s | The patch does not introduce any new Findbugs (version 3.0.0) warnings. |
+1 | yarn tests | 0m 24s | Tests passed in hadoop-yarn-api. |
+1 | yarn tests | 1m 56s | Tests passed in hadoop-yarn-common. |
-1 | yarn tests | 6m 4s | Tests failed in hadoop-yarn-server-nodemanager. |
-1 | yarn tests | 51m 25s | Tests failed in hadoop-yarn-server-resourcemanager. |
108m 58s |
Reason | Tests |
---|---|
Failed unit tests | hadoop.yarn.server.nodemanager.containermanager.localizer.TestResourceLocalizationService |
hadoop.yarn.server.nodemanager.containermanager.container.TestContainer | |
hadoop.yarn.server.nodemanager.containermanager.logaggregation.TestLogAggregationService | |
hadoop.yarn.server.resourcemanager.rmapp.TestRMAppTransitions | |
hadoop.yarn.server.resourcemanager.TestResourceTrackerService | |
hadoop.yarn.server.resourcemanager.applicationsmanager.TestAMRMRPCNodeUpdates |
This message was automatically generated.
I looked at the test failures, they are unrelated to my patch. I get the same test failures when I tested locally on trunk without applying my patch.
zjshen, I am working with kishorch on YARN-2884 and we do not want to expose the containerType to user. We are updating the YARN-2884 patch to use containerInitializationContext to determine if the container is AM or not. Seperately kkaranasos is working on YARN-2882 where he needs to expose the container*Request*Type to the user. That is complementary work & should not affect this patch. So can we get this committed soon as YARN-2884 is blocked on this. Thanks!
-1 overall |
Vote | Subsystem | Runtime | Comment |
---|---|---|---|
0 | pre-patch | 20m 4s | Pre-patch trunk compilation is healthy. |
+1 | @author | 0m 0s | The patch does not contain any @author tags. |
+1 | tests included | 0m 0s | The patch appears to include 5 new or modified test files. |
+1 | javac | 7m 39s | There were no new javac warning messages. |
+1 | javadoc | 9m 37s | There were no new javadoc warning messages. |
+1 | release audit | 0m 23s | The applied patch does not increase the total number of release audit warnings. |
+1 | checkstyle | 2m 46s | There were no new checkstyle issues. |
+1 | whitespace | 0m 7s | The patch has no lines that end in whitespace. |
+1 | install | 1m 25s | mvn install still works. |
+1 | eclipse:eclipse | 0m 34s | The patch built with eclipse:eclipse. |
+1 | findbugs | 5m 42s | The patch does not introduce any new Findbugs (version 3.0.0) warnings. |
+1 | yarn tests | 0m 25s | Tests passed in hadoop-yarn-api. |
+1 | yarn tests | 1m 56s | Tests passed in hadoop-yarn-common. |
-1 | yarn tests | 6m 3s | Tests failed in hadoop-yarn-server-nodemanager. |
-1 | yarn tests | 51m 9s | Tests failed in hadoop-yarn-server-resourcemanager. |
107m 54s |
Reason | Tests |
---|---|
Failed unit tests | hadoop.yarn.server.nodemanager.containermanager.container.TestContainer |
hadoop.yarn.server.nodemanager.containermanager.localizer.TestResourceLocalizationService | |
hadoop.yarn.server.nodemanager.TestDeletionService | |
hadoop.yarn.server.nodemanager.containermanager.logaggregation.TestLogAggregationService | |
hadoop.yarn.server.resourcemanager.applicationsmanager.TestAMRMRPCNodeUpdates | |
hadoop.yarn.server.resourcemanager.rmapp.TestRMAppTransitions | |
hadoop.yarn.server.resourcemanager.TestResourceTrackerService |
This message was automatically generated.
Committed the patch to trunk and branch-2. Thanks for the patch, giovanni.fumarola!
FAILURE: Integrated in Hadoop-trunk-Commit #8150 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8150/)
YARN-3116. RM notifies NM whether a container is an AM container or normal task container. Contributed by Giovanni Matteo Fumarola. (zjshen: rev 1ea36299a47af302379ae0750b571ec021eb54ad)
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairSchedulerTestBase.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/security/TestYARNTokenIdentifier.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/yarn_protos.proto
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestContainerAllocation.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/AuxServices.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/server/api/ContainerContext.java
- hadoop-yarn-project/CHANGES.txt
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/server/api/ContainerTerminationContext.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/Application.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerApplicationAttempt.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/security/ContainerTokenIdentifier.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/server/api/ContainerInitializationContext.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/proto/server/yarn_security_token.proto
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/security/RMContainerTokenSecretManager.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fifo/TestFifoScheduler.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/ProtoUtils.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/server/api/ContainerType.java
FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #253 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/253/)
YARN-3116. RM notifies NM whether a container is an AM container or normal task container. Contributed by Giovanni Matteo Fumarola. (zjshen: rev 1ea36299a47af302379ae0750b571ec021eb54ad)
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/Application.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/server/api/ContainerType.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/security/ContainerTokenIdentifier.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/server/api/ContainerTerminationContext.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fifo/TestFifoScheduler.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/security/TestYARNTokenIdentifier.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/proto/server/yarn_security_token.proto
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/AuxServices.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairSchedulerTestBase.java
- hadoop-yarn-project/CHANGES.txt
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/yarn_protos.proto
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/server/api/ContainerInitializationContext.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerApplicationAttempt.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/security/RMContainerTokenSecretManager.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestContainerAllocation.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/server/api/ContainerContext.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/ProtoUtils.java
FAILURE: Integrated in Hadoop-Yarn-trunk #983 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/983/)
YARN-3116. RM notifies NM whether a container is an AM container or normal task container. Contributed by Giovanni Matteo Fumarola. (zjshen: rev 1ea36299a47af302379ae0750b571ec021eb54ad)
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/yarn_protos.proto
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/server/api/ContainerInitializationContext.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/security/TestYARNTokenIdentifier.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/AuxServices.java
- hadoop-yarn-project/CHANGES.txt
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/server/api/ContainerContext.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/proto/server/yarn_security_token.proto
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerApplicationAttempt.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fifo/TestFifoScheduler.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/security/RMContainerTokenSecretManager.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/security/ContainerTokenIdentifier.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestContainerAllocation.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairSchedulerTestBase.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/Application.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/server/api/ContainerTerminationContext.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/server/api/ContainerType.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/ProtoUtils.java
FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #251 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/251/)
YARN-3116. RM notifies NM whether a container is an AM container or normal task container. Contributed by Giovanni Matteo Fumarola. (zjshen: rev 1ea36299a47af302379ae0750b571ec021eb54ad)
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestContainerAllocation.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/Application.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/ProtoUtils.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/AuxServices.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/yarn_protos.proto
- hadoop-yarn-project/CHANGES.txt
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/security/RMContainerTokenSecretManager.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/server/api/ContainerInitializationContext.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/proto/server/yarn_security_token.proto
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/server/api/ContainerType.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerApplicationAttempt.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/server/api/ContainerContext.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairSchedulerTestBase.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fifo/TestFifoScheduler.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/server/api/ContainerTerminationContext.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/security/TestYARNTokenIdentifier.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/security/ContainerTokenIdentifier.java
FAILURE: Integrated in Hadoop-Hdfs-trunk #2180 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2180/)
YARN-3116. RM notifies NM whether a container is an AM container or normal task container. Contributed by Giovanni Matteo Fumarola. (zjshen: rev 1ea36299a47af302379ae0750b571ec021eb54ad)
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/server/api/ContainerContext.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/security/TestYARNTokenIdentifier.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/server/api/ContainerInitializationContext.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/proto/server/yarn_security_token.proto
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/Application.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/server/api/ContainerType.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestContainerAllocation.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/ProtoUtils.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/AuxServices.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fifo/TestFifoScheduler.java
- hadoop-yarn-project/CHANGES.txt
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairSchedulerTestBase.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/yarn_protos.proto
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerApplicationAttempt.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/server/api/ContainerTerminationContext.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/security/ContainerTokenIdentifier.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/security/RMContainerTokenSecretManager.java
FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #241 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/241/)
YARN-3116. RM notifies NM whether a container is an AM container or normal task container. Contributed by Giovanni Matteo Fumarola. (zjshen: rev 1ea36299a47af302379ae0750b571ec021eb54ad)
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairSchedulerTestBase.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fifo/TestFifoScheduler.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/server/api/ContainerTerminationContext.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/yarn_protos.proto
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/ProtoUtils.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/server/api/ContainerType.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/Application.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/server/api/ContainerContext.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/security/ContainerTokenIdentifier.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/security/TestYARNTokenIdentifier.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestContainerAllocation.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/AuxServices.java
- hadoop-yarn-project/CHANGES.txt
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/server/api/ContainerInitializationContext.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/security/RMContainerTokenSecretManager.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/proto/server/yarn_security_token.proto
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerApplicationAttempt.java
SUCCESS: Integrated in Hadoop-Mapreduce-trunk #2199 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2199/)
YARN-3116. RM notifies NM whether a container is an AM container or normal task container. Contributed by Giovanni Matteo Fumarola. (zjshen: rev 1ea36299a47af302379ae0750b571ec021eb54ad)
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fifo/TestFifoScheduler.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/yarn_protos.proto
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/AuxServices.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairSchedulerTestBase.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/ProtoUtils.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/server/api/ContainerInitializationContext.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/server/api/ContainerTerminationContext.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/server/api/ContainerType.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestContainerAllocation.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerApplicationAttempt.java
- hadoop-yarn-project/CHANGES.txt
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/proto/server/yarn_security_token.proto
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/security/RMContainerTokenSecretManager.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/server/api/ContainerContext.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/security/ContainerTokenIdentifier.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/security/TestYARNTokenIdentifier.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/Application.java
I think we can consider to add an optional field in ContainerLaunchContext, so AMLauncher inside of RM can set this value different from launching normal containers. It then get passed to ContainerImpl, so ContainerInitializationContext in AuxiliaryService can aware it. Thoughts