Details
-
Sub-task
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
None
-
None
-
None
Description
Currently only the ContainerState is sent to the RM in the NMContainerStatus. This lets the restarted RM know if the container is queued or not, but It wont know the ExecutionType. ExecutionType updates (Container promotions) cannot also happen unless the RM knows about opportunistic / QUEUED containers.
Attachments
Attachments
- YARN-6849.001.patch
- 23 kB
- kartheek muthyala
- YARN-6849.002.patch
- 25 kB
- kartheek muthyala
Activity
-1 overall |
Vote | Subsystem | Runtime | Comment |
---|---|---|---|
0 | reexec | 0m 16s | Docker mode activated. |
Prechecks | |||
+1 | @author | 0m 0s | The patch does not contain any @author tags. |
+1 | test4tests | 0m 0s | The patch appears to include 2 new or modified test files. |
trunk Compile Tests | |||
0 | mvndep | 0m 9s | Maven dependency ordering for branch |
+1 | mvninstall | 14m 10s | trunk passed |
+1 | compile | 2m 9s | trunk passed |
+1 | checkstyle | 0m 44s | trunk passed |
+1 | mvnsite | 1m 34s | trunk passed |
-1 | findbugs | 0m 45s | hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager in trunk has 1 extant Findbugs warnings. |
+1 | javadoc | 0m 59s | trunk passed |
Patch Compile Tests | |||
0 | mvndep | 0m 9s | Maven dependency ordering for patch |
+1 | mvninstall | 1m 26s | the patch passed |
+1 | compile | 2m 6s | the patch passed |
+1 | cc | 2m 6s | the patch passed |
+1 | javac | 2m 6s | the patch passed |
-0 | checkstyle | 0m 47s | hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server: The patch generated 4 new + 423 unchanged - 0 fixed = 427 total (was 423) |
+1 | mvnsite | 1m 30s | the patch passed |
+1 | whitespace | 0m 0s | The patch has no whitespace issues. |
+1 | findbugs | 3m 4s | the patch passed |
+1 | javadoc | 0m 56s | the patch passed |
Other Tests | |||
+1 | unit | 1m 38s | hadoop-yarn-server-common in the patch passed. |
+1 | unit | 13m 32s | hadoop-yarn-server-nodemanager in the patch passed. |
-1 | unit | 58m 50s | hadoop-yarn-server-resourcemanager in the patch failed. |
+1 | asflicense | 0m 16s | The patch does not generate ASF License warnings. |
112m 31s |
Reason | Tests |
---|---|
Failed junit tests | hadoop.yarn.server.resourcemanager.scheduler.capacity.TestIncreaseAllocationExpirer |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerAllocation | |
hadoop.yarn.server.resourcemanager.scheduler.TestAbstractYarnScheduler | |
Timed out junit tests | org.apache.hadoop.yarn.server.resourcemanager.reservation.TestFairSchedulerPlanFollower |
Subsystem | Report/Notes |
---|---|
Docker | Image:yetus/hadoop:71bbb86 |
JIRA Issue | |
JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12885797/YARN-6849.001.patch |
Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc |
uname | Linux 411e6f58c439 3.13.0-119-generic #166-Ubuntu SMP Wed May 3 12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
Build tool | maven |
Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh |
git revision | trunk / 6f101e7 |
Default Java | 1.8.0_144 |
findbugs | v3.1.0-RC1 |
findbugs | https://builds.apache.org/job/PreCommit-YARN-Build/17325/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-warnings.html |
checkstyle | https://builds.apache.org/job/PreCommit-YARN-Build/17325/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server.txt |
unit | https://builds.apache.org/job/PreCommit-YARN-Build/17325/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt |
Test Results | https://builds.apache.org/job/PreCommit-YARN-Build/17325/testReport/ |
modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server |
Console output | https://builds.apache.org/job/PreCommit-YARN-Build/17325/console |
Powered by | Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
Thanks for the patch kartheek
Can we also add in the testcase a Guaranteed container.
Also we need to verify that resources are not incremented for queue and node for the opportunistic containers.
Thank you asuresh for the feedback. Have added Guaranteed containers into the testcase, and validated that the consumption is infact only by Guaranteed but not Opportunistic.
-1 overall |
Vote | Subsystem | Runtime | Comment |
---|---|---|---|
0 | reexec | 0m 30s | Docker mode activated. |
Prechecks | |||
+1 | @author | 0m 0s | The patch does not contain any @author tags. |
+1 | test4tests | 0m 0s | The patch appears to include 2 new or modified test files. |
trunk Compile Tests | |||
0 | mvndep | 0m 9s | Maven dependency ordering for branch |
+1 | mvninstall | 14m 51s | trunk passed |
+1 | compile | 2m 16s | trunk passed |
+1 | checkstyle | 0m 45s | trunk passed |
+1 | mvnsite | 1m 35s | trunk passed |
-1 | findbugs | 0m 47s | hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager in trunk has 1 extant Findbugs warnings. |
+1 | javadoc | 1m 2s | trunk passed |
Patch Compile Tests | |||
0 | mvndep | 0m 10s | Maven dependency ordering for patch |
+1 | mvninstall | 1m 26s | the patch passed |
+1 | compile | 2m 12s | the patch passed |
+1 | cc | 2m 12s | the patch passed |
+1 | javac | 2m 12s | the patch passed |
+1 | checkstyle | 0m 43s | the patch passed |
+1 | mvnsite | 1m 30s | the patch passed |
+1 | whitespace | 0m 0s | The patch has no whitespace issues. |
+1 | findbugs | 3m 2s | the patch passed |
+1 | javadoc | 0m 57s | the patch passed |
Other Tests | |||
+1 | unit | 1m 45s | hadoop-yarn-server-common in the patch passed. |
+1 | unit | 14m 36s | hadoop-yarn-server-nodemanager in the patch passed. |
-1 | unit | 47m 20s | hadoop-yarn-server-resourcemanager in the patch failed. |
+1 | asflicense | 0m 18s | The patch does not generate ASF License warnings. |
103m 31s |
Reason | Tests |
---|---|
Failed junit tests | hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerAllocation |
Timed out junit tests | org.apache.hadoop.yarn.server.resourcemanager.TestSubmitApplicationWithRMHA |
org.apache.hadoop.yarn.server.resourcemanager.TestKillApplicationWithRMHA |
Subsystem | Report/Notes |
---|---|
Docker | Image:yetus/hadoop:71bbb86 |
JIRA Issue | |
JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12886082/YARN-6849.002.patch |
Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc |
uname | Linux da9207f513c8 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
Build tool | maven |
Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh |
git revision | trunk / 5bbca80 |
Default Java | 1.8.0_144 |
findbugs | v3.1.0-RC1 |
findbugs | https://builds.apache.org/job/PreCommit-YARN-Build/17362/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-warnings.html |
unit | https://builds.apache.org/job/PreCommit-YARN-Build/17362/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt |
Test Results | https://builds.apache.org/job/PreCommit-YARN-Build/17362/testReport/ |
modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server |
Console output | https://builds.apache.org/job/PreCommit-YARN-Build/17362/console |
Powered by | Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #12824 (See https://builds.apache.org/job/Hadoop-trunk-Commit/12824/)
YARN-6849. NMContainerStatus should have the Container ExecutionType. (arun suresh: rev 1f53ae79728065417c6a99eb6fcc8d3a080ab4cc)
- (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmnode/RMNodeImpl.java
- (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/protocolrecords/NMContainerStatus.java
- (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerApplicationAttempt.java
- (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/ParentQueue.java
- (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/proto/yarn_server_common_service_protos.proto
- (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestRMRestart.java
- (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/protocolrecords/impl/pb/NMContainerStatusPBImpl.java
- (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/AbstractYarnScheduler.java
- (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestResourceTrackerService.java
- (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java
- (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/AppSchedulingInfo.java
- (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/container/ContainerImpl.java
Submitting the first version of the patch