Details
-
Improvement
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
None
-
Reviewed
Description
add localization and container launch time to ContainerMetrics at NM to show these timing information for each active container.
Currently ContainerMetrics has container's actual memory usage(YARN-2984), actual CPU usage(YARN-3122), resource and pid(YARN-3022). It will be better to have localization and container launch time in ContainerMetrics for each active container.
Attachments
Attachments
- YARN-3363.001.patch
- 11 kB
- Zhihai Xu
- YARN-3363.000.patch
- 13 kB
- Zhihai Xu
Issue Links
Activity
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12706461/YARN-3363.000.patch
against trunk revision 0b9f12c.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 1 new or modified test files.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 javadoc. There were no new javadoc warning messages.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed these unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager:
org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.TestLogAggregationService
Test results: https://builds.apache.org/job/PreCommit-YARN-Build/7070//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/7070//console
This message is automatically generated.
The duration do not seem to belong in a ProcessTree class.
Can we instead of flowing the durations through the ProcessTree class, add the metrics directly in the ContainersMonitorImpl#handle by reading the startEvent.getLaunchDuration() directly?
Nit: ContainerMetrics:recordTime maybe could rename to recordStateChangeDurations or something to that effect?
sendContainerMonitorStartEvent has two different ways duration were calculated. Maybe make 2 local variables for launchDuration and localizationDuration and then we do not need the comment.
Hi adhoot, thanks for the thorough review, Your suggestions are reasonable. I uploaded a new patch YARN-3363.001.patch, which addressed all your comments. Please review it. thanks
-1 overall |
Vote | Subsystem | Runtime | Comment |
---|---|---|---|
0 | pre-patch | 14m 39s | Pre-patch trunk compilation is healthy. |
+1 | @author | 0m 0s | The patch does not contain any @author tags. |
+1 | tests included | 0m 0s | The patch appears to include 1 new or modified test files. |
+1 | whitespace | 0m 0s | The patch has no lines that end in whitespace. |
+1 | javac | 7m 30s | There were no new javac warning messages. |
+1 | javadoc | 9m 32s | There were no new javadoc warning messages. |
+1 | release audit | 0m 23s | The applied patch does not increase the total number of release audit warnings. |
-1 | checkstyle | 5m 26s | The applied patch generated 3 additional checkstyle issues. |
+1 | install | 1m 33s | mvn install still works. |
+1 | eclipse:eclipse | 0m 32s | The patch built with eclipse:eclipse. |
+1 | findbugs | 1m 3s | The patch does not introduce any new Findbugs (version 2.0.3) warnings. |
+1 | yarn tests | 6m 0s | Tests passed in hadoop-yarn-server-nodemanager. |
46m 41s |
Subsystem | Report/Notes |
---|---|
Patch URL | http://issues.apache.org/jira/secure/attachment/12727688/YARN-3363.001.patch |
Optional Tests | javadoc javac unit findbugs checkstyle |
git revision | trunk / 416b843 |
checkstyle | https://builds.apache.org/job/PreCommit-YARN-Build/7480/artifact/patchprocess/checkstyle-result-diff.txt |
hadoop-yarn-server-nodemanager test log | https://builds.apache.org/job/PreCommit-YARN-Build/7480/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt |
Test Results | https://builds.apache.org/job/PreCommit-YARN-Build/7480/testReport/ |
Console output | https://builds.apache.org/job/PreCommit-YARN-Build/7480/console |
This message was automatically generated.
-1 overall |
Vote | Subsystem | Runtime | Comment |
---|---|---|---|
0 | pre-patch | 14m 38s | Pre-patch trunk compilation is healthy. |
+1 | @author | 0m 0s | The patch does not contain any @author tags. |
+1 | tests included | 0m 0s | The patch appears to include 1 new or modified test files. |
+1 | whitespace | 0m 0s | The patch has no lines that end in whitespace. |
+1 | javac | 7m 32s | There were no new javac warning messages. |
+1 | javadoc | 9m 37s | There were no new javadoc warning messages. |
+1 | release audit | 0m 23s | The applied patch does not increase the total number of release audit warnings. |
-1 | checkstyle | 5m 22s | The applied patch generated 3 additional checkstyle issues. |
+1 | install | 1m 32s | mvn install still works. |
+1 | eclipse:eclipse | 0m 33s | The patch built with eclipse:eclipse. |
+1 | findbugs | 1m 1s | The patch does not introduce any new Findbugs (version 2.0.3) warnings. |
+1 | yarn tests | 5m 49s | Tests passed in hadoop-yarn-server-nodemanager. |
46m 30s |
Subsystem | Report/Notes |
---|---|
Patch URL | http://issues.apache.org/jira/secure/attachment/12728288/YARN-3363.001.patch |
Optional Tests | javadoc javac unit findbugs checkstyle |
git revision | trunk / 1a2459b |
checkstyle | https://builds.apache.org/job/PreCommit-YARN-Build/7506/artifact/patchprocess/checkstyle-result-diff.txt |
hadoop-yarn-server-nodemanager test log | https://builds.apache.org/job/PreCommit-YARN-Build/7506/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt |
Test Results | https://builds.apache.org/job/PreCommit-YARN-Build/7506/testReport/ |
Java | 1.7.0_55 |
uname | Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
Console output | https://builds.apache.org/job/PreCommit-YARN-Build/7506/console |
This message was automatically generated.
It looks like the checkstyle issues are not related to my change.
I checked the code at the lines shown in the following logs: I didn't see any checkstyle issues,
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainerStartMonitoringEvent.java 19 29 hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainerMetrics.java 82 92 hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/container/ContainerImpl.java 221 223
So it looks like there may be some problems in the checkstyle script.
SUCCESS: Integrated in Hadoop-trunk-Commit #7716 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7716/)
YARN-3363. add localization and container launch time to ContainerMetrics at NM to show these timing information for each active container. (zxu via rkanter) (rkanter: rev ac7d152901e29b1f444507fe4e421eb6e1402b5a)
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainersMonitorImpl.java
- hadoop-yarn-project/CHANGES.txt
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/container/ContainerImpl.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainerStartMonitoringEvent.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainerMetrics.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/TestContainerMetrics.java
FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #181 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/181/)
YARN-3363. add localization and container launch time to ContainerMetrics at NM to show these timing information for each active container. (zxu via rkanter) (rkanter: rev ac7d152901e29b1f444507fe4e421eb6e1402b5a)
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/TestContainerMetrics.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/container/ContainerImpl.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainersMonitorImpl.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainerStartMonitoringEvent.java
- hadoop-yarn-project/CHANGES.txt
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainerMetrics.java
FAILURE: Integrated in Hadoop-Yarn-trunk #915 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/915/)
YARN-3363. add localization and container launch time to ContainerMetrics at NM to show these timing information for each active container. (zxu via rkanter) (rkanter: rev ac7d152901e29b1f444507fe4e421eb6e1402b5a)
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainersMonitorImpl.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/TestContainerMetrics.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainerStartMonitoringEvent.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/container/ContainerImpl.java
- hadoop-yarn-project/CHANGES.txt
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainerMetrics.java
FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #172 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/172/)
YARN-3363. add localization and container launch time to ContainerMetrics at NM to show these timing information for each active container. (zxu via rkanter) (rkanter: rev ac7d152901e29b1f444507fe4e421eb6e1402b5a)
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/TestContainerMetrics.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainersMonitorImpl.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/container/ContainerImpl.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainerStartMonitoringEvent.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainerMetrics.java
- hadoop-yarn-project/CHANGES.txt
SUCCESS: Integrated in Hadoop-Hdfs-trunk #2113 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2113/)
YARN-3363. add localization and container launch time to ContainerMetrics at NM to show these timing information for each active container. (zxu via rkanter) (rkanter: rev ac7d152901e29b1f444507fe4e421eb6e1402b5a)
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainerMetrics.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/TestContainerMetrics.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainerStartMonitoringEvent.java
- hadoop-yarn-project/CHANGES.txt
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/container/ContainerImpl.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainersMonitorImpl.java
FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #182 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/182/)
YARN-3363. add localization and container launch time to ContainerMetrics at NM to show these timing information for each active container. (zxu via rkanter) (rkanter: rev ac7d152901e29b1f444507fe4e421eb6e1402b5a)
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainersMonitorImpl.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainerStartMonitoringEvent.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainerMetrics.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/TestContainerMetrics.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/container/ContainerImpl.java
- hadoop-yarn-project/CHANGES.txt
SUCCESS: Integrated in Hadoop-Mapreduce-trunk #2131 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2131/)
YARN-3363. add localization and container launch time to ContainerMetrics at NM to show these timing information for each active container. (zxu via rkanter) (rkanter: rev ac7d152901e29b1f444507fe4e421eb6e1402b5a)
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/container/ContainerImpl.java
- hadoop-yarn-project/CHANGES.txt
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainerMetrics.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainerStartMonitoringEvent.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainersMonitorImpl.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/TestContainerMetrics.java
I uploaded a new patch
YARN-3363.000.patch for review.