Details
-
Type:
Bug
-
Status: Resolved
-
Priority:
Major
-
Resolution: Fixed
-
Affects Version/s: None
-
Fix Version/s: 2.9.0, 3.0.0-alpha1
-
Component/s: None
-
Labels:None
-
Target Version/s:
Activity
- All
- Comments
- Work Log
- History
- Activity
- Transitions
Thanks for the patch Jian He. Few things -
- Do you think we should add a duration metric? For longer running containers, it would be a useful metric. You'll have to use the MonotonicClock for that though.
long unrgisterDelay = conf.getLong( + YarnConfiguration.NM_CONTAINER_METRICS_UNREGISTER_DELAY_MS, + YarnConfiguration.DEFAULT_NM_CONTAINER_METRICS_UNREGISTER_DELAY_MS);Typo in the spelling of unregisterDelay
- The start time won't be preserved during a work preserving NM restart - this is also true for the other statistical metrics like the histograms, and the advanced actual resource usage metrics. I'm open to fixing this in a follow up JIRA.
Varun Vasudev, thanks for your review !
I think the duration metrics may be derived from finish-start time by the consumer.
Fixed the typo.
 -1 overall |
| Vote | Subsystem | Runtime | Comment |
|---|---|---|---|
| 0 | reexec | 0m 15s | Docker mode activated. |
| +1 | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. |
| +1 | mvninstall | 6m 30s | trunk passed |
| +1 | compile | 0m 22s | trunk passed with JDK v1.8.0_74 |
| +1 | compile | 0m 26s | trunk passed with JDK v1.7.0_95 |
| +1 | checkstyle | 0m 18s | trunk passed |
| +1 | mvnsite | 0m 27s | trunk passed |
| +1 | mvneclipse | 0m 13s | trunk passed |
| +1 | findbugs | 0m 51s | trunk passed |
| +1 | javadoc | 0m 17s | trunk passed with JDK v1.8.0_74 |
| +1 | javadoc | 0m 22s | trunk passed with JDK v1.7.0_95 |
| +1 | mvninstall | 0m 24s | the patch passed |
| +1 | compile | 0m 20s | the patch passed with JDK v1.8.0_74 |
| +1 | javac | 0m 20s | the patch passed |
| +1 | compile | 0m 24s | the patch passed with JDK v1.7.0_95 |
| +1 | javac | 0m 24s | the patch passed |
| -1 | checkstyle | 0m 15s | hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager: patch generated 9 new + 111 unchanged - 0 fixed = 120 total (was 111) |
| +1 | mvnsite | 0m 25s | the patch passed |
| +1 | mvneclipse | 0m 11s | the patch passed |
| +1 | whitespace | 0m 0s | Patch has no whitespace issues. |
| +1 | findbugs | 0m 59s | the patch passed |
| +1 | javadoc | 0m 15s | the patch passed with JDK v1.8.0_74 |
| +1 | javadoc | 0m 19s | the patch passed with JDK v1.7.0_95 |
| -1 | unit | 8m 59s | hadoop-yarn-server-nodemanager in the patch failed with JDK v1.8.0_74. |
| -1 | unit | 9m 32s | hadoop-yarn-server-nodemanager in the patch failed with JDK v1.7.0_95. |
| +1 | asflicense | 0m 17s | Patch does not generate ASF License warnings. |
| 33m 18s |
| Reason | Tests |
|---|---|
| JDK v1.8.0_74 Failed junit tests | hadoop.yarn.server.nodemanager.containermanager.TestAuxServices |
| JDK v1.7.0_95 Failed junit tests | hadoop.yarn.server.nodemanager.containermanager.TestAuxServices |
This message was automatically generated.
| -1 overall |
| Vote | Subsystem | Runtime | Comment |
|---|---|---|---|
| 0 | reexec | 0m 10s | Docker mode activated. |
| +1 | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. |
| +1 | mvninstall | 7m 49s | trunk passed |
| +1 | compile | 0m 28s | trunk passed with JDK v1.8.0_77 |
| +1 | compile | 0m 29s | trunk passed with JDK v1.7.0_95 |
| +1 | checkstyle | 0m 17s | trunk passed |
| +1 | mvnsite | 0m 31s | trunk passed |
| +1 | mvneclipse | 0m 13s | trunk passed |
| +1 | findbugs | 0m 52s | trunk passed |
| +1 | javadoc | 0m 20s | trunk passed with JDK v1.8.0_77 |
| +1 | javadoc | 0m 21s | trunk passed with JDK v1.7.0_95 |
| +1 | mvninstall | 0m 25s | the patch passed |
| +1 | compile | 0m 21s | the patch passed with JDK v1.8.0_77 |
| +1 | javac | 0m 21s | the patch passed |
| +1 | compile | 0m 25s | the patch passed with JDK v1.7.0_95 |
| +1 | javac | 0m 25s | the patch passed |
| -1 | checkstyle | 0m 16s | hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager: patch generated 9 new + 111 unchanged - 0 fixed = 120 total (was 111) |
| +1 | mvnsite | 0m 32s | the patch passed |
| +1 | mvneclipse | 0m 12s | the patch passed |
| +1 | whitespace | 0m 0s | Patch has no whitespace issues. |
| +1 | findbugs | 1m 9s | the patch passed |
| +1 | javadoc | 0m 19s | the patch passed with JDK v1.8.0_77 |
| +1 | javadoc | 0m 19s | the patch passed with JDK v1.7.0_95 |
| -1 | unit | 9m 24s | hadoop-yarn-server-nodemanager in the patch failed with JDK v1.8.0_77. |
| -1 | unit | 9m 48s | hadoop-yarn-server-nodemanager in the patch failed with JDK v1.7.0_95. |
| +1 | asflicense | 0m 19s | Patch does not generate ASF License warnings. |
| 35m 59s |
| Reason | Tests |
|---|---|
| JDK v1.8.0_77 Failed junit tests | hadoop.yarn.server.nodemanager.containermanager.TestNMProxy |
| hadoop.yarn.server.nodemanager.containermanager.TestAuxServices | |
| JDK v1.7.0_95 Failed junit tests | hadoop.yarn.server.nodemanager.containermanager.TestNMProxy |
| hadoop.yarn.server.nodemanager.containermanager.TestAuxServices |
This message was automatically generated.
Fixed TestAuxServices,
TestNMProxy fails on existing code, opened YARN-4922 to track
| -1 overall |
| Vote | Subsystem | Runtime | Comment |
|---|---|---|---|
| 0 | reexec | 0m 12s | Docker mode activated. |
| +1 | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 | test4tests | 0m 0s | The patch appears to include 2 new or modified test files. |
| +1 | mvninstall | 6m 57s | trunk passed |
| +1 | compile | 0m 23s | trunk passed with JDK v1.8.0_77 |
| +1 | compile | 0m 26s | trunk passed with JDK v1.7.0_95 |
| +1 | checkstyle | 0m 18s | trunk passed |
| +1 | mvnsite | 0m 28s | trunk passed |
| +1 | mvneclipse | 0m 13s | trunk passed |
| +1 | findbugs | 0m 50s | trunk passed |
| +1 | javadoc | 0m 17s | trunk passed with JDK v1.8.0_77 |
| +1 | javadoc | 0m 20s | trunk passed with JDK v1.7.0_95 |
| +1 | mvninstall | 0m 24s | the patch passed |
| +1 | compile | 0m 20s | the patch passed with JDK v1.8.0_77 |
| +1 | javac | 0m 20s | the patch passed |
| +1 | compile | 0m 23s | the patch passed with JDK v1.7.0_95 |
| +1 | javac | 0m 23s | the patch passed |
| -1 | checkstyle | 0m 15s | hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager: patch generated 6 new + 111 unchanged - 0 fixed = 117 total (was 111) |
| +1 | mvnsite | 0m 25s | the patch passed |
| +1 | mvneclipse | 0m 10s | the patch passed |
| +1 | whitespace | 0m 0s | Patch has no whitespace issues. |
| +1 | findbugs | 1m 1s | the patch passed |
| +1 | javadoc | 0m 14s | the patch passed with JDK v1.8.0_77 |
| +1 | javadoc | 0m 19s | the patch passed with JDK v1.7.0_95 |
| -1 | unit | 8m 58s | hadoop-yarn-server-nodemanager in the patch failed with JDK v1.8.0_77. |
| -1 | unit | 9m 31s | hadoop-yarn-server-nodemanager in the patch failed with JDK v1.7.0_95. |
| +1 | asflicense | 0m 19s | Patch does not generate ASF License warnings. |
| 33m 40s |
| Reason | Tests |
|---|---|
| JDK v1.8.0_77 Failed junit tests | hadoop.yarn.server.nodemanager.containermanager.TestNMProxy |
| JDK v1.7.0_95 Failed junit tests | hadoop.yarn.server.nodemanager.containermanager.TestNMProxy |
This message was automatically generated.
Latest patch looks good to me. +1.
I'll commit it tomorrow if no one objects.
Committed to trunk and branch-2. Thanks Jian He!
FAILURE: Integrated in Hadoop-trunk-Commit #9566 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9566/)
YARN-4906. Capture container start/finish time in container metrics. (vvasudev: rev b41e65e5bc9459b4d950a2c53860a223f1a0d2ec)
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/container/TestContainer.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/container/ContainerImpl.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainerMetrics.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/TestAuxServices.java
Uploaded a patch which adds start/finish time, exit code into container metrics.
It is done outside of ContainerMonitor, because I think these kind of metrics need not to be dependent on whether resource-calculator plugin or vmem/pmem flag is enabled or not.