Details
-
Type:
Improvement
-
Status: Resolved
-
Priority:
Major
-
Resolution: Fixed
-
Affects Version/s: 2.8.0
-
Fix Version/s: 2.8.0, 3.0.0-alpha1
-
Component/s: nodemanager, resourcemanager
-
Labels:None
-
Target Version/s:
-
Hadoop Flags:Reviewed
Description
We should add the JvmPauseMonitor from HADOOP-9618 to the ResourceManager and NodeManager.
Issue Links
- is related to
-
HADOOP-9618
Add thread which detects JVM pauses
-
- Closed
-
-
MAPREDUCE-6443
Add JvmPauseMonitor to Job History Server
-
- Resolved
-
- relates to
-
HADOOP-12313
NPE in JvmPauseMonitor when calling stop() before start()
-
- Resolved
-
-
YARN-4031
Add JvmPauseMonitor to ApplicationHistoryServer and WebAppProxyServer
-
- Resolved
-
Activity
- All
- Comments
- Work Log
- History
- Activity
- Transitions
 -1 overall |
| Vote | Subsystem | Runtime | Comment |
|---|---|---|---|
| 0 | pre-patch | 18m 8s | Pre-patch trunk compilation is healthy. |
| +1 | @author | 0m 0s | The patch does not contain any @author tags. |
| -1 | tests included | 0m 0s | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. |
| +1 | javac | 7m 57s | There were no new javac warning messages. |
| +1 | javadoc | 10m 27s | There were no new javadoc warning messages. |
| +1 | release audit | 0m 27s | The applied patch does not increase the total number of release audit warnings. |
| -1 | checkstyle | 1m 6s | The applied patch generated 1 new checkstyle issues (total was 49, now 50). |
| -1 | checkstyle | 1m 35s | The applied patch generated 1 new checkstyle issues (total was 50, now 50). |
| -1 | whitespace | 0m 0s | The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix. |
| +1 | install | 1m 24s | mvn install still works. |
| +1 | eclipse:eclipse | 0m 34s | The patch built with eclipse:eclipse. |
| +1 | findbugs | 2m 45s | The patch does not introduce any new Findbugs (version 3.0.0) warnings. |
| -1 | yarn tests | 6m 13s | Tests failed in hadoop-yarn-server-nodemanager. |
| +1 | yarn tests | 53m 12s | Tests passed in hadoop-yarn-server-resourcemanager. |
| 102m 45s |
| Reason | Tests |
|---|---|
| Failed unit tests | hadoop.yarn.server.nodemanager.TestDeletionService |
This message was automatically generated.
The 002 patch fixes the NM checkstyle warning and the trailing whitespace. The RM checkstyle warning is due to a method that is "too long", but it's the serviceInit method, which is setting things up, so I think we can ignore that.
I verified in a cluster that the metrics and log messages that come from the JvmPauseMonitor show up.
Test failure also appears to be unrelated.
| -1 overall |
| Vote | Subsystem | Runtime | Comment |
|---|---|---|---|
| 0 | pre-patch | 21m 7s | Pre-patch trunk compilation is healthy. |
| +1 | @author | 0m 0s | The patch does not contain any @author tags. |
| -1 | tests included | 0m 0s | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. |
| +1 | javac | 9m 37s | There were no new javac warning messages. |
| +1 | javadoc | 11m 11s | There were no new javadoc warning messages. |
| +1 | release audit | 0m 24s | The applied patch does not increase the total number of release audit warnings. |
| -1 | checkstyle | 1m 49s | The applied patch generated 1 new checkstyle issues (total was 50, now 50). |
| +1 | whitespace | 0m 0s | The patch has no lines that end in whitespace. |
| +1 | install | 1m 48s | mvn install still works. |
| +1 | eclipse:eclipse | 0m 43s | The patch built with eclipse:eclipse. |
| +1 | findbugs | 2m 58s | The patch does not introduce any new Findbugs (version 3.0.0) warnings. |
| -1 | yarn tests | 7m 8s | Tests failed in hadoop-yarn-server-nodemanager. |
| -1 | yarn tests | 53m 52s | Tests failed in hadoop-yarn-server-resourcemanager. |
| 110m 42s |
| Reason | Tests |
|---|---|
| Failed unit tests | hadoop.yarn.server.nodemanager.containermanager.logaggregation.TestLogAggregationService |
| Timed out tests | org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesApps |
| Subsystem | Report/Notes |
|---|---|
| Patch URL | http://issues.apache.org/jira/secure/attachment/12748757/YARN-4019.002.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / d540374 |
| checkstyle | https://builds.apache.org/job/PreCommit-YARN-Build/8768/artifact/patchprocess/diffcheckstylehadoop-yarn-server-resourcemanager.txt |
| hadoop-yarn-server-nodemanager test log | https://builds.apache.org/job/PreCommit-YARN-Build/8768/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt |
| hadoop-yarn-server-resourcemanager test log | https://builds.apache.org/job/PreCommit-YARN-Build/8768/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt |
| Test Results | https://builds.apache.org/job/PreCommit-YARN-Build/8768/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output | https://builds.apache.org/job/PreCommit-YARN-Build/8768/console |
This message was automatically generated.
Hi Robert Kanter, thanks for contributing the patch! The Patch LGTM in overall. A small comment is: can we move below line from serviceInit() to serviceStart()?
+ pauseMonitor.start();
Other looks fine to me. I think the test failure is not related to your patch. Isn't it?
Sure. The 003 patch moves the start lines. The test failure was unrelated.
| -1 overall |
| Vote | Subsystem | Runtime | Comment |
|---|---|---|---|
| -1 | pre-patch | 16m 25s | Findbugs (version ) appears to be broken on trunk. |
| +1 | @author | 0m 1s | The patch does not contain any @author tags. |
| -1 | tests included | 0m 0s | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. |
| +1 | javac | 8m 1s | There were no new javac warning messages. |
| +1 | javadoc | 10m 10s | There were no new javadoc warning messages. |
| +1 | release audit | 0m 24s | The applied patch does not increase the total number of release audit warnings. |
| +1 | checkstyle | 0m 44s | There were no new checkstyle issues. |
| +1 | whitespace | 0m 0s | The patch has no lines that end in whitespace. |
| +1 | install | 1m 26s | mvn install still works. |
| +1 | eclipse:eclipse | 0m 34s | The patch built with eclipse:eclipse. |
| +1 | findbugs | 2m 52s | The patch does not introduce any new Findbugs (version 3.0.0) warnings. |
| -1 | yarn tests | 5m 51s | Tests failed in hadoop-yarn-server-nodemanager. |
| -1 | yarn tests | 51m 53s | Tests failed in hadoop-yarn-server-resourcemanager. |
| 98m 27s |
| Reason | Tests |
|---|---|
| Failed unit tests | hadoop.yarn.server.nodemanager.TestNodeManagerReboot |
| hadoop.yarn.server.nodemanager.TestDeletionService | |
| hadoop.yarn.server.nodemanager.TestNodeManagerResync | |
| hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacitySchedulerQueueACLs | |
| hadoop.yarn.server.resourcemanager.scheduler.capacity.TestNodeLabelContainerAllocation | |
| hadoop.yarn.server.resourcemanager.TestRMAdminService |
| Subsystem | Report/Notes |
|---|---|
| Patch URL | http://issues.apache.org/jira/secure/attachment/12748898/YARN-4019.003.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 4ab49a4 |
| hadoop-yarn-server-nodemanager test log | https://builds.apache.org/job/PreCommit-YARN-Build/8774/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt |
| hadoop-yarn-server-resourcemanager test log | https://builds.apache.org/job/PreCommit-YARN-Build/8774/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt |
| Test Results | https://builds.apache.org/job/PreCommit-YARN-Build/8774/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output | https://builds.apache.org/job/PreCommit-YARN-Build/8774/console |
This message was automatically generated.
+1. 03 patch LGTM. Committing it in.
I have commit this patch to trunk and branch-2. Thanks Robert Kanter for the contribution!
FAILURE: Integrated in Hadoop-trunk-Commit #8272 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8272/)
YARN-4019. Add JvmPauseMonitor to ResourceManager and NodeManager. Contributed by Robert Kanter. (junping_du: rev cfee02b3bdd1117370200c9d8ce216676cff8888)
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceManager.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/metrics/NodeManagerMetrics.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeManager.java
- hadoop-yarn-project/CHANGES.txt
Thanks for the review Junping Du
SUCCESS: Integrated in Hadoop-Mapreduce-trunk-Java8 #276 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/276/)
YARN-4019. Add JvmPauseMonitor to ResourceManager and NodeManager. Contributed by Robert Kanter. (junping_du: rev cfee02b3bdd1117370200c9d8ce216676cff8888)
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeManager.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceManager.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/metrics/NodeManagerMetrics.java
- hadoop-yarn-project/CHANGES.txt
FAILURE: Integrated in Hadoop-Mapreduce-trunk #2225 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2225/)
YARN-4019. Add JvmPauseMonitor to ResourceManager and NodeManager. Contributed by Robert Kanter. (junping_du: rev cfee02b3bdd1117370200c9d8ce216676cff8888)
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/metrics/NodeManagerMetrics.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceManager.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeManager.java
- hadoop-yarn-project/CHANGES.txt
FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #280 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/280/)
YARN-4019. Add JvmPauseMonitor to ResourceManager and NodeManager. Contributed by Robert Kanter. (junping_du: rev cfee02b3bdd1117370200c9d8ce216676cff8888)
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceManager.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/metrics/NodeManagerMetrics.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeManager.java
- hadoop-yarn-project/CHANGES.txt
FAILURE: Integrated in Hadoop-Yarn-trunk #1010 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1010/)
YARN-4019. Add JvmPauseMonitor to ResourceManager and NodeManager. Contributed by Robert Kanter. (junping_du: rev cfee02b3bdd1117370200c9d8ce216676cff8888)
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceManager.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeManager.java
- hadoop-yarn-project/CHANGES.txt
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/metrics/NodeManagerMetrics.java
FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #269 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/269/)
YARN-4019. Add JvmPauseMonitor to ResourceManager and NodeManager. Contributed by Robert Kanter. (junping_du: rev cfee02b3bdd1117370200c9d8ce216676cff8888)
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/metrics/NodeManagerMetrics.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeManager.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceManager.java
- hadoop-yarn-project/CHANGES.txt
FAILURE: Integrated in Hadoop-Hdfs-trunk #2207 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2207/)
YARN-4019. Add JvmPauseMonitor to ResourceManager and NodeManager. Contributed by Robert Kanter. (junping_du: rev cfee02b3bdd1117370200c9d8ce216676cff8888)
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeManager.java
- hadoop-yarn-project/CHANGES.txt
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceManager.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/metrics/NodeManagerMetrics.java
Seems like other YARN daemons (like ATS) are also need this, filed YARN-4031 with a quick patch.
I uploaded a proposed patch to correct the hadoop.yarn.server.resourcemanager.TestRMAdminService test failure to YARN-4035.
Most of the code was already there; it was just not hooked up. The patch adds the JvmPauseMonitor to the existing JvmMetrics.