Details
-
Type:
Bug
-
Status: Closed
-
Priority:
Major
-
Resolution: Fixed
-
Affects Version/s: 2.6.0
-
Fix Version/s: 2.8.0, 2.7.3, 2.6.5, 3.0.0-alpha1
-
Component/s: None
-
Labels:None
-
Target Version/s:
-
Hadoop Flags:Reviewed
Description
A fairly common occurrence in RM logs is a string of redundant event-queue logs like the following which does little except bloat the logs:
2016-02-23 08:00:00,948 [IPC Server handler 36 on 8030] INFO event.AsyncDispatcher: Size of event-queue is 1000 2016-02-23 08:00:00,948 [IPC Server handler 36 on 8030] INFO event.AsyncDispatcher: Size of event-queue is 1000 2016-02-23 08:00:00,948 [IPC Server handler 36 on 8030] INFO event.AsyncDispatcher: Size of event-queue is 1000 2016-02-23 08:00:00,948 [IPC Server handler 36 on 8030] INFO event.AsyncDispatcher: Size of event-queue is 1000 2016-02-23 08:00:00,948 [IPC Server handler 36 on 8030] INFO event.AsyncDispatcher: Size of event-queue is 1000 [...]
Activity
- All
- Comments
- Work Log
- History
- Activity
- Transitions
We've been seeing this too, and it looks like a good improvement over the current behavior.
A small question: do you think is volatile is needed for these variables? If these variables are accessed strictly from a single thread, then volatile can be removed? Probably it's not a big deal either way, but it might be good to see whether it's needed or not.
This is indeed accessed from multiple threads. It just so happens in my example they are all from the same thread, but the common case is that an RPC server handler thread will post events to the queue, and there are multiple RPC server handler threads.
+1 pending jenkins.
Kicked off the jenkins build again.
 -1 overall |
| Vote | Subsystem | Runtime | Comment |
|---|---|---|---|
| 0 | reexec | 0m 10s | Docker mode activated. |
| +1 | @author | 0m 0s | The patch does not contain any @author tags. |
| -1 | test4tests | 0m 0s | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. |
| 0 | mvndep | 0m 21s | Maven dependency ordering for branch |
| +1 | mvninstall | 7m 14s | trunk passed |
| +1 | compile | 1m 59s | trunk passed with JDK v1.8.0_72 |
| +1 | compile | 2m 18s | trunk passed with JDK v1.7.0_95 |
| +1 | checkstyle | 0m 35s | trunk passed |
| +1 | mvnsite | 1m 10s | trunk passed |
| +1 | mvneclipse | 0m 29s | trunk passed |
| +1 | findbugs | 2m 20s | trunk passed |
| +1 | javadoc | 0m 53s | trunk passed with JDK v1.8.0_72 |
| +1 | javadoc | 1m 2s | trunk passed with JDK v1.7.0_95 |
| 0 | mvndep | 0m 11s | Maven dependency ordering for patch |
| +1 | mvninstall | 1m 0s | the patch passed |
| +1 | compile | 2m 0s | the patch passed with JDK v1.8.0_72 |
| +1 | javac | 2m 0s | the patch passed |
| +1 | compile | 2m 14s | the patch passed with JDK v1.7.0_95 |
| +1 | javac | 2m 14s | the patch passed |
| +1 | checkstyle | 0m 32s | the patch passed |
| +1 | mvnsite | 1m 5s | the patch passed |
| +1 | mvneclipse | 0m 24s | the patch passed |
| +1 | whitespace | 0m 0s | Patch has no whitespace issues. |
| +1 | findbugs | 2m 49s | the patch passed |
| +1 | javadoc | 0m 52s | the patch passed with JDK v1.8.0_72 |
| +1 | javadoc | 0m 59s | the patch passed with JDK v1.7.0_95 |
| +1 | unit | 2m 4s | hadoop-yarn-common in the patch passed with JDK v1.8.0_72. |
| -1 | unit | 67m 37s | hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.8.0_72. |
| +1 | unit | 2m 15s | hadoop-yarn-common in the patch passed with JDK v1.7.0_95. |
| -1 | unit | 69m 13s | hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.7.0_95. |
| +1 | asflicense | 0m 19s | Patch does not generate ASF License warnings. |
| 173m 23s |
| Reason | Tests |
|---|---|
| JDK v1.8.0_72 Failed junit tests | hadoop.yarn.server.resourcemanager.TestAMAuthorization |
| hadoop.yarn.server.resourcemanager.TestClientRMTokens | |
| JDK v1.7.0_95 Failed junit tests | hadoop.yarn.server.resourcemanager.TestAMAuthorization |
| hadoop.yarn.server.resourcemanager.TestClientRMTokens |
This message was automatically generated.
FAILURE: Integrated in Hadoop-trunk-Commit #9360 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9360/)
YARN-4722. AsyncDispatcher logs redundant event queue sizes (Jason Lowe (sjlee: rev 553b591ba06bbf0b18dca674d25a48218fed0a26)
- hadoop-yarn-project/CHANGES.txt
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceManager.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/event/AsyncDispatcher.java
Committed the patch. Thanks Jason Lowe for your contribution!
Thanks, Sangjin!
Closing the JIRA as part of 2.7.3 release.
Patch that avoids logging the event queue size if it hasn't changed since the last log message. I also updated similar code in the RM scheduler event handler since it can have the same issue.