Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-4722

AsyncDispatcher logs redundant event queue sizes

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.6.0
    • Fix Version/s: 2.8.0, 2.7.3, 2.6.5, 3.0.0-alpha1
    • Component/s: None
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      A fairly common occurrence in RM logs is a string of redundant event-queue logs like the following which does little except bloat the logs:

      2016-02-23 08:00:00,948 [IPC Server handler 36 on 8030] INFO event.AsyncDispatcher: Size of event-queue is 1000
      2016-02-23 08:00:00,948 [IPC Server handler 36 on 8030] INFO event.AsyncDispatcher: Size of event-queue is 1000
      2016-02-23 08:00:00,948 [IPC Server handler 36 on 8030] INFO event.AsyncDispatcher: Size of event-queue is 1000
      2016-02-23 08:00:00,948 [IPC Server handler 36 on 8030] INFO event.AsyncDispatcher: Size of event-queue is 1000
      2016-02-23 08:00:00,948 [IPC Server handler 36 on 8030] INFO event.AsyncDispatcher: Size of event-queue is 1000
      [...]
      

        Activity

        Hide
        jlowe Jason Lowe added a comment -

        Patch that avoids logging the event queue size if it hasn't changed since the last log message. I also updated similar code in the RM scheduler event handler since it can have the same issue.

        Show
        jlowe Jason Lowe added a comment - Patch that avoids logging the event queue size if it hasn't changed since the last log message. I also updated similar code in the RM scheduler event handler since it can have the same issue.
        Hide
        sjlee0 Sangjin Lee added a comment -

        We've been seeing this too, and it looks like a good improvement over the current behavior.

        A small question: do you think is volatile is needed for these variables? If these variables are accessed strictly from a single thread, then volatile can be removed? Probably it's not a big deal either way, but it might be good to see whether it's needed or not.

        Show
        sjlee0 Sangjin Lee added a comment - We've been seeing this too, and it looks like a good improvement over the current behavior. A small question: do you think is volatile is needed for these variables? If these variables are accessed strictly from a single thread, then volatile can be removed? Probably it's not a big deal either way, but it might be good to see whether it's needed or not.
        Hide
        jlowe Jason Lowe added a comment -

        This is indeed accessed from multiple threads. It just so happens in my example they are all from the same thread, but the common case is that an RPC server handler thread will post events to the queue, and there are multiple RPC server handler threads.

        Show
        jlowe Jason Lowe added a comment - This is indeed accessed from multiple threads. It just so happens in my example they are all from the same thread, but the common case is that an RPC server handler thread will post events to the queue, and there are multiple RPC server handler threads.
        Hide
        sjlee0 Sangjin Lee added a comment -

        +1 pending jenkins.

        Show
        sjlee0 Sangjin Lee added a comment - +1 pending jenkins.
        Hide
        sjlee0 Sangjin Lee added a comment -

        Kicked off the jenkins build again.

        Show
        sjlee0 Sangjin Lee added a comment - Kicked off the jenkins build again.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 10s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        0 mvndep 0m 21s Maven dependency ordering for branch
        +1 mvninstall 7m 14s trunk passed
        +1 compile 1m 59s trunk passed with JDK v1.8.0_72
        +1 compile 2m 18s trunk passed with JDK v1.7.0_95
        +1 checkstyle 0m 35s trunk passed
        +1 mvnsite 1m 10s trunk passed
        +1 mvneclipse 0m 29s trunk passed
        +1 findbugs 2m 20s trunk passed
        +1 javadoc 0m 53s trunk passed with JDK v1.8.0_72
        +1 javadoc 1m 2s trunk passed with JDK v1.7.0_95
        0 mvndep 0m 11s Maven dependency ordering for patch
        +1 mvninstall 1m 0s the patch passed
        +1 compile 2m 0s the patch passed with JDK v1.8.0_72
        +1 javac 2m 0s the patch passed
        +1 compile 2m 14s the patch passed with JDK v1.7.0_95
        +1 javac 2m 14s the patch passed
        +1 checkstyle 0m 32s the patch passed
        +1 mvnsite 1m 5s the patch passed
        +1 mvneclipse 0m 24s the patch passed
        +1 whitespace 0m 0s Patch has no whitespace issues.
        +1 findbugs 2m 49s the patch passed
        +1 javadoc 0m 52s the patch passed with JDK v1.8.0_72
        +1 javadoc 0m 59s the patch passed with JDK v1.7.0_95
        +1 unit 2m 4s hadoop-yarn-common in the patch passed with JDK v1.8.0_72.
        -1 unit 67m 37s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.8.0_72.
        +1 unit 2m 15s hadoop-yarn-common in the patch passed with JDK v1.7.0_95.
        -1 unit 69m 13s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.7.0_95.
        +1 asflicense 0m 19s Patch does not generate ASF License warnings.
        173m 23s



        Reason Tests
        JDK v1.8.0_72 Failed junit tests hadoop.yarn.server.resourcemanager.TestAMAuthorization
          hadoop.yarn.server.resourcemanager.TestClientRMTokens
        JDK v1.7.0_95 Failed junit tests hadoop.yarn.server.resourcemanager.TestAMAuthorization
          hadoop.yarn.server.resourcemanager.TestClientRMTokens



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:0ca8df7
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12789227/YARN-4722.001.patch
        JIRA Issue YARN-4722
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 40d149d254b8 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 4343a4c
        Default Java 1.7.0_95
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_72 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
        findbugs v3.0.0
        unit https://builds.apache.org/job/PreCommit-YARN-Build/10617/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_72.txt
        unit https://builds.apache.org/job/PreCommit-YARN-Build/10617/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_95.txt
        unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/10617/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_72.txt https://builds.apache.org/job/PreCommit-YARN-Build/10617/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_95.txt
        JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/10617/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/10617/console
        Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 10s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. 0 mvndep 0m 21s Maven dependency ordering for branch +1 mvninstall 7m 14s trunk passed +1 compile 1m 59s trunk passed with JDK v1.8.0_72 +1 compile 2m 18s trunk passed with JDK v1.7.0_95 +1 checkstyle 0m 35s trunk passed +1 mvnsite 1m 10s trunk passed +1 mvneclipse 0m 29s trunk passed +1 findbugs 2m 20s trunk passed +1 javadoc 0m 53s trunk passed with JDK v1.8.0_72 +1 javadoc 1m 2s trunk passed with JDK v1.7.0_95 0 mvndep 0m 11s Maven dependency ordering for patch +1 mvninstall 1m 0s the patch passed +1 compile 2m 0s the patch passed with JDK v1.8.0_72 +1 javac 2m 0s the patch passed +1 compile 2m 14s the patch passed with JDK v1.7.0_95 +1 javac 2m 14s the patch passed +1 checkstyle 0m 32s the patch passed +1 mvnsite 1m 5s the patch passed +1 mvneclipse 0m 24s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 2m 49s the patch passed +1 javadoc 0m 52s the patch passed with JDK v1.8.0_72 +1 javadoc 0m 59s the patch passed with JDK v1.7.0_95 +1 unit 2m 4s hadoop-yarn-common in the patch passed with JDK v1.8.0_72. -1 unit 67m 37s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.8.0_72. +1 unit 2m 15s hadoop-yarn-common in the patch passed with JDK v1.7.0_95. -1 unit 69m 13s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.7.0_95. +1 asflicense 0m 19s Patch does not generate ASF License warnings. 173m 23s Reason Tests JDK v1.8.0_72 Failed junit tests hadoop.yarn.server.resourcemanager.TestAMAuthorization   hadoop.yarn.server.resourcemanager.TestClientRMTokens JDK v1.7.0_95 Failed junit tests hadoop.yarn.server.resourcemanager.TestAMAuthorization   hadoop.yarn.server.resourcemanager.TestClientRMTokens Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12789227/YARN-4722.001.patch JIRA Issue YARN-4722 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 40d149d254b8 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 4343a4c Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_72 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-YARN-Build/10617/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_72.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/10617/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_95.txt unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/10617/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_72.txt https://builds.apache.org/job/PreCommit-YARN-Build/10617/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_95.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/10617/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn Console output https://builds.apache.org/job/PreCommit-YARN-Build/10617/console Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-trunk-Commit #9360 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9360/)
        YARN-4722. AsyncDispatcher logs redundant event queue sizes (Jason Lowe (sjlee: rev 553b591ba06bbf0b18dca674d25a48218fed0a26)

        • hadoop-yarn-project/CHANGES.txt
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceManager.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/event/AsyncDispatcher.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #9360 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9360/ ) YARN-4722 . AsyncDispatcher logs redundant event queue sizes (Jason Lowe (sjlee: rev 553b591ba06bbf0b18dca674d25a48218fed0a26) hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceManager.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/event/AsyncDispatcher.java
        Hide
        sjlee0 Sangjin Lee added a comment -

        Committed the patch. Thanks Jason Lowe for your contribution!

        Show
        sjlee0 Sangjin Lee added a comment - Committed the patch. Thanks Jason Lowe for your contribution!
        Hide
        jlowe Jason Lowe added a comment -

        Thanks, Sangjin!

        Show
        jlowe Jason Lowe added a comment - Thanks, Sangjin!
        Hide
        vinodkv Vinod Kumar Vavilapalli added a comment -

        Closing the JIRA as part of 2.7.3 release.

        Show
        vinodkv Vinod Kumar Vavilapalli added a comment - Closing the JIRA as part of 2.7.3 release.

          People

          • Assignee:
            jlowe Jason Lowe
            Reporter:
            jlowe Jason Lowe
          • Votes:
            0 Vote for this issue
            Watchers:
            8 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development