Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-12903

IPC Server should allow suppressing exception logging by type, not log 'server too busy' messages

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.7.2
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: ipc
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      HADOOP-10597 added support for RPC congestion control by sending retriable 'server too busy' exceptions to clients.

      However every backoff results in a log message. We've seen these log messages slow down the NameNode.

      2016-03-07 15:02:23,272 INFO org.apache.hadoop.ipc.Server: Socket Reader #1 for port 8020: readAndProcess from client 127.0.0.1 threw exception [org.apache.hadoop.ipc.RetriableException: Server is too busy.]
      

      We already have a metric that tracks the number of backoff events. This log message adds nothing useful.

      The IPC Server should also allow services to skip logging certain exception types altogether.

      1. checkstyle-fixes.patch
        2 kB
        Arpit Agarwal
      2. HADOOP-12903.01.patch
        12 kB
        Arpit Agarwal
      3. HADOOP-12903.02.patch
        12 kB
        Arpit Agarwal
      4. HADOOP-12903.03.patch
        12 kB
        Arpit Agarwal

        Activity

        Hide
        arpitagarwal Arpit Agarwal added a comment -

        v01 patch adds Server#ExceptionsHandler support to avoid logging exceptions by class in . No exceptions are added to the suppress list, but services e.g. NameNode could add certain exceptions to this list in the future.

        Additionally 'Server too busy' errors are completely skipped.

        Show
        arpitagarwal Arpit Agarwal added a comment - v01 patch adds Server#ExceptionsHandler support to avoid logging exceptions by class in . No exceptions are added to the suppress list, but services e.g. NameNode could add certain exceptions to this list in the future. Additionally 'Server too busy' errors are completely skipped.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 0s Docker mode activated.
        -1 patch 0m 4s HADOOP-12903 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help.



        Subsystem Report/Notes
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12791863/HADOOP-12903.01.patch
        JIRA Issue HADOOP-12903
        Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8811/console
        Powered by Apache Yetus 0.3.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. -1 patch 0m 4s HADOOP-12903 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. Subsystem Report/Notes JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12791863/HADOOP-12903.01.patch JIRA Issue HADOOP-12903 Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8811/console Powered by Apache Yetus 0.3.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        arpitagarwal Arpit Agarwal added a comment -

        v2 patch, rebased to current trunk.

        Show
        arpitagarwal Arpit Agarwal added a comment - v2 patch, rebased to current trunk.
        Hide
        cnauroth Chris Nauroth added a comment -

        Hi Arpit Agarwal. This looks good. I just have one suggestion: consider refactoring the common logic of addTerseLoggingExceptions and addSuppressedLoggingExceptions. Then, those two methods could reduce to one-liners, like:

        terseExceptions = addExceptions(terseExceptions, exceptionClass);

        suppressedExceptions = addExceptions(suppressedExceptions, exceptionClass);

        Show
        cnauroth Chris Nauroth added a comment - Hi Arpit Agarwal . This looks good. I just have one suggestion: consider refactoring the common logic of addTerseLoggingExceptions and addSuppressedLoggingExceptions . Then, those two methods could reduce to one-liners, like: terseExceptions = addExceptions(terseExceptions, exceptionClass); suppressedExceptions = addExceptions(suppressedExceptions, exceptionClass);
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 11s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        +1 mvninstall 6m 23s trunk passed
        +1 compile 5m 33s trunk passed with JDK v1.8.0_74
        +1 compile 6m 29s trunk passed with JDK v1.7.0_95
        +1 checkstyle 0m 23s trunk passed
        +1 mvnsite 0m 55s trunk passed
        +1 mvneclipse 0m 14s trunk passed
        +1 findbugs 1m 32s trunk passed
        +1 javadoc 0m 52s trunk passed with JDK v1.8.0_74
        +1 javadoc 1m 2s trunk passed with JDK v1.7.0_95
        +1 mvninstall 0m 39s the patch passed
        +1 compile 5m 30s the patch passed with JDK v1.8.0_74
        +1 javac 5m 30s the patch passed
        +1 compile 6m 31s the patch passed with JDK v1.7.0_95
        +1 javac 6m 31s the patch passed
        -1 checkstyle 0m 22s hadoop-common-project/hadoop-common: patch generated 3 new + 198 unchanged - 1 fixed = 201 total (was 199)
        +1 mvnsite 0m 55s the patch passed
        +1 mvneclipse 0m 13s the patch passed
        -1 whitespace 0m 0s The patch has 11 line(s) that end in whitespace. Use git apply --whitespace=fix.
        +1 findbugs 1m 47s the patch passed
        +1 javadoc 0m 53s the patch passed with JDK v1.8.0_74
        +1 javadoc 1m 4s the patch passed with JDK v1.7.0_95
        +1 unit 7m 45s hadoop-common in the patch passed with JDK v1.8.0_74.
        -1 unit 7m 50s hadoop-common in the patch failed with JDK v1.7.0_95.
        +1 asflicense 0m 21s Patch does not generate ASF License warnings.
        58m 33s



        Reason Tests
        JDK v1.7.0_95 Failed junit tests hadoop.ha.TestZKFailoverController
          hadoop.ipc.TestRPCWaitForProxy



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:0ca8df7
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12792044/HADOOP-12903.02.patch
        JIRA Issue HADOOP-12903
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 2e6a27774b92 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / f86850b
        Default Java 1.7.0_95
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_74 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/8821/artifact/patchprocess/diff-checkstyle-hadoop-common-project_hadoop-common.txt
        whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/8821/artifact/patchprocess/whitespace-eol.txt
        unit https://builds.apache.org/job/PreCommit-HADOOP-Build/8821/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_95.txt
        unit test logs https://builds.apache.org/job/PreCommit-HADOOP-Build/8821/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_95.txt
        JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/8821/testReport/
        modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
        Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8821/console
        Powered by Apache Yetus 0.2.0 http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 11s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 6m 23s trunk passed +1 compile 5m 33s trunk passed with JDK v1.8.0_74 +1 compile 6m 29s trunk passed with JDK v1.7.0_95 +1 checkstyle 0m 23s trunk passed +1 mvnsite 0m 55s trunk passed +1 mvneclipse 0m 14s trunk passed +1 findbugs 1m 32s trunk passed +1 javadoc 0m 52s trunk passed with JDK v1.8.0_74 +1 javadoc 1m 2s trunk passed with JDK v1.7.0_95 +1 mvninstall 0m 39s the patch passed +1 compile 5m 30s the patch passed with JDK v1.8.0_74 +1 javac 5m 30s the patch passed +1 compile 6m 31s the patch passed with JDK v1.7.0_95 +1 javac 6m 31s the patch passed -1 checkstyle 0m 22s hadoop-common-project/hadoop-common: patch generated 3 new + 198 unchanged - 1 fixed = 201 total (was 199) +1 mvnsite 0m 55s the patch passed +1 mvneclipse 0m 13s the patch passed -1 whitespace 0m 0s The patch has 11 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 findbugs 1m 47s the patch passed +1 javadoc 0m 53s the patch passed with JDK v1.8.0_74 +1 javadoc 1m 4s the patch passed with JDK v1.7.0_95 +1 unit 7m 45s hadoop-common in the patch passed with JDK v1.8.0_74. -1 unit 7m 50s hadoop-common in the patch failed with JDK v1.7.0_95. +1 asflicense 0m 21s Patch does not generate ASF License warnings. 58m 33s Reason Tests JDK v1.7.0_95 Failed junit tests hadoop.ha.TestZKFailoverController   hadoop.ipc.TestRPCWaitForProxy Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12792044/HADOOP-12903.02.patch JIRA Issue HADOOP-12903 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 2e6a27774b92 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / f86850b Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_74 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/8821/artifact/patchprocess/diff-checkstyle-hadoop-common-project_hadoop-common.txt whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/8821/artifact/patchprocess/whitespace-eol.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/8821/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_95.txt unit test logs https://builds.apache.org/job/PreCommit-HADOOP-Build/8821/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_95.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/8821/testReport/ modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8821/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
        Hide
        arpitagarwal Arpit Agarwal added a comment -

        Thanks Chris Nauroth, v3 patch addresses your comment. I missed your comment earlier in the jira noise.

        Show
        arpitagarwal Arpit Agarwal added a comment - Thanks Chris Nauroth , v3 patch addresses your comment. I missed your comment earlier in the jira noise.
        Hide
        cnauroth Chris Nauroth added a comment -

        +1 for patch v03, pending pre-commit run. Arpit Agarwal, thank you for the patch.

        Show
        cnauroth Chris Nauroth added a comment - +1 for patch v03, pending pre-commit run. Arpit Agarwal , thank you for the patch.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 15s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        +1 mvninstall 7m 44s trunk passed
        +1 compile 9m 7s trunk passed with JDK v1.8.0_74
        +1 compile 7m 56s trunk passed with JDK v1.7.0_95
        +1 checkstyle 0m 23s trunk passed
        +1 mvnsite 1m 1s trunk passed
        +1 mvneclipse 0m 13s trunk passed
        +1 findbugs 1m 45s trunk passed
        +1 javadoc 1m 7s trunk passed with JDK v1.8.0_74
        +1 javadoc 1m 9s trunk passed with JDK v1.7.0_95
        +1 mvninstall 0m 47s the patch passed
        +1 compile 9m 5s the patch passed with JDK v1.8.0_74
        +1 javac 9m 5s the patch passed
        +1 compile 8m 7s the patch passed with JDK v1.7.0_95
        +1 javac 8m 7s the patch passed
        -1 checkstyle 0m 25s hadoop-common-project/hadoop-common: patch generated 3 new + 198 unchanged - 1 fixed = 201 total (was 199)
        +1 mvnsite 1m 3s the patch passed
        +1 mvneclipse 0m 13s the patch passed
        +1 whitespace 0m 0s Patch has no whitespace issues.
        +1 findbugs 1m 57s the patch passed
        +1 javadoc 1m 1s the patch passed with JDK v1.8.0_74
        +1 javadoc 1m 8s the patch passed with JDK v1.7.0_95
        -1 unit 10m 7s hadoop-common in the patch failed with JDK v1.8.0_74.
        +1 unit 10m 1s hadoop-common in the patch passed with JDK v1.7.0_95.
        +1 asflicense 0m 24s Patch does not generate ASF License warnings.
        76m 11s



        Reason Tests
        JDK v1.8.0_74 Failed junit tests hadoop.fs.shell.find.TestName
          hadoop.ipc.TestIPC



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:0ca8df7
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12792130/HADOOP-12903.03.patch
        JIRA Issue HADOOP-12903
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux f06e6c7e4a30 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 55f73a1
        Default Java 1.7.0_95
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_74 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/8826/artifact/patchprocess/diff-checkstyle-hadoop-common-project_hadoop-common.txt
        unit https://builds.apache.org/job/PreCommit-HADOOP-Build/8826/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_74.txt
        unit test logs https://builds.apache.org/job/PreCommit-HADOOP-Build/8826/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_74.txt
        JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/8826/testReport/
        modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
        Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8826/console
        Powered by Apache Yetus 0.2.0 http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 15s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 7m 44s trunk passed +1 compile 9m 7s trunk passed with JDK v1.8.0_74 +1 compile 7m 56s trunk passed with JDK v1.7.0_95 +1 checkstyle 0m 23s trunk passed +1 mvnsite 1m 1s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 1m 45s trunk passed +1 javadoc 1m 7s trunk passed with JDK v1.8.0_74 +1 javadoc 1m 9s trunk passed with JDK v1.7.0_95 +1 mvninstall 0m 47s the patch passed +1 compile 9m 5s the patch passed with JDK v1.8.0_74 +1 javac 9m 5s the patch passed +1 compile 8m 7s the patch passed with JDK v1.7.0_95 +1 javac 8m 7s the patch passed -1 checkstyle 0m 25s hadoop-common-project/hadoop-common: patch generated 3 new + 198 unchanged - 1 fixed = 201 total (was 199) +1 mvnsite 1m 3s the patch passed +1 mvneclipse 0m 13s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 1m 57s the patch passed +1 javadoc 1m 1s the patch passed with JDK v1.8.0_74 +1 javadoc 1m 8s the patch passed with JDK v1.7.0_95 -1 unit 10m 7s hadoop-common in the patch failed with JDK v1.8.0_74. +1 unit 10m 1s hadoop-common in the patch passed with JDK v1.7.0_95. +1 asflicense 0m 24s Patch does not generate ASF License warnings. 76m 11s Reason Tests JDK v1.8.0_74 Failed junit tests hadoop.fs.shell.find.TestName   hadoop.ipc.TestIPC Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12792130/HADOOP-12903.03.patch JIRA Issue HADOOP-12903 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux f06e6c7e4a30 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 55f73a1 Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_74 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/8826/artifact/patchprocess/diff-checkstyle-hadoop-common-project_hadoop-common.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/8826/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_74.txt unit test logs https://builds.apache.org/job/PreCommit-HADOOP-Build/8826/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_74.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/8826/testReport/ modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8826/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
        Hide
        arpitagarwal Arpit Agarwal added a comment -

        Thank you for the code reviews Chris Nauroth.

        I've committed this for 2.8.0. I fixed the findbugs warnings about line length while committing (delta patch attached).

        Show
        arpitagarwal Arpit Agarwal added a comment - Thank you for the code reviews Chris Nauroth . I've committed this for 2.8.0. I fixed the findbugs warnings about line length while committing ( delta patch attached ).
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-trunk-Commit #9446 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9446/)
        HADOOP-12903. IPC Server should allow suppressing exception logging by (arp: rev 2e040d31c7bba021576e6baf267d937da7ff814a)

        • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Server.java
        • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ipc/TestServer.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #9446 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9446/ ) HADOOP-12903 . IPC Server should allow suppressing exception logging by (arp: rev 2e040d31c7bba021576e6baf267d937da7ff814a) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Server.java hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ipc/TestServer.java

          People

          • Assignee:
            arpitagarwal Arpit Agarwal
            Reporter:
            arpitagarwal Arpit Agarwal
          • Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development