Hadoop Common
  1. Hadoop Common
  2. HADOOP-7121

Exceptions while serializing IPC call response are not handled well

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Critical Critical
    • Resolution: Fixed
    • Affects Version/s: 0.22.0
    • Fix Version/s: 0.22.0
    • Component/s: ipc
    • Labels:
    • Hadoop Flags:
      Reviewed

      Description

      We had a situation where for some reason the serialization of an RPC call's response was throwing OOME. When this happens, the exception is not caught, and the call never gets a response - the client just hangs. Additionally, the OOME propagated all the way to the top of the IPC handler and caused the handler. Plus, the Handler upon exit only logged to stdout and not to the log4j logs.

      1. hadoop-7121.txt
        13 kB
        Todd Lipcon
      2. hadoop-7121.txt
        14 kB
        Todd Lipcon
      3. hadoop-7121.txt
        13 kB
        Todd Lipcon

        Activity

        Todd Lipcon created issue -
        Todd Lipcon made changes -
        Field Original Value New Value
        Labels newbie
        Todd Lipcon made changes -
        Assignee Todd Lipcon [ tlipcon ]
        Todd Lipcon made changes -
        Assignee Todd Lipcon [ tlipcon ]
        Hide
        Todd Lipcon added a comment -

        Attached patch adds coverage to all parts of the IPC lifecycle:

        • client side write param
        • server side read param
        • server side write response
        • client side read response

        In the cases that the error occurs server side, it sends back the exception to the client as well as logging it.

        Show
        Todd Lipcon added a comment - Attached patch adds coverage to all parts of the IPC lifecycle: client side write param server side read param server side write response client side read response In the cases that the error occurs server side, it sends back the exception to the client as well as logging it.
        Todd Lipcon made changes -
        Attachment hadoop-7121.txt [ 12481021 ]
        Todd Lipcon made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12481021/hadoop-7121.txt
        against trunk revision 1129905.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 3 new or modified tests.

        -1 patch. The patch command could not apply the patch.

        Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/549//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12481021/hadoop-7121.txt against trunk revision 1129905. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/549//console This message is automatically generated.
        Hide
        Todd Lipcon added a comment -

        Updated patch against trunk

        Show
        Todd Lipcon added a comment - Updated patch against trunk
        Todd Lipcon made changes -
        Attachment hadoop-7121.txt [ 12481024 ]
        Hide
        Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12481024/hadoop-7121.txt
        against trunk revision 1129905.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 3 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed core unit tests.

        +1 system test framework. The patch passed system test framework compile.

        Test results: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/551//testReport/
        Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/551//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/551//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12481024/hadoop-7121.txt against trunk revision 1129905. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/551//testReport/ Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/551//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/551//console This message is automatically generated.
        Hide
        Eli Collins added a comment -

        +1 lgtm

        Nit: lines 1364, 1366, 1612 don't need to wrap

        Show
        Eli Collins added a comment - +1 lgtm Nit: lines 1364, 1366, 1612 don't need to wrap
        Hide
        Todd Lipcon added a comment -

        nits fixed. Since it was just a whitespace change from previous patch I will commit this version based on Eli's +1.

        Show
        Todd Lipcon added a comment - nits fixed. Since it was just a whitespace change from previous patch I will commit this version based on Eli's +1.
        Todd Lipcon made changes -
        Attachment hadoop-7121.txt [ 12481031 ]
        Hide
        Todd Lipcon added a comment -

        Committed to trunk and 22. Thanks for reviewing, Eli

        Show
        Todd Lipcon added a comment - Committed to trunk and 22. Thanks for reviewing, Eli
        Todd Lipcon made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Hadoop Flags [Reviewed]
        Resolution Fixed [ 1 ]
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12481031/hadoop-7121.txt
        against trunk revision 1129982.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 3 new or modified tests.

        -1 patch. The patch command could not apply the patch.

        Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/552//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12481031/hadoop-7121.txt against trunk revision 1129982. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/552//console This message is automatically generated.
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Common-trunk-Commit #631 (See https://builds.apache.org/hudson/job/Hadoop-Common-trunk-Commit/631/)
        HADOOP-7121. Exceptions while serializing IPC call responses are not handled well. Contributed by Todd Lipcon.

        todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1129982
        Files :

        • /hadoop/common/trunk/CHANGES.txt
        • /hadoop/common/trunk/src/java/org/apache/hadoop/ipc/Server.java
        • /hadoop/common/trunk/src/test/core/org/apache/hadoop/ipc/TestIPC.java
        Show
        Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #631 (See https://builds.apache.org/hudson/job/Hadoop-Common-trunk-Commit/631/ ) HADOOP-7121 . Exceptions while serializing IPC call responses are not handled well. Contributed by Todd Lipcon. todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1129982 Files : /hadoop/common/trunk/CHANGES.txt /hadoop/common/trunk/src/java/org/apache/hadoop/ipc/Server.java /hadoop/common/trunk/src/test/core/org/apache/hadoop/ipc/TestIPC.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Common-22-branch #59 (See https://builds.apache.org/hudson/job/Hadoop-Common-22-branch/59/)
        HADOOP-7121. Exceptions while serializing IPC call responses are not handled well. Contributed by Todd Lipcon.

        todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1129983
        Files :

        • /hadoop/common/branches/branch-0.22/src/java/org/apache/hadoop/ipc/Server.java
        • /hadoop/common/branches/branch-0.22/src/test/core/org/apache/hadoop/ipc/TestIPC.java
        • /hadoop/common/branches/branch-0.22/CHANGES.txt
        Show
        Hudson added a comment - Integrated in Hadoop-Common-22-branch #59 (See https://builds.apache.org/hudson/job/Hadoop-Common-22-branch/59/ ) HADOOP-7121 . Exceptions while serializing IPC call responses are not handled well. Contributed by Todd Lipcon. todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1129983 Files : /hadoop/common/branches/branch-0.22/src/java/org/apache/hadoop/ipc/Server.java /hadoop/common/branches/branch-0.22/src/test/core/org/apache/hadoop/ipc/TestIPC.java /hadoop/common/branches/branch-0.22/CHANGES.txt
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Common-trunk #706 (See https://builds.apache.org/hudson/job/Hadoop-Common-trunk/706/)
        HADOOP-7121. Exceptions while serializing IPC call responses are not handled well. Contributed by Todd Lipcon.

        todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1129982
        Files :

        • /hadoop/common/trunk/CHANGES.txt
        • /hadoop/common/trunk/src/java/org/apache/hadoop/ipc/Server.java
        • /hadoop/common/trunk/src/test/core/org/apache/hadoop/ipc/TestIPC.java
        Show
        Hudson added a comment - Integrated in Hadoop-Common-trunk #706 (See https://builds.apache.org/hudson/job/Hadoop-Common-trunk/706/ ) HADOOP-7121 . Exceptions while serializing IPC call responses are not handled well. Contributed by Todd Lipcon. todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1129982 Files : /hadoop/common/trunk/CHANGES.txt /hadoop/common/trunk/src/java/org/apache/hadoop/ipc/Server.java /hadoop/common/trunk/src/test/core/org/apache/hadoop/ipc/TestIPC.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Common-trunk-Commit #635 (See https://builds.apache.org/hudson/job/Hadoop-Common-trunk-Commit/635/)
        HADOOP-7349. HADOOP-7121 accidentally disabled some tests in TestIPC. Contributed by Todd Lipcon.

        todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1130758
        Files :

        • /hadoop/common/trunk/CHANGES.txt
        • /hadoop/common/trunk/src/test/core/org/apache/hadoop/ipc/TestIPC.java
        Show
        Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #635 (See https://builds.apache.org/hudson/job/Hadoop-Common-trunk-Commit/635/ ) HADOOP-7349 . HADOOP-7121 accidentally disabled some tests in TestIPC. Contributed by Todd Lipcon. todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1130758 Files : /hadoop/common/trunk/CHANGES.txt /hadoop/common/trunk/src/test/core/org/apache/hadoop/ipc/TestIPC.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Common-22-branch #61 (See https://builds.apache.org/hudson/job/Hadoop-Common-22-branch/61/)
        HADOOP-7349. HADOOP-7121 accidentally disabled some tests in TestIPC. Contributed by Todd Lipcon.

        todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1130757
        Files :

        • /hadoop/common/branches/branch-0.22/src/test/core/org/apache/hadoop/ipc/TestIPC.java
        • /hadoop/common/branches/branch-0.22/CHANGES.txt
        Show
        Hudson added a comment - Integrated in Hadoop-Common-22-branch #61 (See https://builds.apache.org/hudson/job/Hadoop-Common-22-branch/61/ ) HADOOP-7349 . HADOOP-7121 accidentally disabled some tests in TestIPC. Contributed by Todd Lipcon. todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1130757 Files : /hadoop/common/branches/branch-0.22/src/test/core/org/apache/hadoop/ipc/TestIPC.java /hadoop/common/branches/branch-0.22/CHANGES.txt
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Common-trunk #708 (See https://builds.apache.org/hudson/job/Hadoop-Common-trunk/708/)
        HADOOP-7349. HADOOP-7121 accidentally disabled some tests in TestIPC. Contributed by Todd Lipcon.

        todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1130758
        Files :

        • /hadoop/common/trunk/CHANGES.txt
        • /hadoop/common/trunk/src/test/core/org/apache/hadoop/ipc/TestIPC.java
        Show
        Hudson added a comment - Integrated in Hadoop-Common-trunk #708 (See https://builds.apache.org/hudson/job/Hadoop-Common-trunk/708/ ) HADOOP-7349 . HADOOP-7121 accidentally disabled some tests in TestIPC. Contributed by Todd Lipcon. todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1130758 Files : /hadoop/common/trunk/CHANGES.txt /hadoop/common/trunk/src/test/core/org/apache/hadoop/ipc/TestIPC.java
        Konstantin Shvachko made changes -
        Status Resolved [ 5 ] Closed [ 6 ]

          People

          • Assignee:
            Todd Lipcon
            Reporter:
            Todd Lipcon
          • Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development