Hadoop Common
  1. Hadoop Common
  2. HADOOP-8986

Server$Call object is never released after it is sent

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Critical Critical
    • Resolution: Fixed
    • Affects Version/s: 2.0.2-alpha, 0.23.4
    • Fix Version/s: 2.0.3-alpha, 0.23.5
    • Component/s: ipc
    • Labels:
      None

      Description

      When an IPC response cannot be returned without blocking an Server$Call object is attached to the SelectionKey of the write selector. However the call object is never removed from the SlectionKey. So for a connection that rarely has large responses but is long lived there is a lot of data.

      1. HADOOP-8986.txt
        0.8 kB
        Robert Joseph Evans

        Activity

        Robert Joseph Evans created issue -
        Hide
        Robert Joseph Evans added a comment -

        This can save several hundred MB on an AM with many reduce tasks.

        Show
        Robert Joseph Evans added a comment - This can save several hundred MB on an AM with many reduce tasks.
        Robert Joseph Evans made changes -
        Field Original Value New Value
        Description When an IPC response cannot be returned without blocking the Server$Call object attached to the SelectionKey of the write selector. However the call object is never removed from the SlectionKey. So for a connection that rarely has large responses but is long lived there is a lot of data. When an IPC response cannot be returned without blocking an Server$Call object is attached to the SelectionKey of the write selector. However the call object is never removed from the SlectionKey. So for a connection that rarely has large responses but is long lived there is a lot of data.
        Hide
        Robert Joseph Evans added a comment -

        There are no tests for this patch. I have tested it manually. All of the old functionality should still work, it is just that more memory will be released to be collected when a call finishes.

        Show
        Robert Joseph Evans added a comment - There are no tests for this patch. I have tested it manually. All of the old functionality should still work, it is just that more memory will be released to be collected when a call finishes.
        Robert Joseph Evans made changes -
        Attachment HADOOP-8986.txt [ 12551227 ]
        Robert Joseph Evans made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12551227/HADOOP-8986.txt
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in hadoop-common-project/hadoop-common.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/1685//testReport/
        Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1685//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12551227/HADOOP-8986.txt against trunk revision . +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-common-project/hadoop-common. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/1685//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1685//console This message is automatically generated.
        Hide
        Daryn Sharp added a comment -

        Looks good. Did you investigate whether a test can be added to TestIPCServerResponder? It causes partial writes which may allow you to check if the response is null out after the call finally completes.

        Show
        Daryn Sharp added a comment - Looks good. Did you investigate whether a test can be added to TestIPCServerResponder ? It causes partial writes which may allow you to check if the response is null out after the call finally completes.
        Hide
        Robert Joseph Evans added a comment -

        I didn't look that closely at adding in a test because I was mostly concerned that it did not break any existing functionality. I will look at TestIPCResponder to see if I can add something in there.

        Show
        Robert Joseph Evans added a comment - I didn't look that closely at adding in a test because I was mostly concerned that it did not break any existing functionality. I will look at TestIPCResponder to see if I can add something in there.
        Hide
        Suresh Srinivas added a comment -

        +1 for the patch. I feel it is unnecessary to add test for this.

        Show
        Suresh Srinivas added a comment - +1 for the patch. I feel it is unnecessary to add test for this.
        Hide
        Robert Joseph Evans added a comment -

        Thanks Daryn and Suresh for the reviews.

        I waited to check it in until I could run one more test to measure how much memory savings this had on my original test case. I ran a sleep job with 20,000 mappers and 3,000 reducers. This was on a cluster large enough and free enough that all of them could be running at the same time. The peek memory usage decreased from about 500MB to 285MB and the final heap as most of the reducers started to get close to completing dropped to 200MB, where as before it was still up at 500MB.

        I put it into trunk, branch-2, and branch-0.23

        Show
        Robert Joseph Evans added a comment - Thanks Daryn and Suresh for the reviews. I waited to check it in until I could run one more test to measure how much memory savings this had on my original test case. I ran a sleep job with 20,000 mappers and 3,000 reducers. This was on a cluster large enough and free enough that all of them could be running at the same time. The peek memory usage decreased from about 500MB to 285MB and the final heap as most of the reducers started to get close to completing dropped to 200MB, where as before it was still up at 500MB. I put it into trunk, branch-2, and branch-0.23
        Robert Joseph Evans made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Fix Version/s 3.0.0 [ 12320357 ]
        Fix Version/s 2.0.3-alpha [ 12323273 ]
        Fix Version/s 0.23.5 [ 12323314 ]
        Resolution Fixed [ 1 ]
        Hide
        Hudson added a comment -

        Integrated in Hadoop-trunk-Commit #2943 (See https://builds.apache.org/job/Hadoop-trunk-Commit/2943/)
        HADOOP-8986. Server$Call object is never released after it is sent (bobby) (Revision 1403745)

        Result = SUCCESS
        bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1403745
        Files :

        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Server.java
        Show
        Hudson added a comment - Integrated in Hadoop-trunk-Commit #2943 (See https://builds.apache.org/job/Hadoop-trunk-Commit/2943/ ) HADOOP-8986 . Server$Call object is never released after it is sent (bobby) (Revision 1403745) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1403745 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Server.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Yarn-trunk #22 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/22/)
        HADOOP-8986. Server$Call object is never released after it is sent (bobby) (Revision 1403745)

        Result = SUCCESS
        bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1403745
        Files :

        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Server.java
        Show
        Hudson added a comment - Integrated in Hadoop-Yarn-trunk #22 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/22/ ) HADOOP-8986 . Server$Call object is never released after it is sent (bobby) (Revision 1403745) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1403745 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Server.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-0.23-Build #421 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/421/)
        svn merge -c 1403745 FIXES: HADOOP-8986. Server$Call object is never released after it is sent (bobby) (Revision 1403750)

        Result = SUCCESS
        bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1403750
        Files :

        • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt
        • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Server.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Build #421 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/421/ ) svn merge -c 1403745 FIXES: HADOOP-8986 . Server$Call object is never released after it is sent (bobby) (Revision 1403750) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1403750 Files : /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Server.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk #1212 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1212/)
        HADOOP-8986. Server$Call object is never released after it is sent (bobby) (Revision 1403745)

        Result = FAILURE
        bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1403745
        Files :

        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Server.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #1212 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1212/ ) HADOOP-8986 . Server$Call object is never released after it is sent (bobby) (Revision 1403745) Result = FAILURE bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1403745 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Server.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk #1242 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1242/)
        HADOOP-8986. Server$Call object is never released after it is sent (bobby) (Revision 1403745)

        Result = FAILURE
        bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1403745
        Files :

        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Server.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1242 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1242/ ) HADOOP-8986 . Server$Call object is never released after it is sent (bobby) (Revision 1403745) Result = FAILURE bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1403745 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Server.java
        Thomas Graves made changes -
        Status Resolved [ 5 ] Closed [ 6 ]
        Allen Wittenauer made changes -
        Fix Version/s 3.0.0 [ 12320357 ]
        Transition Time In Source Status Execution Times Last Executer Last Execution Date
        Open Open Patch Available Patch Available
        2d 20h 55m 1 Robert Joseph Evans 29/Oct/12 18:59
        Patch Available Patch Available Resolved Resolved
        21h 4m 1 Robert Joseph Evans 30/Oct/12 16:04
        Resolved Resolved Closed Closed
        99d 1h 1m 1 Thomas Graves 06/Feb/13 17:05

          People

          • Assignee:
            Robert Joseph Evans
            Reporter:
            Robert Joseph Evans
          • Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development