Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-3373

FileContext HDFS implementation can leak socket caches

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.23.3, 2.0.0-alpha, 3.0.0
    • Fix Version/s: 0.23.4, 2.0.3-alpha
    • Component/s: hdfs-client
    • Labels:
      None

      Description

      As noted by Nicholas in HDFS-3359, FileContext doesn't have a close() method, and thus never calls DFSClient.close(). This means that, until finalizers run, DFSClient will hold on to its SocketCache object and potentially have a lot of outstanding sockets/fds held on to.

      1. HDFS-3373.branch23.patch
        16 kB
        John George
      2. HDFS-3373.trunk.patch.4
        24 kB
        John George
      3. HDFS-3373.trunk.patch.3
        24 kB
        John George
      4. HDFS-3373.trunk.patch.3
        24 kB
        John George
      5. HDFS-3373.trunk.patch.2
        24 kB
        John George
      6. HDFS-3373.trunk.patch.1
        24 kB
        John George
      7. HDFS-3373.trunk.patch
        23 kB
        John George
      8. HDFS-3373.branch-23.patch
        15 kB
        John George

        Issue Links

          Activity

          Hide
          Todd Lipcon added a comment -

          Nicholas had some ideas about this in HDFS-3359. Quoting here:

          I think we should either

          • change it from per DFSClient to per DFSInputStream; or
          • change it to a global static cache (like what we did for LeaseRenewer.)

          Brainstorming: maybe another option would be to have a timer thread which periodically wakes up and scans SocketCaches for old sockets. If any socket has been sitting around longer than a minute or two, we can close it proactively.

          Show
          Todd Lipcon added a comment - Nicholas had some ideas about this in HDFS-3359 . Quoting here: I think we should either change it from per DFSClient to per DFSInputStream; or change it to a global static cache (like what we did for LeaseRenewer.) Brainstorming: maybe another option would be to have a timer thread which periodically wakes up and scans SocketCaches for old sockets. If any socket has been sitting around longer than a minute or two, we can close it proactively.
          Hide
          Suresh Srinivas added a comment -

          Is this a blocker for 2.0 given HDFS-3359 has been committed to 2.0?

          Show
          Suresh Srinivas added a comment - Is this a blocker for 2.0 given HDFS-3359 has been committed to 2.0?
          Hide
          Todd Lipcon added a comment -

          No, I don't think so, because we also now have a fix on the DN side which closes the sockets. So it might result in TIME_WAIT or FIN_WAIT sockets on the client, but shouldn't impact DNs anymore. Unless someone has seen this to be a production issue with 2.0, it shouldn't be considered blocker.

          Show
          Todd Lipcon added a comment - No, I don't think so, because we also now have a fix on the DN side which closes the sockets. So it might result in TIME_WAIT or FIN_WAIT sockets on the client, but shouldn't impact DNs anymore. Unless someone has seen this to be a production issue with 2.0, it shouldn't be considered blocker.
          Hide
          Daryn Sharp added a comment -

          On a related tangent, a similar situation causes the NMs to leak sockets due to unclosed DistributedFileSystems. Each one holds a DFSClient that holds a cache with at least 1 CLOSE_WAIT socket. The socket gets closed if/when the DFSClient looks for a cached socket, but it gets replaced with another CLOSE_WAIT socket.

          With security enabled, a NM leaks 1 socket/user. With security disabled, a NM will leak 1 socket/job since the UGI always appears different which causes multiple instances of the same dfs to pile up in the fs cache.

          The UGI/fs-cache is a separate issue, but there should probably be something like another thread that scans the cached sockets to remove closed ones. Otherwise FileContext will be ill-suited for a daemon that needs to create more than one context since it too will leak sockets.

          Show
          Daryn Sharp added a comment - On a related tangent, a similar situation causes the NMs to leak sockets due to unclosed DistributedFileSystems . Each one holds a DFSClient that holds a cache with at least 1 CLOSE_WAIT socket. The socket gets closed if/when the DFSClient looks for a cached socket, but it gets replaced with another CLOSE_WAIT socket. With security enabled, a NM leaks 1 socket/user. With security disabled, a NM will leak 1 socket/job since the UGI always appears different which causes multiple instances of the same dfs to pile up in the fs cache. The UGI/fs-cache is a separate issue, but there should probably be something like another thread that scans the cached sockets to remove closed ones. Otherwise FileContext will be ill-suited for a daemon that needs to create more than one context since it too will leak sockets.
          Hide
          John George added a comment -

          Attaching a branch-0.23 based patch. I will upload a trunk/branch2 patch in the next day or two.

          Show
          John George added a comment - Attaching a branch-0.23 based patch. I will upload a trunk/branch2 patch in the next day or two.
          Hide
          John George added a comment -

          The approach in this patch is based on the suggestions from Nicholas and Todd:

          • a global static cache to store socket caches
          • a thread to expire cache entries in the cache
          Show
          John George added a comment - The approach in this patch is based on the suggestions from Nicholas and Todd: a global static cache to store socket caches a thread to expire cache entries in the cache
          Hide
          Daryn Sharp added a comment -

          A related issue we need to consider, although tangental is that inability to close doesn't give a fs a chance to cleanup. Ex. closing streams and deleting temp files. I think there might be another jira for that?

          Show
          Daryn Sharp added a comment - A related issue we need to consider, although tangental is that inability to close doesn't give a fs a chance to cleanup. Ex. closing streams and deleting temp files. I think there might be another jira for that?
          Hide
          John George added a comment -

          Daryn,
          I am not able to follow - Inability to close fs is not related to this JIRA because this JIRA is related to caching sockets when an input stream is closed. If I misunderstood your comment, could you elaborate a little more on what you meant?

          Show
          John George added a comment - Daryn, I am not able to follow - Inability to close fs is not related to this JIRA because this JIRA is related to caching sockets when an input stream is closed. If I misunderstood your comment, could you elaborate a little more on what you meant?
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12542653/HDFS-3373.trunk.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          -1 findbugs. The patch appears to introduce 2 new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs:

          org.apache.hadoop.hdfs.TestConnCache
          org.apache.hadoop.hdfs.TestHftpDelegationToken

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/3101//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/3101//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3101//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12542653/HDFS-3373.trunk.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified test files. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. -1 findbugs. The patch appears to introduce 2 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.TestConnCache org.apache.hadoop.hdfs.TestHftpDelegationToken +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/3101//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/3101//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3101//console This message is automatically generated.
          Hide
          John George added a comment -

          TestConnCache failure is related to this JIRA. I had moved testDisableCache() from that test to another test file because now it is not possible to change cache config per DFS.

          TestHftpDelegationToken is unrelated to this patch and has been failing in other builds as well.

          Attaching a patch with testDisableCache() removed from TestConnCache to a new file

          Show
          John George added a comment - TestConnCache failure is related to this JIRA. I had moved testDisableCache() from that test to another test file because now it is not possible to change cache config per DFS. TestHftpDelegationToken is unrelated to this patch and has been failing in other builds as well. Attaching a patch with testDisableCache() removed from TestConnCache to a new file
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12542795/HDFS-3373.trunk.patch.1
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          -1 findbugs. The patch appears to introduce 2 new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs:

          org.apache.hadoop.hdfs.TestHftpDelegationToken

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/3110//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/3110//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3110//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12542795/HDFS-3373.trunk.patch.1 against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified test files. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. -1 findbugs. The patch appears to introduce 2 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.TestHftpDelegationToken +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/3110//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/3110//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3110//console This message is automatically generated.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          Hi John, thanks for working on this. Some comments on the patch:

          • In SocketCache.getInstance(..),
            • If SocketCache is already initialized (i.e. isInitedOnce == true), then the parameters c and e will be igored silently. Would it be a problem?
            • If it throws IllegalStateException, should isInitedOnce not be set to true?
            • typo: "initialzied" => "initialized"
          • In SocketCache.evictExpired(..),
            • remove "throws IOException" since it does not throw it.
            • remove synchronized(this) since it is already synchronized.
          • In SocketCache, add createTime to SocketAndStreams and remove SocketProp.
          • In SocketCache.put(..), move "Preconditions.checkNotNull(sock);" to the very beginning.
          Show
          Tsz Wo Nicholas Sze added a comment - Hi John, thanks for working on this. Some comments on the patch: In SocketCache.getInstance(..), If SocketCache is already initialized (i.e. isInitedOnce == true), then the parameters c and e will be igored silently. Would it be a problem? If it throws IllegalStateException, should isInitedOnce not be set to true? typo: "initialzied" => "initialized" In SocketCache.evictExpired(..), remove "throws IOException" since it does not throw it. remove synchronized(this) since it is already synchronized. In SocketCache, add createTime to SocketAndStreams and remove SocketProp. In SocketCache.put(..), move "Preconditions.checkNotNull(sock);" to the very beginning.
          Hide
          John George added a comment -

          Thanks for looking into the patch Nicholas.

          • In SocketCache.getInstance(..),
            • If SocketCache is already initialized (i.e. isInitedOnce == true), then the parameters c and e will be igored silently. Would it be a problem?

          This was done on purpose to avoid complications. I thought it would be better to keep it simple and allow only one valid setting of c and e.

            • If it throws IllegalStateException, should isInitedOnce not be set to true?
            • typo: "initialzied" => "initialized"
          • In SocketCache.evictExpired(..),
            • remove "throws IOException" since it does not throw it.
            • remove synchronized(this) since it is already synchronized.
          • In SocketCache, add createTime to SocketAndStreams and remove SocketProp.
          • In SocketCache.put(..), move "Preconditions.checkNotNull(sock);" to the very beginning.

          Thanks for these comments - will upload a patch incorporating these soon.

          Show
          John George added a comment - Thanks for looking into the patch Nicholas. In SocketCache.getInstance(..), If SocketCache is already initialized (i.e. isInitedOnce == true), then the parameters c and e will be igored silently. Would it be a problem? This was done on purpose to avoid complications. I thought it would be better to keep it simple and allow only one valid setting of c and e. If it throws IllegalStateException, should isInitedOnce not be set to true? typo: "initialzied" => "initialized" In SocketCache.evictExpired(..), remove "throws IOException" since it does not throw it. remove synchronized(this) since it is already synchronized. In SocketCache, add createTime to SocketAndStreams and remove SocketProp. In SocketCache.put(..), move "Preconditions.checkNotNull(sock);" to the very beginning. Thanks for these comments - will upload a patch incorporating these soon.
          Hide
          John George added a comment -
            • If SocketCache is already initialized (i.e. isInitedOnce == true), then the parameters c and e will be igored silently. Would it be a problem?

          I suppose you actually meant if we should LOG a message in cases where someone tries to change the value after it has been set once? I agree that it is a good idea.

          Show
          John George added a comment - If SocketCache is already initialized (i.e. isInitedOnce == true), then the parameters c and e will be igored silently. Would it be a problem? I suppose you actually meant if we should LOG a message in cases where someone tries to change the value after it has been set once? I agree that it is a good idea.
          Hide
          John George added a comment -

          Attaching a new patch with Nicholas's comment taken into consideration

          Show
          John George added a comment - Attaching a new patch with Nicholas's comment taken into consideration
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12545606/HDFS-3373.trunk.patch.2
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/3202//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/3202//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3202//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12545606/HDFS-3373.trunk.patch.2 against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified test files. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/3202//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/3202//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3202//console This message is automatically generated.
          Hide
          John George added a comment -

          Canceling patch to fix findbugs issue.

          Show
          John George added a comment - Canceling patch to fix findbugs issue.
          Hide
          John George added a comment -

          Cancelling patch and uploading another file to trigger patch test run

          Show
          John George added a comment - Cancelling patch and uploading another file to trigger patch test run
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12546297/HDFS-3373.trunk.patch.3
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/3227//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3227//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12546297/HDFS-3373.trunk.patch.3 against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified test files. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/3227//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3227//console This message is automatically generated.
          Hide
          Tsz Wo Nicholas Sze added a comment -
          +  private synchronized void evictExpired(long expiryPeriod) 
          +      throws IOException {
          

          evictExpired(..) still has "throws IOException" but IOException is never thrown in the code. Patch looks good other than that.

          Show
          Tsz Wo Nicholas Sze added a comment - + private synchronized void evictExpired( long expiryPeriod) + throws IOException { evictExpired(..) still has "throws IOException" but IOException is never thrown in the code. Patch looks good other than that.
          Hide
          John George added a comment -

          Thanks Nicholas and sorry for overlooking that review comment in the previous patches.

          Attaching a new patch that removes the IOException.

          Show
          John George added a comment - Thanks Nicholas and sorry for overlooking that review comment in the previous patches. Attaching a new patch that removes the IOException.
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12546571/HDFS-3373.trunk.patch.4
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/3232//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3232//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12546571/HDFS-3373.trunk.patch.4 against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 3 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/3232//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3232//console This message is automatically generated.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          +1 patch looks good.

          Show
          Tsz Wo Nicholas Sze added a comment - +1 patch looks good.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          I have committed this. Thanks, John!

          Show
          Tsz Wo Nicholas Sze added a comment - I have committed this. Thanks, John!
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk-Commit #2775 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2775/)
          HDFS-3373. Change DFSClient input stream socket cache to global static and add a thread to cleanup expired cache entries. Contributed by John George (Revision 1390466)

          Result = SUCCESS
          szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1390466
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/SocketCache.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestConnCache.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDistributedFileSystem.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestSocketCache.java
          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #2775 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2775/ ) HDFS-3373 . Change DFSClient input stream socket cache to global static and add a thread to cleanup expired cache entries. Contributed by John George (Revision 1390466) Result = SUCCESS szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1390466 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/SocketCache.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestConnCache.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDistributedFileSystem.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestSocketCache.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #2838 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2838/)
          HDFS-3373. Change DFSClient input stream socket cache to global static and add a thread to cleanup expired cache entries. Contributed by John George (Revision 1390466)

          Result = SUCCESS
          szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1390466
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/SocketCache.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestConnCache.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDistributedFileSystem.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestSocketCache.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #2838 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2838/ ) HDFS-3373 . Change DFSClient input stream socket cache to global static and add a thread to cleanup expired cache entries. Contributed by John George (Revision 1390466) Result = SUCCESS szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1390466 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/SocketCache.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestConnCache.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDistributedFileSystem.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestSocketCache.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk-Commit #2797 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2797/)
          HDFS-3373. Change DFSClient input stream socket cache to global static and add a thread to cleanup expired cache entries. Contributed by John George (Revision 1390466)

          Result = FAILURE
          szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1390466
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/SocketCache.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestConnCache.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDistributedFileSystem.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestSocketCache.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #2797 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2797/ ) HDFS-3373 . Change DFSClient input stream socket cache to global static and add a thread to cleanup expired cache entries. Contributed by John George (Revision 1390466) Result = FAILURE szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1390466 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/SocketCache.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestConnCache.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDistributedFileSystem.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestSocketCache.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #1178 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1178/)
          HDFS-3373. Change DFSClient input stream socket cache to global static and add a thread to cleanup expired cache entries. Contributed by John George (Revision 1390466)

          Result = SUCCESS
          szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1390466
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/SocketCache.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestConnCache.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDistributedFileSystem.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestSocketCache.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #1178 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1178/ ) HDFS-3373 . Change DFSClient input stream socket cache to global static and add a thread to cleanup expired cache entries. Contributed by John George (Revision 1390466) Result = SUCCESS szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1390466 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/SocketCache.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestConnCache.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDistributedFileSystem.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestSocketCache.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #1209 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1209/)
          HDFS-3373. Change DFSClient input stream socket cache to global static and add a thread to cleanup expired cache entries. Contributed by John George (Revision 1390466)

          Result = SUCCESS
          szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1390466
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/SocketCache.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestConnCache.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDistributedFileSystem.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestSocketCache.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1209 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1209/ ) HDFS-3373 . Change DFSClient input stream socket cache to global static and add a thread to cleanup expired cache entries. Contributed by John George (Revision 1390466) Result = SUCCESS szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1390466 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/SocketCache.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestConnCache.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDistributedFileSystem.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestSocketCache.java
          Hide
          John George added a comment -

          Thanks Nicholas.

          Show
          John George added a comment - Thanks Nicholas.
          Hide
          John George added a comment -

          Attaching a branch 23 patch based on Nicholas's review for the patch for trunk

          Show
          John George added a comment - Attaching a branch 23 patch based on Nicholas's review for the patch for trunk
          Hide
          John George added a comment -

          Reopening for branch 23

          Show
          John George added a comment - Reopening for branch 23
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12546899/HDFS-3373.branch23.patch
          against trunk revision .

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3246//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12546899/HDFS-3373.branch23.patch against trunk revision . -1 patch . The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3246//console This message is automatically generated.
          Hide
          Robert Joseph Evans added a comment -

          The 0.23 patch looks like a fairly straight forward port of the trunk version, but what happened to hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestSocketCache.java?

          Show
          Robert Joseph Evans added a comment - The 0.23 patch looks like a fairly straight forward port of the trunk version, but what happened to hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestSocketCache.java?
          Hide
          John George added a comment -

          In 2.0, setting cache capacity to 0 disabled the cache, this (along with some other changes) was not in 0.23. TestSocketCache was a one test file to test if cache disabling worked. Since, 0.23 does not have that, I did not port the test.

          Show
          John George added a comment - In 2.0, setting cache capacity to 0 disabled the cache, this (along with some other changes) was not in 0.23. TestSocketCache was a one test file to test if cache disabling worked. Since, 0.23 does not have that, I did not port the test.
          Hide
          Robert Joseph Evans added a comment -

          Makes since. Because it is such a straight forward patch I feel OK checking the code in. Thanks for the work John.

          Show
          Robert Joseph Evans added a comment - Makes since. Because it is such a straight forward patch I feel OK checking the code in. Thanks for the work John.
          Hide
          Robert Joseph Evans added a comment -

          I pulled this into branch-0.23 too

          Show
          Robert Joseph Evans added a comment - I pulled this into branch-0.23 too
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-0.23-Build #389 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/389/)
          HDFS-3373. FileContext HDFS implementation can leak socket caches (John George via bobby) (Revision 1391542)

          Result = UNSTABLE
          bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1391542
          Files :

          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/SocketCache.java
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestConnCache.java
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDistributedFileSystem.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Build #389 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/389/ ) HDFS-3373 . FileContext HDFS implementation can leak socket caches (John George via bobby) (Revision 1391542) Result = UNSTABLE bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1391542 Files : /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/SocketCache.java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestConnCache.java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDistributedFileSystem.java

            People

            • Assignee:
              John George
              Reporter:
              Todd Lipcon
            • Votes:
              0 Vote for this issue
              Watchers:
              15 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development