Commons VFS
  1. Commons VFS
  2. VFS-200

[SFTP] Failure when files are very large

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: 1.0
    • Fix Version/s: 2.1
    • Labels:
      None
    • Environment:

      Java EE 5

      Description

      We only get this error with very large files. We are using the latest release of JSch (0.1.37).

      org.apache.commons.vfs.FileSystemException: Could not read file "sftp://[useridRemoved]@[ipAddrRemoved]/returned/ccp_allusers.20080219010439.01.noe.clean_C_.txt".
      at org.apache.commons.vfs.provider.AbstractFileObject.getInputStream(AbstractFileObject.java:1052)
      at org.apache.commons.vfs.provider.DefaultFileContent.getInputStream(DefaultFileContent.java:312)
      at com.ibm.account.batch.jobs.gecs.SecureFTPHelper.retrieveGECSResultsFiles(SecureFTPHelper.java:155)
      Caused by: 4:
      at com.jcraft.jsch.ChannelSftp._get(ChannelSftp.java(Compiled Code))
      at com.jcraft.jsch.ChannelSftp.get(ChannelSftp.java:855)
      at com.jcraft.jsch.ChannelSftp.get(ChannelSftp.java:834)
      at org.apache.commons.vfs.provider.sftp.SftpFileObject.doGetInputStream(SftpFileObject.java:362)
      at org.apache.commons.vfs.provider.AbstractFileObject.getInputStream(AbstractFileObject.java:1048)
      ... 14 more
      Caused by: java.io.IOException: inputstream is closed
      at com.jcraft.jsch.ChannelSftp.fill(ChannelSftp.java(Inlined Compiled Code))
      at com.jcraft.jsch.ChannelSftp.header(ChannelSftp.java(Compiled Code))
      ... 19 more

        Issue Links

          Activity

          Hide
          Michael Manion added a comment -

          Maybe it would help if I provide more information on this issue. We are using Commons-VFS to login to an sftp server and download very 2 or 3 large files (several hundred MB each) every week. The download fails consistently with the error message seen in my original problem description. We have tested with smaller files and do not see the error. We are using the official 1.0 version of the Commons-VFS jarfile, along with the latest JSch jarfile (0.1.37).

          We tried breaking one of the large files up into 20MB "chunks" on the remote sftp server. The result was that the first 4 of these 20MB files would download correctly, and then we get the same error while downloading the 5th one. Again, this is a consistent result.

          Show
          Michael Manion added a comment - Maybe it would help if I provide more information on this issue. We are using Commons-VFS to login to an sftp server and download very 2 or 3 large files (several hundred MB each) every week. The download fails consistently with the error message seen in my original problem description. We have tested with smaller files and do not see the error. We are using the official 1.0 version of the Commons-VFS jarfile, along with the latest JSch jarfile (0.1.37). We tried breaking one of the large files up into 20MB "chunks" on the remote sftp server. The result was that the first 4 of these 20MB files would download correctly, and then we get the same error while downloading the 5th one. Again, this is a consistent result.
          Hide
          Trey Massingill added a comment -

          I think this is happening because the sftp provider for VFS is calling JSch methods that read the entire contents of the remote file into memory before providing the caller an input stream. In my situation, this caused an out of memory error.

          The fix for this is for the VFS sftp provider to call the JSch method that return InputStream (and OutputStream) objects.

          Show
          Trey Massingill added a comment - I think this is happening because the sftp provider for VFS is calling JSch methods that read the entire contents of the remote file into memory before providing the caller an input stream. In my situation, this caused an out of memory error. The fix for this is for the VFS sftp provider to call the JSch method that return InputStream (and OutputStream) objects.
          Hide
          Trey Massingill added a comment -

          I think these issues are probably related.

          Show
          Trey Massingill added a comment - I think these issues are probably related.
          Hide
          Ralph Goers added a comment -

          I see comments in SftpFileObject.getInputStream related to this. Unfortunately, I don't have a server to test this nor am I all that familiar with JSch. If someone could provide instructions on how to set up a functional test for this, provide a test that can be included to test it and then provide a patch I'll gladly apply it.

          Show
          Ralph Goers added a comment - I see comments in SftpFileObject.getInputStream related to this. Unfortunately, I don't have a server to test this nor am I all that familiar with JSch. If someone could provide instructions on how to set up a functional test for this, provide a test that can be included to test it and then provide a patch I'll gladly apply it.
          Hide
          Gary Gregory added a comment -

          Yes, this was addressed by VFS-410[SFTP]: SftpFileObject getInputStream(long) reads the whole file into memory.

          It is in SVN as of revision 1328346.

          Show
          Gary Gregory added a comment - Yes, this was addressed by VFS-410 [SFTP] : SftpFileObject getInputStream(long) reads the whole file into memory. It is in SVN as of revision 1328346.
          Hide
          Gary Gregory added a comment -

          In SVN as of revision 1328346.

          Show
          Gary Gregory added a comment - In SVN as of revision 1328346.

            People

            • Assignee:
              Unassigned
              Reporter:
              Michael Manion
            • Votes:
              3 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development