Commons Net
  1. Commons Net
  2. NET-46

[FTP] retrieveFileStream fails randomly or hangs

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 1.4, 3.1
    • Fix Version/s: 3.2
    • Component/s: FTP
    • Labels:
      None
    • Environment:

      Operating System: Windows XP
      Platform: PC

      Description

      For my application I need a way to get the InputStream of a binary file on a
      FTPServer. What I did was :

      // connect and get ftpFiles as an array
      // for each ftpFile ...

      InputStream is = ftp.retrieveFileStream(ftpFiles[i].getName());

      However, this behaves erratically : sometimes the inputstream is correct and
      sometimes it is null (and the ftpFile exists, no weird name or anything odd
      about it).

      After first blaming my FTPServer (I use GuildFTPd 0.9.9.13) I tried another
      FTPServer (Serv-U 6.1), but this also had the same behavior.

      Then I thought I might have to do with timing. So I tried Thread.sleep(xxx) on a
      couple of locations but to no avail. In a last attempt (was getting pretty
      desperate ) I rewrote my original line and replaced it by this :

      ByteArrayOutputStream out = new ByteArrayOutputStream();
      ftp.retrieveFile(ftpFiles[i].getName(),out);
      InputStream is = new ByteArrayInputStream(out.toByteArray());

      And much to my surprise, it worked like a charm. Tested it a couple of times (on
      both FTPServer products) and works perfectly.

      So I'm guessing something is going wrong in your retrieveFileStream
      implementation. Maybe something worth looking into ? (easiest fix : use the
      ByteArrayOut/InputStream swap ).

      kind regards,

      Dennis

        Activity

        Hide
        Daniel Savarese added a comment -

        retrieveFileStream has been working essentially unchanged since 1997. Keep
        in mind that retrieveFile performs the exact same operation as retrieveFileStream
        to initiate a fiile transfer. If retrieveFileStream is returning null, then
        retrieveFile would return false under the same circumstances. It's
        more likely the issue you've encountered lies elsewhere. If you
        can provide a self-contained program (class with main()) that we can
        compile and run, we can better determine if there is something in FTPClient
        that requires fixing. Thanks.

        Show
        Daniel Savarese added a comment - retrieveFileStream has been working essentially unchanged since 1997. Keep in mind that retrieveFile performs the exact same operation as retrieveFileStream to initiate a fiile transfer. If retrieveFileStream is returning null, then retrieveFile would return false under the same circumstances. It's more likely the issue you've encountered lies elsewhere. If you can provide a self-contained program (class with main()) that we can compile and run, we can better determine if there is something in FTPClient that requires fixing. Thanks.
        Hide
        Anders Mårtensson added a comment -

        I'm experiencing the same problem as Dennis.

        Out of 100 file transfers 2 InputStream's were returned as null.
        And, which was my original problem, on one occation the application hung. See
        the Java thread below.

        Seems like the FTPClient.retrieveFileStream() call now and then returns a null
        InputStream and occationally hangs...

        I'm currently using version 1.3.0 so I'll try Dennis fix with the 1.4.0.

        Regards,
        Anders M

        Current Java thread:
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Object.java:429)
        at org.apache.commons.net.telnet.TelnetInputStream.read(TelnetInputStream.java:339)

        • locked <0x447b0098> (a [I)
          at org.apache.commons.net.telnet.TelnetInputStream.read(TelnetInputStream.java:466)
          at java.io.BufferedInputStream.read1(BufferedInputStream.java:220)
          at java.io.BufferedInputStream.read(BufferedInputStream.java:277)
        • locked <0x447b6270> (a java.io.BufferedInputStream)
          at sun.nio.cs.StreamDecoder$CharsetSD.readBytes(StreamDecoder.java:408)
          at sun.nio.cs.StreamDecoder$CharsetSD.implRead(StreamDecoder.java:450)
          at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:182)
        • locked <0x447b60e0> (a java.io.InputStreamReader)
          at java.io.InputStreamReader.read(InputStreamReader.java:167)
          at java.io.BufferedReader.fill(BufferedReader.java:136)
          at java.io.BufferedReader.readLine(BufferedReader.java:299)
        • locked <0x447b60e0> (a java.io.InputStreamReader)
          at java.io.BufferedReader.readLine(BufferedReader.java:362)
          at org.apache.commons.net.ftp.FTP.__getReply(FTP.java:260)
          at org.apache.commons.net.ftp.FTP.sendCommand(FTP.java:456)
          at org.apache.commons.net.ftp.FTP.sendCommand(FTP.java:484)
          at org.apache.commons.net.ftp.FTPClient.openDataConnection(FTPClient.java:475)
          at org.apache.commons.net.ftp.FTPClient.retrieveFileStream(FTPClient.java:1327)
          at spar_vo.SparFTPConnectionVO.getFile(SparFTPConnectionVO.java:119)
          at spar_uc.SparGetFileUC.performUseCase(SparGetFileUC.java:63)
          at spar.SparApplication.main(SparApplication.java:97)
        Show
        Anders Mårtensson added a comment - I'm experiencing the same problem as Dennis. Out of 100 file transfers 2 InputStream's were returned as null. And, which was my original problem, on one occation the application hung. See the Java thread below. Seems like the FTPClient.retrieveFileStream() call now and then returns a null InputStream and occationally hangs... I'm currently using version 1.3.0 so I'll try Dennis fix with the 1.4.0. Regards, Anders M Current Java thread: at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:429) at org.apache.commons.net.telnet.TelnetInputStream.read(TelnetInputStream.java:339) locked <0x447b0098> (a [I) at org.apache.commons.net.telnet.TelnetInputStream.read(TelnetInputStream.java:466) at java.io.BufferedInputStream.read1(BufferedInputStream.java:220) at java.io.BufferedInputStream.read(BufferedInputStream.java:277) locked <0x447b6270> (a java.io.BufferedInputStream) at sun.nio.cs.StreamDecoder$CharsetSD.readBytes(StreamDecoder.java:408) at sun.nio.cs.StreamDecoder$CharsetSD.implRead(StreamDecoder.java:450) at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:182) locked <0x447b60e0> (a java.io.InputStreamReader) at java.io.InputStreamReader.read(InputStreamReader.java:167) at java.io.BufferedReader.fill(BufferedReader.java:136) at java.io.BufferedReader.readLine(BufferedReader.java:299) locked <0x447b60e0> (a java.io.InputStreamReader) at java.io.BufferedReader.readLine(BufferedReader.java:362) at org.apache.commons.net.ftp.FTP.__getReply(FTP.java:260) at org.apache.commons.net.ftp.FTP.sendCommand(FTP.java:456) at org.apache.commons.net.ftp.FTP.sendCommand(FTP.java:484) at org.apache.commons.net.ftp.FTPClient. openDataConnection (FTPClient.java:475) at org.apache.commons.net.ftp.FTPClient.retrieveFileStream(FTPClient.java:1327) at spar_vo.SparFTPConnectionVO.getFile(SparFTPConnectionVO.java:119) at spar_uc.SparGetFileUC.performUseCase(SparGetFileUC.java:63) at spar.SparApplication.main(SparApplication.java:97)
        Hide
        Sebb added a comment -

        No recent reports.

        If this occurs with the NET 2.2 or later, please re-open with details

        Show
        Sebb added a comment - No recent reports. If this occurs with the NET 2.2 or later, please re-open with details
        Hide
        Dheeraj V.S. added a comment -

        I faced a similar hang in retrieveFileStream() when using commons-net-3.1. And the callstack is essentially similar to that posted by Anders Mårtensson:

        libcore.io.Posix recvfromBytes Posix.java -2 true
        libcore.io.Posix recvfrom Posix.java 131 false
        libcore.io.BlockGuardOs recvfrom BlockGuardOs.java 164 false
        libcore.io.IoBridge recvfrom IoBridge.java 513 false
        java.net.PlainSocketImpl read PlainSocketImpl.java 488 false
        java.net.PlainSocketImpl access$000 PlainSocketImpl.java 46 false
        java.net.PlainSocketImpl$PlainSocketInputStream read PlainSocketImpl.java 240 false
        java.io.InputStreamReader read InputStreamReader.java 244 false
        java.io.BufferedReader fillBuf BufferedReader.java 130 false
        java.io.BufferedReader read BufferedReader.java 238 false
        org.apache.commons.net.io.CRLFLineReader readLine CRLFLineReader.java 58 false
        org.apache.commons.net.ftp.FTP __getReply FTP.java 310 false
        org.apache.commons.net.ftp.FTP __getReply FTP.java 290 false
        org.apache.commons.net.ftp.FTP sendCommand FTP.java 479 false
        org.apache.commons.net.ftp.FTPClient openDataConnection FTPClient.java 769 false
        org.apache.commons.net.ftp.FTPClient _retrieveFileStream FTPClient.java 1747 false
        org.apache.commons.net.ftp.FTPClient retrieveFileStream FTPClient.java 1739 false

        My guess is that in _openDataConnection_(), the following line to set the dataTimeOut should have been set before calling sendCommand():
        socket.setSoTimeout(__dataTimeout);

        Otherwise, the client would be waiting indefinitely for the server's reply to the RETR command.

        FWIW, I'm running it on Android 4.x

        Show
        Dheeraj V.S. added a comment - I faced a similar hang in retrieveFileStream() when using commons-net-3.1. And the callstack is essentially similar to that posted by Anders Mårtensson : libcore.io.Posix recvfromBytes Posix.java -2 true libcore.io.Posix recvfrom Posix.java 131 false libcore.io.BlockGuardOs recvfrom BlockGuardOs.java 164 false libcore.io.IoBridge recvfrom IoBridge.java 513 false java.net.PlainSocketImpl read PlainSocketImpl.java 488 false java.net.PlainSocketImpl access$000 PlainSocketImpl.java 46 false java.net.PlainSocketImpl$PlainSocketInputStream read PlainSocketImpl.java 240 false java.io.InputStreamReader read InputStreamReader.java 244 false java.io.BufferedReader fillBuf BufferedReader.java 130 false java.io.BufferedReader read BufferedReader.java 238 false org.apache.commons.net.io.CRLFLineReader readLine CRLFLineReader.java 58 false org.apache.commons.net.ftp.FTP __getReply FTP.java 310 false org.apache.commons.net.ftp.FTP __getReply FTP.java 290 false org.apache.commons.net.ftp.FTP sendCommand FTP.java 479 false org.apache.commons.net.ftp.FTPClient openDataConnection FTPClient.java 769 false org.apache.commons.net.ftp.FTPClient _retrieveFileStream FTPClient.java 1747 false org.apache.commons.net.ftp.FTPClient retrieveFileStream FTPClient.java 1739 false My guess is that in _ openDataConnection _(), the following line to set the dataTimeOut should have been set before calling sendCommand(): socket.setSoTimeout(__dataTimeout); Otherwise, the client would be waiting indefinitely for the server's reply to the RETR command. FWIW, I'm running it on Android 4.x
        Hide
        Sebb added a comment -

        Good catch.

        In the case of ACTIVE_LOCAL_DATA_CONNECTION_MODE, the code uses the __dataTimeout to protect against the server.accept() timing out.

        However otherwise (i.e. PASSIVE_LOCAL_DATA_CONNECTION_MODE), the socket timeout is not set until the end of the method, after several commands have been issued.

        The code has been changed a bit since 3.1, however there are still some paths where the newly created socket is used before the timeout has been set up.

        Show
        Sebb added a comment - Good catch. In the case of ACTIVE_LOCAL_DATA_CONNECTION_MODE, the code uses the __dataTimeout to protect against the server.accept() timing out. However otherwise (i.e. PASSIVE_LOCAL_DATA_CONNECTION_MODE), the socket timeout is not set until the end of the method, after several commands have been issued. The code has been changed a bit since 3.1, however there are still some paths where the newly created socket is used before the timeout has been set up.
        Hide
        Sebb added a comment -

        URL: http://svn.apache.org/viewvc?rev=1413545&view=rev
        Log:
        NET-46 retrieveFileStream fails randomly or hangs

        Modified:
        commons/proper/net/trunk/src/changes/changes.xml
        commons/proper/net/trunk/src/main/java/org/apache/commons/net/ftp/FTPClient.java

        Show
        Sebb added a comment - URL: http://svn.apache.org/viewvc?rev=1413545&view=rev Log: NET-46 retrieveFileStream fails randomly or hangs Modified: commons/proper/net/trunk/src/changes/changes.xml commons/proper/net/trunk/src/main/java/org/apache/commons/net/ftp/FTPClient.java

          People

          • Assignee:
            Unassigned
            Reporter:
            Dennis Meerveld
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development