FtpServer
  1. FtpServer
  2. FTPSERVER-287

NLST: Implementation only supports listing files in working directory [patch provided]

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Won't Fix
    • Affects Version/s: 1.1.0
    • Fix Version/s: 1.0.2, 1.1.0
    • Component/s: Core
    • Labels:
      None
    • Environment:
      Fedora 10-64bit and RH 5.2-64bit, Java 1.6.0_12-64

      Description

      The NLST formatter, as implemented on trunk is insufficient to handle any request other than a file within in the current working directory. Some examples:

      ftp> passive
      Passive mode on.
      ftp> nlist /directory/file.txt
      227 Entering Passive Mode (127,0,0,1,179,241)
      150 File status okay; about to open data connection.
      file.txt
      226 Closing data connection.

      Other FTP servers return the following:

      ftp> passive
      Passive mode on.
      ftp> nlist /directory/file.txt
      227 Entering Passive Mode (127,0,0,1,179,241)
      150 File status okay; about to open data connection.
      /directory/file.txt
      226 Closing data connection.

      Upon investigating, I found that the formatter will not handle absolute file requests, parent directory request or non-absolute child directory requests. It does not error, it just doesn't give useful output.

      I've modified the code to handle the cases that I could come up with, but there may be other situations that need to be covered. I'm not an expert on the FTP specification (but what I could find was not impressive), so there many be additional cases that need to be covered.

      Please consider the attached patch, with accompanying test cases

      1. nlst.patch
        43 kB
        Dennis Keller

        Activity

        Hide
        Dennis Keller added a comment -

        FTPServer NLST appears to be working as designed.

        Show
        Dennis Keller added a comment - FTPServer NLST appears to be working as designed.
        Hide
        Dennis Keller added a comment -

        Interesting discussion. Thanks to all for the evaluation. It seems that there is much disparity amongst the various FTP server implementations and that the logic in the Apache FTP server is as correct as the other server's implementations I suppose this is what happens when specifications are vague!

        Your suggestions not to rely on the output of NLIST are good and we will change our code so that it does not use the output of NLST as input to other requests.

        Show
        Dennis Keller added a comment - Interesting discussion. Thanks to all for the evaluation. It seems that there is much disparity amongst the various FTP server implementations and that the logic in the Apache FTP server is as correct as the other server's implementations I suppose this is what happens when specifications are vague! Your suggestions not to rely on the output of NLIST are good and we will change our code so that it does not use the output of NLST as input to other requests.
        Hide
        David Latorre added a comment -

        By the way, PureFTPD and vsFTPD in Linux also seem to behave as described ( I didn't check the $HOME option though).

        Still I also think that clients shouldn't rely on the output of the FTP server ... since you're expecting that the output filename be equal to the nlist argument , why don't you use it in your program?

        With your patch, what's the behaviour you get when you list a directory ? For example: NLIST /myDir1/mydDir2

        Would return just the filenames ? or results of the kind: /myDir1/myDir2/file.txt ?

        " ls" and Pure-FTPD would return just the filenames ... so it is not very useful to have NLIST return the full paths only sometimes.

        Show
        David Latorre added a comment - By the way, PureFTPD and vsFTPD in Linux also seem to behave as described ( I didn't check the $HOME option though). Still I also think that clients shouldn't rely on the output of the FTP server ... since you're expecting that the output filename be equal to the nlist argument , why don't you use it in your program? With your patch, what's the behaviour you get when you list a directory ? For example: NLIST /myDir1/mydDir2 Would return just the filenames ? or results of the kind: /myDir1/myDir2/file.txt ? " ls" and Pure-FTPD would return just the filenames ... so it is not very useful to have NLIST return the full paths only sometimes.
        Hide
        David Latorre added a comment -

        By Niklas Gustavsson:

        One addition to this list, FileZilla works as described by Dennis.

        Another one, proftpd prints full path, but does not support parent
        directories (".." seems to be ignored). Does support home directory
        paths, but translated them to absolute paths (from the root of the
        file system).

        All of this seems to be a mess, not sure that we conclude that we do
        the wrong, nor the right thing at the moment. At least clients can not
        assume anything and thus probably have to find the file names (without
        the path) in whatever gets returned. Of course, the client knows the
        path already, since it sent it.

        Dennis, could you maybe describe some further on why you need to full
        path and how your client interoperates with servers that do not full
        support this (like proftpd which is a major player in this area).

        Show
        David Latorre added a comment - By Niklas Gustavsson: One addition to this list, FileZilla works as described by Dennis. Another one, proftpd prints full path, but does not support parent directories (".." seems to be ignored). Does support home directory paths, but translated them to absolute paths (from the root of the file system). All of this seems to be a mess, not sure that we conclude that we do the wrong, nor the right thing at the moment. At least clients can not assume anything and thus probably have to find the file names (without the path) in whatever gets returned. Of course, the client knows the path already, since it sent it. Dennis, could you maybe describe some further on why you need to full path and how your client interoperates with servers that do not full support this (like proftpd which is a major player in this area).
        Hide
        David Latorre added a comment -

        Comments by Sai in the mailing list:

        I would like to say that the results you have indicated only come from FTP
        servers that actually run the UNIX's ls command when a NLST command is
        received. Other servers probably adhere to the RFC by just returning names.

        Here are a few tests I did with various FTP servers:

        Results from Netscape's anonymous FTP site did match up with what Dennis
        described. (ftp.netscape.com)

        Results from GlobalScape's anonymous FTP site always returned just names. (
        ftp.globalscape.com)

        Results from an AS/400 FTP server (IBM's System i) always returned just
        names. (Private)

        Results from IPSwitch's WS_FTP server always returned just names. (
        ftp.ipswitch.com)

        Results from Microsoft's anonymous web site running MS FTP service also
        retruns just names (ftp.microsoft.com)

        Hopefully this might help making a decision.

        Sai Pullabhotla
        www.jMethods.com

        Show
        David Latorre added a comment - Comments by Sai in the mailing list: I would like to say that the results you have indicated only come from FTP servers that actually run the UNIX's ls command when a NLST command is received. Other servers probably adhere to the RFC by just returning names. Here are a few tests I did with various FTP servers: Results from Netscape's anonymous FTP site did match up with what Dennis described. (ftp.netscape.com) Results from GlobalScape's anonymous FTP site always returned just names. ( ftp.globalscape.com) Results from an AS/400 FTP server (IBM's System i) always returned just names. (Private) Results from IPSwitch's WS_FTP server always returned just names. ( ftp.ipswitch.com) Results from Microsoft's anonymous web site running MS FTP service also retruns just names (ftp.microsoft.com) Hopefully this might help making a decision. Sai Pullabhotla www.jMethods.com
        Hide
        Dennis Keller added a comment -

        Hi Niklas - I suppose my comment would be that regardless of the text of the RFC, if you compare the Apache FTPServer response against that of other FTP servers, you will see the difference (they return paths when the request includes a path).

        I included an example in my initial posting. I suppose you could interpret the spec literally, however I urge to you test the cases I've provided using other ftp servers. I believe the intent of the RFC is to behave similarly to a list command on a UNIX machine. If you include the path in a list command, the response will include the target path. The RFC says "... return a stream of names of files...", which does not exclude the path, if the file was requested with a path. For example:

        If I execute "nlst file.txt", I expect a response of "file.txt"
        If I execute "nlst directory/file.txt", I expect a response of "directory/file.txt"
        If I execute "nlst /directory/file.txt", I expect a response of "/directory/file.txt"
        If I execute "nlst ../parentdir/subdir/file.txt", I expect a response of "../parentdir/subdir/file.txt"

        This is how other FTP servers work and this is how unix works. The RFC is vague and open to interpretation, therefore we need to look to other implementations to find the consensus implementation. As it stands now, we are unable to use the apache ftp server (before the patch) because of the significant implementation difference.

        Note that my patch may not be the best solution, however there is an implementation issue now. So at the very least the test cases should be included and the implementation constructed around them. I will provide more examples if you require.

        Show
        Dennis Keller added a comment - Hi Niklas - I suppose my comment would be that regardless of the text of the RFC, if you compare the Apache FTPServer response against that of other FTP servers, you will see the difference (they return paths when the request includes a path). I included an example in my initial posting. I suppose you could interpret the spec literally, however I urge to you test the cases I've provided using other ftp servers. I believe the intent of the RFC is to behave similarly to a list command on a UNIX machine. If you include the path in a list command, the response will include the target path. The RFC says "... return a stream of names of files...", which does not exclude the path, if the file was requested with a path. For example: If I execute "nlst file.txt", I expect a response of "file.txt" If I execute "nlst directory/file.txt", I expect a response of "directory/file.txt" If I execute "nlst /directory/file.txt", I expect a response of "/directory/file.txt" If I execute "nlst ../parentdir/subdir/file.txt", I expect a response of "../parentdir/subdir/file.txt" This is how other FTP servers work and this is how unix works. The RFC is vague and open to interpretation, therefore we need to look to other implementations to find the consensus implementation. As it stands now, we are unable to use the apache ftp server (before the patch) because of the significant implementation difference. Note that my patch may not be the best solution, however there is an implementation issue now. So at the very least the test cases should be included and the implementation constructed around them. I will provide more examples if you require.
        Hide
        Niklas Gustavsson added a comment -

        I've worked some on this and wrote some test cases (I had some troubles following along in the provided patch). Comparing with your examples above, we seem to return the correct reply for whatever path I throw at it. However, return only the file name, not the full path. The RFC says "The server will return a stream of names of files and no other information." which seem like we might be doing the correct (or at least acceptable) thing when returning only the file names.

        The test case has been added in trunk in core/src/test/java/org/apache/ftpserver/clienttests/NLSTTest.java.

        I'm assuming I'm missing something so please help me along

        Show
        Niklas Gustavsson added a comment - I've worked some on this and wrote some test cases (I had some troubles following along in the provided patch). Comparing with your examples above, we seem to return the correct reply for whatever path I throw at it. However, return only the file name, not the full path. The RFC says "The server will return a stream of names of files and no other information." which seem like we might be doing the correct (or at least acceptable) thing when returning only the file names. The test case has been added in trunk in core/src/test/java/org/apache/ftpserver/clienttests/NLSTTest.java. I'm assuming I'm missing something so please help me along
        Hide
        Dennis Keller added a comment -

        Patch to support more NLST request types.

        Show
        Dennis Keller added a comment - Patch to support more NLST request types.

          People

          • Assignee:
            Unassigned
            Reporter:
            Dennis Keller
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development