Issue Details (XML | Word | Printable)

Key: NET-90
Type: Bug Bug
Status: Closed Closed
Resolution: Fixed
Priority: Major Major
Assignee: Unassigned
Reporter: Mark Ravenscroft
Votes: 0
Watchers: 0
Operations

If you were logged in you would be able to see more operations.
Commons Net

[net] FTP "get" and "list" tasks fail

Created: 04/Mar/04 10:56 PM   Updated: 20/Sep/07 05:31 AM
Return to search
Component/s: None
Affects Version/s: 1.1
Fix Version/s: None

Time Tracking:
Not Specified

File Attachments:
  Size
Java Source File AIXFTPEntryParser.java 2004-09-23 08:13 PM Rory Winston 9 kB
Java Source File AIXFTPEntryParserTest.java 2004-09-23 08:14 PM Rory Winston 9 kB
Text File somefile.txt 2004-03-08 07:28 PM Mark Ravenscroft 3 kB
Environment:
Operating System: All
Platform: PC

Bugzilla Id: 27437


 Description  « Hide
When attempting to get or list files via FTP, ANT returns the filenames
corrupted.

Running Ant 1.6.1 under MS Windows 2000, client is Unix on AIX RS6000.

To reproduce:
-------------------------------------------------------------------------
<?xml version='1.0'?>
<project name='INGDirect' basedir='.' default='FTPList'>

<description>
Use FTP to list files on Unix box
</description>

<property name='libhome' location='D:\apache-ant-1.6.1\lib' />
<property name='java.classpath' location='${libhome}\ant-starteam-1.6.jar;{libhome}\commons-net.jar;{libhome}\jakarta-oro.jar'/>
<taskdef name="ftp"
classname="org.apache.tools.ant.taskdefs.optional.net.FTP" />

<target name='FTPList'>

<ftp action="list"
server="abcd"
userid="****"
password="****"
remotedir="mirtest"
listing='tmp.list'
verbose="yes">
<fileset>
<include name="**.m"/>
</fileset>
</ftp>


</target>

</project>

-------------------------------------------------------------
Output is:


D:\ant-scripts\source>set PATH=D:\apache-ant-1.6.1\bin;
rdgswas1
\StarGate_Lib;C:\WINNT\System32;D:\apache-ant-1.6.1\lib

D:\ant-scripts\source>set CLASSPATH=\\rdgswas1\StarGate_Lib\starteam54.jar

D:\ant-scripts\source>ANT -d
Apache Ant version 1.6.1 compiled on February 12 2004
Buildfile: build.xml
Adding reference: ant.PropertyHelper
Detected Java version: 1.4 in: C:\j2sdk1.4.1_01\jre
Detected OS: Windows 2000
Adding reference: ant.ComponentHelper
Setting ro project property: ant.version -> Apache Ant version 1.6.1 compiled
on February 12 2004
Setting ro project property: ant.file -> D:\ant-scripts\source\build.xml
Adding reference: ant.projectHelper
Adding reference: ant.parsing.context
Adding reference: ant.targets
parsing buildfile D:\ant-scripts\source\build.xml with URI = file:///D:/ant-
scripts/source/build.xml
Setting ro project property: ant.project.name -> INGDirect
Adding reference: INGDirect
Setting ro project property: ant.file.INGDirect > D:\ant
scripts\source\build.xml
Project base dir set to: D:\ant-scripts\source
+Target:
+Target: FTPList
Setting project property: libhome -> D:\apache-ant-1.6.1\lib
Setting project property: java.classpath > D:\apache-ant-1.6.1\lib\ant
starteam-1.6.jar;{libhome}\commons-net.jar;{libhome}\jakarta-oro.jar
[taskdef] dropping rdgswas1\StarGate_Lib\starteam54.jar from path as it
doesn't exist
Class org.apache.tools.ant.taskdefs.optional.net.FTP loaded from parent loader
(parentFirst)
Build sequence for target `FTPList' is [FTPList]
Complete build sequence is [FTPList, ]

FTPList:
[ftp] Opening FTP connection to dbos
[ftp] connected
[ftp] logging in to FTP server
[ftp] login succeeded
[ftp] changing the remote directory
[ftp] listing files
Could not load a dependent class (com/sun/media/jai/codec/FileSeekableStream)
for type image
Could not load a dependent class (com/jcraft/jsch/UserInfo) for type sshexec
Could not load a dependent class (com/jcraft/jsch/UserInfo) for type scp
Could not load class (org.apache.tools.ant.tasksdefs.cvslib.CvsVersion) for
type cvsversion
Could not load a dependent class (jdepend/xmlui/JDepend) for type jdepend
Could not load a dependent class (junit/framework/TestListener) for type junit
fileset: Setup scanner in dir null with patternSet{ includes: [**.m] excludes: [] }
[ftp] listing ar 16:27 ZMIR2.m
[ftp] listing ar 16:27 ZMIR69.m
[ftp] listing ar 16:27 ZMIRNEW.m
[ftp] listing ar 16:27 ZMRPC03A.m
[ftp] listing ar 16:27 ZMRPC03B.m
[ftp] listing ar 16:27 ZMRPC61A.m
[ftp] listing ar 16:27 ZMRPC800.m
[ftp] listing ar 16:27 ZMRPC801.m
[ftp] listing ar 16:27 ZMRPC802.m
[ftp] listing ar 16:27 ZMRPC803.m
[ftp] listing ar 16:27 ZMRPC804.m
[ftp] listing ar 16:27 ZMRPC805.m
[ftp] listing ar 16:27 ZMRPC806.m
[ftp] listing ar 16:27 ZMRPC808.m
[ftp] listing ar 16:27 ZMRPC809.m
[ftp] listing ar 16:27 ZMRPC810.m
[ftp] listing ar 16:27 ZMRPC812.m
[ftp] listing ar 16:27 ZMRPC901.m
[ftp] listing ar 16:27 ZMRPC902.m
[ftp] listing ar 16:27 ZMRPC903.m
[ftp] listing ar 16:27 ZMRPC904.m
[ftp] listing ar 16:27 ZMRPC905.m
[ftp] listing ar 16:27 ZMRPC906.m
[ftp] listing ar 16:27 ZMRPC907.m
[ftp] listing ar 16:27 ZMRPC908.m
[ftp] listing ar 16:27 ZMRPC909.m
[ftp] listing ar 16:27 ZMRPC910.m
[ftp] listing ar 16:27 ZMRPC913.m
[ftp] listing ar 16:27 ZMRPC914.m
[ftp] listing ar 16:27 ZMRPC921.m
[ftp] listing ar 16:27 ZMRPC925.m
[ftp] listing ar 16:27 ZMRPC926.m
[ftp] listing ar 16:27 ZMRPC927.m
[ftp] listing ar 16:27 ZMRPC929.m
[ftp] listing ar 16:27 ZMRPC930.m
[ftp] listing ar 16:27 ZMRPC935.m
[ftp] listing ar 16:27 ZMRPC938.m
[ftp] listing ar 16:27 ZMRPC940.m
[ftp] listing ar 16:27 ZMRPC941.m
[ftp] listing ar 16:27 ZMRPC942.m
[ftp] listing ar 16:27 ZMRPC961.m
[ftp] listing ar 16:27 ZMRPC962.m
[ftp] listing ar 16:27 ZMRPC963.m
[ftp] listing ar 16:27 ZMRPC964.m
[ftp] listing ar 16:27 ZMRPC979.m
[ftp] listing ar 16:27 ZMRPC991.m
[ftp] listing ar 16:27 ZMRPCP29.m
[ftp] listing ar 16:27 ZMRPCUTL.m
[ftp] 48 files listed
[ftp] disconnecting

BUILD SUCCESSFUL
Total time: 2 seconds
---------------------------------------------------------------------------
Contents of "listing" file

rw-r---- 1 ravensm sca 814 02 Mar 16:27 ZMIR2.m
rw-r---- 1 ravensm sca 780 02 Mar 16:27 ZMIR69.m
rw-r---- 1 ravensm sca 2120 02 Mar 16:27 ZMIRNEW.m
rw-r---- 1 ravensm sca 15347 02 Mar 16:27 ZMRPC03A.m
rw-r---- 1 ravensm sca 12172 02 Mar 16:27 ZMRPC03B.m
rw-r---- 1 ravensm sca 6242 02 Mar 16:27 ZMRPC61A.m
rw-r---- 1 ravensm sca 3259 02 Mar 16:27 ZMRPC800.m
rw-r---- 1 ravensm sca 3264 02 Mar 16:27 ZMRPC801.m
rw-r---- 1 ravensm sca 5145 02 Mar 16:27 ZMRPC802.m
rw-r---- 1 ravensm sca 5071 02 Mar 16:27 ZMRPC803.m
rw-r---- 1 ravensm sca 5096 02 Mar 16:27 ZMRPC804.m
rw-r---- 1 ravensm sca 4683 02 Mar 16:27 ZMRPC805.m
rw-r---- 1 ravensm sca 5009 02 Mar 16:27 ZMRPC806.m
rw-r---- 1 ravensm sca 2425 02 Mar 16:27 ZMRPC808.m
rw-r---- 1 ravensm sca 9150 02 Mar 16:27 ZMRPC809.m
rw-r---- 1 ravensm sca 18007 02 Mar 16:27 ZMRPC810.m
rw-r---- 1 ravensm sca 3923 02 Mar 16:27 ZMRPC812.m
rw-r---- 1 ravensm sca 7023 02 Mar 16:27 ZMRPC901.m
rw-r---- 1 ravensm sca 15221 02 Mar 16:27 ZMRPC902.m
rw-r---- 1 ravensm sca 16943 02 Mar 16:27 ZMRPC903.m
rw-r---- 1 ravensm sca 11454 02 Mar 16:27 ZMRPC904.m
rw-r---- 1 ravensm sca 5926 02 Mar 16:27 ZMRPC905.m
rw-r---- 1 ravensm sca 14167 02 Mar 16:27 ZMRPC906.m
rw-r---- 1 ravensm sca 31350 02 Mar 16:27 ZMRPC907.m
rw-r---- 1 ravensm sca 26765 02 Mar 16:27 ZMRPC908.m
rw-r---- 1 ravensm sca 40841 02 Mar 16:27 ZMRPC909.m
rw-r---- 1 ravensm sca 3493 02 Mar 16:27 ZMRPC910.m
rw-r---- 1 ravensm sca 4636 02 Mar 16:27 ZMRPC913.m
rw-r---- 1 ravensm sca 5633 02 Mar 16:27 ZMRPC914.m
rw-r---- 1 ravensm sca 12887 02 Mar 16:27 ZMRPC921.m
rw-r---- 1 ravensm sca 8606 02 Mar 16:27 ZMRPC925.m
rw-r---- 1 ravensm sca 9463 02 Mar 16:27 ZMRPC926.m
rw-r---- 1 ravensm sca 5786 02 Mar 16:27 ZMRPC927.m
rw-r---- 1 ravensm sca 12513 02 Mar 16:27 ZMRPC929.m
rw-r---- 1 ravensm sca 6603 02 Mar 16:27 ZMRPC930.m
rw-r---- 1 ravensm sca 4194 02 Mar 16:27 ZMRPC935.m
rw-r---- 1 ravensm sca 7216 02 Mar 16:27 ZMRPC938.m
rw-r---- 1 ravensm sca 4466 02 Mar 16:27 ZMRPC940.m
rw-r---- 1 ravensm sca 23840 02 Mar 16:27 ZMRPC941.m
rw-r---- 1 ravensm sca 3964 02 Mar 16:27 ZMRPC942.m
rw-r---- 1 ravensm sca 6405 02 Mar 16:27 ZMRPC961.m
rw-r---- 1 ravensm sca 41925 02 Mar 16:27 ZMRPC962.m
rw-r---- 1 ravensm sca 9280 02 Mar 16:27 ZMRPC963.m
rw-r---- 1 ravensm sca 10757 02 Mar 16:27 ZMRPC964.m
rw-r---- 1 ravensm sca 3907 02 Mar 16:27 ZMRPC979.m
rw-r---- 1 ravensm sca 3592 02 Mar 16:27 ZMRPC991.m
rw-r---- 1 ravensm sca 44899 02 Mar 16:27 ZMRPCP29.m
rw-r---- 1 ravensm sca 21039 02 Mar 16:27 ZMRPCUTL.m

----------------------------------------------------------------------

i.e. The FTP command is corrupting the incoming filenames. A similar example
using FTP "get" results in a DOS file being created named "ar 16".



 All   Comments   Work Log   Change History   Subversion Commits      Sort Order: Ascending order - Click to sort in descending order
Antoine Levy-Lambert added a comment - 05/Mar/04 04:09 AM
Can you write down :
  • which version of commons-net you use ?
  • which version of jakarta-oro you use ?
  • what is the exact operating system version of the FTP server ?
  • what is the langage installed in the operating system of the FTP server ?
  • please attach a file listing done under a command line ftp client as an
    attachment to this bug report.

I do not think that the ftp task corrupts the name of the files. There is
probably not the right parser in commons-net for the type of listings produced
by your AIX ftp server (langage ?)

Cheers,
Antoine


Mark Ravenscroft added a comment - 05/Mar/04 07:12 PM
Antoine,

commons-net.jar + jakarta-oro.jar in use are those supplied as part of ant
release 1.6.1 and are dated 12/02/2004

O/S of FTP client is MS Windows 2000 v 5.00.2195 SP 2
O/S of FTP server :

uname -v returns 4
uname -r returns 3

Language installed is English (United Kingdom)

Command line ftp client listing:

ZMIR69.m
ZMIRNEW.m
ZMRPC03A.m
ZMRPC03B.m
ZMRPC61A.m
ZMRPC800.m
ZMRPC801.m
ZMRPC802.m
ZMRPC803.m
ZMRPC804.m
ZMRPC805.m
ZMRPC806.m
ZMRPC808.m
ZMRPC809.m
ZMRPC810.m
ZMRPC812.m
ZMRPC901.m
ZMRPC902.m
ZMRPC903.m
ZMRPC904.m
ZMRPC905.m
ZMRPC906.m
ZMRPC907.m
ZMRPC908.m
ZMRPC909.m
ZMRPC910.m
ZMRPC913.m
ZMRPC914.m
ZMRPC921.m
ZMRPC925.m
ZMRPC926.m
ZMRPC927.m
ZMRPC929.m
ZMRPC930.m
ZMRPC935.m
ZMRPC938.m
ZMRPC940.m
ZMRPC941.m
ZMRPC942.m
ZMRPC961.m
ZMRPC962.m
ZMRPC963.m
ZMRPC964.m
ZMRPC979.m
ZMRPC991.m
ZMRPCP29.m
ZMRPCUTL.m
226 Transfer complete.
ftp: 570 bytes received in 0.03Seconds 17.81Kbytes/sec.
ftp>


Antoine Levy-Lambert added a comment - 06/Mar/04 06:07 AM
commons-net and jakarta-oro are not bundled with ant.
You need both of them to execute the FTP task.
ant-commons-net and ant-jakarta-oro are two jar files containing tasks dependent
upon commons-net and jakarta-oro respectively, but they are not jakarta-oro
and commons-net.
So again, which version of commons-net and jakarta-oro are you using ?
Also do a directory listing with ls -l through the ftp client and send us the
output (I think ls -l somefile.txt redirects the output directly into a file
called somefile.txt, ===> then you can attach somefile.txt to this bug report.

Mark Ravenscroft added a comment - 08/Mar/04 07:28 PM
Created an attachment (id=10703)
ls-l performed from the ftp client as requested

Mark Ravenscroft added a comment - 08/Mar/04 07:32 PM
Antoine,

I am using the following:

commons-net-1.1.0.jar and jakarta-oro-2.0.8.jar

I've also sent you an attachment, as requested, which is an 'ls-l' from the ftp
client.

Mark


Steve Cohen added a comment - 09/Mar/04 09:27 AM
Antoine is basically correct. There is (currently) no support for this ftp format in commons.net.
The format is CLOSE TO but not identical to the Standard unix ftp server list format that is the
default in commons-net.

Here is a typical unix ftp standard listing followed by one of yours:
-rwxr-xr-x 2 root root 4096 Mar 2 15:13 zxbox
rw-r---- 1 ravensm sca 9280 02 Mar 16:27 ZMRPC963.m

Clearly, the problem seems to be one of locale (reversal of the month and day fields). The
standard unix ftp parser is the default in commons-net and the format on your server is
confusing the parser.

The basic problem is that the FTP specification (RFC959) defines the FTP listing format in a
very loose way. The protocol was intended for manual scanning by a user and definitely not
for automation purposes. Similar problems crop up with servers in other languages such as a
German one that might use the abbreviation okt for october, etc.

I have never heard this complaint about unix servers in non-American locales, but I have heard
it about AIX servers. We could write a parser that expected your format, but I have a feeling it
would break American AIX ftp servers.

Additionally, commons-net is moving toward an autodetection scheme for ftp server types. It
does a pretty good job telling NT servers from Unix ones, but your situation brings up a new
wrinkle. Which locale should a parser assume for a server if there is no way to make the
server define its listing format? Assuming there is some way to differentiate AIX ftp servers
from Unix ones in an automated fashion, we would have to come up with a way of
differentiating the date-format locales as well, and I'm not sure that's possible.

It's a real mess, frankly, due to the vagueness of the ftp spec on this matter. As a committer
for commons-net, I would be willing to work with you to get this resolved, but I need your input
and that of other aix ftp server users to determine the RIGHTway to solve this problem. I am
guessing that the problem is not very widespread, or we would have heard many more
complaints about this. The more instances we can get of this pattern the more likely it will be
to be solved.

However, it's totally out of Ant's hands to be able to fix this.


Antoine Levy-Lambert added a comment - 09/Mar/04 10:24 PM
bug report reassigned to commons-dev

Mark Ravenscroft added a comment - 10/Mar/04 04:18 PM
No problem. I'd be happy to assist with any future work you may need to do on
commons-net (work-schedules permitting!).

If you do go ahead, let me know what additional information you need.

Mark


Steve Cohen added a comment - 10/Mar/04 08:11 PM
To start with I could use some data.

As you appear to be in the UK, are you in a position to do a little experimenting, contacting
several different FTP servers there with commons-net to see where the problems lie? Are only
AIX servers affected or are regular unix servers affected too? Are all AIX servers affected or
are some affected and some not?

That would be most helpful. I need to know two things.
1. Is the FTP server's listing format determined by its locale in any determinable way?
2. If it is, is there any way to determine that locale from the FTP process.

If, as I suspect the answer to both of these, but especially #2, is negative, then the only way
to determine this automatedly would be trial and error. And once you expand the scope of the
problem to cover not only the date format, but also the language of the locale for the month
names, the combinatorial possibilities start to overwhelm.


Mark Ravenscroft added a comment - 10/Mar/04 10:35 PM
The only servers we have here are AIX so I cannot confirm whether the problem
exists with other server types.

We did change the locale within the Unix shell and can confirm that the output
of an "ls -l" changed:

With LANG=en_US

rw-rw-r- 1 reedd sca 0 10 Mar 10:21 THR9737272.mje

With LANG=en_UK

rw-rw-r- 1 reedd sca 0 Mar 10 10:21 THR9737272.mje

An "ls -l" within FTP remains resolutely the same whether
conducted from the FTP server or the client. (I'm not aware of a way in which
we can change the locale within the FTP client.)

Similarly, I don't think there is a way of determining the locale of the FTP
process.


Steve Cohen added a comment - 26/Jun/04 12:08 PM
reopen after 1.2.2 release

Rory Winston added a comment - 23/Sep/04 08:13 PM
Created an attachment (id=12848)
AIXFTPEntryParser.java

Rory Winston added a comment - 23/Sep/04 08:14 PM
Created an attachment (id=12849)
AIXFTPEntryParserTest.java

Rory Winston added a comment - 23/Sep/04 08:15 PM
I agree with Steve that there may be a more elegant long-term solution to this,
however, in the interests of the interim, I have created an AIX parser and
testcase that successfully parses the output that Mark sent previously. Steve
et. al, let me know if you approve of this approach, and I can commit it.

Rory Winston added a comment - 23/Sep/04 10:00 PM
Sorry, my bad. I didn't realize that Daniel has an AIX parser in there already.

Steve Cohen added a comment - 24/Sep/04 09:11 AM
Rory - I favor your submission far above the old thing in the proposal tree which I think must
be what you're talking about. Your submission will work with the current setup and the old one
needs significant rework.

However, before we commit it, I would like to get clarity on my main question, which nobody
has ever answered, ever since this bug (and similar ones before it) was first posted. So, let
me reiterate:

Is the d/m/y ordering of the date a particular attribute of AIX or the AIX ftp server, anywhere
and everywhere, or is it an artifact of LOCALE? Do American AIX servers format the date in
the American traditional m/d/y way - or do they follow the d/m/y model?

Because if it's the former, I am opposed to creating an AIX parser that is only good for a
particular locale. Of course, non-Americans may complain, and rightfully so, IMHO, that that's
what THEY have had to put up with all these years. But the extremely low volume of such
complaints, plus the utter lack of response I've received to multiple requests for ANY actual
data on the prevalence of the various date schemes on FTP servers around the world, inclines
me to believe that there must be many FTP servers in Europe that are following the American
model. If I am wrong about this, PLEASE, let someone correct me!

In other words, I am opposed to saying "AIX" when what we really mean is "AIX in a British
locale" or "AIX in a European locale". That will lead to a jumble. What I'm really worried about
is that we might start to see requests for UnixFTPFileEntryParserDeutsch or
NTFTPFileEntryParserEspanol. We need to SOLVE the locale thing, not just paper over it.

Also, we need to now think about the autodetection mechanism. Do you know how AIX FTP
servers identify themselves to the SYST command?


Mario Ivankovits added a comment - 24/Sep/04 12:33 PM
This is what i got on an AIX 4.2:

UNIX Type: L8 Version: BSD-44

Not much information to gather a locale.


Rory Winston added a comment - 24/Sep/04 05:12 PM
I agree with Steve - we need a generic problem to deal with the date formatting
issues that occur. I think finding an elegant solution is very, very
difficult. A lot of FTP servers can not be reliably identified (I've tried a few
AIX servers this morning, from which they returned nothing useful or consistent
from SYST), and the date format can vary wildly based on server configuration.
Witness the IIS FTP server options:

http://www.microsoft.com/windows2000/en/server/iis/default.asp?url=/windows2000/en/server/iis/htm/core/iiprsovr.htm

and

http://www.microsoft.com/windows2000/en/server/iis/htm/asp/apro1k9x.htm

Two possibilities spring to mind:

1. We could match an entry from an initial 'ls' against a couple of regexes
designed exclusively to isolate and identify the date component, repeatedly
attempting to match until we find one that fits (sort of like an autodetection
of date formats). I'm not sure how reliable this would be in practise, but feel
it should work.

2. We could explicitly add a setDateFormat() to the FTPClient class, and allow a
user to plug in a date format of choice. This could also be accessed via an
optional property from the Ant task.


Steve Cohen added a comment - 25/Sep/04 10:21 AM
Rory:
Your report on AIX's SYST command response confims my suspicion that this is just a flavor
of the UnixFTPFileEntryParser. The date formatting is another issue entirely

As for solutions to THAT problem, of the two that you present, I much prefer #2. It's realistic
and hopes for finding an automated way are not, IMHO.

It's pretty obvious to me that we should not expect to find any automated solution using SYST
to the date problem. There is enough standardization to enable the system to detect a server
type - at least for the cases that use US format dates. But given that the FTP RFC has
nothing whatever to say about locale, our chances of success that way are slim to none.

So let's look at what we'd need to do to define a date format. It's going to be more complicated
than a SimpleDateFormat, but I still think quite doable. It may be as simple as specifying a
server locale or a language locale but maybe more complicated. Perhaps a long and a short
format must be allowed to be specified as well. That part of the regex that lists the month
abbreviations could be replaced by a method call returning a delimited list of months. Clients
like Ant could be enhanced with a parameter allowing this to be specified.

I think we need a bit of upfront design and then we'll be ready to work on a solution. So let's
keep the discussion going. I suggest the mailing list would be a better forum for this than this
defect report.


Rory Winston added a comment - 16/Apr/05 12:39 AM
The mechanisms to fix this are in the trunk and will be released in 1.4. I have
added a TestCase to the repository that shows how you can solve this issue using
the FTPClientConfig class.