[net] Solution for ant ftp fails with Nullpointerexception when a symlink is evaluated in method checkRemoteSensitivity in FTP.java for a z/OS ftp server

• Key: NET-39
• Type: Bug
• Status: Closed
• Resolution: Fixed
• Priority: Major
• Assignee: Unassigned
• Reporter: Denis Gaebler
• Affects Version/s: 1.4
• Fix Version/s: 2.0

Environment

Operating System: Windows 2000
Platform: PC

Description

The following code is trying to download a file from a z/OS FTP server:
<ftp action="get" server="${server}" userid="${userid}" password="${password}"
remotedir="/usr/lpp/ims/imsjava91">
<fileset dir="C:\temp">
<include name="imsjava.jar"/>
</fileset>
</ftp>

The directory in my case also contains symlinks to non Unix File Systems. For
those symlinks somehow the FTP classes are not able to obtain file information,
e.g. the following listing prints String target and array[pcounter] from method
checkRemoteSensitivity in org/apache/tools/ant/taskdefs/optional/net/FTP.java:

[ftp] getting files
[ftp] ibm drwxr-xr-x 2 OMVSKERN SYS1 8192 Feb 14 2005 IBM
[ftp] ibm rw-rr- 2 OMVSKERN SYS1 1963 Feb 14 2005 README
[ftp] ibm drwxr-xr-x 3 OMVSKERN SYS1 8192 Feb 14 2005 cics
[ftp] ibm drwxr-xr-x 4 OMVSKERN SYS1 8192 Jun 16 13:50 dlimodel
[ftp] ibm rw-rr- 2 OMVSKERN SYS1 267217 Jun 16 13:50 imsjava.ja
r
[ftp] ibm rw-rr- 2 OMVSKERN SYS1 5545 Jun 16 13:50 imsjava91.
rar
[ftp] ibm drwxr-xr-x 3 OMVSKERN SYS1 8192 Feb 14 2005 lib
[ftp] ibm null

In FTP.java at line 536 it is not checked if array[pcounter] is null.

Commons-net is at 1.4.0. Apache Ant version is 1.6.2 compiled on July 16 2004. I
did not try to research the cause in the code, why the file information is null.
For me I added an if clause around lines 536-538 to check if array[pcounter] is
null and this worked great.

So I replaced lines:
if (array[pcounter].getName().equals(target) &&
pcounter != icounter) { candidateFound = false; }

with:
if (array[pcounter] != null) {
if (array[pcounter].getName().equals(target) &&
pcounter != icounter) { candidateFound = false; } }
}

I would highly appreciate to add that code to Ant, otherwise my customers will
not be able to use any Ant version for some z/OS directories.
Thank you in advance.

Comments

[Denis Gaebler]

Created an attachment (id=16103)
diff file for the solution

It should be easy to forward fit this solution for newer versions of FTP.java

1. please try against CVS_HEAD or Ant1.6.5 and verify that the problem is still
there. I suspect it is, but, we need to be sure, and diffs against latest code
are easier to apply.

2, the patch hides a symptom, not the underlying cause, and we should fix it
properly. Assuming the defect is in this code (now moved a bit down the files)

for (int icounter = 0; icounter < array.length; icounter++) {
if (array[icounter].isDirectory()) {
if (!array[icounter].getName().equals(".")
&& !array[icounter].getName().equals("..")) {
candidateFound = true;
target = fiddleName(array[icounter].getName());
getProject().log("will try to cd to "
+ target + " where a directory called " +
array[icounter].getName()
+ " exists", Project.MSG_DEBUG);
for (int pcounter = 0; pcounter < array.length;
pcounter++) {
if (array[pcounter].getName().equals(target) &&
pcounter != icounter) { candidateFound = false; }
}
if (candidateFound) { break; }
}
}
}

then we are iterating twice through the array. the outer loop needs to handle
nullness too, or we should strip null pointers out earlier.

[Denis Gaebler]

(In reply to comment #2)
Hi Steve,

the array[icounter].isDirectory() for some reason returns always true, even if
array[icounter] is null and although the files are no directory.

Buildfile: build.xml

all:
[ftp] getting files
[ftp] isdirectory: true array: drwxr-xr-x 2 OMVSKERN SYS1 8192
Feb 14 2005 IBM
[ftp] isdirectory: true array: rw-rr- 2 OMVSKERN SYS1 1963
Feb 14 2005 README
[ftp] isdirectory: true array: drwxr-xr-x 3 OMVSKERN SYS1 8192
Feb 14 2005 cics
[ftp] isdirectory: true array: drwxr-xr-x 4 OMVSKERN SYS1 8192
Jun 16 13:50 dlimodel
[ftp] isdirectory: true array: rw-rr- 2 OMVSKERN SYS1 267217
Jun 16 13:50 imsjava.jar
[ftp] isdirectory: true array: rw-rr- 2 OMVSKERN SYS1 5545
Jun 16 13:50 imsjava91.rar
[ftp] isdirectory: true array: drwxr-xr-x 3 OMVSKERN SYS1 8192
Feb 14 2005 lib
[ftp] isdirectory: true array: null
[ftp] isdirectory: true array: drwxr-xr-x 8 OMVSKERN SYS1 8192
Jul 20 19:23 samples
[ftp] isdirectory: true array: rw-rr- 1 OMVSKERN SYS1 87809
Sep 24 2004 samples.jar

In addition, you are right. For my current directory it's luck, that the
NullPointerException does not occur in the outer loop.

The root error seams to be in commons net classes, since with a wu-ftp server
the isdirectory() method works correct and only returns true for directories.
I'll try to find the cause of the array being null in case of a symbolic link in
the z/OS directory.

[Steve Loughran]

Maybe isDirectory() is final and the JVM is not checking things.

If this is a recurrent problem, the best tactic would be to strip null fields
from the array. And/Or we track it down to commons-net problem and have them fix
it. Best to do both, to deal with older versions of the c-net stack.

[Denis Gaebler]

(In reply to comment #4)
I found the error in commons-net UnixFTPEntryParser.java
This class is unable to handle the external symlink permission bit in z/OS which
is e.
erwxrwxrwx 1 OMVSKERN SYS1 7 Feb 14 2005 libJavTDLI.so -> DFSCLIB
The regular expression which is defined there is not able to handle the e which
leeds to skip all processing and return a null FTPFile Object.

I don't know, if you think it is still required to catch null entries. I try to
change the UnixFTPEntryParser.java to work.
Please let me know, if I should create a bug for the commons-net classes.

[Denis Gaebler]

Here is the description of the e bit from the z/OS Unix System Services Manual:

-e
Specifies that the link created by ln be an external link. One purpose for
creating an external link is to create a mount point that an NFS client can use
to access a data set through the DFSMS/MVS® Network File System feature. The
normal content of an external link is a name that refers to an object outside
the hierarchical file system, such as a data set. The data set that the
DFSMS/MVS Network File System feature uses can be any type of MVS data set. For
a partitioned data set, however, you specify a fully qualified name in all caps.
For example:

ln -e NOLL.PLIB.PGMA /u/noll/plib/pgma

External links can also be used to map an HFS file name to a PDS or PDSE
member name for an executable load module. An example of how you would define
the external link is:

ln -e MYPGM /u/smorg/mylongpgmname

[Denis Gaebler]

Created an attachment (id=16118)
diff for UnixFTPEntryParser for commons-net-src-20050819

It adds support for the z/OS external link bit e, which is so far not included
in the Regular Expression REGEX and sets the type for e to
FTPFile.SYMBOLIC_LINK_TYPE.
Its the org.apache.commons.net.ftp.parser.UnixFTPEntryParser and not the ftp2
one

[Steve Cohen]

Pardon my ignorance, but I don't know very much about z/OS. I do know something
about commons-net and the ftp parsers.

Is this external link bit "e" something that is z/OS specific or do other
flavors of unix support it? Would this break the parser on non-z/OS systems?

Are there other differences between z/OS and regular unix FTP servers that we
need to take account of?

Where I'm going with this is:

Would it be better to accept this patch or to create an independent parser for
z/OS? To help answer this question, here's another:

When connecting to a z/OS FTP server, what does the SYST ftp command return?

Sorry to take so long in answering, but would like to have these questions
answered before I jump one way or the other.

[Denis Gaebler]

(In reply to comment #8)
Hi Steve,

sorry for the delay, I did not get an email for your update. We have been using
a modified version so far, but that is not really an ultimate solution.
For the first question, the e flag is something specific to z/OS since its to
only way to define links from the HFS UNIX file system to the traditional z/OS
filesystems which are called Datasets (sequential or partitioned).
I don't know of any other Unix that uses the e flag. Don't know if it is
specified somewhere else.

I think it is better to have a parser for z/OS FTP Server, here is the SYST Output:
220-FTPD1 IBM FTP CS V1R7 at FLEXES.DFW.IBM.COM, 16:34:51 on 2006-04-11.
220 Connection will close if idle for more than 5 minutes.
Name (flexes.ehningen.de.ibm.com:gaebler):
331 Send password please.
Password:
230 GAEBLER is logged on. Working directory is "GAEBLER.".
Remote system type is MVS.
ftp> syst
215 MVS is the operating system of this server. FTP Server is running on z/OS.
ftp>

Hope that helps, Denis.

[Rory Winston]

Added to 2.0 branch