[net] The FTPClient is unable to list files on Japanese servers

• Key: NET-2
• Type: Bug
• Status: Closed
• Resolution: Fixed
• Priority: Major
• Assignee: Unassigned
• Reporter: Leif Mortenson
• Affects Version/s: Nightly Builds

Environment

Operating System: All
Platform: All

Description

I ran into some problems attempting to get the FTPClient to be able to list
files on a Solaris server which is configured to use the Japanese locale. The
problem turned out to be caused by the regular expressions used to parse the
individual lines received from the server.

It works great with the English locale, but on Japanese servers, every single
line of output was failing to match.

I modified the UnixFTPEntryParser to work correctly with either English or
Japanese locales. I also added appropriate test cases to make sure every thing
works.

Obviously other languages are going to most likely have similar problems. The
way I modified the regex, should work correctly with any language that uses
numerical months and adds units after the month, day, and/or years.

I also found that there were some encoding problems. The FTP class allows the
user to configure a control encoding using the setControlEncoding() method. But
this encoding was not being used is several places throughout the code.

I had to add a couple new methods to handle encoding, but all existing public
methods were left unchanged and function the same as they did before.

In addition, if the user specified an invalid control encoding when used with
the Ant task, it was kicking out some some nasty NPEs because the ant task always
tries to log off. That could be viewed as a problem with the Ant task, but I made
the FTP class handle this case more gracefully. The problem is that the
isConnected method returns true when the socket has been opened, but the
protocol is not yet all the way initialized. The logout and disconnect methods
now handle these cases correctly.

I will post right back with the patch. This message was also posted on the dev
list.

Cheers,
Leif

Comments

[Leif Mortenson]

Here is the full patch:

cvs diff -u
cvs server: Diffing .
cvs server: Diffing proposal
cvs server: Diffing proposal/ftp2
cvs server: Diffing proposal/ftp2/src
cvs server: Diffing proposal/ftp2/src/java
cvs server: Diffing proposal/ftp2/src/java/org
cvs server: Diffing proposal/ftp2/src/java/org/apache
cvs server: Diffing proposal/ftp2/src/java/org/apache/commons
cvs server: Diffing proposal/ftp2/src/java/org/apache/commons/net
cvs server: Diffing proposal/ftp2/src/java/org/apache/commons/net/ftp
cvs server: Diffing proposal/ftp2/src/java/org/apache/commons/net/ftp/ftp2
cvs server: Diffing proposal/ftp2/src/java/org/apache/commons/net/ftp/ftp2/parser
cvs server: Diffing proposal/ftp2/src/test
cvs server: Diffing proposal/ftp2/src/test/org
cvs server: Diffing proposal/ftp2/src/test/org/apache
cvs server: Diffing proposal/ftp2/src/test/org/apache/commons
cvs server: Diffing proposal/ftp2/src/test/org/apache/commons/net
cvs server: Diffing proposal/ftp2/src/test/org/apache/commons/net/ftp
cvs server: Diffing proposal/ftp2/src/test/org/apache/commons/net/ftp/ftp2
cvs server: Diffing proposal/ftp2/src/test/org/apache/commons/net/ftp/ftp2/parser
cvs server: Diffing src
cvs server: Diffing src/java
cvs server: Diffing src/java/examples
cvs server: Diffing src/java/org
cvs server: Diffing src/java/org/apache
cvs server: Diffing src/java/org/apache/commons
cvs server: Diffing src/java/org/apache/commons/net
cvs server: Diffing src/java/org/apache/commons/net/bsd
cvs server: Diffing src/java/org/apache/commons/net/ftp
Index: src/java/org/apache/commons/net/ftp/FTP.java
===================================================================
RCS file:
/home/cvspublic/jakarta-commons/net/src/java/org/apache/commons/net/ftp/FTP.java,v
retrieving revision 1.16
diff -u -r1.16 FTP.java
— src/java/org/apache/commons/net/ftp/FTP.java 8 Aug 2004 20:38:35 -0000 1.16
+++ src/java/org/apache/commons/net/ftp/FTP.java 18 Aug 2004 06:11:31 -0000
@@ -420,6 +420,11 @@
***/
public int sendCommand(String command, String args) throws IOException
{
+ if ((__commandBuffer == null) || (_controlOutput == null))
+ { + throw new IOException("Not connected."); + }
+
String message;

__commandBuffer.setLength(0);
Index: src/java/org/apache/commons/net/ftp/FTPClient.java
===================================================================
RCS file:
/home/cvspublic/jakarta-commons/net/src/java/org/apache/commons/net/ftp/FTPClient.java,v
retrieving revision 1.38
diff -u -r1.38 FTPClient.java
— src/java/org/apache/commons/net/ftp/FTPClient.java 29 Jun 2004 04:54:30
-0000 1.38
+++ src/java/org/apache/commons/net/ftp/FTPClient.java 18 Aug 2004 06:11:32 -0000
@@ -1931,8 +1931,8 @@
if ((socket = openDataConnection(FTPCommand.NLST, pathname)) == null)
return null;

• reader =
• new BufferedReader(new InputStreamReader(socket.getInputStream()));
  + reader = new BufferedReader(
  + new InputStreamReader(socket.getInputStream(), getControlEncoding()));

results = new Vector();
while ((line = reader.readLine()) != null)
@@ -2338,7 +2338,7 @@
}

• engine.readServerList(socket.getInputStream());
  + engine.readServerList(socket.getInputStream(), getControlEncoding());

socket.close();

@@ -2425,7 +2425,7 @@
if ((socket = openDataConnection(FTPCommand.LIST, pathname)) == null)
return new FTPFile[0];

• results = parser.parseFileList(socket.getInputStream());
  + results = parser.parseFileList(socket.getInputStream(),
  getControlEncoding());

socket.close();

Index: src/java/org/apache/commons/net/ftp/FTPFileEntryParserImpl.java
===================================================================
RCS file:
/home/cvspublic/jakarta-commons/net/src/java/org/apache/commons/net/ftp/FTPFileEntryParserImpl.java,v
retrieving revision 1.8
diff -u -r1.8 FTPFileEntryParserImpl.java
— src/java/org/apache/commons/net/ftp/FTPFileEntryParserImpl.java 22 Apr 2004
00:48:07 -0000 1.8
+++ src/java/org/apache/commons/net/ftp/FTPFileEntryParserImpl.java 18 Aug 2004
06:11:32 -0000
@@ -34,7 +34,7 @@

• The constructor for a FTPFileEntryParserImpl object.
  */
  public FTPFileEntryParserImpl()

• {
  + {
  }

@@ -47,16 +47,37 @@

• <p>
• @param listStream The InputStream from which the file list should be
• read.
  + * @param encoding The encoding to use, null for system default.
• @return The list of file information contained in the given path. null
• if the list could not be obtained or if there are no files in
• the directory.
• @exception java.io.IOException If an I/O error occurs reading the
  listStream.
  ***/

• public FTPFile[] parseFileList(InputStream listStream) throws IOException
  + public FTPFile[] parseFileList(InputStream listStream, String encoding)
  throws IOException { - FTPFileList ffl = FTPFileList.create(listStream, this); + FTPFileList ffl = FTPFileList.create(listStream, this, encoding); return ffl.getFiles(); + }

+ /***
+ * Parses an FTP server file listing and converts it into a usable format
+ * in the form of an array of <code> FTPFile </code> instances. If the
+ * file list contains no files, <code> null </code> should be
+ * returned, otherwise an array of <code> FTPFile </code> instances
+ * representing the files in the directory is returned.
+ * <p>
+ * @param listStream The InputStream from which the file list should be
+ * read.
+ * @return The list of file information contained in the given path. null
+ * if the list could not be obtained or if there are no files in
+ * the directory.
+ * @exception java.io.IOException If an I/O error occurs reading the
listStream.
+ *
+ * @deprecated The version of this method which takes an encoding should be
used.
+ ***/
+ public FTPFile[] parseFileList(InputStream listStream) throws IOException
+ { + return parseFileList(listStream, null); }

/**
Index: src/java/org/apache/commons/net/ftp/FTPFileList.java
===================================================================
RCS file:
/home/cvspublic/jakarta-commons/net/src/java/org/apache/commons/net/ftp/FTPFileList.java,v
retrieving revision 1.13
diff -u -r1.13 FTPFileList.java
— src/java/org/apache/commons/net/ftp/FTPFileList.java 21 Apr 2004 23:30:33
-0000 1.13
+++ src/java/org/apache/commons/net/ftp/FTPFileList.java 18 Aug 2004 06:11:32 -0000
@@ -82,6 +82,7 @@

• the output of the LIST command was returned
• @param parser the default <code>FTPFileEntryParser</code> to be used
• by this object. This may later be changed using the init() method.
  + * @param encoding The encoding to use, null for system default.
  *
• @return the <code>FTPFileList</code> created, with an initialized
• of unparsed lines of output. Will be null if the listing cannot
  @@ -90,26 +91,61 @@
• Thrown on any failure to read from the socket.
  */
  public static FTPFileList create(InputStream stream,

• FTPFileEntryParser parser)
  + FTPFileEntryParser parser,
  + String encoding)
  throws IOException { FTPFileList list = new FTPFileList(parser); - list.readStream(stream); + list.readStream(stream, encoding); parser.preParse(list.lines); return list; }

/**
+ * The only way to create an <code>FTPFileList</code> object. Invokes
+ * the private constructor and then reads the stream supplied stream to
+ * build the intermediate array of "lines" which will later be parsed
+ * into <code>FTPFile</code> object.
+ *
+ * @param stream The input stream created by reading the socket on which
+ * the output of the LIST command was returned
+ * @param parser the default <code>FTPFileEntryParser</code> to be used
+ * by this object. This may later be changed using the init() method.
+ *
+ * @return the <code>FTPFileList</code> created, with an initialized
+ * of unparsed lines of output. Will be null if the listing cannot
+ * be read from the stream.
+ * @exception IOException
+ * Thrown on any failure to read from the socket.
+ *
+ * @deprecated The version of this method which takes an encoding should be
used.
+ */
+ public static FTPFileList create(InputStream stream,
+ FTPFileEntryParser parser)
+ throws IOException
+ { + return create(stream, parser, null); + }
+
+ /**

• internal method for reading the input into the <code>lines</code> vector.
  *
• @param stream The socket stream on which the input will be read.
  + * @param encoding The encoding to use, null for system default.
  *
• @exception IOException thrown on any failure to read the stream
  */

• public void readStream(InputStream stream) throws IOException
  + public void readStream(InputStream stream, String encoding) throws IOException
  {
• BufferedReader reader =
• new BufferedReader(new InputStreamReader(stream));
  + BufferedReader reader;
  + if (encoding == null)
  + { + reader = new BufferedReader(new InputStreamReader(stream)); + }
  + else
  + { + reader = new BufferedReader(new InputStreamReader(stream, encoding)); + }
  
  String line = this.parser.readNextEntry(reader);
  
  @@ -119,6 +155,20 @@
  line = this.parser.readNextEntry(reader);
  }
  reader.close();
  + }
  +
  + /**
  + * internal method for reading the input into the <code>lines</code> vector.
  + *
  + * @param stream The socket stream on which the input will be read.
  + *
  + * @exception IOException thrown on any failure to read the stream
  + *
  + * @deprecated The version of this method which takes an encoding should be
  used.
  + */
  + public void readStream(InputStream stream) throws IOException
  + { + readStream(stream, null); }
  
  /**
  Index: src/java/org/apache/commons/net/ftp/FTPFileListParser.java
  ===================================================================
  RCS file:
  /home/cvspublic/jakarta-commons/net/src/java/org/apache/commons/net/ftp/FTPFileListParser.java,v
  retrieving revision 1.12
  diff -u -r1.12 FTPFileListParser.java
  — src/java/org/apache/commons/net/ftp/FTPFileListParser.java 29 Jun 2004
  02:16:33 -0000 1.12
  +++ src/java/org/apache/commons/net/ftp/FTPFileListParser.java 18 Aug 2004
  06:11:32 -0000
  @@ -45,10 +45,29 @@
  * <p>
  * @param listStream The InputStream from which the file list should be
  * read.
  + * @param encoding The encoding to use, null for system default.
  * @return The list of file information contained in the given path. null
  * if the list could not be obtained or if there are no files in
  * the directory.
  * @exception IOException If an I/O error occurs reading the listStream.
  + ***/
  + FTPFile[] parseFileList(InputStream listStream, String encoding) throws
  IOException;
  +
  + /***
  + * Parses an FTP server file listing and converts it into a usable format
  + * in the form of an array of <code> FTPFile </code> instances. If the
  + * file list contains no files, <code> null </code> should be
  + * returned, otherwise an array of <code> FTPFile </code> instances
  + * representing the files in the directory is returned.
  + * <p>
  + * @param listStream The InputStream from which the file list should be
  + * read.
  + * @return The list of file information contained in the given path. null
  + * if the list could not be obtained or if there are no files in
  + * the directory.
  + * @exception IOException If an I/O error occurs reading the listStream.
  + *
  + * @deprecated The version of this method which takes an encoding should be
  used.
  ***/
  FTPFile[] parseFileList(InputStream listStream) throws IOException;
  
  Index: src/java/org/apache/commons/net/ftp/FTPListParseEngine.java
  ===================================================================
  RCS file:
  /home/cvspublic/jakarta-commons/net/src/java/org/apache/commons/net/ftp/FTPListParseEngine.java,v
  retrieving revision 1.7
  diff -u -r1.7 FTPListParseEngine.java
  — src/java/org/apache/commons/net/ftp/FTPListParseEngine.java 22 Apr 2004
  00:48:07 -0000 1.7
  +++ src/java/org/apache/commons/net/ftp/FTPListParseEngine.java 18 Aug 2004
  06:11:33 -0000
  @@ -87,19 +87,39 @@
  * on the server.
  *
  * @param stream input stream provided by the server socket.
  + * @param encoding The encoding to use, null for system default.
  *
  * @exception IOException
  * thrown on any failure to read from the sever.
  */
  - public void readServerList(InputStream stream)
  + public void readServerList(InputStream stream, String encoding)
  throws IOException
  { this.entries = new LinkedList(); - readStream(stream); + readStream(stream, encoding); this.parser.preParse(this.entries); resetIterator(); }
  
  + /**
  + * handle the iniitial reading and preparsing of the list returned by
  + * the server. After this method has completed, this object will contain
  + * a list of unparsed entries (Strings) each referring to a unique file
  + * on the server.
  + *
  + * @param stream input stream provided by the server socket.
  + *
  + * @exception IOException
  + * thrown on any failure to read from the sever.
  + *
  + * @deprecated The version of this method which takes an encoding should be
  used.
  + */
  + public void readServerList(InputStream stream)
  + throws IOException
  + { + readServerList(stream, null); + }
  +
  
  /**
  * Internal method for reading the input into the <code>entries</code> list.
  @@ -110,14 +130,22 @@
  * and other data that will not be part of the final listing.
  *
  * @param stream The socket stream on which the input will be read.
  + * @param encoding The encoding to use, null for system default.
  *
  * @exception IOException
  * thrown on any failure to read the stream
  */
  - private void readStream(InputStream stream) throws IOException
  + private void readStream(InputStream stream, String encoding) throws IOException
  {
  - BufferedReader reader =
  - new BufferedReader(new InputStreamReader(stream));
  + BufferedReader reader;
  + if (encoding == null)
  + {+ reader = new BufferedReader(new InputStreamReader(stream));+ } }
  + else
  + { + reader = new BufferedReader(new InputStreamReader(stream, encoding)); + }

String line = this.parser.readNextEntry(reader);

cvs server: Diffing src/java/org/apache/commons/net/ftp/parser
Index: src/java/org/apache/commons/net/ftp/parser/UnixFTPEntryParser.java
===================================================================
RCS file:
/home/cvspublic/jakarta-commons/net/src/java/org/apache/commons/net/ftp/parser/UnixFTPEntryParser.java,v
retrieving revision 1.18
diff -u -r1.18 UnixFTPEntryParser.java
— src/java/org/apache/commons/net/ftp/parser/UnixFTPEntryParser.java 28 Jul
2004 05:01:47 -0000 1.18
+++ src/java/org/apache/commons/net/ftp/parser/UnixFTPEntryParser.java 18 Aug
2004 06:11:33 -0000
@@ -35,7 +35,7 @@

• to determine which month is matched by the parser
  */
  private static final String MONTHS =

• "(Jan|Feb|Mar|Apr|May|Jun|Jul|Aug|Sep|Oct|Nov|Dec)";
  + "Jan|Feb|Mar|Apr|May|Jun|Jul|Aug|Sep|Oct|Nov|Dec";

/**

• this is the regular expression used by this parser.
  @@ -55,6 +55,15 @@
• execution is on
• T the 1000 bit is turned on, and execution is off (undefined bit-
• state)
  + *
  + * Japanese formatted listings use numerical months, and the month, day
  + * and year are each followed by a single kanji character. Rather than
  + * testing for the specific kanji characters, this expression allows any
  + * non-numerical, non-space character(s). This should make other languages
  + * which use a similar syntax work as well.
  + *
  + * (For Japanese, the matched characters are '\u6708' after month,
  + * '\u65e5' after day, and '\u5e74' after year.)
  */
  private static final String REGEX =
  "([bcdlfmpSs-])"
  @@ -63,9 +72,9 @@
  + "(\\S+)
  s+"
  + "(?\\S+)
  s+)?"
  + "(\\d+)
  s+"

• + MONTHS + "
  s+"
• + "((?:[0-9])|(?:[0-2][0-9])|(?:3[0-1]))
  s+"
• + "((\\d\\d\\d\\d)|((?:[01]\\d)|(?:2[0123])|(?:[1-9])):([012345]\\d))
  s+"
  + + "((?:" + MONTHS + ")|(???:[0-9])|(?:[0-1][0-9]))))[^0-9\\s]*?
  s+"
  + + "(?(?:[0-9])|(?:[0-2][0-9])|(?:3[0-1]))(?:[^0-9\\s]*?))
  s+"
  + +
  "((?\\d\\d\\d\\d)[^0-9\\s]*?)|((?:[01]\\d)|(?:2[0123])|(?:[1-9])):([012345]\\d))
  s+"
  + "(\\S+)(
  s*.*)";

@@ -94,7 +103,6 @@
*/
public FTPFile parseFTPEntry(String entry)
{
-
FTPFile file = new FTPFile();
file.setRawListing(entry);
int type;
@@ -114,7 +122,7 @@
String min = group(24);
String name = group(25);
String endtoken = group(26);
-
+
// bcdlfmpSs-
switch (typeStr.charAt(0))

{ @@ -130,8 +138,8 @@ // break; - fall through case 'f': case '-': - type = FTPFile.FILE_TYPE; - break; + type = FTPFile.FILE_TYPE; + break; default: type = FTPFile.UNKNOWN_TYPE; }

@@ -189,8 +197,17 @@

try
{
+ int month;
int pos = MONTHS.indexOf(mo);

• int month = pos / 4;
  + if ( pos >= 0 )
  + { + month = pos / 4; + }
  + else
  + { + // Expected to be base-0 + month = Integer.parseInt( mo ) - 1; + }

if (null != yr)
{
Index: src/java/org/apache/commons/net/ftp/parser/VMSFTPEntryParser.java
===================================================================
RCS file:
/home/cvspublic/jakarta-commons/net/src/java/org/apache/commons/net/ftp/parser/VMSFTPEntryParser.java,v
retrieving revision 1.24
diff -u -r1.24 VMSFTPEntryParser.java
— src/java/org/apache/commons/net/ftp/parser/VMSFTPEntryParser.java 28 Jul
2004 05:01:47 -0000 1.24
+++ src/java/org/apache/commons/net/ftp/parser/VMSFTPEntryParser.java 18 Aug
2004 06:11:33 -0000
@@ -94,18 +94,17 @@

• <p>
• @param listStream The InputStream from which the file list should be
• read.
  + * @param encoding The encoding to use, null for system default.
• @return The list of file information contained in the given path. null
• if the list could not be obtained or if there are no files in
• the directory.
• @exception IOException If an I/O error occurs reading the listStream.
  ***/

• public FTPFile[] parseFileList(InputStream listStream) throws IOException {
  + public FTPFile[] parseFileList(InputStream listStream, String encoding)
  throws IOException { FTPListParseEngine engine = new FTPListParseEngine(this); - engine.readServerList(listStream); + engine.readServerList(listStream, encoding); return engine.getFiles(); }
  -
  -

/**

• Parses a line of a VMS FTP server file listing and converts it into a
  cvs server: Diffing src/java/org/apache/commons/net/io
  cvs server: Diffing src/java/org/apache/commons/net/nntp
  cvs server: Diffing src/java/org/apache/commons/net/pop3
  cvs server: Diffing src/java/org/apache/commons/net/smtp
  cvs server: Diffing src/java/org/apache/commons/net/telnet
  cvs server: Diffing src/java/org/apache/commons/net/tftp
  cvs server: Diffing src/java/org/apache/commons/net/util
  cvs server: Diffing src/test
  cvs server: Diffing src/test/org
  cvs server: Diffing src/test/org/apache
  cvs server: Diffing src/test/org/apache/commons
  cvs server: Diffing src/test/org/apache/commons/net
  cvs server: Diffing src/test/org/apache/commons/net/ftp
  cvs server: Diffing src/test/org/apache/commons/net/ftp/parser
  Index: src/test/org/apache/commons/net/ftp/parser/UnixFTPEntryParserTest.java
  ===================================================================
  RCS file:
  /home/cvspublic/jakarta-commons/net/src/test/org/apache/commons/net/ftp/parser/UnixFTPEntryParserTest.java,v
  retrieving revision 1.14
  diff -u -r1.14 UnixFTPEntryParserTest.java
  • • src/test/org/apache/commons/net/ftp/parser/UnixFTPEntryParserTest.java 29
      Jul 2004 11:38:36 -0000 1.14
      +++ src/test/org/apache/commons/net/ftp/parser/UnixFTPEntryParserTest.java 18
      Aug 2004 06:11:33 -0000
      @@ -14,7 +14,6 @@
• limitations under the License.
  */
  package org.apache.commons.net.ftp.parser;
  -
  import java.util.Calendar;

import junit.framework.TestSuite;
@@ -69,7 +68,9 @@
"-rwxr-xr-t 1 500 500 0 Mar 25 08:21 testStickyExec",
"rwSr-Sr- 1 500 500 0 Mar 25 08:22 testSuid",
"rwsr-sr- 1 500 500 0 Mar 25 08:23 testSuidExec",

• "rw-rr- 1 1 3518644 May 25 12:12 std"
  + "rw-rr- 1 1 3518644 May 25 12:12 std",
  + "rw-rw--- 1 ep1adm sapsys 0 8\u6708 17\u65e5 20:10
  caerrinf",
  + "rw-rw--- 1 ep1adm sapsys 0 6\u6708 3\u65e5 2003\u5e74
  \u8a66\u9a13\u30d5\u30a1\u30a4\u30eb.csv"
  };

/**
@@ -215,6 +216,78 @@
cal.set(Calendar.DATE,2);
cal.set(Calendar.HOUR_OF_DAY, 15);
cal.set(Calendar.MINUTE, 13);
+ assertEquals(df.format(cal.getTime()),
+ df.format(f.getTimestamp().getTime()));
+ }
+
+ /**
+ * @see
org.apache.commons.net.ftp.parser.FTPParseTestFramework#testParseFieldsOnFile()
+ */
+ public void testParseFieldsOnFileJapaneseTime() throws Exception
+ {
+ FTPFile f = getParser().parseFTPEntry("-rwxr-xr-x 2 user group
4096 3\u6708 2\u65e5 15:13 zxbox");
+ assertNotNull("Could not parse entry.",
+ f);
+ assertTrue("Should have been a file.",
+ f.isFile());
+ checkPermissions(f);
+ assertEquals(2,
+ f.getHardLinkCount());
+ assertEquals("user",
+ f.getUser());
+ assertEquals("group",
+ f.getGroup());
+ assertEquals("zxbox",
+ f.getName());
+ assertEquals(4096,
+ f.getSize());
+
+ Calendar cal = Calendar.getInstance();
+ cal.set(Calendar.MONTH, Calendar.MARCH);
+
+ cal.set(Calendar.DATE,1);
+ cal.set(Calendar.HOUR_OF_DAY, 0);
+ cal.set(Calendar.MINUTE, 0);
+ cal.set(Calendar.SECOND, 0);
+ if (f.getTimestamp().getTime().before(cal.getTime())) { + cal.add(Calendar.YEAR, -1); + }
+ cal.set(Calendar.DATE,2);
+ cal.set(Calendar.HOUR_OF_DAY, 15);
+ cal.set(Calendar.MINUTE, 13);
+ assertEquals(df.format(cal.getTime()),
+ df.format(f.getTimestamp().getTime()));
+ }
+
+ /**
+ * @see
org.apache.commons.net.ftp.parser.FTPParseTestFramework#testParseFieldsOnFile()
+ */
+ public void testParseFieldsOnFileJapaneseYear() throws Exception
+ { + FTPFile f = getParser().parseFTPEntry("-rwxr-xr-x 2 user group 4096 3\u6708 2\u65e5 2003\u5e74 \u8a66\u9a13\u30d5\u30a1\u30a4\u30eb.csv"); + assertNotNull("Could not parse entry.", + f); + assertTrue("Should have been a file.", + f.isFile()); + checkPermissions(f); + assertEquals(2, + f.getHardLinkCount()); + assertEquals("user", + f.getUser()); + assertEquals("group", + f.getGroup()); + assertEquals("\u8a66\u9a13\u30d5\u30a1\u30a4\u30eb.csv", + f.getName()); + assertEquals(4096, + f.getSize()); + + Calendar cal = Calendar.getInstance(); + cal.set(Calendar.YEAR, 2003); + cal.set(Calendar.MONTH, Calendar.MARCH); + cal.set(Calendar.DATE,2); + cal.set(Calendar.HOUR_OF_DAY, 0); + cal.set(Calendar.MINUTE, 0); + cal.set(Calendar.SECOND, 0); assertEquals(df.format(cal.getTime()), df.format(f.getTimestamp().getTime())); }
Index: src/test/org/apache/commons/net/ftp/parser/VMSFTPEntryParserTest.java
===================================================================
RCS file:
/home/cvspublic/jakarta-commons/net/src/test/org/apache/commons/net/ftp/parser/VMSFTPEntryParserTest.java,v
retrieving revision 1.16
diff -u -r1.16 VMSFTPEntryParserTest.java
— src/test/org/apache/commons/net/ftp/parser/VMSFTPEntryParserTest.java 29 Jun
2004 04:54:32 -0000 1.16
+++ src/test/org/apache/commons/net/ftp/parser/VMSFTPEntryParserTest.java 18 Aug
2004 06:11:33 -0000
@@ -91,7 +91,7 @@
VMSFTPEntryParser parser = new VMSFTPEntryParser();
FTPListParseEngine engine = new FTPListParseEngine(parser);
engine.readServerList(

• new ByteArrayInputStream(fullListing.getBytes()));
  + new ByteArrayInputStream(fullListing.getBytes()), "ISO-8859-1");
  FTPFile[] files = engine.getFiles();
  assertEquals(6, files.length);
  assertFileInListing(files, "2-JUN.LIS");
  @@ -111,7 +111,7 @@
  VMSFTPEntryParser parser = new VMSVersioningFTPEntryParser();
  FTPListParseEngine engine = new FTPListParseEngine(parser);
  engine.readServerList(
• new ByteArrayInputStream(fullListing.getBytes()));
  + new ByteArrayInputStream(fullListing.getBytes()), "ISO-8859-1");
  FTPFile[] files = engine.getFiles();
  assertEquals(3, files.length);
  assertFileInListing(files, "1-JUN.LIS;1");
  cvs server: Diffing src/test/org/apache/commons/net/telnet
  cvs server: Diffing xdocs
  cvs server: Diffing xdocs/images

[Rory Winston]

I have applied the parts of this patch that deal keeping the encoding consistent
throughout the codebase. Thanks, Leif.

[Rory Winston]

Many thanks for the patch Leif. I applied parts of it for release 1.3, however I
have not applied the whole patch, so I felt it better to mark as WONTFIX rather
than FIXED.

[Steve Cohen]

Which parts have already been applied and which parts have you decide not to
apply, Rory?

The business about numeric date formats can handled by the new FTPClientConfig
system. I don't think we should be entertaining any more patches dealing with
Regexes to handle date formats. The new approach, using FTPClientConfigs uses
SimpleDateFormats to handle the dates.

The stuff with encodings is another matter. There may be merit to it but I
haven't had time to look at it yet, either.

[Steve Cohen]

Oh, duh. You DID say what parts of the patch you applied. That part of my last
post should be ignored. You've applied the encoding stuff. That is great.

But the date formats are now handled a different way as I indicated in my
previous post. So WONTFIX is the appropriate designation here. There is a
solution to the Date Format problem. It is the new FTPClientConfig system.
Therefore I am changing this to FIXED. Your committing of Mr. Mortenson's patch
fixes the encoding problem. And the FTPClientConfig date parsing fixes the date
problem.

[Steve Cohen]

Aargh. My last post was garbled too. Here is what I meant to say:

But the date formats are now handled a different way as I indicated in my
previous post. So WONTFIX is NOT the appropriate designation here. There is
now a solution to the date formatting problem in the new FTPClientConfig system
with its Date Format options. Your committing of Mr. Mortenson's patch fixes
the encoding problem. And the FTPClientConfig date parsing fixes the date
problem. Since both pieces of this are now fixed, I have changed the status
accordingly, to FIXED.

[Rory Winston]

Steve,

I applied the part solely dealing with encosng for 1.3. I didnt apply any
changes to the regex portion.

[Steve Cohen]

(In reply to comment #7)
> Steve,
>
> I applied the part solely dealing with encosng for 1.3. I didnt apply any
> changes to the regex portion.

I think you did, didn't you? Isn't that what change 161502 is about in
UnixFTPEntryParser? You may have intended it for another issue, but I think it
solves this one.

[Steve Cohen]

Well, it may not have been fixed at the time I marked it so, but I believe that
with my most recent changes of this weekend, it is.

Mr. Mortenson:
Here is the story as far as the numeric date format goes. Back in January, in
response to several infrequent but persistent complaints about being unable to
parse dates written with month names in languages other than English (leaving
aside for the moment languages like Japanese which require encoding changes as
you know), a new system based on the new class FTPClientConfig was created.
This allows, among other things, a user to pass to the parser a SimpleDateFormat
string to parse a date not formatted in the "standard" way, as well as to pass
in a set of month names other than those of the locale in which the client is
operating.

Somewhat later, a user informed us (it was a first for us) that there now
existed unix FTP servers that used a numeric date format. Thinking that I had
fortuitously solved that problem without even realizing it, I pointed him at the
new code. Turned out, it didn't quite work - the user could pass in the new
date format, but there was another piece of code that pulled out the Date
portion of the listing and sent it to the DateFormat for parsing. That part did
not work with the all-numeric date format. After the fixes of this weekend, it
now does. This is why the bug is marked fixed.

However, it's not seamless. The default date format is still the classic "MMM
dd yyyy" one, not the yyyy-MM-dd that you need to handle all-numeric formatted
dates. There is a backward-compatibility problem with the servers that do not
use the all-numeric format. It is my fond wish that the all-numeric format will
be adopted soon and become ubiquitous. Then we can think about changing the
default behavior. But for now, if you construct the parser with the constructor
that takes an FTPClientConfig properly initialized you should be able to get
this to work.