[net] FTP timestamp: year recognition

• Key: NET-83
• Type: Bug
• Status: Closed
• Resolution: Fixed
• Priority: Major
• Assignee: Unassigned
• Reporter: Satoshi Ishigami
• Affects Version/s: 1.4

Environment

Operating System: All
Platform: All

Description

org.apache.commons.net.ftp.parser.FTPTimestampParserImpl

public Calendar parseTimestamp(String timestampStr)
:
if (working.after(now)) { working.add(Calendar.YEAR, -1); }

If system date of the remote machine is ahead of local machine, the code above
does not work well. For example, when the system date of local machine is five
minutes later than date of file at the remote machine, it is recognized as the
file a year ago.

LOCAL SYSDATE:Jun/02/2005 14:00
File TimeStamp
Jun/02 14:05 -recognized as-> Jun/02/2004 14:05

The date format is switched after six months, so I think that the code above
shuold be changed as following:

// s m h d half-year
// 1000*60*60*24*183=15811200000ms
long wkt = working.getTimeInMillis();
long nwt = now.getTimeInMillis();
if (wkt > (nwt+15811200000L)) { working.add(Calendar.YEAR, -1); } }

Comments

[Steve Cohen]

Synchronize your machines if you're dealing in intervals this fine!

How do you KNOW "the date format is switched after six months"? Is that the way
your server does it? The problem with FTP is that there is NO standard on these
matters. On the other hand if you DO have some authoritative statement of what
you're saying, I'd love to see it. Please send it.

Or, if timezone differences are the reason for your discrepancy, 1.4.0 offers a
way to handle it. It's in the documentation. See FTPClientConfig.

[Steve Cohen]

On further review – you're right - or mostly so!

I checked a few popular FTP servers (including several Apache mirrors) and all
seem to follow your pattern. They show the year after six months.

Then I looked back at earlier versions of our code.

Prior to the introduction of
org.apache.commons.net.ftp.parser.FTPTimestampParserImpl, the default
implementation simply compared months. If the month on the timestamp was
greater than the current month, it must be last year. That was the test. It
eliminates most "slop". I suppose it might fail when the current time is Jan 31
23:59:59 and file timestamp is Feb 1 00:00, but it basically gave the right
results except in these edge cases.

The new code "improved" things by delegating to java.util.Date's "simple"
after() method but made matters worse in the "slop" case.

Your suggestion probably works best of all, but I'm still a little bothered by
the six month thing. Suppose there is an FTP server that displays the time for
all files less than a year old - which is what the old code, written by Daniel
Savarese, seems to imply. Then your code breaks there.

Perhaps Daniel might like to chime in on how he remembers this?

[Daniel Savarese]

I think sometimes the year would be missing for
6 months ago < date < 12 months ago
but I can't be sure. I dug up an archive of the original oroinc.com
CVS repository and looked through the log comments but couldn't come
up with anything definitive. The looser test may have simply been
defensive programming. 1997 was oh so long ago.

[Steve Cohen]

OK - we're in uncharted waters here and I am not entirely sure what is the right
thing to do.

Daniel Savarese, the original author of this code base does not remember the
reasons for his original code which was more tolerant but not completely so of
the client and server clocks being out of synch.

The current implementation is more "accurate" but at the cost of being
completely intolerant of "slop".

I could for example, without too much trouble, recode this so that any server
timestamps that appear to be more than one day in the future be considered last
year. That would handle this for all timezones in the world as well as slop,
and it may be the most faithful rendition of what appears to be Daniel's
original intent. But I'm still uncomfortable introducing inaccuracies to
account for slop, unless I can be reasonably sure it's harmless.

Mr. Ishigami: I need more information from you.
1) Can you tell me your exact use case? Are you having this problem because
a) the clocks are out of synch and you could fix the problem
b) the clocks are out of synch but you can't do anything about it beacuse the
server is out of your control?
c) the server is in a different time zone from the client which you could handle
by using FTPClientConfig?

2) Please indicate whether you have any information other than observation that
says that the rollover from MMM dd HH:mm to MMM dd yyyy display occurs when a
file becomes six months old.

Parenthetically, I'll note that this whole business is a powerful argument in
favor of those FTP servers, such as the one that Neeme Praks indicates is now
standard on Debian GNU Linux (not to mention Windows), which display all
timestamps as yyyy-MM-dd HH:mm, instead of making us guess.

[Steve Cohen]

(In reply to comment #4)

> Parenthetically, I'll note that this whole business is a powerful argument in
> favor of those FTP servers, such as the one that Neeme Praks indicates is now
> standard on Debian GNU Linux (not to mention Windows), which display all
> timestamps as yyyy-MM-dd HH:mm, instead of making us guess.

Out of curiosity, I took a look at some of Debian's FTP mirrors and it appears
they do not use their numeric format yet.

[Steve Cohen]

(In reply to comment #4)
>> The current implementation is more "accurate" but at the cost of being
> completely intolerant of "slop".
>
> I could for example, without too much trouble, recode this so that any server
> timestamps that appear to be more than one day in the future be considered last
> year.

I have had a go at testing this idea. It breaks our JUnit tests, but that's
just because the JUnit tests were written with a "no-slop" approach. Since I'm
having trouble deciding whether it's a good thing or a bad thing to allow for
"slop", maybe that's a sign that we need a "slop-mode" option in FTPClientConfig
(probably named differently)? Perhaps something similar to how
SimpleDateFormat.isLenient() exists because one size does not fit all?

[Satoshi Ishigami]

> 1) Can you tell me your exact use case? Are you having this problem because
> a) the clocks are out of synch and you could fix the problem
> b) the clocks are out of synch but you can't do anything about it beacuse the
> server is out of your control?
> c) the server is in a different time zone from the client which you could handle
> by using FTPClientConfig?

b)

In my case, two servers are running on same time zone and the clocks are out of
synch.

I use the Commons Net for the purpose of B2B. In case of intranet, several
servers can synchronize clocks, for instance, by using time server. On the other
hand, in case of B2B (or internet), clocks are out of synch because each server
is controled under each organization. All organizations do not always do
synchronization. This says some server is out of my control.

This problem is reporductable as following for my case:

Server A=14:00 Server B=14:05
1) create file on Server B (timestamp is 14:05)
2) connect from Sercer A to Server B and execute LIST.

I tried to quick-fix this problem to described code. But, I thought that this is
not only my problem, so I reported.

> 2) Please indicate whether you have any information other than observation
> that says that the rollover from MMM dd HH:mm to MMM dd yyyy display occurs
> when a file becomes six months old.

Six month came from my observation. I do not have further information.
I check in my environments:

Solaris 9 FTP Server rollover six months.
HP-UX 11.11 FTP server (Version 1.1.214.4) rollover six months.
Tru64 UNIX FTP server (Version 5.60) rollover six months.
RHL, ES3 vsFTPd 1.2.0 rollover six months.
Win2003 Microsoft FTP Service rollover by year?

I think that the problem is how long local server can accept when timestamp of
file on remote server is ahead of system date of local server: For a day, a week,
a month or a year?

My suggestion is six months becase of observation above.

[Daniel Savarese]

(In reply to comment #6)
> having trouble deciding whether it's a good thing or a bad thing to allow for
> "slop", maybe that's a sign that we need a "slop-mode" option in FTPClientConfig
> (probably named differently)? Perhaps something similar to how
> SimpleDateFormat.isLenient() exists because one size does not fit all?

That sounds like a good compromise. Every time we've tried to impose
a single way to handle this sort of thing, we always seem to get bitten
by some special case (curse FTP and its unspecified listing output! .
Letting the API user decide if he wants a looser "slop"-handling
interpretation of dates provides sufficient flexibility
for programmers to work around special cases without forcing the API
to be all-knowing.

[Steve Cohen]

A new method, FTPClientConfig.setLenientFutureDates() allows a "slop" factor of
one day, which sounds like it will fit with your use case.

You'll have to get source from CVS or use nightly builds until this is released.