We're transitioning to Hadoop. Until then, we're parsing the files that Flume drops on S3.
S3's API says that keys will be returned in order. It's easy to ask S3:
"Given I am on 2011-03-17/0400/flume-1.seq, give me one file."
Assuming the next lexicographically ordered file is 2011-03-17/0400/flume-2.seq, then you don't have to do any cumbersome faux-directory sweeping (since S3 doesn't know about directories per se). You can let Amazon do that work for you.
We don't have any requirements about sprintf-style formatting of the filename; just that they're written in order