Attached is first cut at adding s3 handling to CopyFiles.

Here's list of changes:

+ Allow hdfs or dfs URI schemes (Used to be dfs only).
+ Changed the usage message so filesystem is generic URI (rather than namenode:port | local).
+ getFileSysName was removed. Use Filesystem.get with fs URI instead.
+ getMapCount: Moved duplicated code for figuring number of maps here.
+ toURI: Added. Have (duplicated) tests of URIness go via here instead.
+ CopyFilesReducer: Removed two instances. Does nothing.
+ Added testing of URIness to members of file-of-source URIs.
+ Minor javadoc and formatting changes.

Its lightly tested.

Updated patch.

+ Renamed DFSCopyFilesMapper as FSCopyFilesMapper
+ If no scheme, use 'default' (the value of 'fs.default.name' in hadoop-site.xml).

I ran more extensive tests going from hdfs to s3 and back again and copying from http into s3 and hdfs (distcp is a nice tool). For example, here is output from a copy of a small nutch segment from hdfs to s3 (in the below hdfs was set as the fs.default.name filesystem):

stack@debord:~/checkouts/hadoop$ ./bin/hadoop fs -lsr outputs/segments
/user/stack/outputs/segments/20070108213341-test <dir>
/user/stack/outputs/segments/20070108213341-test/crawl_fetch <dir>
/user/stack/outputs/segments/20070108213341-test/crawl_fetch/part-00000 <dir>
/user/stack/outputs/segments/20070108213341-test/crawl_fetch/part-00000/data <r 1> 1187
/user/stack/outputs/segments/20070108213341-test/crawl_fetch/part-00000/index <r 1> 234
/user/stack/outputs/segments/20070108213341-test/crawl_parse <dir>
/user/stack/outputs/segments/20070108213341-test/crawl_parse/part-00000 <r 1> 9010
/user/stack/outputs/segments/20070108213341-test/parse_data <dir>
/user/stack/outputs/segments/20070108213341-test/parse_data/part-00000 <dir>
/user/stack/outputs/segments/20070108213341-test/parse_data/part-00000/data <r 1> 4630
/user/stack/outputs/segments/20070108213341-test/parse_data/part-00000/index <r 1> 234
/user/stack/outputs/segments/20070108213341-test/parse_text <dir>
/user/stack/outputs/segments/20070108213341-test/parse_text/part-00000 <dir>
/user/stack/outputs/segments/20070108213341-test/parse_text/part-00000/data <r 1> 6180
/user/stack/outputs/segments/20070108213341-test/parse_text/part-00000/index <r 1> 234

Here's copy to an s3 directory named segments-bkup:

% ./bin/hadoop distcp /user/stack/outputs/segments s3://KEY:SECRET@BUCKET/segments-bkup

Here's listing of s3 content:

stack@debord:~/checkouts/hadoop$ ./bin/hadoop fs -fs s3://KEY:SECRET@BUCKET/segments-bkup -lsr /segments-bkup/
/segments-bkup/20070108213341-test <dir>
/segments-bkup/20070108213341-test/crawl_fetch <dir>
/segments-bkup/20070108213341-test/crawl_fetch/part-00000 <dir>
/segments-bkup/20070108213341-test/crawl_fetch/part-00000/data <r 1> 1187
/segments-bkup/20070108213341-test/crawl_fetch/part-00000/index <r 1> 234
/segments-bkup/20070108213341-test/crawl_parse <dir>
/segments-bkup/20070108213341-test/crawl_parse/part-00000 <r 1> 9010
/segments-bkup/20070108213341-test/parse_data <dir>
/segments-bkup/20070108213341-test/parse_data/part-00000 <dir>
/segments-bkup/20070108213341-test/parse_data/part-00000/data <r 1> 4630
/segments-bkup/20070108213341-test/parse_data/part-00000/index <r 1> 234
/segments-bkup/20070108213341-test/parse_text <dir>
/segments-bkup/20070108213341-test/parse_text/part-00000 <dir>
/segments-bkup/20070108213341-test/parse_text/part-00000/data <r 1> 6180
/segments-bkup/20070108213341-test/parse_text/part-00000/index <r 1> 234

I just tried using this patch, and I managed to copy some local files to the S3 file system without trouble.

Looking at the code I noticed that the -fs option doesn't seem to be used any longer so it can be dropped. Other than that, it looks fine to me.

Fix usage string (suggested by Tom White review)

Marking issue with 'patch available'.

Thanks for the review Tom.

-1, because 3 attempts failed to build and test the latest attachment (http://issues.apache.org/jira/secure/attachment/12350196/copyfiles-s3-3.diff) against trunk revision r502402. Please note that this message is automatically generated and may represent a problem with the automation system and not the patch.

New patch to fix broken unit test. Removes 'dfs' scheme. Only 'hdfs' allowed from here on out.

Mr 'Hadoop QA', do I have to do anything special to re-trigger your auto-application and test of version 4 of the patch? Thanks.

> Mr 'Hadoop QA' [ ... ]

Please, call him "Nigel".

+1, because http://issues.apache.org/jira/secure/attachment/12350237/copyfiles-s3-4.diff applied and successfully tested against trunk revision r502694.

I just committed this. Thanks, Michael!