[HADOOP-930] Add support for reading regular (non-block-based) files from S3 in S3FileSystem

• Type: New Feature
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.10.1
• Fix Version/s: 0.18.0
• Component/s: fs
• Hadoop Flags: Reviewed
• Release Note: Added support for reading and writing native S3 files. Native S3 files are referenced using s3n URIs. See http://wiki.apache.org/hadoop/AmazonS3 for more details.
• Assignee: Thomas White
• Reporter: Thomas White
• Created: 25/Jan/07 14:48
• Updated: 02/May/13 02:29
• Resolved: 06/Jun/08 21:07

Description

People often have input data on S3 that they want to use for a Map Reduce job and the current S3FileSystem implementation cannot read it since it assumes a block-based format.

We would add the following metadata to files written by S3FileSystem: an indication that it is block oriented ("S3FileSystem.type=block") and a filesystem version number ("S3FileSystem.version=1.0"). Regular S3 files would not have the type metadata so S3FileSystem would not try to interpret them as inodes.

An extension to write regular files to S3 would not be covered by this change - we could do this as a separate piece of work (we still need to decide whether to introduce another scheme - e.g. rename block-based S3 to "s3fs" and call regular S3 "s3" - or whether to just use a configuration property to control block-based vs. regular writes).

Issue Links

• is depended upon by: HADOOP-3361 Implement renames for NativeS3FileSystem
• relates to: HADOOP-3494 Improve S3FileSystem data integrity using MD5 checksums