Hadoop Common
  1. Hadoop Common
  2. HADOOP-930

Add support for reading regular (non-block-based) files from S3 in S3FileSystem

    Details

    • Type: New Feature New Feature
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.10.1
    • Fix Version/s: 0.18.0
    • Component/s: fs
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Added support for reading and writing native S3 files. Native S3 files are referenced using s3n URIs. See http://wiki.apache.org/hadoop/AmazonS3 for more details.

      Description

      People often have input data on S3 that they want to use for a Map Reduce job and the current S3FileSystem implementation cannot read it since it assumes a block-based format.

      We would add the following metadata to files written by S3FileSystem: an indication that it is block oriented ("S3FileSystem.type=block") and a filesystem version number ("S3FileSystem.version=1.0"). Regular S3 files would not have the type metadata so S3FileSystem would not try to interpret them as inodes.

      An extension to write regular files to S3 would not be covered by this change - we could do this as a separate piece of work (we still need to decide whether to introduce another scheme - e.g. rename block-based S3 to "s3fs" and call regular S3 "s3" - or whether to just use a configuration property to control block-based vs. regular writes).

      1. hadoop-930.patch
        96 kB
        Tom White
      2. hadoop-930-v2.patch
        98 kB
        Tom White
      3. hadoop-930-v3.patch
        99 kB
        Tom White
      4. hadoop-930-v4.patch
        99 kB
        Tom White
      5. hadoop-930-v5.patch
        99 kB
        Tom White
      6. jets3t-0.6.0.jar
        282 kB
        Tom White

        Issue Links

          Activity

          Tom White created issue -
          Tom White made changes -
          Field Original Value New Value
          Attachment hadoop-930.patch [ 12381527 ]
          Tom White made changes -
          Attachment jets3t-0.6.0.jar [ 12381528 ]
          Tom White made changes -
          Link This issue blocks HADOOP-3361 [ HADOOP-3361 ]
          Tom White made changes -
          Attachment hadoop-930-v2.patch [ 12381601 ]
          Tom White made changes -
          Assignee Tom White [ tomwhite ]
          Tom White made changes -
          Fix Version/s 0.18.0 [ 12312972 ]
          Status Open [ 1 ] Patch Available [ 10002 ]
          Tom White made changes -
          Attachment hadoop-930-v3.patch [ 12381659 ]
          Tom White made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Tom White made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Tom White made changes -
          Attachment hadoop-930-v4.patch [ 12383208 ]
          Tom White made changes -
          Link This issue relates to HADOOP-3494 [ HADOOP-3494 ]
          Tom White made changes -
          Attachment hadoop-930-v5.patch [ 12383445 ]
          Doug Cutting made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Release Note Add support for native S3 files.
          Resolution Fixed [ 1 ]
          Hadoop Flags [Reviewed]
          Tom White made changes -
          Release Note Add support for native S3 files. Add support for reading and writing native S3 files. Native S3 files are referenced using s3n URIs. See http://wiki.apache.org/hadoop/AmazonS3 for more details.
          Robert Chansler made changes -
          Release Note Add support for reading and writing native S3 files. Native S3 files are referenced using s3n URIs. See http://wiki.apache.org/hadoop/AmazonS3 for more details. Added support for reading and writing native S3 files. Native S3 files are referenced using s3n URIs. See http://wiki.apache.org/hadoop/AmazonS3 for more details.
          Nigel Daley made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Gavin made changes -
          Link This issue blocks HADOOP-3361 [ HADOOP-3361 ]
          Gavin made changes -
          Link This issue is depended upon by HADOOP-3361 [ HADOOP-3361 ]

            People

            • Assignee:
              Tom White
              Reporter:
              Tom White
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development