Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-9384

Update S3 native fs implementation to use AWS SDK to support authorization through roles

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Closed
    • Minor
    • Resolution: Won't Fix
    • None
    • None
    • fs/s3
    • Locally: RHEL 6, AWS S3
      Remotely: AWS EC2 (RHEL 6), AWS S3

    Description

      Currently the S3 native implementation org.apache.hadoop.fs.s3native.Jets3tNativeFileSystemStore requires credentials to be set explicitly. Amazon allows setting credentials for instances instead of users, via roles. Such are rotated frequently and kept in a local cache all of which is handled by the AWS SDK in this case the AmazonS3Client. The SDK follows a specific order to establish whether credentials are set explicitly or via a role:

      To support this feature the current NativeFileSystemStore implementation needs to be altered to use the AWS SDK instead of the JetS3t S3 libraries.

      A request for this feature has previously been raised as part of the Flume project (FLUME-1691) where the HDFS on top of S3 implementation is used as a manner of logging into S3 via an HDFS Sink.

      Attachments

        1. HADOOP-9384.patch
          13 kB
          D. Granit
        2. HADOOP-9384-v2.patch
          14 kB
          D. Granit

        Issue Links

          Activity

            People

              Unassigned Unassigned
              drgranit D. Granit
              Votes:
              1 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: