Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-9384

Update S3 native fs implementation to use AWS SDK to support authorization through roles

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Minor
    • Resolution: Won't Fix
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: fs/s3
    • Labels:
    • Environment:

      Locally: RHEL 6, AWS S3
      Remotely: AWS EC2 (RHEL 6), AWS S3

    • Target Version/s:

      Description

      Currently the S3 native implementation org.apache.hadoop.fs.s3native.Jets3tNativeFileSystemStore requires credentials to be set explicitly. Amazon allows setting credentials for instances instead of users, via roles. Such are rotated frequently and kept in a local cache all of which is handled by the AWS SDK in this case the AmazonS3Client. The SDK follows a specific order to establish whether credentials are set explicitly or via a role:

      To support this feature the current NativeFileSystemStore implementation needs to be altered to use the AWS SDK instead of the JetS3t S3 libraries.

      A request for this feature has previously been raised as part of the Flume project (FLUME-1691) where the HDFS on top of S3 implementation is used as a manner of logging into S3 via an HDFS Sink.

        Attachments

        1. HADOOP-9384-v2.patch
          14 kB
          D. Granit
        2. HADOOP-9384.patch
          13 kB
          D. Granit

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                drgranit D. Granit
              • Votes:
                1 Vote for this issue
                Watchers:
                8 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: