Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-12723

S3A: Add ability to plug in any AWSCredentialsProvider

    Details

    • Type: New Feature
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.7.1
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: fs/s3
    • Labels:
      None
    • Target Version/s:
    • Release Note:
      Users can integrate a custom credential provider with S3A. See documentation of configuration property fs.s3a.aws.credentials.provider for further details.

      Description

      Although S3A currently has built-in support for org.apache.hadoop.fs.s3a.BasicAWSCredentialsProvider, com.amazonaws.auth.InstanceProfileCredentialsProvider, and org.apache.hadoop.fs.s3a.AnonymousAWSCredentialsProvider, it does not support any other credentials provider that implements the com.amazonaws.auth.AWSCredentialsProvider interface. Supporting the ability to plug in any com.amazonaws.auth.AWSCredentialsProvider instance will expand the options for S3 credentials, such as:

      • temporary credentials from STS, e.g. via com.amazonaws.auth.STSSessionCredentialsProvider
      • IAM role-based credentials, e.g. via com.amazonaws.auth.STSAssumeRoleSessionCredentialsProvider
      • a custom credentials provider that satisfies one's own needs, e.g. bucket-specific credentials, user-specific credentials, etc.

      To support this, we can add a configuration for the fully qualified class name of a credentials provider, to be loaded by S3AFileSystem.initialize(URI, Configuration).

      The configured credentials provider should implement com.amazonaws.auth.AWSCredentialsProvider and have a constructor that accepts (URI uri, Configuration conf).

        Attachments

        1. HADOOP-12723-branch-2-006.patch
          13 kB
          Steve Loughran
        2. HADOOP-12723.5.patch
          13 kB
          Steven K. Wong
        3. HADOOP-12723.4.patch
          10 kB
          Steven K. Wong
        4. HADOOP-12723.3.patch
          10 kB
          Steven K. Wong
        5. HADOOP-12723.2.patch
          9 kB
          Steven K. Wong
        6. HADOOP-12723.1.patch
          9 kB
          Steven K. Wong
        7. HADOOP-12723.0.patch
          9 kB
          Steven K. Wong

          Issue Links

            Activity

              People

              • Assignee:
                slider Steven K. Wong
                Reporter:
                slider Steven K. Wong
              • Votes:
                0 Vote for this issue
                Watchers:
                9 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: