[HADOOP-12723] S3A: Add ability to plug in any AWSCredentialsProvider

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.7.1
• Fix Version/s: 2.8.0, 3.0.0-alpha1
• Component/s: fs/s3
• Target Version/s: 2.8.0
• Release Note: Users can integrate a custom credential provider with S3A. See documentation of configuration property fs.s3a.aws.credentials.provider for further details.
• Assignee: Steven K. Wong
• Reporter: Steven K. Wong
• Created: 20/Jan/16 21:02
• Updated: 30/Aug/16 01:19
• Resolved: 20/May/16 12:53

Description

Although S3A currently has built-in support for org.apache.hadoop.fs.s3a.BasicAWSCredentialsProvider, com.amazonaws.auth.InstanceProfileCredentialsProvider, and org.apache.hadoop.fs.s3a.AnonymousAWSCredentialsProvider, it does not support any other credentials provider that implements the com.amazonaws.auth.AWSCredentialsProvider interface. Supporting the ability to plug in any com.amazonaws.auth.AWSCredentialsProvider instance will expand the options for S3 credentials, such as:

• temporary credentials from STS, e.g. via com.amazonaws.auth.STSSessionCredentialsProvider
• IAM role-based credentials, e.g. via com.amazonaws.auth.STSAssumeRoleSessionCredentialsProvider
• a custom credentials provider that satisfies one's own needs, e.g. bucket-specific credentials, user-specific credentials, etc.

To support this, we can add a configuration for the fully qualified class name of a credentials provider, to be loaded by S3AFileSystem.initialize(URI, Configuration).

The configured credentials provider should implement com.amazonaws.auth.AWSCredentialsProvider and have a constructor that accepts (URI uri, Configuration conf).

Issue Links

• is depended upon by: HADOOP-12537 S3A to support Amazon STS temporary credentials
• is depended upon by: HADOOP-11694 Über-jira: S3a phase II: robustness, scale and performance
• is duplicated by: HADOOP-13277 Need To Support IAM role based access for supporting Amazon S3