[HADOOP-12807] S3AFileSystem should read AWS credentials from environment variables

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 2.7.2
• Fix Version/s: 2.8.0, 3.0.0-alpha1
• Component/s: fs/s3
• Release Note: Adds support to S3AFileSystem for reading AWS credentials from environment variables.
• Assignee: Tobin Baker
• Reporter: Tobin Baker
• Created: 16/Feb/16 01:58
• Updated: 30/Aug/16 01:18
• Resolved: 06/Jun/16 21:52

Description

Unlike the DefaultAWSCredentialsProviderChain in the AWS SDK, the AWSCredentialsProviderChain constructed by S3AFileSystem does not include an EnvironmentVariableCredentialsProvider instance. This prevents users from supplying AWS credentials in the environment variables AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY, which is the only alternative in some scenarios.

In my scenario, I need to access S3 from within a test running in a CI environment that does not support IAM roles but does allow me to supply encrypted environment variables. Thus, the only secure approach I can use is to supply my AWS credentials in environment variables (plaintext configuration files are out of the question).

Issue Links

• is depended upon by: HADOOP-11694 Über-jira: S3a phase II: robustness, scale and performance
• is duplicated by: HADOOP-13277 Need To Support IAM role based access for supporting Amazon S3
• relates to: HADOOP-12548 Read s3a creds from a Credential Provider
• relates to: HADOOP-12806 Hadoop fs s3a lib not working with temporary credentials in AWS Lambda