[FLUME-1691] Allow use of EC2 roles with S3 sink

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Won't Fix
• Affects Version/s: 0.9.4
• Fix Version/s: 0.9.5
• Component/s: Sinks+Sources
• Assignee: Unassigned
• Reporter: Steve Stogner
• Created: 10/Nov/12 00:30
• Updated: 05/Nov/14 11:34
• Resolved: 05/Nov/14 11:34

Description

If you assign an IAM role to an EC2 instance, then AWS exposes role credentials through the metadata interface. These credentials are temporary credentials that AWS rolls periodically. When making calls to AWS with temporary credentials, you have to use a token in addition to the access ID and secret key. Flume would impress if it would default to the EC2 role credentials when using an S3 sink with no credentials configuration required. Flume would either refresh the credentials from the metadata with every call to S3 or when it detects that the credentials have expired. Users could still override the use of role credentials with user credentials via the current configuration method (fs.s3.awsAccessKeyId, fs.s3.awsSecretAccessKey, fs.s3n.awsAccessKeyId, fs.s3n.awsSecretAccessKey).

Issue Links

• requires: HADOOP-9384 Update S3 native fs implementation to use AWS SDK to support authorization through roles