[IMPALA-6067] S3: Impala should be able to use IAM roles to access s3 storage

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: Impala 2.7.0, Impala 2.8.0, Impala 2.9.0, Impala 2.10.0
• Fix Version/s: Impala 2.11.0
• Component/s: Infrastructure
• Environment: EC2
• Assignee: Laszlo Gaal
• Reporter: Laszlo Gaal
• Created: 17/Oct/17 15:02
• Updated: 11/Dec/17 19:10
• Resolved: 11/Dec/17 19:10

Description

Amazon VMs can access s3 storage using supplied IAM roles for authentication, bypassing the need to carry around AWS access keys in environment variables, which are vulnerable to accidental disclosure.

Impala should be able to utilize this mechanism for s3 access in addition to the usual AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY credential pair.

This Amazon page explains how to obtain the temporary role credentials from within an EC2 instance.

HADOOP-13277 explains how to configure HDFS to automatically authenticate to S3 using the instance's role.