[FLINK-8439] Add Flink shading to AWS credential provider s3 hadoop config

• Type: Improvement
• Status: Closed
• Priority: Critical
• Resolution: Fixed
• Fix Version/s: 1.6.0
• Component/s: Documentation
• Labels: pull-request-available
• Assignee: Andrey Zagrebin
• Reporter: Dyana Rose
• Created: 16/Jan/18 15:17
• Updated: 01/Jul/19 12:41
• Resolved: 27/Jul/18 11:53

Description

This came up when using the s3 for the file system backend and running under ECS.

With no credentials in the container, hadoop-aws will default to EC2 instance level credentials when accessing S3. However when running under ECS, you will generally want to default to the task definition's IAM role.

In this case you need to set the hadoop property

fs.s3a.aws.credentials.provider

to a fully qualified class name(s). see hadoop-aws docs

This works as expected when you add this setting to flink-conf.yaml but there is a further 'gotcha.'  Because the AWS sdk is shaded, the actual full class name for, in this case, the ContainerCredentialsProvider is

org.apache.flink.fs.s3hadoop.shaded.com.amazonaws.auth.ContainerCredentialsProvider

 

meaning the full setting is:

fs.s3a.aws.credentials.provider: org.apache.flink.fs.s3hadoop.shaded.com.amazonaws.auth.ContainerCredentialsProvider

If you instead set it to the unshaded class name you will see a very confusing error stating that the ContainerCredentialsProvider doesn't implement AWSCredentialsProvider (which it most certainly does.)

Adding this information (how to specify alternate Credential Providers, and the name space gotcha) to the AWS deployment docs would be useful to anyone else using S3.

Issue Links

• is related to: FLINK-13044 Shading of AWS SDK in flink-s3-fs-hadoop results in ClassNotFoundExceptions
• links to: GitHub Pull Request #6405
• links to: GitHub Pull Request #6405