[HADOOP-18481] AWS v2 SDK upgrade log to not warn of use standard AWS Credential Providers

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.4.0
• Fix Version/s: 3.3.5, 3.3.9
• Component/s: fs/s3
• Labels: pull-request-available
• Assignee: Ahmar Suhail
• Reporter: Steve Loughran
• Created: 05/Oct/22 14:10
• Updated: 14/Oct/22 10:54
• Resolved: 14/Oct/22 10:54

Description

looking at test output with the sdk warnings enabled, it is now always warning of a v1 provider reference, even if the user hasn't set any fs.s3a.credential.provider option

2022-10-05 14:09:09,733 [setup] DEBUG s3a.S3AUtils (S3AUtils.java:createAWSCredentialProvider(691)) - Credential provider class is org.apache.hadoop.fs.s3a.TemporaryAWSCredentialsProvider
2022-10-05 14:09:09,733 [setup] DEBUG s3a.S3AUtils (S3AUtils.java:createAWSCredentialProvider(691)) - Credential provider class is org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider
2022-10-05 14:09:09,734 [setup] WARN  s3a.SDKV2Upgrade (LogExactlyOnce.java:warn(39)) - Directly referencing AWS SDK V1 credential provider com.amazonaws.auth.EnvironmentVariableCredentialsProvider. AWS SDK V1 credential providers will be removed once S3A is upgraded to SDK V2
2022-10-05 14:09:09,734 [setup] DEBUG s3a.S3AUtils (S3AUtils.java:createAWSCredentialProvider(691)) - Credential provider class is com.amazonaws.auth.EnvironmentVariableCredentialsProvider
2022-10-05 14:09:09,734 [setup] DEBUG s3a.S3AUtils (S3AUtils.java:createAWSCredentialProvider(691)) - Credential provider class is org.apache.hadoop.fs.s3a.auth.IAMInstanceCredentialsProvider

This is because the EnvironmentVariableCredentialsProvider provider is on the default list of providers.

Everybody who is using the S3 a connector and who has not explicitly declared a new set of providers excluding this one will be seeing the error message.

Proposed:

Don't warn on this provider. Instead with the v2 move the classname can be patched to switch to a modified one.

The alternative would be to provide an s3a specific env var provider subclass of this; and while that is potentially good in future it is a bit more effort for the forthcoming 3.3.5 release.
And especially because and it will not be in previous versions people cannot explicitly switch to it in their configs and be confident it will always be there,

Issue Links

• is caused by: HADOOP-18382 Upgrade AWS SDK to V2 - Prerequisites
• links to: GitHub Pull Request #4973