[HADOOP-18382] Upgrade AWS SDK to V2 - Prerequisites

• Type: Sub-task
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 3.4.0, 3.3.5
• Fix Version/s: 3.4.0, 3.3.5
• Component/s: fs/s3
• Labels: pull-request-available
• Target Version/s: 3.4.0, 3.3.5
• Hadoop Flags: Incompatible change

Release Note

In preparation for an (incompatible but necessary) move to the AWS SDK v2, some uses of internal/deprecated uses of AWS classes/interfaces are logged as warnings, though only once during the life of a JVM. Set the log "org.apache.hadoop.fs.s3a.SDKV2Upgrade" to only log at INFO to hide these.

• Assignee: Ahmar Suhail
• Reporter: Ahmar Suhail
• Created: 01/Aug/22 12:15
• Updated: 28/Jan/24 00:17
• Resolved: 05/Oct/22 14:11

Description

We want to update the AWS SDK to V2, before we do this we should warn on things that will no longer supported. The following changes should be made:

 

• getAmazonS3Client() - Warn that this method will be removed 
• initCustomSigners() - Warn that the interface is changing, any custom signers will need to be updated

• bindAWSClient - If DT is enabled, warn that credential providers interface is changing, any custom cred providers used in binding classes will need to be updated
•  buildAWSProviderList - if any SDK V1 cred providers are in this list, warn that these will be removed
• S3ClientFactory - Update javadocs to say this interface will be replaced by a V2 client factory, mark interface deprecated?

Issue Links

• causes: HADOOP-18481 AWS v2 SDK upgrade log to not warn of use standard AWS Credential Providers
• links to: GitHub Pull Request #4698