[HADOOP-18073] S3A: Upgrade AWS SDK to V2

• Type: Task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.3.1
• Fix Version/s: 3.4.0
• Component/s: auth, fs/s3
• Labels: pull-request-available
• Target Version/s: 3.4.0
• Hadoop Flags: Incompatible change

Release Note

The S3A connector now uses the V2 AWS SDK.
This is a significant change at the source code level.
Any applications using the internal extension/override points in
the filesystem connector are likely to break.
Consult the document aws_sdk_upgrade for the full details.

• Assignee: Ahmar Suhail Assign to me
• Reporter: xiaowei sun
• Created: 08/Jan/22 00:22
• Updated: 24/Oct/24 17:43
• Resolved: 11/Sep/23 13:38

Description

This task tracks upgrading Hadoop's AWS connector S3A from AWS SDK for Java V1 to AWS SDK for Java V2.

Original use case:

We would like to access s3 with AWS SSO, which is supported in software.amazon.awssdk:sdk-core:2.*.

In particular, from https://hadoop.apache.org/docs/stable/hadoop-aws/tools/hadoop-aws/index.html, when to set 'fs.s3a.aws.credentials.provider', it must be "com.amazonaws.auth.AWSCredentialsProvider". We would like to support "software.amazon.awssdk.auth.credentials.ProfileCredentialsProvider" which supports AWS SSO, so users only need to authenticate once.

Issue Links

• breaks: SPARK-38958 Override S3 Client in Spark Write/Read calls
• causes: HBASE-28056 [HBoss] add support for AWS v2 SDK
• is a child of: HADOOP-18995 S3A: Upgrade AWS SDK version to 2.21.33 for Amazon S3 Express One Zone support
• is depended upon by: HADOOP-18886 S3A: AWS SDK V2 Migration: stabilization and S3Express
• is depended upon by: HADOOP-18477 Über-jira: S3A Hadoop 3.3.9 features
• is related to: HADOOP-18352 Support AWS IAM Identity Centre (prev. AWS SSO) for providing credentials to S3A
• is related to: SPARK-45393 Upgrade Hadoop to 3.4.0
• is related to: SPARK-44124 Upgrade AWS SDK to v2
• relates to: HADOOP-18286 S3a: allow custom retry policies
• relates to: SPARK-48571 Reduce the number of accesses to S3 object storage
• links to: GitHub Pull Request #4706
• links to: GitHub Pull Request #4944
• links to: GitHub Pull Request #5163
• links to: GitHub Pull Request #5684
• links to: GitHub Pull Request #5707
• links to: GitHub Pull Request #5857
• links to: GitHub Pull Request #5858
• links to: GitHub Pull Request #5980
• links to: GitHub Pull Request #5981
• links to: GitHub Pull Request #6303

Sub-Tasks

• Upgrade AWS SDK to V2 - Prerequisites: Resolved
• AWS v2 SDK upgrade log to not warn of use standard AWS Credential Providers: Resolved
• AWS SDK V2 - Complete outstanding items: Resolved
• AWS SDK V2 - Update region logic: Resolved
• AWS SDK V2 - Qualify the upgrade.: Resolved
• AWS SDK V2 - Fix failing tests: Resolved
• AWS SDK v2 code tuning: Resolved
• AWS v2 SDK: stabilise dependencies with rest of hadoop libraries: Resolved
• convert declarations of AWS v1 SDK EnvironmentVariableCredentialsProvider to v2 version: Resolved
• AWS SDK V2 - ITestS3AHugeFilesNoMultipart failure: Resolved
• s3a DelegationToken plugin to expand return type of deploy/binding: Resolved
• LogExactlyOnce to add a debug() method: Resolved
• list AWS SDK v2 libraries in LICENSE-binary: Resolved
• Merge aws v2 upgrade feature branch into trunk: Resolved
• AWS SDK v2 build complaints: Resolved
• AWS SDK v2: make the v1 bridging support optional: Resolved
• AWS SDK V2 - Upgrade SDK to 2.20.28 and restores multipart copy: Resolved
• AWS SDK v2 typo in aws.evenstream.version: Resolved
• AWS SDK V2 - AuditFailureExceptions aren't being translated properly: Resolved