[SPARK-45721] Upgrade AWS SDK to v2 for Hadoop dependency

• Type: Sub-task
• Status: Closed
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: 3.5.0
• Component/s: Build
• Target Version/s: 4.0.0
• Assignee: Unassigned
• Reporter: Lantao Jin
• Created: 30/Oct/23 09:36
• Updated: 21/Mar/24 05:17
• Resolved: 21/Mar/24 05:17

Description

Hadoop is planning to ship the SDKv2 upgrade in 3.4.0, as shown in this HADOOP-18073. One of the Hadoop modules that Spark relies on is hadoop-aws, which comes with S3A connector that allows Spark to access data in S3 buckets. Hadoop-aws contains dependency on AWS SDKv1, thus we should also update the Hadoop version to 3.4.0.

Issue Links

• is superceded by: SPARK-45393 Upgrade Hadoop to 3.4.0