[SPARK-46793] Revert S3A endpoint fixup logic of SPARK-35878

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: 3.5.0, 3.4.3
• Component/s: Spark Core
• Assignee: Unassigned
• Reporter: Steve Loughran
• Created: 22/Jan/24 11:58
• Updated: 02/Apr/24 18:20
• Resolved: 21/Mar/24 05:18

Description

The v2 SDK does its region resolution "differently", and the changes of SPARK-35878 actually create problems.

That PR went in to fix a regression in Hadoop 3.3.1 which has been fixed since 3.3.2; removing it is not going to cause problems on anyone not using the 3.3.1 release, which is 3 years old and replaced by multiple follow on 3.3.x releases

Issue Links

• duplicates: SPARK-47113 Revert S3A endpoint fixup logic of SPARK-35878
• relates to: HADOOP-19044 AWS SDK V2 - Update S3A region logic
• relates to: SPARK-35878 add fs.s3a.endpoint if unset and fs.s3a.endpoint.region is null