[HADOOP-18328] S3A supports S3 on Outposts

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.4.0, 3.3.9
• Fix Version/s: 3.4.0, 3.3.9
• Component/s: fs/s3
• Labels: pull-request-available
• Target Version/s: 3.4.0, 3.3.9
• Hadoop Flags: Reviewed
• Assignee: Sotetsu Suzugamine
• Reporter: Sotetsu Suzugamine
• Created: 07/Jul/22 02:48
• Updated: 28/Jan/24 00:27
• Resolved: 31/Aug/23 14:03

Description

Currently, the endpoint for using S3 accesspoint is set as "s3-accesspoint.%s.amazonaws.com" as follows.
https://github.com/apache/hadoop/blob/3ec4b932c179d9ec6c4e465f25e35b3d7eded08b/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/ArnResource.java#L29

However, "s3-outposts.%s.amazonaws.com" is the preferred endpoint when accessing S3 on Outposts bucket by accesspoint.
This ticket improves them.

Issue Links

• is related to: HADOOP-18095 S3A connector to improve support for all AWS partitions
• is related to: SPARK-45393 Upgrade Hadoop to 3.4.0
• links to: GitHub Pull Request #4533
• links to: GitHub Pull Request #5976