[HADOOP-18889] S3A: V2 SDK client does not work with third-party store

• Type: Sub-task
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 3.4.0
• Fix Version/s: 3.4.0, 3.3.7-aws
• Component/s: fs/s3
• Labels: pull-request-available
• Target Version/s: 3.4.0, 3.3.7-aws
• Hadoop Flags: Reviewed
• Assignee: Steve Loughran
• Reporter: Steve Loughran
• Created: 12/Sep/23 14:58
• Updated: 27/Jan/24 06:59
• Resolved: 12/Oct/23 16:48

Description

testing against an external store without specifying region now blows up because the region is queried off eu-west-1.

What are we do to here? require the region setting *which wasn't needed before? what even region do we provide for third party stores?

Issue Links

• contains: HADOOP-18857 AWS v2 SDK: fail meaningfully if legacy v2 signing requested
• contains: HADOOP-18927 S3ARetryHandler to treat SocketExceptions as connectivity failures
• is related to: HADOOP-18673 AWS SDK V2 - Refactor getS3Region & other follow up items
• is related to: HADOOP-18915 Tune/extend S3A http connection and thread pool settings
• relates to: HADOOP-18908 Improve s3a region handling, including determining from endpoint
• links to: GitHub Pull Request #6141
• links to: GitHub Pull Request #6180
• links to: GitHub Pull Request #6186