[HADOOP-18095] S3A connector to improve support for all AWS partitions

• Type: Sub-task
• Status: Open
• Priority: Minor
• Resolution: Unresolved
• Affects Version/s: 3.3.2
• Component/s: fs/s3
• Assignee: Unassigned
• Reporter: Steve Loughran
• Created: 24/Jan/22 16:52
• Updated: 11/Aug/23 09:38

Description

There are some minor issues in using the S3A connector's more advanced features in china

see https://docs.aws.amazon.com/general/latest/gr/aws-arns-and-namespaces.html

Specifically, that "arn:aws:" prefix we use for all arns needs to be configurable so that aws-cn can be used instead.

This means finding where we create and use these in production code (dynamically creating IAM role policies) and in tests, and making it configurable.

proposed

• add an option fs.s3a.aws.partition, default aws.
• new StoreContext methods to query this, and create the arn for the current bucket (string concat or from the bucket's ARN if created with an AP ARN)
• docs

I remember ABFS had a problem with oauth endpoints, that was a lot more serious.

Can't think of real tests for this, other than verifying that if you create an invalid partition "aws-mars" some things break.

someone needs to run all our existing tests in china, including those with IAM roles and SSE-KMS.

Issue Links

• relates to: HADOOP-18194 Public dataset class for S3A integration tests
• relates to: HADOOP-18328 S3A supports S3 on Outposts