[HADOOP-17705] S3A to add option fs.s3a.endpoint.region to set AWS region

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 3.3.2
• Component/s: fs/s3
• Labels: pull-request-available
• Assignee: Mehakmeet Singh
• Reporter: Mehakmeet Singh
• Created: 17/May/21 11:21
• Updated: 28/Apr/23 10:33
• Resolved: 24/May/21 12:14

Description

Currently, AWS region is either constructed via the endpoint URL, by making an assumption that the 2nd component after delimiter "." is the region in endpoint URL, which doesn't work for private links and sets the default to us-east-1 thus causing authorization issue w.r.t the private link.

The option fs.s3a.endpoint.region allows this to be explicitly set

how to set the s3 region on older hadoop releases

For anyone who needs to set the signing region on older versions of the s3a client you do not need this festure. instead just provide a custom endpoint to region mapping json file

1. Download the default region mapping file awssdk_config_default.json
2. Add a new regular expression to map the endpoint/hostname to the target region
3. Save the file as /etc/hadoop/conf/awssdk_config_override.json
4. verify basic hadop fs -ls commands work
5. copy to the rest of the cluster.
6. There should be no need to restart any services

Issue Links

• is depended upon by: HADOOP-15603 S3A to support configuring various AWS S3 client extended options
• is depended upon by: HADOOP-17677 Distcp is unable to determine region with S3 PrivateLink endpoints
• is required by: HADOOP-18338 Unable to access data from S3 bucket over a vpc endpoint - 400 bad request
• relates to: HADOOP-18448 s3a endpoint per bucket configuration in pyspark is ignored
• links to: GitHub Pull Request #3020
• links to: GitHub Pull Request #5046
• links to: https://github.com/aws/aws-sdk-java/blob/master/aws-java-sdk-core/src/main/resources/com/amazonaws/internal/config/awssdk_config_default.json