[SPARK-35878] add fs.s3a.endpoint if unset and fs.s3a.endpoint.region is null

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.2.0
• Fix Version/s: 3.2.0
• Component/s: Spark Core
• Labels: pull-request-available
• Target Version/s: 3.2.0
• Assignee: Steve Loughran
• Reporter: Steve Loughran
• Created: 24/Jun/21 12:45
• Updated: 21/Feb/24 06:27
• Resolved: 25/Jun/21 12:25

Description

People working with S3A and hadoop 3.3.1 outside of EC2, and without the AWS CLI setup, are likely to hit: HADOOP-17771

It should be straightforward to fix up the config similar to SPARK-35868...this will be backwards (harmless) and forwards compatible (it's the recommended workaround)

Issue Links

• is related to: SPARK-46793 Revert S3A endpoint fixup logic of SPARK-35878
• relates to: HADOOP-17771 S3AFS creation fails "Unable to find a region via the region provider chain."
• relates to: SPARK-29250 Upgrade to Hadoop 3.3.1
• links to: [Github] Pull Request #33064 (steveloughran)
• links to: GitHub Pull Request #44834