Details
-
Sub-task
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
3.4.0
Description
If both fs.s3a.endpoint & fs.s3a.endpoint.region are empty, Spark will set fs.s3a.endpoint to
s3.amazonaws.com here:
HADOOP-18908, updated the region logic such that if fs.s3a.endpoint.region is set, or if a region can be parsed from fs.s3a.endpoint (which will happen in this case, region will be US_EAST_1), cross region access is not enabled. This will cause 400 errors if the bucket is not in US_EAST_1.
Proposed: Updated the logic so that if the endpoint is the global s3.amazonaws.com , cross region access is enabled.
Attachments
Issue Links
- causes
-
HADOOP-19066 AWS SDK V2 - Enabling FIPS should be allowed with central endpoint
- Resolved
- is related to
-
SPARK-46793 Revert S3A endpoint fixup logic of SPARK-35878
- Closed
- links to