Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-18886 S3A: AWS SDK V2 Migration: stabilization and S3Express
  3. HADOOP-18908

Improve s3a region handling, including determining from endpoint

VotersWatch issueWatchersLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    Description

      S3A region logic improved for better inference and
      to be compatible with previous releases

      1. If you are using an AWS S3 AccessPoint, its region is determined
      from the ARN itself.
      2. If fs.s3a.endpoint.region is set and non-empty, it is used.
      3. If fs.s3a.endpoint is an s3.*.amazonaws.com url,
      the region is determined by by parsing the URL
      Note: vpce endpoints are not handled by this.
      4. If fs.s3a.endpoint.region==null, and none could be determined
      from the endpoint, use us-east-2 as default.
      5. If fs.s3a.endpoint.region=="" then it is handed off to
      The default AWS SDK resolution process.

      Consult the AWS SDK documentation for the details on its resolution
      process, knowing that it is complicated and may use environment variables,
      entries in ~/.aws/config, IAM instance information within
      EC2 deployments and possibly even JSON resources on the classpath.
      Put differently: it is somewhat brittle across deployments.

      Attachments

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            ahmar Ahmar Suhail
            stevel@apache.org Steve Loughran
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment