Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-15620 Über-jira: S3A phase VI: Hadoop 3.3 features
  3. HADOOP-14391

s3a: auto-detect region for bucket and use right endpoint

    XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 3.0.0-alpha2
    • Fix Version/s: None
    • Component/s: fs/s3
    • Labels:
      None
    • Target Version/s:

      Description

      Specifying the S3A endpoint (fs.s3a.endpoint) is

      • required for regions which only support v4 authentication
      • A good practice for all regions.

      The user experience of having to configure endpoints is not great. Often it is neglected and leads to additional cost, reduced performance, or failures for v4 auth regions.

      I want to explore an option which, when enabled, auto-detects the region for an s3 bucket and uses the proper endpoint. Not sure if this is possible or anyone has looked into it yet.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                fabbri Aaron Fabbri
                Reporter:
                fabbri Aaron Fabbri
              • Votes:
                0 Vote for this issue
                Watchers:
                7 Start watching this issue

                Dates

                • Created:
                  Updated: