Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-15620 Über-jira: S3A phase VI: Hadoop 3.3 features
  3. HADOOP-15503

strip s3.amazonaws.com off hostnames before making s3a calls

    XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Open
    • Priority: Minor
    • Resolution: Unresolved
    • Affects Version/s: 3.1.0
    • Fix Version/s: None
    • Component/s: fs/s3
    • Labels:
      None

      Description

      If you copy an http URL https://bucketname.s3.amazonaws.com/ and convert to an s3a one by replacing the schema, you can try to list

      bin/hadoop fs -ls -r s3a://hwdev-steve-new.s3.amazonaws.com/
      

      But do that and you get told there's no such bucket

      ls: Bucket hwdev-steve-new.s3.amazonaws.com does not exist
      

      This is non-intuitive, and catches me out.

      We could strip this automatically during initialization to produce the actual bucket, which would need to be done before any per-bucket init is done.

      I do worry about what could break though

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              stevel@apache.org Steve Loughran
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated: