[HADOOP-15503] strip s3.amazonaws.com off hostnames before making s3a calls

• Type: Sub-task
• Status: Open
• Priority: Minor
• Resolution: Unresolved
• Affects Version/s: 3.1.0
• Component/s: fs/s3
• Assignee: Unassigned
• Reporter: Steve Loughran
• Created: 30/May/18 12:41
• Updated: 04/Oct/22 16:23

Description

If you copy an http URL https://bucketname.s3.amazonaws.com/ and convert to an s3a one by replacing the schema, you can try to list

bin/hadoop fs -ls -r s3a://hwdev-steve-new.s3.amazonaws.com/

But do that and you get told there's no such bucket

ls: Bucket hwdev-steve-new.s3.amazonaws.com does not exist

This is non-intuitive, and catches me out.

We could strip this automatically during initialization to produce the actual bucket, which would need to be done before any per-bucket init is done.

I do worry about what could break though