[HADOOP-13779] s3a to skip s3 bucket existence check in initialize() for faster creation

• Type: Sub-task
• Status: Resolved
• Priority: Minor
• Resolution: Duplicate
• Component/s: fs/s3
• Assignee: Unassigned
• Reporter: Rajesh Balamohan
• Created: 01/Nov/16 11:31
• Updated: 16/Jun/17 09:54
• Resolved: 16/Jun/17 09:54

Description

The verifyBucketExists() makes an HTTPS HEAD request of a bucket when creating an FS, adding an extra HTTPS call of a few hundred millis. (as this is the first call, cost may be amplified for DNS lookup, thread pool creation, etc, so may appear more expensive).

If a bucket doesn't exist, the first actual client-initiated operation (get, list, put) will trigger a failure, so the call could potentially be stripped out.

1. it will complicate failure reporting, if you want to distinguish "unauthed" and "not found" on a blob from those on a bucket.
2. the fact that the first HTTPS request probably includes first-HTTPS call overhead (pool creation, DNS, any certificate checking) may make that first call appear more expensive than it is...those actions would just take place on the first client-initiated call.

Issue Links

• duplicates: HADOOP-13757 Remove verifyBuckets overhead in S3AFileSystem::initialize()