[HADOOP-17454] [s3a] Disable bucket existence check - set fs.s3a.bucket.probe to 0

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.3.0
• Fix Version/s: 3.3.1
• Labels: pull-request-available

Release Note

S3A bucket existence check is disabled (fs.s3a.bucket.probe is 0), so there will be no existence check on the bucket during the S3AFileSystem initialization. The first operation which attempts to interact with the bucket which will fail if the bucket does not exist.

• Assignee: Gabor Bota
• Reporter: Gabor Bota
• Created: 05/Jan/21 10:38
• Updated: 21/Oct/22 11:52
• Resolved: 05/Jan/21 14:48

Description

Set the value of fs.s3a.bucket.probe to 0 by default.
Bucket existence checks are done in the initialization phase of the S3AFileSystem. It's not required to run this check: the operation itself will fail if the bucket does not exist instead of the check.

Some points on why do we want to set this to 0:

• When it's set to 0, bucket existence checks won't be done during initialization thus making it faster.
• Avoid the additional one or two requests on the bucket root, so the user does not need rights to read or list that folder.

Issue Links

• causes: HADOOP-16995 ITestS3AConfiguration proxy tests fail when bucket probes == 0
• causes: HADOOP-18233 Initialization race condition with TemporaryAWSCredentialsProvider
• supercedes: HADOOP-16711 S3A bucket existence checks to support v2 API and "no checks at all"
• links to: GitHub Pull Request #2593