Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-17454

[s3a] Disable bucket existence check - set fs.s3a.bucket.probe to 0

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 3.3.0
    • Fix Version/s: None
    • Component/s: None
    • Release Note:
      Hide
      S3A bucket existence check is disabled (fs.s3a.bucket.probe is 0), so there will be no existence check on the bucket during the S3AFileSystem initialization. The first operation which attempts to interact with the bucket which will fail if the bucket does not exist.
      Show
      S3A bucket existence check is disabled (fs.s3a.bucket.probe is 0), so there will be no existence check on the bucket during the S3AFileSystem initialization. The first operation which attempts to interact with the bucket which will fail if the bucket does not exist.

      Description

      Set the value of fs.s3a.bucket.probe to 0 by default.
      Bucket existence checks are done in the initialization phase of the S3AFileSystem. It's not required to run this check: the operation itself will fail if the bucket does not exist instead of the check.

      Some points on why do we want to set this to 0:

      • When it's set to 0, bucket existence checks won't be done during initialization thus making it faster.
      • Avoid the additional one or two requests on the bucket root, so the user does not need rights to read or list that folder.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                gabor.bota Gabor Bota
                Reporter:
                gabor.bota Gabor Bota
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 1.5h
                  1.5h