Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-15620 Über-jira: S3A phase VI: Hadoop 3.3 features
  3. HADOOP-15460

S3A FS to add "s3a:no-existence-checks" to the builder file creation option set

    XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 3.1.0
    • Fix Version/s: None
    • Component/s: fs/s3
    • Labels:
      None

      Description

      As promised to [~StephanEwen]: add and s3a-specific option to the builder-API to create files for all existence checks to be skipped.

      This

      1. eliminates a few hundred milliseconds
      2. avoids any caching of negative HEAD/GET responses in the S3 load balancers.

      Callers will be expected to know what what they are doing.

      FWIW, we are doing some PUT calls in the committer which bypass this stuff, for the same reason. If you've just created a directory, you know there's nothing underneath, so no need to check.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                stevel@apache.org Steve Loughran
              • Votes:
                0 Vote for this issue
                Watchers:
                6 Start watching this issue

                Dates

                • Created:
                  Updated: