[HADOOP-15460] S3A FS to add "fs.s3a.create.performance" to the builder file creation option set

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.1.0
• Fix Version/s: 3.3.5
• Component/s: fs/s3
• Assignee: Steve Loughran
• Reporter: Steve Loughran
• Created: 14/May/18 11:53
• Updated: 04/Oct/22 17:01
• Resolved: 04/Oct/22 17:00

Description

As promised to StephanEwen: add and s3a-specific option to the builder-API to create files for all existence checks to be skipped.

This

1. eliminates a few hundred milliseconds
2. avoids any caching of negative HEAD/GET responses in the S3 load balancers.

Callers will be expected to know what what they are doing.

FWIW, we are doing some PUT calls in the committer which bypass this stuff, for the same reason. If you've just created a directory, you know there's nothing underneath, so no need to check.

adding this inside HADOOP-17833 as we are effectively doing this under the magic dir tree. having it as an option and using it to save all manifests/success files also saves one LIST per manifest write (task commit) and the LIST when saving a _SUCCESS file.

Issue Links

• Is contained by: HADOOP-17833 Improve Magic Committer Performance
• is duplicated by: HADOOP-18278 Do not perform a LIST call when creating a file
• is related to: BEAM-5934 FileSink affected by S3 eventual consistency
• is superceded by: HADOOP-16490 Avoid/handle cached 404s during S3A file creation
• relates to: HADOOP-15525 s3a: clarify / improve support for mixed ACL buckets