[HADOOP-18278] Do not perform a LIST call when creating a file

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Duplicate
• Affects Version/s: 3.3.3
• Component/s: fs/s3
• Target Version/s: 3.4.0
• Assignee: Unassigned
• Reporter: Sam Kramer
• Created: 07/Jun/22 16:35
• Updated: 10/Oct/23 18:43
• Resolved: 08/Jun/22 10:57

Description

Hello,

We've noticed that when creating a file, which does not exist in S3, we see an extra LIST call gets issued to see if it's a directory (i.e. if key = "bar", it will issue an object list request for "bar/"). 

Is this really necessary, shouldn't a HEAD request be sufficient to determine if it actually exists or not? As we're creating 1000s of files, this is quite expensive, as we're effectively doubling our costs for file creation. Curious if others have experienced similar or identical issues, or if there are any workarounds. 

https://github.com/apache/hadoop/blob/516a2a8e440378c868ddb02cb3ad14d0d879037f/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java#L3359-L3369

 

Thanks,

Sam

Issue Links

• duplicates: HADOOP-15460 S3A FS to add "fs.s3a.create.performance" to the builder file creation option set
• relates to: HADOOP-18930 S3A: make fs.s3a.create.performance an option you can set for the entire bucket