[HADOOP-18544] S3A: add option to disable probe for dir marker recreation on delete/rename.

• Type: Sub-task
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.3.4
• Component/s: fs/s3
• Labels: pull-request-available
• Assignee: Harshit Gupta
• Reporter: Steve Loughran
• Created: 28/Nov/22 17:48
• Updated: 29/Jul/24 15:09

Description

In applications which do many single-file deletions on the same dir, a lot of time is wasted in maybeCreateFakeParentDirectory().

Proposed: add an option to disable the probe, for use by applications which are happy for parent dirs to sometimes disappear after a cleanup.

file by file delete is still woefully inefficient because of the HEAD request on every file, but there's no need to amplify the damage.

Issue Links

• is related to: HADOOP-19161 S3A: option "fs.s3a.performance.flags" to take list of performance flags
• links to: GitHub Pull Request #5354