[HADOOP-18113] remove filtering of directory markers in s3a RenameOperation

• Type: Sub-task
• Status: Open
• Priority: Minor
• Resolution: Unresolved
• Affects Version/s: 3.4.0
• Component/s: fs/s3
• Assignee: Unassigned
• Reporter: Steve Loughran
• Created: 03/Feb/22 11:22
• Updated: 03/Dec/24 13:39

Description

S3A RenameOperation tries to filter out renaming intermediate directory markers so as to avoid retaining them when copying from an authoritiative dir to a non-auth one. we did this to support installations where dir marker retention was only enabled for auth dirs

• nobody, AFAIK, does this..it is all or nothing
• all recent hadoop releases, including earlier branches, is now marker aware
• the extras complexity makes debugging rename problems harder.

let's just cut it out.

Issue Links

• is related to: HADOOP-18112 Implement paging during S3 multi object delete.
• is related to: HADOOP-18752 Change fs.s3a.directory.marker.retention to "keep"