[HADOOP-18752] Change fs.s3a.directory.marker.retention to "keep"

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.3.5
• Fix Version/s: 3.4.0
• Component/s: fs/s3
• Labels: pull-request-available
• Target Version/s: 3.4.0, 3.3.9
• Hadoop Flags: Incompatible change

Release Note

The s3a connector no longer deletes directory markers by default, which speeds up write operations, reduces iO throttling and saves money. this can cause problems with older hadoop releases trying to write to the same bucket. (Hadoop 3.3.0; Hadoop 3.2.x before 3.2.2, and all previous releases). Set "fs.s3a.directory.marker.retention" to "delete" for backwards compatibility

• Assignee: Steve Loughran
• Reporter: Steve Loughran
• Created: 24/May/23 17:53
• Updated: 16/Jan/24 09:06
• Resolved: 08/Jun/23 11:22

Description

Change the default value of "fs.s3a.directory.marker.retention" to keep; update docs to match.

maybe include with HADOOP-17802 so we don't blow up with fewer markers being created.

Issue Links

• relates to: HADOOP-17802 S3A Xattr/getXAttr to handle directories without markers
• relates to: HADOOP-18113 remove filtering of directory markers in s3a RenameOperation
• relates to: HADOOP-13230 S3A to optionally retain directory markers
• links to: GitHub Pull Request #5689
• links to: GitHub Pull Request #5859