[HADOOP-17199] Backport HADOOP-13230 list/getFileStatus changes for preserved directory markers

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.2.1, 3.1.3
• Fix Version/s: 3.2.2, 2.10.1
• Component/s: fs/s3
• Labels: pull-request-available
• Target Version/s: 3.2.2
• Assignee: Steve Loughran
• Reporter: Steve Loughran
• Created: 10/Aug/20 18:54
• Updated: 10/Nov/23 11:44
• Resolved: 25/Aug/20 21:53

Description

Our supported Hadoop releases all need to handle s3a stores where the s3a client does not delete directory markers.

This is only the minimal set of list changes; no intent of changing marker retention policy. It should be transparent to all clients

• list/getFileStatus changes
• backwards compatible test of new algorithm
• new tests to verify marker handling
• marker rename bug (HADOOP-17200) surfacing in testing

Issue Links

• causes: HADOOP-18968 S3A FileSystem does not correctly list empty directories after HADOOP-17199
• contains: HADOOP-17200 Renaming a file under a sibling empty directory doesn't delete dest dir's marker
• depends upon: HADOOP-17210 backport HADOOP-15691 PathCapabilities API to branch-3.2
• is depended upon by: HADOOP-17217 S3A FileSystem does not correctly delete directories with fake entries
• is depended upon by: SPARK-35299 Dataframe overwrite on S3A does not delete old files with S3 object-put to table path/
• is related to: HADOOP-17228 Backport HADOOP-13230 listing changes for preserved directory markers to 3.1.x
• relates to: HADOOP-13230 S3A to optionally retain directory markers
• links to: GitHub Pull Request #2247
• links to: GitHub Pull Request #2262
• links to: GitHub Pull Request #2269
• links to: https://github.com/apache/hadoop/pull/2210