[HADOOP-16380] S3A tombstones can confuse empty directory status

• Type: Sub-task
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 3.3.0
• Fix Version/s: 3.3.0
• Component/s: fs/s3, test
• Assignee: Steve Loughran
• Reporter: Steve Loughran
• Created: 18/Jun/19 10:00
• Updated: 05/Aug/19 19:02
• Resolved: 23/Jul/19 13:53

Description

If S3AFileSystem does an S3 LIST restricted to a single object to see if a directory is empty, and the single entry found has a tombstone marker (either from an inconsistent DDB Table or from an eventually consistent LIST) then it will consider the directory empty, even if there is 1+ entry which is not deleted

We need to make sure the calculation of whether a directory is empty or not is resilient to this, efficiently.

It surfaces as an issue two places

• delete(path) (where it may make things worse)
• rename(src, dest), where a check is made for dest != an empty directory.

Issue Links

• blocks: HADOOP-14936 S3Guard: remove "experimental" from documentation
• is caused by: HADOOP-16279 S3Guard: Implement time-based (TTL) expiry for entries (and tombstones)
• is related to: HADOOP-16427 Downgrade INFO message on rm s3a root dir to DEBUG
• relates to: HADOOP-15183 S3Guard store becomes inconsistent after partial failure of rename
• relates to: HADOOP-16384 S3A: Avoid inconsistencies between DDB and S3
• links to: GitHub Pull Request #985
• links to: GitHub Pull Request #1061
• links to: GitHub Pull Request #1079
• links to: GitHub Pull Request #1106
• links to: GitHub Pull Request #1123
• links to: PR1079