If S3AFileSystem does an S3 LIST restricted to a single object to see if a directory is empty, and the single entry found has a tombstone marker (either from an inconsistent DDB Table or from an eventually consistent LIST) then it will consider the directory empty, even if there is 1+ entry which is not deleted
We need to make sure the calculation of whether a directory is empty or not is resilient to this, efficiently.
It surfaces as an issue two places
- delete(path) (where it may make things worse)
- rename(src, dest), where a check is made for dest != an empty directory.