[HADOOP-17200] Renaming a file under a sibling empty directory doesn't delete dest dir's marker

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: 3.3.0, 3.2.1, 3.1.3
• Fix Version/s: 3.3.1
• Component/s: fs/s3
• Assignee: Steve Loughran
• Reporter: Steve Loughran
• Created: 10/Aug/20 19:03
• Updated: 15/Aug/20 19:26
• Resolved: 15/Aug/20 19:26

Description

This has probably existed for a long time -the fact that nobody has noticed is probably just luck.

Probably came with the fix of HADOOP-15079.

Given

• a base directory /base
• A file /base/file
• an empty directory /base/empty (with empty dir marker)
• if you rename base/file under base/empty (key: with the dir passed in, not the full filename) the file is moved, but the marker /base/empty/ is not

The issue is that the delete/recreate markers compares src and dest parent for equivalence when skipping marker delete, but if the final path of the rename is under the path "dst" then the comparison is wrong -it must be on the parent of the final path, not the path handed to rename()

    if (!src.getParent().equals(dst.getParent())) {
      LOG.debug("source & dest parents are different; fix up dir markers");
      deleteUnnecessaryFakeDirectories(dst.getParent());
      maybeCreateFakeParentDirectory(src);
    }

Issue Links

• Dependency: HADOOP-13230 S3A to optionally retain directory markers
• Is contained by: HADOOP-13230 S3A to optionally retain directory markers
• Is contained by: HADOOP-17199 Backport HADOOP-13230 list/getFileStatus changes for preserved directory markers