[HADOOP-14255] S3A to delete unnecessary fake directory objects in mkdirs()

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.9.0, 3.0.0-alpha4
• Component/s: fs/s3
• Target Version/s: 2.9.0, 3.0.0-alpha4
• Hadoop Flags: Reviewed
• Assignee: Mingliang Liu
• Reporter: Mingliang Liu
• Created: 29/Mar/17 21:57
• Updated: 26/Jun/17 23:48
• Resolved: 12/Apr/17 21:33

Description

In S3AFileSystem, as an optimization, we delete unnecessary fake directory objects if that directory contains at least one (nested) file. That is done in closing stream of newly created file. However, if the directory becomes non-empty after we just create an empty subdirectory, we do not delete its fake directory object though that fake directory object becomes "unnecessary".

So in S3AFileSystem::mkdirs(), we have a pending TODO:

// TODO: If we have created an empty file at /foo/bar and we then call
// mkdirs for /foo/bar/baz/roo what happens to the empty file /foo/bar/?
private boolean innerMkdirs(Path p, FsPermission permission)

This JIRA is to fix the TODO: provide consistent behavior for a fake directory object between its nested subdirectory and nested file by deleting it.

See related discussion in HADOOP-14236. Thanks [e-mail] for discussion.

Issue Links

• breaks: HADOOP-14428 s3a: mkdir appears to be broken
• is duplicated by: HADOOP-13222 s3a.mkdirs() to delete empty fake parent directories
• is related to: HADOOP-14236 S3Guard: S3AFileSystem::rename() should move non-listed sub-directory entries in metadata store
• is related to: HADOOP-14594 ITestS3AFileOperationCost::testFakeDirectoryDeletion to uncomment metric assertions
• relates to: HADOOP-13230 S3A to optionally retain directory markers