[HADOOP-13221] s3a create() doesn't check for an ancestor path being a file

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.7.2
• Fix Version/s: 3.4.0
• Component/s: fs/s3
• Assignee: Sean Mackrory
• Reporter: Steve Loughran
• Created: 31/May/16 11:25
• Updated: 27/Jul/20 09:37
• Resolved: 27/Jul/20 09:37

Description

Seen in a code review. Notable that if true, this got by all the FS contract tests —showing we missed a couple.

S3AFilesystem.create() does not examine its parent paths to verify that there does not exist one which is a file. It looks for the destination path if overwrite=false (see HADOOP-13188 for issues there), but it doesn't check the parent for not being a file, or the parent of that path.

It must go up the tree, verifying that either a path does not exist, or that the path is a directory. The scan can stop at the first entry which is is a directory, thus the operation is O(empty-directories) and not O(directories).

Issue Links

• is related to: HADOOP-13188 S3A file-create should throw error rather than overwrite directories
• is related to: HADOOP-15542 S3AFileSystem - FileAlreadyExistsException when prefix is a file and part of a directory tree
• relates to: HADOOP-12667 s3a: Support createNonRecursive API
• relates to: HADOOP-14506 Add create() contract test that verifies ancestor dir creation