[HDFS-303] Make contracts of LocalFileSystem and DistributedFileSystem consistent

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Done
• Assignee: Andras Bokor
• Reporter: Thomas White
• Created: 08/Sep/08 13:43
• Updated: 29/Nov/16 14:06
• Resolved: 29/Nov/16 14:06

Description

There are a number of edge cases that the two file system implementations handle differently. In particular:

• When trying to make a directory under an existing file, HDFS throws an IOException while LocalFileSystem doesn't.
• The FileSytem#listStatus(Path) method returns null for a non-existent file on HDFS, while LocalFileSytem returns an empty FileStatus array.
• When trying to rename a non-existent path, LocalFileSystem throws an IOException, while HDFS returns false.
• When renaming a file or directory to a non-existent directory (e.g. /a/b to /c/d, where /c doesn't exist) LocalFileSystem succeeds (returns true) while HDFS fails (false).
• When renaming a file (or directory) as an existing file (or directory) LocalFileSystem succeeds (returns true) while HDFS fails (false).
  We should document the expected behaviour for these cases in FileSystem's javadoc, and make sure all implementations conform to it.

Issue Links

• incorporates: HDFS-6262 HDFS doesn't raise FileNotFoundException if the source of a rename() is missing
• incorporates: HADOOP-10375 Local FS doesn't raise an error on mkdir() over a file
• incorporates: HADOOP-6229 Atempt to make a directory under an existing file on LocalFileSystem should throw an Exception.
• incorporates: HADOOP-6240 Rename operation is not consistent between different implementations of FileSystem
• incorporates: HDFS-538 DistributedFileSystem::listStatus incorrectly returns null for empty result sets
• incorporates: HDFS-590 When trying to rename a non-existent path, LocalFileSystem throws an FileNotFoundException, while HDFS returns false
• incorporates: HDFS-602 Atempt to make a directory under an existing file on DistributedFileSystem should throw an FileAlreadyExistsException instead of FileNotFoundException
• incorporates: MAPREDUCE-6814 mapred's FileAlreadyExistsException should be deprecated in favor of hadoop-common's one.
• incorporates: MAPREDUCE-963 mapred's FileAlreadyExistsException should be deprecated in favor of hadoop-common's one.
• is blocked by: HADOOP-7363 TestRawLocalFileSystemContract is needed
• is related to: HADOOP-7352 FileSystem#listStatus should throw IOE upon access error
• is related to: HADOOP-9361 Strictly define the expected behavior of filesystem APIs and write tests to verify compliance
• relates to: HIVE-1893 ConditionalResolverSkewJoin.getTasks may throw NullPointerException