[HADOOP-9361] Strictly define the expected behavior of filesystem APIs and write tests to verify compliance

• Type: Improvement
• Status: Closed
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 2.4.0, 3.0.0-alpha1
• Fix Version/s: 2.5.0
• Component/s: fs, test
• Target Version/s: 2.5.0
• Assignee: Steve Loughran
• Reporter: Steve Loughran
• Created: 05/Mar/13 11:47
• Updated: 13/Aug/16 17:11
• Resolved: 03/Jul/14 13:16

Description

FileSystem and FileContract aren't tested rigorously enough -while HDFS gets tested downstream, other filesystems, such as blobstore bindings, don't.

The only tests that are common are those of FileSystemContractTestBase, which HADOOP-9258 shows is incomplete.

I propose

1. writing more tests which clarify expected behavior
2. testing operations in the interface being in their own JUnit4 test classes, instead of one big test suite.
3. Having each FS declare via a properties file what behaviors they offer, such as atomic-rename, atomic-delete, umask, immediate-consistency -test methods can downgrade to skipped test cases if a feature is missing.

Issue Links

• contains: HADOOP-10375 Local FS doesn't raise an error on mkdir() over a file
• contains: HADOOP-10589 NativeS3FileSystem throw NullPointerException when the file is empty
• depends upon: HDFS-4258 Rename of Being Written Files
• depends upon: HADOOP-6240 Rename operation is not consistent between different implementations of FileSystem
• depends upon: HADOOP-9432 Add support for markdown .md files in site documentation
• incorporates: HADOOP-10419 BufferedFSInputStream NPEs on getPos() on a closed stream
• incorporates: HADOOP-10533 S3 input stream NPEs in MapReduce job
• incorporates: HADOOP-9495 Define behaviour of Seekable.seek(), write tests, fix all hadoop implementations for compliance
• incorporates: HADOOP-10458 swifts should throw FileAlreadyExistsException on attempt to overwrite file
• is depended upon by: HADOOP-10400 Incorporate new S3A FileSystem implementation
• is related to: HDFS-6803 Documenting DFSClient#DFSInputStream expectations reading and preading in concurrent context
• is related to: HADOOP-9597 FileSystem open() API is not clear if FileNotFoundException is thrown when the path does not exist
• is related to: HADOOP-11708 CryptoOutputStream synchronization differences from DFSOutputStream break HBase
• is related to: HADOOP-3257 Path should handle all characters
• is related to: HADOOP-9702 FileSystem should fail create call with OverWrite and Append flags
• is related to: HADOOP-10440 HarFsInputStream of HarFileSystem, when reading data, computing the position has bug
• is related to: HADOOP-10457 S3N NPEs if you do a read() after a seek() past the EOF
• is related to: HDFS-6193 HftpFileSystem open should throw FileNotFoundException for non-existing paths
• is related to: MAPREDUCE-5264 FileAlreadyExistsException is assumed to be thrown by FileSystem#mkdirs or FileContext#mkdir in the codebase
• is related to: HADOOP-8545 Filesystem Implementation for OpenStack Swift
• is related to: HDFS-4872 Idempotent delete operation.
• is related to: HDFS-353 DFSClient does not always throw a FileNotFound exception when a file could not be opened
• is related to: HDFS-6735 A minor optimization to avoid pread() be blocked by read() inside the same DFSInputStream
• is related to: HADOOP-9565 Add a Blobstore interface to add to blobstore FileSystems
• relates to: HADOOP-10801 Fix dead link in site.xml
• relates to: HADOOP-9438 LocalFileContext does not throw an exception on mkdir for already existing directory
• relates to: HADOOP-12177 [Umbrella] Update and extend filesystem specification
• relates to: HADOOP-7352 FileSystem#listStatus should throw IOE upon access error
• relates to: HADOOP-12994 Specify PositionedReadable, add contract tests, fix problems
• relates to: HDFS-303 Make contracts of LocalFileSystem and DistributedFileSystem consistent
• relates to: HADOOP-7363 TestRawLocalFileSystemContract is needed
• relates to: HADOOP-11417 review filesystem seek logic, clarify/confirm spec, test & fix compliance
• relates to: HBASE-8722 Specify and document the FileSystem level behaviors required by HBase

Sub-Tasks

• Write FSWrapper class to wrap FileSystem and FileContext for better test coverage: Resolved
• Define Semantics of FileSystem more rigorously: Closed
• Add stricter tests to FileSystemContractTestBase: Closed
• FileSystem open() API is not clear if FileNotFoundException is thrown when the path does not exist: Closed
• Write contract tests for S3Native; fix places where it breaks: Closed
• Write contract tests for FTP filesystem, fix places where it breaks: Closed