[HADOOP-9590] Move to JDK7 improved APIs for file operations when available

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Assignee: Unassigned
• Reporter: Ivan Mitic
• Created: 22/May/13 02:49
• Updated: 27/Apr/16 09:39

Description

JDK6 does not have a complete support for local file system file operations. Specifically:

• There is no symlink/hardlink APIs what forced Hadoop to defer to shell based tooling
• No error information returned when File#mkdir/mkdirs or File#renameTo fails, making it unnecessary hard to troubleshoot some issues
• File#canRead/canWrite/canExecute do not perform any access checks on Windows making APIs inconsistent with the Unix behavior
• File#setReadable/setWritable/setExecutable do not change access rights on Windows making APIs inconsistent with the Unix behavior
• File#length does not work as expected on symlinks on Windows
• File#renameTo does not work as expected on symlinks on Windows

All above resulted in Hadoop community having to fill in the gaps by providing equivalent native implementations or applying workarounds.

JDK7 addressed (as far as I know) all (or most) of the above problems, either thru the newly introduced Files class or thru bug fixes.

This is a tracking Jira to revisit above mediations once JDK7 becomes the supported platform by the Hadoop community. This work would allow significant portion of the native platform-dependent code to be replaced with Java equivalents what is goodness w.r.t. Hadoop cross-platform support.

Issue Links

• is related to: HDFS-6531 Create a native api to delete file, like the renameTo method in NativeIO, for better error reporting
• is related to: HADOOP-9061 Java6+Windows does not work well with symlinks
• is related to: HADOOP-9413 Introduce common utils for File#setReadable/Writable/Executable and File#canRead/Write/Execute that work cross-platform
• is related to: HDFS-4428 FsDatasetImpl should disclose what the error is when a rename fails
• is related to: HDFS-6695 Investigate using Java 7's nonblocking file I/O in BlockReaderLocal to implement read timeouts
• is related to: HDFS-4839 add NativeIO#mkdirs, that provides an error message on failure
• is related to: HADOOP-3518 Want NIO.2 (JSR 203) file system provider for Hadoop FileSystem
• relates to: HDFS-7118 Improve diagnostics on storage directory rename operations by using NativeIO#renameTo in Storage#rename.
• relates to: HDFS-7119 Split error checks in AtomicFileOutputStream#close into separate conditions to improve diagnostics.