Want NIO.2 (JSR 203) file system provider for Hadoop FileSystem

• Key: HADOOP-3518
• Type: Wish
• Status: Open
• Priority: Major
• Assignee: Unassigned
• Reporter: Tom White
• Component/s: fs

Description

JSR 203 (aka "NIO.2" or "more NIO") is defining a rich set of classes for interacting with files and file systems (as well as other NIO enhancements). It is scheduled to be released as a part of Java 7.

This motivation behind this issue is to see if NIO.2 can be used as an interface to Hadoop's FileSystem class before NIO.2 is finalized, thus giving Hadoop developers an opportunity to influence NIO's design (if necessary). Also, learning more about NIO.2 may inform design decisions for Hadoop filesystems.

The starting point for this work should be the java.nio.file.spi package (http://openjdk.java.net/projects/nio/javadoc/java/nio/file/spi/package-summary.html). There is an example of a filesystem provider (for ZIP files) linked from the OpenJDK page for NIO.2: http://openjdk.java.net/projects/nio/. This page also has other useful links, such as a JavaOne talk, javadoc and source code.

Comments

[Tom White]

Like Hadoop, NIO.2 identifies filesystems by URI scheme (see http://openjdk.java.net/projects/nio/javadoc/java/nio/file/spi/FileSystemProvider.html). A minor difference with Hadoop is that in NIO.2 the file operations are on the Path class (http://openjdk.java.net/projects/nio/javadoc/java/nio/file/Path.html), not the FileSystem class (but this doesn't preclude writing a NIO.2 provider for Hadoop, of course).

[Sanjay Radia]

I have taken a look at NIO2.
NIO2's path class interface does not quite match with ours ( Please see my comments in HADOOP-4952.)
However, the provider interface looks very promising.
We have an opportunity to work with the JSR203 team to see the provider interface is sufficient to express Hadoop FS.
Alan Bateman, the spec lead would like to work with the Hadoop community to prototype the Hadoop FS using NIO2's provider interface.
In particular, if we get this done quickly Alan can present the hadoop fs as a example at his JavaOne talk this year (May/June 2009?)
Any volunteers for this?

[Johan Liesén]

I proposed this as a GSoC project (http://mail-archives.apache.org/mod_mbox/hadoop-core-dev/200903.mbox/browser) and am willing to chip in. However, the JavaOne time frame is too short for this.

I believe it's suitable as a GSoC project and can be run side-by-side with both the developments in JSR203 and Hadoop.