Hadoop Common
  1. Hadoop Common
  2. HADOOP-3518

Want NIO.2 (JSR 203) file system provider for Hadoop FileSystem

    Details

    • Type: Wish Wish
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: fs
    • Labels:
      None

      Description

      JSR 203 (aka "NIO.2" or "more NIO") is defining a rich set of classes for interacting with files and file systems (as well as other NIO enhancements). It is scheduled to be released as a part of Java 7.

      This motivation behind this issue is to see if NIO.2 can be used as an interface to Hadoop's FileSystem class before NIO.2 is finalized, thus giving Hadoop developers an opportunity to influence NIO's design (if necessary). Also, learning more about NIO.2 may inform design decisions for Hadoop filesystems.

      The starting point for this work should be the java.nio.file.spi package (http://openjdk.java.net/projects/nio/javadoc/java/nio/file/spi/package-summary.html). There is an example of a filesystem provider (for ZIP files) linked from the OpenJDK page for NIO.2: http://openjdk.java.net/projects/nio/. This page also has other useful links, such as a JavaOne talk, javadoc and source code.

        Issue Links

          Activity

          Hide
          Johan Liesén added a comment -

          I proposed this as a GSoC project (http://mail-archives.apache.org/mod_mbox/hadoop-core-dev/200903.mbox/browser) and am willing to chip in. However, the JavaOne time frame is too short for this.

          I believe it's suitable as a GSoC project and can be run side-by-side with both the developments in JSR203 and Hadoop.

          Show
          Johan Liesén added a comment - I proposed this as a GSoC project ( http://mail-archives.apache.org/mod_mbox/hadoop-core-dev/200903.mbox/browser ) and am willing to chip in. However, the JavaOne time frame is too short for this. I believe it's suitable as a GSoC project and can be run side-by-side with both the developments in JSR203 and Hadoop.
          Hide
          Sanjay Radia added a comment -

          I have taken a look at NIO2.
          NIO2's path class interface does not quite match with ours ( Please see my comments in HADOOP-4952.)
          However, the provider interface looks very promising.
          We have an opportunity to work with the JSR203 team to see the provider interface is sufficient to express Hadoop FS.
          Alan Bateman, the spec lead would like to work with the Hadoop community to prototype the Hadoop FS using NIO2's provider interface.
          In particular, if we get this done quickly Alan can present the hadoop fs as a example at his JavaOne talk this year (May/June 2009?)
          Any volunteers for this?

          Show
          Sanjay Radia added a comment - I have taken a look at NIO2. NIO2's path class interface does not quite match with ours ( Please see my comments in HADOOP-4952 .) However, the provider interface looks very promising. We have an opportunity to work with the JSR203 team to see the provider interface is sufficient to express Hadoop FS. Alan Bateman, the spec lead would like to work with the Hadoop community to prototype the Hadoop FS using NIO2's provider interface. In particular, if we get this done quickly Alan can present the hadoop fs as a example at his JavaOne talk this year (May/June 2009?) Any volunteers for this?
          Hide
          Tom White added a comment -

          Like Hadoop, NIO.2 identifies filesystems by URI scheme (see http://openjdk.java.net/projects/nio/javadoc/java/nio/file/spi/FileSystemProvider.html). A minor difference with Hadoop is that in NIO.2 the file operations are on the Path class (http://openjdk.java.net/projects/nio/javadoc/java/nio/file/Path.html), not the FileSystem class (but this doesn't preclude writing a NIO.2 provider for Hadoop, of course).

          Show
          Tom White added a comment - Like Hadoop, NIO.2 identifies filesystems by URI scheme (see http://openjdk.java.net/projects/nio/javadoc/java/nio/file/spi/FileSystemProvider.html ). A minor difference with Hadoop is that in NIO.2 the file operations are on the Path class ( http://openjdk.java.net/projects/nio/javadoc/java/nio/file/Path.html ), not the FileSystem class (but this doesn't preclude writing a NIO.2 provider for Hadoop, of course).

            People

            • Assignee:
              Unassigned
              Reporter:
              Tom White
            • Votes:
              2 Vote for this issue
              Watchers:
              17 Start watching this issue

              Dates

              • Created:
                Updated:

                Development