Details

    • Type: New Feature New Feature
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      WebDAV stands for Distributed Authoring and Versioning. It is a set of extensions to the HTTP protocol that lets users collaboratively edit and manage files on a remote web server. It is often considered as a replacement for NFS or SAMBA

      HDFS (Hadoop Distributed File System) needs a friendly file system interface. DFSShell commands are unfamiliar. Instead it is more convenient for Hadoop users to use a mountable network drive. A friendly interface to HDFS will be used both for casual browsing of data and for bulk import/export.

      The FUSE provider for HDFS is already available ( http://issues.apache.org/jira/browse/HADOOP-17 ) but it had scalability problems. WebDAV is a popular alternative.

      The typical licensing terms for WebDAV tools are also attractive:
      GPL for Linux client tools that Hadoop would not redistribute anyway.
      More importantly, Apache Project/Apache license for Java tools and for server components.
      This allows for a tighter integration with the HDFS code base.

      There are some interesting Apache projects that support WebDAV.
      But these are probably too heavyweight for the needs of Hadoop:
      Tomcat servlet: http://tomcat.apache.org/tomcat-4.1-doc/catalina/docs/api/org/apache/catalina/servlets/WebdavServlet.html
      Slide: http://jakarta.apache.org/slide/

      Being HTTP-based and "backwards-compatible" with Web Browser clients, the WebDAV server protocol could even be piggy-backed on the existing Web UI ports of the Hadoop name node / data nodes. WebDAV can be hosted as (Jetty) servlets. This minimizes server code bloat and this avoids additional network traffic between HDFS and the WebDAV server.

      General Clients (read-only):
      Any web browser

      Linux Clients:
      Mountable GPL davfs2 http://dav.sourceforge.net/
      FTP-like GPL Cadaver http://www.webdav.org/cadaver/

      Server Protocol compliance tests:
      http://www.webdav.org/neon/litmus/
      A goal is for Hadoop HDFS to pass this test (minus support for Properties)

      Pure Java clients:
      DAV Explorer Apache lic. http://www.ics.uci.edu/~webdav/

      WebDAV also makes it convenient to add advanced features in an incremental fashion:
      file locking, access control lists, hard links, symbolic links.
      New WebDAV standards get accepted and more or less featured WebDAV clients exist.
      core http://www.webdav.org/specs/rfc2518.html
      ACLs http://www.webdav.org/specs/rfc3744.html
      redirects "soft links" http://greenbytes.de/tech/webdav/rfc4437.html
      BIND "hard links" http://www.webdav.org/bind/
      quota http://tools.ietf.org/html/rfc4331

      1. jetty-slide.xml
        2 kB
        Albert Strasheim
      2. slideusers.properties
        0.0 kB
        Albert Strasheim
      3. hadoop-webdav.zip
        15 kB
        Albert Strasheim
      4. lib.webdav.tar.gz
        2.14 MB
        Enis Soztutar
      5. webdav_wip1.patch
        41 kB
        Enis Soztutar
      6. webdav_wip2.patch
        43 kB
        Enis Soztutar
      7. screenshot-1.jpg
        54 kB
        Pete Wyckoff
      8. hadoop-496-3.patch
        43 kB
        Michael Bieniosek
      9. hadoop-496-spool-cleanup.patch
        43 kB
        Michael Bieniosek
      10. hadoop-496-4.patch
        38 kB
        Michael Bieniosek
      11. hadoop-496-5.tgz
        7 kB
        Michael Bieniosek

        Issue Links

          Activity

          Gavin made changes -
          Link This issue depends upon HADOOP-1654 [ HADOOP-1654 ]
          Gavin made changes -
          Link This issue depends on HADOOP-1654 [ HADOOP-1654 ]
          Konstantin Boudnik made changes -
          Link This issue duplicates HDFS-2178 [ HDFS-2178 ]
          Owen O'Malley made changes -
          Project Hadoop Common [ 12310240 ] HDFS [ 12310942 ]
          Key HADOOP-496 HDFS-225
          Component/s dfs [ 12310710 ]
          Michael Bieniosek made changes -
          Attachment hadoop-496-5.tgz [ 12374118 ]
          Anurag Sharma made changes -
          Attachment fuse-j-hadoopfs-0.zip [ 12370716 ]
          Anurag Sharma made changes -
          Attachment fuse-j-patch.zip [ 12370717 ]
          Anurag Sharma made changes -
          Attachment fuse-j-patch.zip [ 12370717 ]
          Attachment fuse-j-hadoopfs-0.zip [ 12370716 ]
          Michael Bieniosek made changes -
          Attachment hadoop-496-4.patch [ 12369698 ]
          Michael Bieniosek made changes -
          Attachment hadoop-496-spool-cleanup.patch [ 12369561 ]
          Michael Bieniosek made changes -
          Attachment hadoop-496-3.patch [ 12369559 ]
          Pete Wyckoff made changes -
          Attachment screenshot-1.jpg [ 12369066 ]
          Albert Strasheim made changes -
          Link This issue depends on HADOOP-1654 [ HADOOP-1654 ]
          Enis Soztutar made changes -
          Attachment webdav_wip2.patch [ 12362625 ]
          Enis Soztutar made changes -
          Assignee Sameer Paranjpye [ sameerp ] Enis Soztutar [ enis ]
          Enis Soztutar made changes -
          Attachment webdav_wip1.patch [ 12362527 ]
          Attachment lib.webdav.tar.gz [ 12362526 ]
          Albert Strasheim made changes -
          Attachment hadoop-webdav.zip [ 12361351 ]
          Albert Strasheim made changes -
          Attachment slideusers.properties [ 12361350 ]
          Albert Strasheim made changes -
          Attachment jetty-slide.xml [ 12361349 ]
          Doug Cutting made changes -
          Assignee Sameer Paranjpye [ sameerp ]
          Michel Tourn made changes -
          Field Original Value New Value
          Link This issue is related to HADOOP-512 [ HADOOP-512 ]
          Michel Tourn created issue -

            People

            • Assignee:
              Enis Soztutar
              Reporter:
              Michel Tourn
            • Votes:
              13 Vote for this issue
              Watchers:
              32 Start watching this issue

              Dates

              • Created:
                Updated:

                Development