[#HDFS-225] Expose HDFS as a WebDAV store

Hadoop HDFS

Expose HDFS as a WebDAV store

Created: 30/Aug/06 07:28 PM Updated: 03/Aug/09 01:30 PM

Return to search

Component/s:

None

Affects Version/s:

None

Fix Version/s:

None

Time Tracking:

Not Specified

File Attachments:

File Name	Date Attached	Attached By	Size
hadoop-496-3.patch	2007-11-15 12:33 AM	Michael Bieniosek	43 kB
hadoop-496-4.patch	2007-11-16 10:25 PM	Michael Bieniosek	38 kB
hadoop-496-5.tgz	2008-01-26 07:18 AM	Michael Bieniosek	7 kB
hadoop-496-spool-cleanup.patch	2007-11-15 01:43 AM	Michael Bieniosek	43 kB
hadoop-webdav.zip	2007-07-07 01:23 PM	Albert Strasheim	15 kB
jetty-slide.xml	2007-07-07 01:11 PM	Albert Strasheim	2 kB
lib.webdav.tar.gz	2007-07-25 01:38 PM	Enis Soztutar	2.14 MB
slideusers.properties	2007-07-07 01:12 PM	Albert Strasheim	0.0 kB
webdav_wip1.patch	2007-07-25 01:38 PM	Enis Soztutar	41 kB
webdav_wip2.patch	2007-07-26 03:34 PM	Enis Soztutar	43 kB

Image Attachments:

1. screenshot-1.jpg
(54 kB)

Issue Links:

Dependants

This issue depends on:

~~HADOOP-1654~~ IOUtils class

Reference

This issue is related to:

~~HADOOP-512~~ Write and integrate HDFS webdav provider

Description

« Hide

HDFS (Hadoop Distributed File System) needs a friendly file system interface. DFSShell commands are unfamiliar. Instead it is more convenient for Hadoop users to use a mountable network drive. A friendly interface to HDFS will be used both for casual browsing of data and for bulk import/export.

The FUSE provider for HDFS is already available ( http://issues.apache.org/jira/browse/HADOOP-17 ) but it had scalability problems. WebDAV is a popular alternative.

The typical licensing terms for WebDAV tools are also attractive:
GPL for Linux client tools that Hadoop would not redistribute anyway.
More importantly, Apache Project/Apache license for Java tools and for server components.
This allows for a tighter integration with the HDFS code base.

There are some interesting Apache projects that support WebDAV.
But these are probably too heavyweight for the needs of Hadoop:
Tomcat servlet: http://tomcat.apache.org/tomcat-4.1-doc/catalina/docs/api/org/apache/catalina/servlets/WebdavServlet.html
Slide: http://jakarta.apache.org/slide/

Being HTTP-based and "backwards-compatible" with Web Browser clients, the WebDAV server protocol could even be piggy-backed on the existing Web UI ports of the Hadoop name node / data nodes. WebDAV can be hosted as (Jetty) servlets. This minimizes server code bloat and this avoids additional network traffic between HDFS and the WebDAV server.

General Clients (read-only):
Any web browser

Linux Clients:
Mountable GPL davfs2 http://dav.sourceforge.net/
FTP-like GPL Cadaver http://www.webdav.org/cadaver/

Server Protocol compliance tests:
http://www.webdav.org/neon/litmus/
A goal is for Hadoop HDFS to pass this test (minus support for Properties)

Pure Java clients:
DAV Explorer Apache lic. http://www.ics.uci.edu/~webdav/

WebDAV also makes it convenient to add advanced features in an incremental fashion:
file locking, access control lists, hard links, symbolic links.
New WebDAV standards get accepted and more or less featured WebDAV clients exist.
core http://www.webdav.org/specs/rfc2518.html
ACLs http://www.webdav.org/specs/rfc3744.html
redirects "soft links" http://greenbytes.de/tech/webdav/rfc4437.html
BIND "hard links" http://www.webdav.org/bind/
quota http://tools.ietf.org/html/rfc4331

Description

WebDAV stands for Distributed Authoring and Versioning. It is a set of extensions to the HTTP protocol that lets users collaboratively edit and manage files on a remote web server. It is often considered as a replacement for NFS or SAMBA HDFS (Hadoop Distributed File System) needs a friendly file system interface. DFSShell commands are unfamiliar. Instead it is more convenient for Hadoop users to use a mountable network drive. A friendly interface to HDFS will be used both for casual browsing of data and for bulk import/export. The FUSE provider for HDFS is already available ( http://issues.apache.org/jira/browse/HADOOP-17 ) but it had scalability problems. WebDAV is a popular alternative. The typical licensing terms for WebDAV tools are also attractive: GPL for Linux client tools that Hadoop would not redistribute anyway. More importantly, Apache Project/Apache license for Java tools and for server components. This allows for a tighter integration with the HDFS code base. There are some interesting Apache projects that support WebDAV. But these are probably too heavyweight for the needs of Hadoop: Tomcat servlet: http://tomcat.apache.org/tomcat-4.1-doc/catalina/docs/api/org/apache/catalina/servlets/WebdavServlet.html Slide: http://jakarta.apache.org/slide/ Being HTTP-based and "backwards-compatible" with Web Browser clients, the WebDAV server protocol could even be piggy-backed on the existing Web UI ports of the Hadoop name node / data nodes. WebDAV can be hosted as (Jetty) servlets. This minimizes server code bloat and this avoids additional network traffic between HDFS and the WebDAV server. General Clients (read-only): Any web browser Linux Clients: Mountable GPL davfs2 http://dav.sourceforge.net/ FTP-like GPL Cadaver http://www.webdav.org/cadaver/ Server Protocol compliance tests: http://www.webdav.org/neon/litmus/ A goal is for Hadoop HDFS to pass this test (minus support for Properties) Pure Java clients: DAV Explorer Apache lic. http://www.ics.uci.edu/~webdav/ WebDAV also makes it convenient to add advanced features in an incremental fashion: file locking, access control lists, hard links, symbolic links. New WebDAV standards get accepted and more or less featured WebDAV clients exist. core http://www.webdav.org/specs/rfc2518.html ACLs http://www.webdav.org/specs/rfc3744.html redirects "soft links" http://greenbytes.de/tech/webdav/rfc4437.html BIND "hard links" http://www.webdav.org/bind/ quota http://tools.ietf.org/html/rfc4331

Show »

All

Comments

Work Log

Change History

Subversion Commits

Sort Order:

No work has yet been logged on this issue.