[HDFS-2178] HttpFS - a read/write Hadoop file system proxy

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.23.0
• Fix Version/s: 0.23.1
• Hadoop Flags: Reviewed
• Assignee: Alejandro Abdelnur
• Reporter: Alejandro Abdelnur
• Created: 20/Jul/11 21:15
• Updated: 04/Sep/18 23:46
• Resolved: 08/Dec/11 19:31

Description

We'd like to contribute Hoop to Hadoop HDFS as a replacement (an improvement) for HDFS Proxy.

Hoop provides access to all Hadoop Distributed File System (HDFS) operations (read and write) over HTTP/S.

The Hoop server component is a REST HTTP gateway to HDFS supporting all file system operations. It can be accessed using standard HTTP tools (i.e. curl and wget), HTTP libraries from different programing languages (i.e. Perl, Java Script) as well as using the Hoop client. The Hoop server component is a standard Java web-application and it has been implemented using Jersey (JAX-RS).

The Hoop client component is an implementation of Hadoop FileSystem client that allows using the familiar Hadoop filesystem API to access HDFS data through a Hoop server.

Repo: https://github.com/cloudera/hoop

Docs: http://cloudera.github.com/hoop

Blog: http://www.cloudera.com/blog/2011/07/hoop-hadoop-hdfs-over-http/

Hoop is a Maven based project that depends on Hadoop HDFS and Alfredo (for Kerberos HTTP SPNEGO authentication).

To make the integration easy, HDFS Mavenization (HDFS-2096) would have to be done first, as well as the Alfredo contribution (HADOOP-7119).

Issue Links

• depends upon: HADOOP-7119 add Kerberos HTTP SPNEGO authentication support to Hadoop JT/NN/DN/TT web-consoles
• depends upon: HDFS-2096 Mavenization of hadoop-hdfs
• is blocked by: HDFS-2606 webhdfs client filesystem impl must set the content-type header for create/append
• is duplicated by: HDFS-225 Expose HDFS as a WebDAV store
• is related to: HDFS-2284 Write Http access to HDFS
• is related to: HDFS-2646 Hadoop HttpFS introduced 4 findbug warnings.
• is related to: HDFS-2649 eclipse:eclipse build fails for hadoop-hdfs-httpfs
• is related to: HDFS-2657 TestHttpFSServer and TestServerWebApp are failing on trunk
• is related to: HDFS-2658 HttpFS introduced 70 javadoc warnings
• is related to: HDFS-2645 Webhdfs & HttpFS (Hoop) should share the same codebase
• relates to: HDFS-13894 Access HDFS through a proxy and natively