[HADOOP-10741] A lightweight WebHDFS client library

• Type: New Feature
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: tools
• Assignee: Mohammad Islam Assign to me
• Reporter: Tsz-wo Sze
• Created: 23/Jun/14 23:00
• Updated: 02/Oct/14 04:03

Description

One of the motivations for creating WebHDFS is for applications connecting to HDFS from outside the cluster. In order to do so, users have to either

1. install Hadoop and use WebHdfsFileSsytem, or
2. develop their own client using the WebHDFS REST API.

For #1, it is very difficult to manage and unnecessarily complicated for other applications since Hadoop is not a lightweight library. For #2, it is not easy to deal with security and handle transient errors.

Therefore, we propose adding a lightweight WebHDFS client as a separated library which does not depend on Common and HDFS. The client can be packaged as a standalone jar. Other applications simply add the jar to their classpath for using it.

Issue Links

• is related to: HDFS-6200 Create a separate jar for hdfs-client