[HDFS-2316] [umbrella] WebHDFS: a complete FileSystem implementation for accessing HDFS over HTTP

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.23.1, 1.0.0
• Component/s: webhdfs
• Labels: critical-0.22.0

Release Note

Provide WebHDFS as a complete FileSystem implementation for accessing HDFS over HTTP.
Previous hftp feature was a read-only FileSystem and does not provide "write" accesses.

• Assignee: Tsz-wo Sze
• Reporter: Tsz-wo Sze
• Created: 07/Sep/11 10:02
• Updated: 19/Dec/17 18:50
• Resolved: 24/May/13 19:04

Description

We current have hftp for accessing HDFS over HTTP. However, hftp is a read-only FileSystem and does not provide "write" accesses.

In HDFS-2284, we propose to have WebHDFS for providing a complete FileSystem implementation for accessing HDFS over HTTP. The is the umbrella JIRA for the tasks.

Issue Links

• duplicates: HDFS-843 Add HTTP POST/PUT/DELETE support for web servers in datanodes
• is related to: HDFS-2645 Webhdfs & HttpFS (Hoop) should share the same codebase
• relates to: HDFS-12512 RBF: Add WebHDFS
• relates to: HDFS-2361 hftp is broken
• relates to: HADOOP-7721 dfs.web.authentication.kerberos.principal expects the full hostname and does not replace _HOST with the hostname
• relates to: HDFS-2368 defaults created for web keytab and principal, these properties should not have defaults
• relates to: HDFS-2409 _HOST in dfs.web.authentication.kerberos.principal.
• relates to: HDFS-2545 Webhdfs: Support multiple namenodes in federation
• relates to: HDFS-2631 Add optional libwebhdfs support to fuse-dfs
• relates to: HDFS-2656 Implement a pure c client based on webhdfs
• relates to: HDFS-2552 Add WebHdfs Forrest doc
• relates to: HDFS-2587 Add WebHDFS apt doc

Sub-Tasks

• Write Http access to HDFS: Closed
• Read access to HDFS using HTTP REST: Closed
• Provide authentication to webhdfs using SPNEGO: Closed
• Configuration option to enable/disable webhdfs.: Closed
• Support getFileBlockLocations and getDelegationToken in webhdfs: Closed
• Support getContentSummary and getFileChecksum in webhdfs: Closed
• webhdfs: support case insensitive query parameter names: Closed
• Support delegation token renewal in webhdfs: Closed
• webhdfs api parameter validation should be better: Closed
• webhdfs returns two content-type headers: Closed
• webhdfs open an invalid path leads to a 500 which states a npe, we should return a 404 with appropriate error message: Closed
• webhdfs liststatus json does not convert to a valid xml document: Closed
• webhdfs setpermission,settimes,setowner api does not return any content upon success: Resolved
• webhdfs GETFILESTATUS calls shows the localName as empty: Resolved
• tail using a webhdfs uri throws an error: Closed
• webhdfs mkdirs api call creates path with 777 permission, we should default it to 755: Closed
• distcp with a webhdfs uri on a secure cluster fails: Closed
• [webhdfs] When Getting the file using OP=OPEN with DN http address, ESTABLISHED sockets are growing.: Closed
• add version prefix and root methods to webhdfs: Closed
• webhdfs api's should return a root element in the json response: Closed
• webhdfs liststatus json response is not correct: Closed
• webhdfs setreplication api should return a 403 when called on a directory: Closed
• The renewer in NamenodeWebHdfsMethods.generateDelegationToken(..) is not used: Closed
• webhdfs throws a npe when ugi is null from getDelegationToken: Closed
• Remove the use of Range header from webhdfs: Closed
• webhdfs rest call to a secure dn fails when a token is sent: Closed
• Support doAs and GETHOMEDIRECTORY in webhdfs: Closed
• Change WebHdfsFileSystem to two-step create/append: Closed