[HDDS-5447] HttpFS support in Ozone

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Implemented
• Fix Version/s: 1.4.0
• Component/s: HttpFS GateWay, Ozone Client, Ozone Manager
• Labels: pull-request-available
• Assignee: István Fajth
• Reporter: Aravindan Vijayan
• Created: 15/Jul/21 16:21
• Updated: 07/Mar/23 23:08
• Resolved: 07/Mar/23 07:05

Description

There are several tools out there mainly written in Python, that uses the webhdfs interface to connect to HDFS. Even there are quite a few other filesystem implementations that provide access via the same rest interface that HDFS provides.
HUE also implements the HDFS file browser by accessing HDFS via the REST API either on the NameNodes or on HTTPFS Server instances added to the HDFS service.

This gave the inspiration to check and experiment what is required to support a similar REST endpoint over Ozone.

The advantage is that we can ease the migration of tools developed in-house that are using this interface of HDFS, while we can add the possibility to browse Ozone from HUE.
There is literally no disadvantage of having such an interface, as we can implement it as a separate module which does not have any interference with the rest of the code, as the REST endpoint as with HTTPFS will use a regular Java based Ozone client to serve any requests.

Issue Links

• links to: GitHub Pull Request #4117
• links to: GitHub Pull Request #4356

Sub-Tasks

• Copy over HttpFS module from Hadoop 3.3.1 to Ozone: Resolved
• Remove unnecessary hadoop dependencies from httpfs module: Resolved
• Add lifecycle management to HttpFS server: Resolved
• Add a simple test suite for HTTPFS GW: Resolved
• [HTTPFSGW] Remove or replace Hadoop shaded guava dependencies.: Resolved
• [HTTPFSGGW] junit.jar and json-simple in jar report: Resolved
• [HTTPFSGW] Move to org.apache.ozone package from org.apache.hadoop: Resolved
• [HTTPFSGW] Remove non-server side related code from Ozone: Resolved
• [HTTPFSGW] Fix dependency issues after master merge: Resolved
• [HTTPFSGW] Add proper handling for unsupported operations: Resolved
• [HTTPFSGW] Port HTTPFS node and robot tests to ozone-ha, and ozonesecure(-ha): Resolved
• [HTTPFSGW] Merge master into HDDS-5447-httpfs branch: Resolved
• Ozone Runner Image: Create httpfs user and group: Resolved
• [HTTPFSGW] Fix secure integration tests for the HttpFS module: Resolved
• [HTTPFSGW] Clean up dependencies: Resolved
• Replace Hadoop annotations with Ozone-specific ones: Resolved
• [HTTPFSGW] Review ZK and Curator dependencies, and get rid of them.: Resolved
• Replace import from shaded Guava: Resolved
• [HTTPFSGW] Change ozone-filesystem to a runtime dependency for HttpFS: Resolved
• Add okio as dependency to the root pom: Resolved
• [HTTPFSGW] Remove support for satisfyStoragePolicy operation: Resolved
• [HTTPFSGW] Update module doc, and place it in Ozone project docs: Resolved
• Dependency convergence error for zookeeper: Resolved
• [HTTPFSGW] Conform with new checkstyle rule in master: Resolved
• [HTTPFSGW] Add JvmPauseMonitor for HttpFSServerWebServer: Resolved