[HADOOP-13037] Refactor Azure Data Lake Store as an independent FileSystem

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.8.0, 3.0.0-alpha2
• Component/s: fs/adl
• Target Version/s: 2.8.0
• Hadoop Flags: Reviewed
• Release Note: Hadoop now supports integration with Azure Data Lake as an alternative Hadoop-compatible file system. Please refer to the Hadoop site documentation of Azure Data Lake for details on usage and configuration.
• Assignee: Vishwajeet Dusane
• Reporter: Shrikant Naidu
• Created: 19/Apr/16 13:21
• Updated: 25/Oct/19 20:25
• Resolved: 09/Mar/17 23:45

Description

The jira proposes an improvement over HADOOP-12666 to remove webhdfs dependencies from the ADL file system client and build out a standalone client. At a high level, this approach would extend the Hadoop file system class to provide an implementation for accessing Azure Data Lake. The scheme used for accessing the file system will continue to be

adl://<accountname>.azuredatalake.net/path/to/file.

The Azure Data Lake Cloud Store will continue to provide a webHDFS rest interface. The client will access the ADLS store using WebHDFS Rest APIs provided by the ADLS store.

Issue Links

• breaks: HADOOP-14205 No FileSystem for scheme: adl
• Dependent: HADOOP-12666 Support Microsoft Azure Data Lake - as a file system in Hadoop
• is related to: HADOOP-13687 Provide a unified dependency artifact that transitively includes the cloud storage modules shipped with Hadoop.
• relates to: HDFS-9938 Support extensions to WebHdfsFileSystem

Sub-Tasks

• Improve Azure Data Lake contract tests.: Resolved