[HDFS-14805] Mounting external stores in HDFS on-the-fly

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Assignee: Unassigned
• Reporter: Virajith Jalaparti
• Created: 30/Aug/19 22:04
• Updated: 08/Dec/20 01:51

Description

Provided storage (HDFS-9806) allows HDFS to address data in external storage systems, including cloud stores. Data mounted in this manner, seamlessly, appears to be part of HDFS for applications/clients. The external data can also be cached by HDFS on local disks and SSDs, accelerating remote data reads (HDFS-13069).

However, Provided storage was originally targeted at ephemeral HDFS deployments in the cloud (e.g., Azure HDInsight). Long running HDFS clusters are common in many other scenarios which can benefit from accessing data in remote stores. This JIRA targets such scenarios and aims to provide the ability to:
(a) Dynamically mount external stores in a HDFS cluster while supporting high availability.
(b) Mount multiple remote stores simultaneously.
(c) Reduce deployment overheads and simplify usability of Provided storage.

Issue Links

• is duplicated by: HDFS-12478 [PROVIDED Phase 2] Command line tools for managing Provided Storage Backup mounts
• is part of: HDFS-15714 HDFS Provided Storage Read/Write Mount Support On-the-fly
• relates to: HDFS-12090 Handling writes from HDFS to Provided storages
• relates to: HDFS-9806 Allow HDFS block replicas to be provided by an external storage system
• relates to: HDFS-13069 Enable HDFS to cache data read from external storage systems

Sub-Tasks

• Add ability to import file ACLs from remote store: Resolved
• Ability to check if a block has a replica on provided storage: Resolved