[HDFS-15289] Allow viewfs mounts with HDFS/HCFS scheme and centralized mount table

• Type: New Feature
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.2.0
• Component/s: fs
• Target Version/s: 3.3.9
• Assignee: Uma Maheswara Rao G
• Reporter: Uma Maheswara Rao G
• Created: 20/Apr/20 18:57
• Updated: 29/Nov/22 15:51

Description

ViewFS provides flexibility to mount different filesystem types with mount points configuration table. This approach is solving the scalability problems, but users need to reconfigure the filesystem to ViewFS and to its scheme.  This will be problematic in the case of paths persisted in meta stores, ex: Hive. In systems like Hive, it will store uris in meta store. So, changing the file system scheme will create a burden to upgrade/recreate meta stores. In our experience many users are not ready to change that.  

Router based federation is another implementation to provide coordinated mount points for HDFS federation clusters. Even though this provides flexibility to handle mount points easily, this will not allow other(non-HDFS) file systems to mount. So, this does not solve the purpose when users want to mount external(non-HDFS) filesystems.

So, the problem here is: Even though many users want to adapt to the scalable fs options available, technical challenges of changing schemes (ex: in meta stores) in deployments are obstructing them.

So, we propose to allow hdfs scheme in ViewFS like client side mount system and provision user to create mount links without changing URI paths. 
I will upload detailed design doc shortly.

Issue Links

• is related to: HADOOP-17024 ListStatus on ViewFS root (ls "/") should list the linkFallBack root (configured target root).
• is related to: HADOOP-17028 ViewFS should initialize target filesystems lazily
• is related to: HADOOP-17029 ViewFS does not return correct user/group and ACL
• is related to: HADOOP-17032 Handle an internal dir in viewfs having multiple children mount points pointing to different filesystems
• is related to: HADOOP-17060 listStatus and getFileStatus behave inconsistent in the case of ViewFs implementation for isDirectory
• is related to: HADOOP-17999 No-op implementation of setWriteChecksum and setVerifyChecksum in ViewFileSystem
• is related to: HDFS-15396 Fix TestViewFileSystemOverloadSchemeHdfsFileSystemContract#testListStatusRootDir
• relates to: HADOOP-18129 Change URI[] in INodeLink to String[] to reduce memory footprint of ViewFileSystem
• relates to: RANGER-3058 [ranger-hive] create table fails when ViewDFS(client side HDFS mounting fs) mount points are targeting to Ozone/S3 FS
• relates to: HIVE-24342 isPathEncrypted should make sure resolved path also from HDFS
• links to: ViewFsOverloadScheme User Guide

Sub-Tasks

• Provide FileContext based ViewFSOverloadScheme implementation: In Progress
• mkdir should not create dir in fallback if the dir already in mount Path: Open
• DFS cacheadmin, ECAdmin, StoragePolicyAdmin commands should handle ViewFSOverloadScheme: Open
• Implement ViewFsAdmin to list the mount points, target fs for path etc.: Open
• DistCP fails with ViewHDFS and preserveEC options if the actual target path is non HDFS: Open
• Provide documentation for ViewHDFS: Open
• Add resolveMountPath API in FileSystem: Open