[HDFS-15289] Allow viewfs mounts with HDFS/HCFS scheme and centralized mount table

• Type: New Feature
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.2.0
• Component/s: fs
• Target Version/s: 3.3.9
• Assignee: Uma Maheswara Rao G
• Reporter: Uma Maheswara Rao G
• Created: 20/Apr/20 18:57
• Updated: 29/Nov/22 15:51

Description

ViewFS provides flexibility to mount different filesystem types with mount points configuration table. This approach is solving the scalability problems, but users need to reconfigure the filesystem to ViewFS and to its scheme.  This will be problematic in the case of paths persisted in meta stores, ex: Hive. In systems like Hive, it will store uris in meta store. So, changing the file system scheme will create a burden to upgrade/recreate meta stores. In our experience many users are not ready to change that.  

Router based federation is another implementation to provide coordinated mount points for HDFS federation clusters. Even though this provides flexibility to handle mount points easily, this will not allow other(non-HDFS) file systems to mount. So, this does not solve the purpose when users want to mount external(non-HDFS) filesystems.

So, the problem here is: Even though many users want to adapt to the scalable fs options available, technical challenges of changing schemes (ex: in meta stores) in deployments are obstructing them.

So, we propose to allow hdfs scheme in ViewFS like client side mount system and provision user to create mount links without changing URI paths. 
I will upload detailed design doc shortly.

Issue Links

• is related to: HADOOP-17024 ListStatus on ViewFS root (ls "/") should list the linkFallBack root (configured target root).
• is related to: HADOOP-17028 ViewFS should initialize target filesystems lazily
• is related to: HADOOP-17029 ViewFS does not return correct user/group and ACL
• is related to: HADOOP-17032 Handle an internal dir in viewfs having multiple children mount points pointing to different filesystems
• is related to: HADOOP-17060 listStatus and getFileStatus behave inconsistent in the case of ViewFs implementation for isDirectory
• is related to: HADOOP-17999 No-op implementation of setWriteChecksum and setVerifyChecksum in ViewFileSystem
• is related to: HDFS-15396 Fix TestViewFileSystemOverloadSchemeHdfsFileSystemContract#testListStatusRootDir
• relates to: HADOOP-18129 Change URI[] in INodeLink to String[] to reduce memory footprint of ViewFileSystem
• relates to: RANGER-3058 [ranger-hive] create table fails when ViewDFS(client side HDFS mounting fs) mount points are targeting to Ozone/S3 FS
• relates to: HIVE-24342 isPathEncrypted should make sure resolved path also from HDFS
• links to: ViewFsOverloadScheme User Guide

Sub-Tasks

• Extend ViewFS and provide ViewFSOverloadScheme implementation with scheme configurable.: Resolved
• Make mount-table to read from central place ( Let's say from HDFS): Resolved
• Make NflyFS to work when ViewFsOverloadScheme's scheme and target uris schemes are same.: Resolved
• Make DFSAdmin tool to work with ViewFSOverloadScheme: Resolved
• Document the ViewFSOverloadScheme details in ViewFS guide: Resolved
• DFSAdmin should close filesystem and dfsadmin -setBalancerBandwidth should work with ViewFSOverloadScheme: Resolved
• Add all available fs.viewfs.overload.scheme.target.<scheme>.impl classes in core-default.xml bydefault.: Resolved
• Add fs.viewfs.overload.scheme.target.ofs.impl to core-default.xml: Resolved
• FSUsage$DF should consider ViewFSOverloadScheme in processPath: Resolved
• ViewFileSystemOverloadScheme should represent mount links as non symlinks: Resolved
• Fix ContentSummary for mount links in ViewFileSystemOverloadScheme: Resolved
• Merged ListStatus with Fallback target filesystem and InternalDirViewFS.: Resolved
• mkdirs should work when parent dir is internalDir and fallback configured.: Resolved
• Default mount table name used by ViewFileSystem should be configurable: Resolved
• create should work when parent dir is internalDir and fallback configured.: Resolved
• Fix NN trash emptier to work if ViewFSOveroadScheme enabled: Resolved
• Optionally ignore port number in mount-table name when picking from initialized uri: Resolved
• ViewFsOverloadScheme should work when -fs option pointing to remote cluster without mount links: Resolved
• ViewFsOverloadScheme should not display error message with "viewfs://" even when it's initialized with other fs.: Resolved
• When Empty mount points, we are assigning fallback link to self. But it should not use full URI for target fs.: Resolved
• mkdirs on fallback should throw IOE out instead of suppressing and returning false: Resolved
• Provide DFS API compatible class(ViewDistributedFileSystem), but use ViewFileSystemOverloadScheme inside: Resolved
• getChildFilesystems should include fallback fs as well: Resolved
• ViewDistributedFileSystem#recoverLease should call super.recoverLease when there are no mounts configured: Resolved
• listFiles on root/InternalDir will fail if fallback root has file: Resolved
• Fix the rename issues with fallback fs enabled: Resolved
• ViewDFS#getDelegationToken should not throw UnsupportedOperationException.: Resolved
• ViewHDFS#create(f, permission, cflags, bufferSize, replication, blockSize, progress, checksumOpt) should not be restricted to DFS only.: Resolved
• ViewHDFS#canonicalizeUri should not be restricted to DFS only API.: Resolved
• Namenode trashEmptier should not init ViewFs on startup: Resolved
• ViewFileSystemOverloadScheme support specifying mount table loader imp through conf: Resolved
• ViewDistributedFileSystem#rename wrongly using src in the place of dst.: Resolved
• Fix ViewDFS with mount points for HDFS only API: Resolved
• Provide FileContext based ViewFSOverloadScheme implementation: In Progress
• mkdir should not create dir in fallback if the dir already in mount Path: Open
• DFS cacheadmin, ECAdmin, StoragePolicyAdmin commands should handle ViewFSOverloadScheme: Open
• Implement ViewFsAdmin to list the mount points, target fs for path etc.: Open
• DistCP fails with ViewHDFS and preserveEC options if the actual target path is non HDFS: Open
• Provide documentation for ViewHDFS: Open
• Add resolveMountPath API in FileSystem: Open