[HDFS-13891] HDFS RBF stabilization phase I

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.2.0
• Labels: RBF
• Assignee: Unassigned
• Reporter: Brahma Reddy Battula
• Created: 03/Sep/18 19:22
• Updated: 02/Sep/20 07:54
• Resolved: 02/Sep/20 07:54

Description

RBF(Router Based Federation) shipped in 3.0+ and 2.9..

now its out various corner cases, scale and error handling issues are surfacing.

And we are targeting security feaiure (HDFS-13532) also.

this umbrella to fix all those issues and support missing protocols(HDFS-13655) before next 3.3 release.

Issue Links

• incorporates: HDFS-13532 RBF: Adding security
• incorporates: HDFS-13655 RBF: Add missing ClientProtocol APIs to RBF
• is related to: HDFS-12615 Router-based HDFS federation phase 2
• relates to: HDFS-14603 Über-JIRA: HDFS RBF stabilization phase II

Sub-Tasks

• RBF: TestRouterWebHDFSContractAppend.testRenameFileBeingAppended fails: Resolved
• RBF: RouterAdmin update cmd is overwriting the entry not updating the existing: Resolved
• RBF: Update mount table cache immediately after changing (add/update/remove) mount table entries.: Resolved
• RBF: Fail when try to remove mount point paths: Resolved
• RBF: Connection creator thread should catch Throwable: Resolved
• RBF: Add multiple paths for dfsrouteradmin "rm" and "clrquota" commands: Resolved
• RBF: The DN_REPORT_TIME_OUT and DN_REPORT_CACHE_EXPIRE should be configured in RBFConfigKeys.: Resolved
• RBF: The default MountTableResolver should fail resolving multi-destination paths: Resolved
• RBF: Handle NPE for NamenodeBeanMetrics#getFederationMetrics: Resolved
• RBF: Add more information to HdfsFileStatus for a mount point: Resolved
• RBF: ProvidedCapacityTotal json exception in NamenodeHeartbeatService: Resolved
• RBF: LS command for root shows wrong owner and permission information.: Resolved
• RBF: Add option to fail operations when a subcluster is unavailable: Resolved
• RBF: MIN_ACTIVE_RATIO should be configurable: Resolved
• RBF: Make the read-only column of Mount Table clearly understandable: Resolved
• RBF: Fix a typo in RouterAdmin usage: Resolved
• RBF: Throw StandbyException instead of IOException so that client can retry when can not get connection: Resolved
• RBF: RouterAdmin should support dfsrouteradmin -refreshRouterArgs command: Resolved
• RBF: Add stale nodes to federation metrics: Resolved
• RBF: Quotas of the sub-cluster should be removed when removing the mount point: Resolved
• RBF: Remove hard coded router status from FederationMetrics.: Resolved
• RBF: Inconsistency with the Default Namespace: Resolved
• RBF: Cleanup quota modules: Resolved
• RBF: rollEdit() command fails with Router: Resolved
• RBF: setQuota() through router is working for only the mount Points under the Source column in MountTable: Resolved
• RBF: ACL commands should work over all the destinations: Resolved
• RBF: Remove dependency on availability of default namespace: Resolved
• RBF: Add configuration documents for using multiple sub-clusters: Resolved
• RBF: NPE in getContentSummary() for getEcPolicy() in case of multiple destinations: Resolved
• RBF : MiniRouterDFSCluster should configure the failover proxy provider for namespace: Resolved
• RBF: Setting attributes should set on all subclusters' directories.: Resolved
• RBF: Tooling to identify the subcluster location of a file: Resolved
• RBF : Exceptions are exposing the actual sub cluster path: Resolved
• RBF: Fix safemode message for Router: Resolved
• RBF: Fix the location of the DNs in getDatanodeReport(): Resolved
• RBF: Throw RetriableException instead of IOException when no namenodes available: Resolved
• RBF: Support unavailable subclusters for mount points with multiple destinations: Resolved
• RBF: Add maintenance nodes to federation metrics: Resolved
• RBF: IOE While Removing Mount Entry: Resolved
• RBF: Use human readable format for long numbers in the Router UI: Resolved
• RBF: Fix renaming folders spread across multiple subclusters: Resolved
• RBF: Optimize configuration item resolving for monitor namenode: Resolved
• RBF: Fix trailing "/" for webhdfs: Resolved
• RBF: Prevent loading metric system when disabled: Resolved
• RBF: Router shouldn't allow READ operations in safe mode: Resolved
• RBF: Optimize the file write process in case of multiple destinations.: Resolved
• RBF: Add safemode to Router UI: Resolved
• RBF: Router should support RefreshUserMappingsProtocol: Resolved
• RBF: getContentSummary() should allow non-existing folders: Resolved
• RBF: Add order text SPACE in CLI command 'hdfs dfsrouteradmin': Resolved
• RBF: Remove unnecessary quota checks: Resolved
• RBF: Clean-up and refactor UI components: Resolved
• RBF: Create hdfs-rbf-site.xml for RBF specific properties: Resolved
• RBF: Separate namenodeHeartbeat and routerHeartbeat to different config key: Resolved
• RBF: Update the document of RBF related metrics: Resolved
• RBF: Router should support GetUserMappingsProtocol: Resolved
• RBF: Failed to get statistics from NameNodes before 2.9.0: Resolved
• RBF: updateMount should validate the newEntry in RouterAdmin: Resolved
• RBF: Remove FSCK from Router Web UI: Resolved
• RBF: Add FSCK to the Router: Resolved