[ATLAS-2444] HDFS NameNode Federation support

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.0.0
• Assignee: Apoorv Naik
• Reporter: Srikanth Venkat
• Created: 14/Feb/18 04:42
• Updated: 06/Jun/18 02:15
• Resolved: 06/Jun/18 02:15

Description

With Hadoop 3, HDFS supports multiple namespaces; each namespace is serviced by a cluster of name-nodes identified by a nameServiceId. HDFS path entity in Atlas should be updated to capture the the nameServiceId of the namenode cluster for the path. Also, references to individual namenode in the path should be replaced with corresponding nameServiceID.

Changes:

1. HDFS model to include a new attribute called nameServiceId, default will be empty
2. hdfs path's qualified name will include the cluster info as well
3. All Hooks dealing with HDFS path will resolve the host:port to the respective nameServiceId

Issue Links

• is required by: ATLAS-2381 Disabled Integration Test: HiveHookIT.java:196
• is required by: ATLAS-2382 Disabled Integration Test: HiveHookIT.java:662
• is required by: ATLAS-2384 Disabled Integration Test: HiveHookIT.java:967
• is required by: ATLAS-2385 Disabled Integration Test: HiveHookIT.java:1111