[HDFS-14118] Support using DNS to resolve nameservices to IP addresses

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 3.3.0
• Hadoop Flags: Reviewed
• Release Note: HDFS clients can use a single domain name to discover servers (namenodes/routers/observers) instead of explicitly listing out all hosts in the config
• Assignee: Fengnan Li
• Reporter: Fengnan Li
• Created: 30/Nov/18 21:56
• Updated: 10/Feb/21 14:08
• Resolved: 23/Feb/19 21:09

Description

In router based federation (RBF), clients will need to know about routers to talk to the HDFS cluster (obviously), and having routers updating (adding/removing) will have to make config change in every client, which is a painful process.

DNS can be used here to resolve the single domain name clients knows to a list of routers in the current config. However, DNS won't be able to consider only resolving to the working router based on certain health thresholds.

There are some ways about how this can be solved. One way is to have a separate script to regularly check the status of the router and update the DNS records if a router fails the health thresholds. In this way, security might be carefully considered for this way. Another way is to have the client do the normal connecting/failover after they get the list of routers, which requires the change of current failover proxy provider.

See the attached design document for details about the proposed solution.

Issue Links

• is a child of: HDFS-15831 Adopt more DNS resolving for HDFS
• is related to: HDFS-14600 DataNodes using DNS NameNode discovery will not send block reports when failover
• is related to: HDFS-14601 NameNode HA with single DNS record for NameNode discovery prevent running ZKFC
• is related to: HADOOP-16168 mvn clean site is not compiling in trunk
• is related to: HDFS-15343 TestConfiguredFailoverProxyProvider is failing
• is related to: HDFS-14327 Using FQDN instead of IP to access servers with DNS resolving
• is related to: HDFS-15785 Datanode to support using DNS to resolve nameservices to IP addresses to get list of namenodes
• is related to: HDFS-13312 NameNode High Availability ZooKeeper based discovery rather than explicit nn1,nn2 configs
• relates to: HADOOP-16168 mvn clean site is not compiling in trunk
• relates to: HDFS-14310 Improve documents for using DNS to resolve namenodes and routers