[HDFS-17333] DFSClient supports lazy resolution from hostname to IP.

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Done
• Fix Version/s: 3.4.1, 3.5.0
• Labels: pull-request-available
• Assignee: Jian Zhang
• Reporter: Jian Zhang
• Created: 10/Jan/24 03:05
• Updated: 02/Mar/24 12:43
• Resolved: 02/Mar/24 12:42

Description

Currently, when dfsclient is started, it will resolve all hosts of all namservices:
at DFSUtilClient#getAddresses(conf, null, addressKey)
at AbstractNNFailoverProxyProvider#getProxyAddresses(URI uri, String addressKey)
If the current environment where the dfsClient is located causes resolution of host->ip to be very slow, the existing logic will undoubtedly take a long time when there are too many nameservices.

Now, each dfsclient only needs the IPs of all namenodes of a certain nameservice at most. A better situation is that if the namenode selected by dfsclient for the first time can provide the required services normally, then the client only needs to know the IP of this namenode. Therefore, it is not necessary to resolve all namenodes of all nameservices in the configuration file, when dfsclient is started.

This patch supports lazy resolution of host->ip, which will only be resolved when the host needs to be accessed.

Issue Links

• links to: GitHub Pull Request #6430