Example stacktrace:

	at java.net.Inet4AddressImpl.lookupAllHostAddr(Native Method)
	at java.net.InetAddress$1.lookupAllHostAddr(InetAddress.java:901)
	at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1295)
	at java.net.InetAddress.getAllByName0(InetAddress.java:1248)
	at java.net.InetAddress.getAllByName(InetAddress.java:1164)
	at java.net.InetAddress.getAllByName(InetAddress.java:1098)
	at java.net.InetAddress.getByName(InetAddress.java:1048)
	at org.apache.hadoop.security.SecurityUtil$StandardHostResolver.getByName(SecurityUtil.java:507)
	at org.apache.hadoop.security.SecurityUtil.getByName(SecurityUtil.java:494)
	at org.apache.hadoop.net.NetUtils.createSocketAddrForHost(NetUtils.java:237)
	at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:217)
	at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:164)
	at org.apache.hadoop.hdfs.DFSUtilClient.getNNAddress(DFSUtilClient.java:612)
	at org.apache.hadoop.hdfs.DFSUtilClient.getNNAddress(DFSUtilClient.java:638)
	at org.apache.hadoop.hdfs.NameNodeProxiesClient.createHAProxy(NameNodeProxiesClient.java:341)
	at org.apache.hadoop.hdfs.NameNodeProxiesClient.createProxyWithClientProtocol(NameNodeProxiesClient.java:144)
	at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:339)
	at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:283)
	at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:148)
	at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2711)
	at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:95)
	at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2748)
	at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2730)
	at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:375)
	at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:174)
	at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:359)
	at org.apache.hadoop.fs.Path.getFileSystem(Path.java:294)
	at org.apache.hadoop.fs.shell.PathData.expandAsGlob(PathData.java:325)
	at org.apache.hadoop.fs.shell.Command.expandArgument(Command.java:239)
	at org.apache.hadoop.fs.shell.Command.expandArguments(Command.java:222)
	at org.apache.hadoop.fs.shell.FsCommand.processRawArguments(FsCommand.java:103)
	at org.apache.hadoop.fs.shell.Command.run(Command.java:166)
	at org.apache.hadoop.fs.FsShell.run(FsShell.java:309)
	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
	at org.apache.hadoop.fs.FsShell.main(FsShell.java:367)

Patch 001 fixes the unnecessary DNS resolution by checking against configured service name. Added a test similar to what HADOOP-9150 did to guarantee the URI is not DNS-resolved.
The original DFSUtilClient#getNNAddress(URI) is untouched, given that: 1. It's public; 2. We need configuration to check URI logicality.

-1 overall

This message was automatically generated.

Xiao Chen ,thanks for raising this issue.Patch looks good to me.
I have few review comments.

1.
Following place also requires a check on whether NNURI is logical or not.
DFSUtilClient#getNNAddress(Configuration conf) --> should internally call DFSUtilClient#getNNAddressCheckLogical()

2.
As the check on NNURI is logical, is not necessary for Non-HA cluster,in patch checks at following places is not required
NameNodeProxies#createProxy
NameNodeProxiesClient#createProxyWithClientProtocol

Thanks for the review J.Andreina! Attached patch 002.
1. Makes sense, updated. Good catch!
2. I think it's theoretically possible for a misconfigured cluster to reach there and silently try to resolve in said methods. Since we don't have check in configuration to eliminate this possibility, it's better to handle it than silently resolving. Another option is to do a precondition check here instead of handling it, but I feel that may be too aggressive... What do you think?

I think it's theoretically possible for a misconfigured cluster to reach there and silently try to resolve in said methods....Another option is to do a precondition check here instead of handling it, but I feel that may be too aggressive... What do you think?

Xiao Chen, As u have mentioned, there is already a check for failoverProxyProvider as below

    if (failoverProxyProvider == null) {
      return createNonHAProxy(conf, DFSUtilClient.getNNAddress(nameNodeUri),
          xface, UserGroupInformation.getCurrentUser(), true,
          fallbackToSimpleAuth);
    }

Here for any Non-HA cluster/for HA cluster where NameNode URI is configured to an ip:port(illogical URI), there is no necessity for configuring failoverProxyProvider(hence failoverProxyProvider would be null )

So i feel, here it is not required to handle cases with misconfigurations .

Thanks for the quick response J.Andreina. I initially meant a more 'wrong' configuration (e.g. HA service, but no failoverproxyprovider is configured etc.). But seems in this case a exception is thrown later in buildTokenService, so I agree we don't need the check here.
noformat
java.lang.IllegalArgumentException: java.net.UnknownHostException: nameservice1
at org.apache.hadoop.security.SecurityUtil.buildTokenService(SecurityUtil.java:406)
at org.apache.hadoop.hdfs.NameNodeProxies.createNonHAProxy(NameNodeProxies.java:166)
at org.apache.hadoop.hdfs.NameNodeProxies.createProxy(NameNodeProxies.java:118)
at org.apache.hadoop.hdfs.TestDFSClientFailover.testCreateProxyDoesntDnsResolveLogicalURI(TestDFSClientFailover.java:321)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
Caused by: java.net.UnknownHostException: nameservice1
noformat
Uploaded patch 3 to reflex this.

-1 overall

This message was automatically generated.

Thanks Xiao for the patch and J.Andreina for the review.

Should we add the isLogicalUri logic to the existing getNNAddress(URI filesystemURI) method? Is there a downside of always doing this check? It's cleaner to have one util method to handle all cases, assuming the check doesn't harm non-HA cases.

Another minor point in the 03 patch is that DFSUtilClient. is redundant:

retAddr = DFSUtilClient.getNNAddress(filesystemURI);

Thanks Zhe Zhang for the review!

Is there a downside of always doing this check?

I didn't add isLogicalUri because it requires both the Configuration and the URI. The method getNNAddress(URI) doesn't have Configuration currently, and is declared public. So adding conf would make it backwards incompatible. I investigated in our code base, all places invoking getNNAddress(URI) we can get the conf. But we can't assume no client code is using it. Do you think mark getNNAddress(URI) as deprecated and add getNNAddress(Configuration, URI) is the way to go?

Another minor point in the 03 patch is that DFSUtilClient. is redundant.

Good catch. I'll fix it after receiving the above feedback.

Thanks for clarifying this Xiao. I agree with the approach in 03 patch. So +1 pending the minor fix.

Thanks Zhe Zhang, attached patch 4 with the fix.

Thanks Xiao. +1 pending Jenkins. I'm not sure why Jenkins didn't start on v4 patch. I just manually triggered.

-1 overall

This message was automatically generated.

Thanks Zhe.
The findbugs and asflicense warnings are not introduced by this patch.
The test failures seems not related and passed locally. (TestRequestHedgingProxyProvider#testHedgingWhenOneFails appears to be failing for both jdk7 and jdk8, but looks unrelated and passed locally.)
The javac warning is expected, see original comment here.

Thanks Xiao for updating the patch and verifying Jenkins results. Agreed that testHedgingWhenOneFails is the only seemingly related failure. I verified that the test doesn't trigger the changed code.

+1, I will commit the patch shortly.

Thanks Zhe!

FAILURE: Integrated in Hadoop-trunk-Commit #8785 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8785/)
HDFS-9364. Unnecessary DNS resolution attempts when creating (zhz: rev 73b94d789969354bb9a6872d99976763ca8470d7)

• hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSUtilClient.java
• hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
• hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/NameNodeProxiesClient.java
• hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSClientFailover.java

FAILURE: Integrated in Hadoop-Yarn-trunk #1386 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1386/)
HDFS-9364. Unnecessary DNS resolution attempts when creating (zhz: rev 73b94d789969354bb9a6872d99976763ca8470d7)

• hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSClientFailover.java
• hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
• hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSUtilClient.java
• hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/NameNodeProxiesClient.java

FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #662 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/662/)
HDFS-9364. Unnecessary DNS resolution attempts when creating (zhz: rev 73b94d789969354bb9a6872d99976763ca8470d7)

• hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSClientFailover.java
• hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
• hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/NameNodeProxiesClient.java
• hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSUtilClient.java

FAILURE: Integrated in Hadoop-Mapreduce-trunk #2591 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2591/)
HDFS-9364. Unnecessary DNS resolution attempts when creating (zhz: rev 73b94d789969354bb9a6872d99976763ca8470d7)

• hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSClientFailover.java
• hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/NameNodeProxiesClient.java
• hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
• hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSUtilClient.java

FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #651 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/651/)
HDFS-9364. Unnecessary DNS resolution attempts when creating (zhz: rev 73b94d789969354bb9a6872d99976763ca8470d7)

• hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
• hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/NameNodeProxiesClient.java
• hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSUtilClient.java
• hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSClientFailover.java

FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #592 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/592/)
HDFS-9364. Unnecessary DNS resolution attempts when creating (zhz: rev 73b94d789969354bb9a6872d99976763ca8470d7)

• hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/NameNodeProxiesClient.java
• hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
• hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSUtilClient.java
• hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSClientFailover.java

FAILURE: Integrated in Hadoop-Hdfs-trunk #2531 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2531/)
HDFS-9364. Unnecessary DNS resolution attempts when creating (zhz: rev 73b94d789969354bb9a6872d99976763ca8470d7)

• hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/NameNodeProxiesClient.java
• hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSClientFailover.java
• hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
• hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSUtilClient.java