Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-9364

Unnecessary DNS resolution attempts when creating NameNodeProxies

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: ha, performance
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      When creating NameNodeProxies, we always try to DNS-resolve namenode URIs. This is unnecessary if the URI is logical, and may be significantly slow if the DNS is having problems.

      1. HDFS-9364.001.patch
        6 kB
        Xiao Chen
      2. HDFS-9364.002.patch
        7 kB
        Xiao Chen
      3. HDFS-9364.003.patch
        5 kB
        Xiao Chen
      4. HDFS-9364.004.patch
        5 kB
        Xiao Chen

        Issue Links

          Activity

          Hide
          xiaochen Xiao Chen added a comment -

          Example stacktrace:

          	at java.net.Inet4AddressImpl.lookupAllHostAddr(Native Method)
          	at java.net.InetAddress$1.lookupAllHostAddr(InetAddress.java:901)
          	at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1295)
          	at java.net.InetAddress.getAllByName0(InetAddress.java:1248)
          	at java.net.InetAddress.getAllByName(InetAddress.java:1164)
          	at java.net.InetAddress.getAllByName(InetAddress.java:1098)
          	at java.net.InetAddress.getByName(InetAddress.java:1048)
          	at org.apache.hadoop.security.SecurityUtil$StandardHostResolver.getByName(SecurityUtil.java:507)
          	at org.apache.hadoop.security.SecurityUtil.getByName(SecurityUtil.java:494)
          	at org.apache.hadoop.net.NetUtils.createSocketAddrForHost(NetUtils.java:237)
          	at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:217)
          	at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:164)
          	at org.apache.hadoop.hdfs.DFSUtilClient.getNNAddress(DFSUtilClient.java:612)
          	at org.apache.hadoop.hdfs.DFSUtilClient.getNNAddress(DFSUtilClient.java:638)
          	at org.apache.hadoop.hdfs.NameNodeProxiesClient.createHAProxy(NameNodeProxiesClient.java:341)
          	at org.apache.hadoop.hdfs.NameNodeProxiesClient.createProxyWithClientProtocol(NameNodeProxiesClient.java:144)
          	at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:339)
          	at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:283)
          	at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:148)
          	at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2711)
          	at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:95)
          	at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2748)
          	at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2730)
          	at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:375)
          	at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:174)
          	at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:359)
          	at org.apache.hadoop.fs.Path.getFileSystem(Path.java:294)
          	at org.apache.hadoop.fs.shell.PathData.expandAsGlob(PathData.java:325)
          	at org.apache.hadoop.fs.shell.Command.expandArgument(Command.java:239)
          	at org.apache.hadoop.fs.shell.Command.expandArguments(Command.java:222)
          	at org.apache.hadoop.fs.shell.FsCommand.processRawArguments(FsCommand.java:103)
          	at org.apache.hadoop.fs.shell.Command.run(Command.java:166)
          	at org.apache.hadoop.fs.FsShell.run(FsShell.java:309)
          	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
          	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
          	at org.apache.hadoop.fs.FsShell.main(FsShell.java:367)
          
          Show
          xiaochen Xiao Chen added a comment - Example stacktrace: at java.net.Inet4AddressImpl.lookupAllHostAddr(Native Method) at java.net.InetAddress$1.lookupAllHostAddr(InetAddress.java:901) at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1295) at java.net.InetAddress.getAllByName0(InetAddress.java:1248) at java.net.InetAddress.getAllByName(InetAddress.java:1164) at java.net.InetAddress.getAllByName(InetAddress.java:1098) at java.net.InetAddress.getByName(InetAddress.java:1048) at org.apache.hadoop.security.SecurityUtil$StandardHostResolver.getByName(SecurityUtil.java:507) at org.apache.hadoop.security.SecurityUtil.getByName(SecurityUtil.java:494) at org.apache.hadoop.net.NetUtils.createSocketAddrForHost(NetUtils.java:237) at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:217) at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:164) at org.apache.hadoop.hdfs.DFSUtilClient.getNNAddress(DFSUtilClient.java:612) at org.apache.hadoop.hdfs.DFSUtilClient.getNNAddress(DFSUtilClient.java:638) at org.apache.hadoop.hdfs.NameNodeProxiesClient.createHAProxy(NameNodeProxiesClient.java:341) at org.apache.hadoop.hdfs.NameNodeProxiesClient.createProxyWithClientProtocol(NameNodeProxiesClient.java:144) at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:339) at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:283) at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:148) at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2711) at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:95) at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2748) at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2730) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:375) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:174) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:359) at org.apache.hadoop.fs.Path.getFileSystem(Path.java:294) at org.apache.hadoop.fs.shell.PathData.expandAsGlob(PathData.java:325) at org.apache.hadoop.fs.shell.Command.expandArgument(Command.java:239) at org.apache.hadoop.fs.shell.Command.expandArguments(Command.java:222) at org.apache.hadoop.fs.shell.FsCommand.processRawArguments(FsCommand.java:103) at org.apache.hadoop.fs.shell.Command.run(Command.java:166) at org.apache.hadoop.fs.FsShell.run(FsShell.java:309) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84) at org.apache.hadoop.fs.FsShell.main(FsShell.java:367)
          Hide
          xiaochen Xiao Chen added a comment -

          Patch 001 fixes the unnecessary DNS resolution by checking against configured service name. Added a test similar to what HADOOP-9150 did to guarantee the URI is not DNS-resolved.
          The original DFSUtilClient#getNNAddress(URI) is untouched, given that: 1. It's public; 2. We need configuration to check URI logicality.

          Show
          xiaochen Xiao Chen added a comment - Patch 001 fixes the unnecessary DNS resolution by checking against configured service name. Added a test similar to what HADOOP-9150 did to guarantee the URI is not DNS-resolved. The original DFSUtilClient#getNNAddress(URI) is untouched, given that: 1. It's public; 2. We need configuration to check URI logicality.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 6s docker + precommit patch detected.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 2m 56s trunk passed
          +1 compile 1m 6s trunk passed with JDK v1.8.0_60
          +1 compile 1m 3s trunk passed with JDK v1.7.0_79
          +1 checkstyle 0m 23s trunk passed
          +1 mvneclipse 0m 28s trunk passed
          -1 findbugs 2m 8s hadoop-hdfs-project/hadoop-hdfs in trunk cannot run convertXmlToText from findbugs
          +1 javadoc 1m 44s trunk passed with JDK v1.8.0_60
          +1 javadoc 2m 30s trunk passed with JDK v1.7.0_79
          -1 mvninstall 0m 27s hadoop-hdfs in the patch failed.
          +1 compile 1m 7s the patch passed with JDK v1.8.0_60
          -1 javac 8m 26s hadoop-hdfs-project-jdk1.8.0_60 with JDK v1.8.0_60 has problems.
          +1 javac 1m 7s the patch passed
          +1 compile 1m 4s the patch passed with JDK v1.7.0_79
          -1 javac 9m 30s hadoop-hdfs-project-jdk1.7.0_79 with JDK v1.7.0_79 has problems.
          +1 javac 1m 4s the patch passed
          +1 checkstyle 0m 22s the patch passed
          +1 mvneclipse 0m 27s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 4m 44s the patch passed
          +1 javadoc 1m 43s the patch passed with JDK v1.8.0_60
          +1 javadoc 2m 29s the patch passed with JDK v1.7.0_79
          -1 unit 63m 21s hadoop-hdfs in the patch failed with JDK v1.8.0_60.
          +1 unit 1m 1s hadoop-hdfs-client in the patch passed with JDK v1.8.0_60.
          -1 unit 61m 39s hadoop-hdfs in the patch failed with JDK v1.7.0_79.
          +1 unit 1m 2s hadoop-hdfs-client in the patch passed with JDK v1.7.0_79.
          -1 asflicense 0m 24s Patch generated 58 ASF License warnings.
          158m 1s



          Reason Tests
          JDK v1.8.0_60 Failed junit tests hadoop.hdfs.server.namenode.ha.TestEditLogTailer
          JDK v1.7.0_79 Failed junit tests hadoop.hdfs.shortcircuit.TestShortCircuitCache
            hadoop.hdfs.TestDFSStripedOutputStreamWithFailure090
            hadoop.hdfs.server.blockmanagement.TestBlocksWithNotEnoughRacks
            hadoop.hdfs.TestRecoverStripedFile
            hadoop.hdfs.TestDFSStripedOutputStreamWithFailure
            hadoop.hdfs.TestDFSClientRetries



          Subsystem Report/Notes
          Docker Client=1.7.1 Server=1.7.1 Image:test-patch-base-hadoop-date2015-11-03
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12770267/HDFS-9364.001.patch
          JIRA Issue HDFS-9364
          Optional Tests asflicense javac javadoc mvninstall unit findbugs checkstyle compile
          uname Linux f46b44f33e43 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /home/jenkins/jenkins-slave/workspace/PreCommit-HDFS-Build/patchprocess/apache-yetus-1a9afee/precommit/personality/hadoop.sh
          git revision trunk / 957f031
          Default Java 1.7.0_79
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_60 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_79
          findbugs v3.0.0
          findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/13355/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs.html
          mvninstall https://builds.apache.org/job/PreCommit-HDFS-Build/13355/artifact/patchprocess/patch-mvninstall-hadoop-hdfs-project_hadoop-hdfs.txt
          javac hadoop-hdfs-project-jdk1.8.0_60: https://builds.apache.org/job/PreCommit-HDFS-Build/13355/artifact/patchprocess/javac-hadoop-hdfs-project-jdk1.8.0_60-diff.txt
          javac hadoop-hdfs-project-jdk1.7.0_79: https://builds.apache.org/job/PreCommit-HDFS-Build/13355/artifact/patchprocess/javac-hadoop-hdfs-project-jdk1.7.0_79-diff.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/13355/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_60.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/13355/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_79.txt
          unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/13355/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_60.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13355/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_79.txt
          JDK v1.7.0_79 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13355/testReport/
          asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/13355/artifact/patchprocess/patch-asflicense-problems.txt
          modules C: hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-client U: hadoop-hdfs-project
          Max memory used 227MB
          Powered by Apache Yetus http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13355/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 6s docker + precommit patch detected. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 2m 56s trunk passed +1 compile 1m 6s trunk passed with JDK v1.8.0_60 +1 compile 1m 3s trunk passed with JDK v1.7.0_79 +1 checkstyle 0m 23s trunk passed +1 mvneclipse 0m 28s trunk passed -1 findbugs 2m 8s hadoop-hdfs-project/hadoop-hdfs in trunk cannot run convertXmlToText from findbugs +1 javadoc 1m 44s trunk passed with JDK v1.8.0_60 +1 javadoc 2m 30s trunk passed with JDK v1.7.0_79 -1 mvninstall 0m 27s hadoop-hdfs in the patch failed. +1 compile 1m 7s the patch passed with JDK v1.8.0_60 -1 javac 8m 26s hadoop-hdfs-project-jdk1.8.0_60 with JDK v1.8.0_60 has problems. +1 javac 1m 7s the patch passed +1 compile 1m 4s the patch passed with JDK v1.7.0_79 -1 javac 9m 30s hadoop-hdfs-project-jdk1.7.0_79 with JDK v1.7.0_79 has problems. +1 javac 1m 4s the patch passed +1 checkstyle 0m 22s the patch passed +1 mvneclipse 0m 27s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 4m 44s the patch passed +1 javadoc 1m 43s the patch passed with JDK v1.8.0_60 +1 javadoc 2m 29s the patch passed with JDK v1.7.0_79 -1 unit 63m 21s hadoop-hdfs in the patch failed with JDK v1.8.0_60. +1 unit 1m 1s hadoop-hdfs-client in the patch passed with JDK v1.8.0_60. -1 unit 61m 39s hadoop-hdfs in the patch failed with JDK v1.7.0_79. +1 unit 1m 2s hadoop-hdfs-client in the patch passed with JDK v1.7.0_79. -1 asflicense 0m 24s Patch generated 58 ASF License warnings. 158m 1s Reason Tests JDK v1.8.0_60 Failed junit tests hadoop.hdfs.server.namenode.ha.TestEditLogTailer JDK v1.7.0_79 Failed junit tests hadoop.hdfs.shortcircuit.TestShortCircuitCache   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure090   hadoop.hdfs.server.blockmanagement.TestBlocksWithNotEnoughRacks   hadoop.hdfs.TestRecoverStripedFile   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure   hadoop.hdfs.TestDFSClientRetries Subsystem Report/Notes Docker Client=1.7.1 Server=1.7.1 Image:test-patch-base-hadoop-date2015-11-03 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12770267/HDFS-9364.001.patch JIRA Issue HDFS-9364 Optional Tests asflicense javac javadoc mvninstall unit findbugs checkstyle compile uname Linux f46b44f33e43 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /home/jenkins/jenkins-slave/workspace/PreCommit-HDFS-Build/patchprocess/apache-yetus-1a9afee/precommit/personality/hadoop.sh git revision trunk / 957f031 Default Java 1.7.0_79 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_60 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_79 findbugs v3.0.0 findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/13355/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs.html mvninstall https://builds.apache.org/job/PreCommit-HDFS-Build/13355/artifact/patchprocess/patch-mvninstall-hadoop-hdfs-project_hadoop-hdfs.txt javac hadoop-hdfs-project-jdk1.8.0_60: https://builds.apache.org/job/PreCommit-HDFS-Build/13355/artifact/patchprocess/javac-hadoop-hdfs-project-jdk1.8.0_60-diff.txt javac hadoop-hdfs-project-jdk1.7.0_79: https://builds.apache.org/job/PreCommit-HDFS-Build/13355/artifact/patchprocess/javac-hadoop-hdfs-project-jdk1.7.0_79-diff.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/13355/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_60.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/13355/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_79.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/13355/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_60.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13355/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_79.txt JDK v1.7.0_79 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13355/testReport/ asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/13355/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-client U: hadoop-hdfs-project Max memory used 227MB Powered by Apache Yetus http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13355/console This message was automatically generated.
          Hide
          andreina J.Andreina added a comment -

          Xiao Chen ,thanks for raising this issue.Patch looks good to me.
          I have few review comments.

          1.
          Following place also requires a check on whether NNURI is logical or not.
          DFSUtilClient#getNNAddress(Configuration conf) --> should internally call DFSUtilClient#getNNAddressCheckLogical()

          2.
          As the check on NNURI is logical, is not necessary for Non-HA cluster,in patch checks at following places is not required
          NameNodeProxies#createProxy
          NameNodeProxiesClient#createProxyWithClientProtocol

          Show
          andreina J.Andreina added a comment - Xiao Chen ,thanks for raising this issue.Patch looks good to me. I have few review comments. 1. Following place also requires a check on whether NNURI is logical or not. DFSUtilClient#getNNAddress(Configuration conf) --> should internally call DFSUtilClient#getNNAddressCheckLogical() 2. As the check on NNURI is logical, is not necessary for Non-HA cluster,in patch checks at following places is not required NameNodeProxies#createProxy NameNodeProxiesClient#createProxyWithClientProtocol
          Hide
          xiaochen Xiao Chen added a comment -

          Thanks for the review J.Andreina! Attached patch 002.
          1. Makes sense, updated. Good catch!
          2. I think it's theoretically possible for a misconfigured cluster to reach there and silently try to resolve in said methods. Since we don't have check in configuration to eliminate this possibility, it's better to handle it than silently resolving. Another option is to do a precondition check here instead of handling it, but I feel that may be too aggressive... What do you think?

          Show
          xiaochen Xiao Chen added a comment - Thanks for the review J.Andreina ! Attached patch 002. 1. Makes sense, updated. Good catch! 2. I think it's theoretically possible for a misconfigured cluster to reach there and silently try to resolve in said methods. Since we don't have check in configuration to eliminate this possibility, it's better to handle it than silently resolving. Another option is to do a precondition check here instead of handling it, but I feel that may be too aggressive... What do you think?
          Hide
          andreina J.Andreina added a comment -

          I think it's theoretically possible for a misconfigured cluster to reach there and silently try to resolve in said methods....Another option is to do a precondition check here instead of handling it, but I feel that may be too aggressive... What do you think?

          Xiao Chen, As u have mentioned, there is already a check for failoverProxyProvider as below

              if (failoverProxyProvider == null) {
                return createNonHAProxy(conf, DFSUtilClient.getNNAddress(nameNodeUri),
                    xface, UserGroupInformation.getCurrentUser(), true,
                    fallbackToSimpleAuth);
              }
          

          Here for any Non-HA cluster/for HA cluster where NameNode URI is configured to an ip:port(illogical URI), there is no necessity for configuring failoverProxyProvider(hence failoverProxyProvider would be null )

          So i feel, here it is not required to handle cases with misconfigurations .

          Show
          andreina J.Andreina added a comment - I think it's theoretically possible for a misconfigured cluster to reach there and silently try to resolve in said methods....Another option is to do a precondition check here instead of handling it, but I feel that may be too aggressive... What do you think? Xiao Chen , As u have mentioned, there is already a check for failoverProxyProvider as below if (failoverProxyProvider == null ) { return createNonHAProxy(conf, DFSUtilClient.getNNAddress(nameNodeUri), xface, UserGroupInformation.getCurrentUser(), true , fallbackToSimpleAuth); } Here for any Non-HA cluster/for HA cluster where NameNode URI is configured to an ip:port(illogical URI), there is no necessity for configuring failoverProxyProvider(hence failoverProxyProvider would be null ) So i feel, here it is not required to handle cases with misconfigurations .
          Hide
          xiaochen Xiao Chen added a comment -

          Thanks for the quick response J.Andreina. I initially meant a more 'wrong' configuration (e.g. HA service, but no failoverproxyprovider is configured etc.). But seems in this case a exception is thrown later in buildTokenService, so I agree we don't need the check here.
          noformat
          java.lang.IllegalArgumentException: java.net.UnknownHostException: nameservice1
          at org.apache.hadoop.security.SecurityUtil.buildTokenService(SecurityUtil.java:406)
          at org.apache.hadoop.hdfs.NameNodeProxies.createNonHAProxy(NameNodeProxies.java:166)
          at org.apache.hadoop.hdfs.NameNodeProxies.createProxy(NameNodeProxies.java:118)
          at org.apache.hadoop.hdfs.TestDFSClientFailover.testCreateProxyDoesntDnsResolveLogicalURI(TestDFSClientFailover.java:321)
          at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
          at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
          at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
          at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
          at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
          at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
          Caused by: java.net.UnknownHostException: nameservice1
          noformat
          Uploaded patch 3 to reflex this.

          Show
          xiaochen Xiao Chen added a comment - Thanks for the quick response J.Andreina . I initially meant a more 'wrong' configuration (e.g. HA service, but no failoverproxyprovider is configured etc.). But seems in this case a exception is thrown later in buildTokenService , so I agree we don't need the check here. noformat java.lang.IllegalArgumentException: java.net.UnknownHostException: nameservice1 at org.apache.hadoop.security.SecurityUtil.buildTokenService(SecurityUtil.java:406) at org.apache.hadoop.hdfs.NameNodeProxies.createNonHAProxy(NameNodeProxies.java:166) at org.apache.hadoop.hdfs.NameNodeProxies.createProxy(NameNodeProxies.java:118) at org.apache.hadoop.hdfs.TestDFSClientFailover.testCreateProxyDoesntDnsResolveLogicalURI(TestDFSClientFailover.java:321) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74) Caused by: java.net.UnknownHostException: nameservice1 noformat Uploaded patch 3 to reflex this.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 6s docker + precommit patch detected.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 3m 24s trunk passed
          +1 compile 0m 59s trunk passed with JDK v1.8.0_60
          +1 compile 0m 56s trunk passed with JDK v1.7.0_79
          +1 checkstyle 0m 20s trunk passed
          +1 mvneclipse 0m 25s trunk passed
          -1 findbugs 1m 51s hadoop-hdfs-project/hadoop-hdfs in trunk cannot run convertXmlToText from findbugs
          +1 javadoc 1m 29s trunk passed with JDK v1.8.0_60
          +1 javadoc 2m 7s trunk passed with JDK v1.7.0_79
          +1 mvninstall 1m 7s the patch passed
          +1 compile 0m 55s the patch passed with JDK v1.8.0_60
          -1 javac 7m 26s hadoop-hdfs-project-jdk1.8.0_60 with JDK v1.8.0_60 generated 9 new issues (was 29, now 30).
          +1 javac 0m 55s the patch passed
          +1 compile 0m 54s the patch passed with JDK v1.7.0_79
          -1 javac 8m 21s hadoop-hdfs-project-jdk1.7.0_79 with JDK v1.7.0_79 generated 9 new issues (was 29, now 30).
          +1 javac 0m 54s the patch passed
          +1 checkstyle 0m 20s the patch passed
          +1 mvneclipse 0m 26s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 3m 58s the patch passed
          +1 javadoc 1m 24s the patch passed with JDK v1.8.0_60
          +1 javadoc 2m 8s the patch passed with JDK v1.7.0_79
          -1 unit 50m 49s hadoop-hdfs in the patch failed with JDK v1.8.0_60.
          +1 unit 0m 49s hadoop-hdfs-client in the patch passed with JDK v1.8.0_60.
          -1 unit 50m 20s hadoop-hdfs in the patch failed with JDK v1.7.0_79.
          +1 unit 0m 54s hadoop-hdfs-client in the patch passed with JDK v1.7.0_79.
          -1 asflicense 0m 20s Patch generated 56 ASF License warnings.
          130m 38s



          Reason Tests
          JDK v1.8.0_60 Failed junit tests hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes
            hadoop.hdfs.server.balancer.TestBalancerWithMultipleNameNodes
            hadoop.hdfs.server.namenode.ha.TestDNFencing
          JDK v1.7.0_79 Failed junit tests hadoop.hdfs.TestDFSStripedOutputStreamWithFailure
            hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots



          Subsystem Report/Notes
          Docker Client=1.7.1 Server=1.7.1 Image:test-patch-base-hadoop-date2015-11-04
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12770623/HDFS-9364.003.patch
          JIRA Issue HDFS-9364
          Optional Tests asflicense javac javadoc mvninstall unit findbugs checkstyle compile
          uname Linux 6f6d7b425006 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /home/jenkins/jenkins-slave/workspace/PreCommit-HDFS-Build/patchprocess/apache-yetus-d0f6847/precommit/personality/hadoop.sh
          git revision trunk / 5667129
          Default Java 1.7.0_79
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_60 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_79
          findbugs v3.0.0
          findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/13385/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs.html
          javac hadoop-hdfs-project-jdk1.8.0_60: https://builds.apache.org/job/PreCommit-HDFS-Build/13385/artifact/patchprocess/diff-compile-javac-hadoop-hdfs-project-jdk1.8.0_60.txt
          javac hadoop-hdfs-project-jdk1.7.0_79: https://builds.apache.org/job/PreCommit-HDFS-Build/13385/artifact/patchprocess/diff-compile-javac-hadoop-hdfs-project-jdk1.7.0_79.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/13385/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_60.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/13385/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_79.txt
          unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/13385/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_60.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13385/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_79.txt
          JDK v1.7.0_79 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13385/testReport/
          asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/13385/artifact/patchprocess/patch-asflicense-problems.txt
          modules C: hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-client U: hadoop-hdfs-project
          Max memory used 227MB
          Powered by Apache Yetus http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13385/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 6s docker + precommit patch detected. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 3m 24s trunk passed +1 compile 0m 59s trunk passed with JDK v1.8.0_60 +1 compile 0m 56s trunk passed with JDK v1.7.0_79 +1 checkstyle 0m 20s trunk passed +1 mvneclipse 0m 25s trunk passed -1 findbugs 1m 51s hadoop-hdfs-project/hadoop-hdfs in trunk cannot run convertXmlToText from findbugs +1 javadoc 1m 29s trunk passed with JDK v1.8.0_60 +1 javadoc 2m 7s trunk passed with JDK v1.7.0_79 +1 mvninstall 1m 7s the patch passed +1 compile 0m 55s the patch passed with JDK v1.8.0_60 -1 javac 7m 26s hadoop-hdfs-project-jdk1.8.0_60 with JDK v1.8.0_60 generated 9 new issues (was 29, now 30). +1 javac 0m 55s the patch passed +1 compile 0m 54s the patch passed with JDK v1.7.0_79 -1 javac 8m 21s hadoop-hdfs-project-jdk1.7.0_79 with JDK v1.7.0_79 generated 9 new issues (was 29, now 30). +1 javac 0m 54s the patch passed +1 checkstyle 0m 20s the patch passed +1 mvneclipse 0m 26s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 3m 58s the patch passed +1 javadoc 1m 24s the patch passed with JDK v1.8.0_60 +1 javadoc 2m 8s the patch passed with JDK v1.7.0_79 -1 unit 50m 49s hadoop-hdfs in the patch failed with JDK v1.8.0_60. +1 unit 0m 49s hadoop-hdfs-client in the patch passed with JDK v1.8.0_60. -1 unit 50m 20s hadoop-hdfs in the patch failed with JDK v1.7.0_79. +1 unit 0m 54s hadoop-hdfs-client in the patch passed with JDK v1.7.0_79. -1 asflicense 0m 20s Patch generated 56 ASF License warnings. 130m 38s Reason Tests JDK v1.8.0_60 Failed junit tests hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes   hadoop.hdfs.server.balancer.TestBalancerWithMultipleNameNodes   hadoop.hdfs.server.namenode.ha.TestDNFencing JDK v1.7.0_79 Failed junit tests hadoop.hdfs.TestDFSStripedOutputStreamWithFailure   hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots Subsystem Report/Notes Docker Client=1.7.1 Server=1.7.1 Image:test-patch-base-hadoop-date2015-11-04 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12770623/HDFS-9364.003.patch JIRA Issue HDFS-9364 Optional Tests asflicense javac javadoc mvninstall unit findbugs checkstyle compile uname Linux 6f6d7b425006 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /home/jenkins/jenkins-slave/workspace/PreCommit-HDFS-Build/patchprocess/apache-yetus-d0f6847/precommit/personality/hadoop.sh git revision trunk / 5667129 Default Java 1.7.0_79 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_60 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_79 findbugs v3.0.0 findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/13385/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs.html javac hadoop-hdfs-project-jdk1.8.0_60: https://builds.apache.org/job/PreCommit-HDFS-Build/13385/artifact/patchprocess/diff-compile-javac-hadoop-hdfs-project-jdk1.8.0_60.txt javac hadoop-hdfs-project-jdk1.7.0_79: https://builds.apache.org/job/PreCommit-HDFS-Build/13385/artifact/patchprocess/diff-compile-javac-hadoop-hdfs-project-jdk1.7.0_79.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/13385/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_60.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/13385/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_79.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/13385/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_60.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13385/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_79.txt JDK v1.7.0_79 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13385/testReport/ asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/13385/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-client U: hadoop-hdfs-project Max memory used 227MB Powered by Apache Yetus http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13385/console This message was automatically generated.
          Hide
          zhz Zhe Zhang added a comment -

          Thanks Xiao for the patch and J.Andreina for the review.

          Should we add the isLogicalUri logic to the existing getNNAddress(URI filesystemURI) method? Is there a downside of always doing this check? It's cleaner to have one util method to handle all cases, assuming the check doesn't harm non-HA cases.

          Another minor point in the 03 patch is that DFSUtilClient. is redundant:

          retAddr = DFSUtilClient.getNNAddress(filesystemURI);
          
          Show
          zhz Zhe Zhang added a comment - Thanks Xiao for the patch and J.Andreina for the review. Should we add the isLogicalUri logic to the existing getNNAddress(URI filesystemURI) method? Is there a downside of always doing this check? It's cleaner to have one util method to handle all cases, assuming the check doesn't harm non-HA cases. Another minor point in the 03 patch is that DFSUtilClient. is redundant: retAddr = DFSUtilClient.getNNAddress(filesystemURI);
          Hide
          xiaochen Xiao Chen added a comment -

          Thanks Zhe Zhang for the review!

          Is there a downside of always doing this check?

          I didn't add isLogicalUri because it requires both the Configuration and the URI. The method getNNAddress(URI) doesn't have Configuration currently, and is declared public. So adding conf would make it backwards incompatible. I investigated in our code base, all places invoking getNNAddress(URI) we can get the conf. But we can't assume no client code is using it. Do you think mark getNNAddress(URI) as deprecated and add getNNAddress(Configuration, URI) is the way to go?

          Another minor point in the 03 patch is that DFSUtilClient. is redundant.

          Good catch. I'll fix it after receiving the above feedback.

          Show
          xiaochen Xiao Chen added a comment - Thanks Zhe Zhang for the review! Is there a downside of always doing this check? I didn't add isLogicalUri because it requires both the Configuration and the URI . The method getNNAddress(URI) doesn't have Configuration currently, and is declared public . So adding conf would make it backwards incompatible. I investigated in our code base, all places invoking getNNAddress(URI) we can get the conf. But we can't assume no client code is using it. Do you think mark getNNAddress(URI) as deprecated and add getNNAddress(Configuration, URI) is the way to go? Another minor point in the 03 patch is that DFSUtilClient. is redundant. Good catch. I'll fix it after receiving the above feedback.
          Hide
          zhz Zhe Zhang added a comment -

          Thanks for clarifying this Xiao. I agree with the approach in 03 patch. So +1 pending the minor fix.

          Show
          zhz Zhe Zhang added a comment - Thanks for clarifying this Xiao. I agree with the approach in 03 patch. So +1 pending the minor fix.
          Hide
          xiaochen Xiao Chen added a comment -

          Thanks Zhe Zhang, attached patch 4 with the fix.

          Show
          xiaochen Xiao Chen added a comment - Thanks Zhe Zhang , attached patch 4 with the fix.
          Hide
          zhz Zhe Zhang added a comment -

          Thanks Xiao. +1 pending Jenkins. I'm not sure why Jenkins didn't start on v4 patch. I just manually triggered.

          Show
          zhz Zhe Zhang added a comment - Thanks Xiao. +1 pending Jenkins. I'm not sure why Jenkins didn't start on v4 patch. I just manually triggered.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 8s docker + precommit patch detected.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 3m 18s trunk passed
          +1 compile 1m 17s trunk passed with JDK v1.8.0_66
          +1 compile 1m 7s trunk passed with JDK v1.7.0_79
          +1 checkstyle 0m 26s trunk passed
          +1 mvneclipse 0m 30s trunk passed
          -1 findbugs 2m 18s hadoop-hdfs-project/hadoop-hdfs in trunk has 1 extant Findbugs warnings.
          +1 javadoc 1m 48s trunk passed with JDK v1.8.0_66
          +1 javadoc 2m 47s trunk passed with JDK v1.7.0_79
          +1 mvninstall 1m 21s the patch passed
          +1 compile 1m 15s the patch passed with JDK v1.8.0_66
          -1 javac 9m 14s hadoop-hdfs-project-jdk1.8.0_66 with JDK v1.8.0_66 generated 9 new issues (was 29, now 30).
          +1 javac 1m 15s the patch passed
          +1 compile 1m 7s the patch passed with JDK v1.7.0_79
          -1 javac 10m 21s hadoop-hdfs-project-jdk1.7.0_79 with JDK v1.7.0_79 generated 9 new issues (was 29, now 30).
          +1 javac 1m 7s the patch passed
          +1 checkstyle 0m 26s the patch passed
          +1 mvneclipse 0m 29s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 4m 49s the patch passed
          +1 javadoc 1m 53s the patch passed with JDK v1.8.0_66
          +1 javadoc 2m 39s the patch passed with JDK v1.7.0_79
          -1 unit 76m 27s hadoop-hdfs in the patch failed with JDK v1.8.0_66.
          +1 unit 1m 7s hadoop-hdfs-client in the patch passed with JDK v1.8.0_66.
          -1 unit 73m 53s hadoop-hdfs in the patch failed with JDK v1.7.0_79.
          +1 unit 1m 3s hadoop-hdfs-client in the patch passed with JDK v1.7.0_79.
          -1 asflicense 0m 21s Patch generated 56 ASF License warnings.
          185m 49s



          Reason Tests
          JDK v1.8.0_66 Failed junit tests hadoop.hdfs.server.namenode.ha.TestEditLogTailer
            hadoop.hdfs.TestWriteReadStripedFile
            hadoop.hdfs.server.datanode.TestBlockReplacement
            hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes
            hadoop.hdfs.TestReplication
            hadoop.hdfs.server.namenode.TestNameNodeResourceChecker
            hadoop.hdfs.server.namenode.TestNamenodeCapacityReport
            hadoop.hdfs.server.namenode.ha.TestRequestHedgingProxyProvider
            hadoop.hdfs.server.namenode.TestQuotaWithStripedBlocks
            hadoop.hdfs.TestDFSClientRetries
          JDK v1.7.0_79 Failed junit tests hadoop.hdfs.web.TestWebHdfsTimeouts
            hadoop.hdfs.TestDFSStripedOutputStreamWithFailure
            hadoop.hdfs.TestDecommission
            hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure
            hadoop.hdfs.TestRollingUpgrade
            hadoop.hdfs.server.namenode.ha.TestHASafeMode
            hadoop.hdfs.server.namenode.ha.TestRequestHedgingProxyProvider



          Subsystem Report/Notes
          Docker Client=1.7.0 Server=1.7.0 Image:test-patch-base-hadoop-date2015-11-09
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12771145/HDFS-9364.004.patch
          JIRA Issue HDFS-9364
          Optional Tests asflicense javac javadoc mvninstall unit findbugs checkstyle compile
          uname Linux 6744e13a01a1 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /home/jenkins/jenkins-slave/workspace/PreCommit-HDFS-Build/patchprocess/apache-yetus-ee5baeb/precommit/personality/hadoop.sh
          git revision trunk / 94a1833
          Default Java 1.7.0_79
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_79
          findbugs v3.0.0
          findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/13449/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html
          javac hadoop-hdfs-project-jdk1.8.0_66: https://builds.apache.org/job/PreCommit-HDFS-Build/13449/artifact/patchprocess/diff-compile-javac-hadoop-hdfs-project-jdk1.8.0_66.txt
          javac hadoop-hdfs-project-jdk1.7.0_79: https://builds.apache.org/job/PreCommit-HDFS-Build/13449/artifact/patchprocess/diff-compile-javac-hadoop-hdfs-project-jdk1.7.0_79.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/13449/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/13449/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_79.txt
          unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/13449/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13449/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_79.txt
          JDK v1.7.0_79 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13449/testReport/
          asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/13449/artifact/patchprocess/patch-asflicense-problems.txt
          modules C: hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-client U: hadoop-hdfs-project
          Max memory used 226MB
          Powered by Apache Yetus http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13449/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 8s docker + precommit patch detected. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 3m 18s trunk passed +1 compile 1m 17s trunk passed with JDK v1.8.0_66 +1 compile 1m 7s trunk passed with JDK v1.7.0_79 +1 checkstyle 0m 26s trunk passed +1 mvneclipse 0m 30s trunk passed -1 findbugs 2m 18s hadoop-hdfs-project/hadoop-hdfs in trunk has 1 extant Findbugs warnings. +1 javadoc 1m 48s trunk passed with JDK v1.8.0_66 +1 javadoc 2m 47s trunk passed with JDK v1.7.0_79 +1 mvninstall 1m 21s the patch passed +1 compile 1m 15s the patch passed with JDK v1.8.0_66 -1 javac 9m 14s hadoop-hdfs-project-jdk1.8.0_66 with JDK v1.8.0_66 generated 9 new issues (was 29, now 30). +1 javac 1m 15s the patch passed +1 compile 1m 7s the patch passed with JDK v1.7.0_79 -1 javac 10m 21s hadoop-hdfs-project-jdk1.7.0_79 with JDK v1.7.0_79 generated 9 new issues (was 29, now 30). +1 javac 1m 7s the patch passed +1 checkstyle 0m 26s the patch passed +1 mvneclipse 0m 29s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 4m 49s the patch passed +1 javadoc 1m 53s the patch passed with JDK v1.8.0_66 +1 javadoc 2m 39s the patch passed with JDK v1.7.0_79 -1 unit 76m 27s hadoop-hdfs in the patch failed with JDK v1.8.0_66. +1 unit 1m 7s hadoop-hdfs-client in the patch passed with JDK v1.8.0_66. -1 unit 73m 53s hadoop-hdfs in the patch failed with JDK v1.7.0_79. +1 unit 1m 3s hadoop-hdfs-client in the patch passed with JDK v1.7.0_79. -1 asflicense 0m 21s Patch generated 56 ASF License warnings. 185m 49s Reason Tests JDK v1.8.0_66 Failed junit tests hadoop.hdfs.server.namenode.ha.TestEditLogTailer   hadoop.hdfs.TestWriteReadStripedFile   hadoop.hdfs.server.datanode.TestBlockReplacement   hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes   hadoop.hdfs.TestReplication   hadoop.hdfs.server.namenode.TestNameNodeResourceChecker   hadoop.hdfs.server.namenode.TestNamenodeCapacityReport   hadoop.hdfs.server.namenode.ha.TestRequestHedgingProxyProvider   hadoop.hdfs.server.namenode.TestQuotaWithStripedBlocks   hadoop.hdfs.TestDFSClientRetries JDK v1.7.0_79 Failed junit tests hadoop.hdfs.web.TestWebHdfsTimeouts   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure   hadoop.hdfs.TestDecommission   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure   hadoop.hdfs.TestRollingUpgrade   hadoop.hdfs.server.namenode.ha.TestHASafeMode   hadoop.hdfs.server.namenode.ha.TestRequestHedgingProxyProvider Subsystem Report/Notes Docker Client=1.7.0 Server=1.7.0 Image:test-patch-base-hadoop-date2015-11-09 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12771145/HDFS-9364.004.patch JIRA Issue HDFS-9364 Optional Tests asflicense javac javadoc mvninstall unit findbugs checkstyle compile uname Linux 6744e13a01a1 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /home/jenkins/jenkins-slave/workspace/PreCommit-HDFS-Build/patchprocess/apache-yetus-ee5baeb/precommit/personality/hadoop.sh git revision trunk / 94a1833 Default Java 1.7.0_79 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_79 findbugs v3.0.0 findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/13449/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html javac hadoop-hdfs-project-jdk1.8.0_66: https://builds.apache.org/job/PreCommit-HDFS-Build/13449/artifact/patchprocess/diff-compile-javac-hadoop-hdfs-project-jdk1.8.0_66.txt javac hadoop-hdfs-project-jdk1.7.0_79: https://builds.apache.org/job/PreCommit-HDFS-Build/13449/artifact/patchprocess/diff-compile-javac-hadoop-hdfs-project-jdk1.7.0_79.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/13449/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/13449/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_79.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/13449/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13449/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_79.txt JDK v1.7.0_79 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13449/testReport/ asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/13449/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-client U: hadoop-hdfs-project Max memory used 226MB Powered by Apache Yetus http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13449/console This message was automatically generated.
          Hide
          xiaochen Xiao Chen added a comment -

          Thanks Zhe.
          The findbugs and asflicense warnings are not introduced by this patch.
          The test failures seems not related and passed locally. (TestRequestHedgingProxyProvider#testHedgingWhenOneFails appears to be failing for both jdk7 and jdk8, but looks unrelated and passed locally.)
          The javac warning is expected, see original comment here.

          Show
          xiaochen Xiao Chen added a comment - Thanks Zhe. The findbugs and asflicense warnings are not introduced by this patch. The test failures seems not related and passed locally. ( TestRequestHedgingProxyProvider#testHedgingWhenOneFails appears to be failing for both jdk7 and jdk8, but looks unrelated and passed locally.) The javac warning is expected, see original comment here .
          Hide
          zhz Zhe Zhang added a comment -

          Thanks Xiao for updating the patch and verifying Jenkins results. Agreed that testHedgingWhenOneFails is the only seemingly related failure. I verified that the test doesn't trigger the changed code.

          +1, I will commit the patch shortly.

          Show
          zhz Zhe Zhang added a comment - Thanks Xiao for updating the patch and verifying Jenkins results. Agreed that testHedgingWhenOneFails is the only seemingly related failure. I verified that the test doesn't trigger the changed code. +1, I will commit the patch shortly.
          Hide
          xiaochen Xiao Chen added a comment -

          Thanks Zhe!

          Show
          xiaochen Xiao Chen added a comment - Thanks Zhe!
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #8785 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8785/)
          HDFS-9364. Unnecessary DNS resolution attempts when creating (zhz: rev 73b94d789969354bb9a6872d99976763ca8470d7)

          • hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSUtilClient.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/NameNodeProxiesClient.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSClientFailover.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #8785 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8785/ ) HDFS-9364 . Unnecessary DNS resolution attempts when creating (zhz: rev 73b94d789969354bb9a6872d99976763ca8470d7) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSUtilClient.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/NameNodeProxiesClient.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSClientFailover.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk #1386 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1386/)
          HDFS-9364. Unnecessary DNS resolution attempts when creating (zhz: rev 73b94d789969354bb9a6872d99976763ca8470d7)

          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSClientFailover.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSUtilClient.java
          • hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/NameNodeProxiesClient.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #1386 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1386/ ) HDFS-9364 . Unnecessary DNS resolution attempts when creating (zhz: rev 73b94d789969354bb9a6872d99976763ca8470d7) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSClientFailover.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSUtilClient.java hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/NameNodeProxiesClient.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #662 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/662/)
          HDFS-9364. Unnecessary DNS resolution attempts when creating (zhz: rev 73b94d789969354bb9a6872d99976763ca8470d7)

          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSClientFailover.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/NameNodeProxiesClient.java
          • hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSUtilClient.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #662 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/662/ ) HDFS-9364 . Unnecessary DNS resolution attempts when creating (zhz: rev 73b94d789969354bb9a6872d99976763ca8470d7) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSClientFailover.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/NameNodeProxiesClient.java hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSUtilClient.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk #2591 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2591/)
          HDFS-9364. Unnecessary DNS resolution attempts when creating (zhz: rev 73b94d789969354bb9a6872d99976763ca8470d7)

          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSClientFailover.java
          • hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/NameNodeProxiesClient.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSUtilClient.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #2591 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2591/ ) HDFS-9364 . Unnecessary DNS resolution attempts when creating (zhz: rev 73b94d789969354bb9a6872d99976763ca8470d7) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSClientFailover.java hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/NameNodeProxiesClient.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSUtilClient.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #651 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/651/)
          HDFS-9364. Unnecessary DNS resolution attempts when creating (zhz: rev 73b94d789969354bb9a6872d99976763ca8470d7)

          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/NameNodeProxiesClient.java
          • hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSUtilClient.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSClientFailover.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #651 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/651/ ) HDFS-9364 . Unnecessary DNS resolution attempts when creating (zhz: rev 73b94d789969354bb9a6872d99976763ca8470d7) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/NameNodeProxiesClient.java hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSUtilClient.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSClientFailover.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #592 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/592/)
          HDFS-9364. Unnecessary DNS resolution attempts when creating (zhz: rev 73b94d789969354bb9a6872d99976763ca8470d7)

          • hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/NameNodeProxiesClient.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSUtilClient.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSClientFailover.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #592 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/592/ ) HDFS-9364 . Unnecessary DNS resolution attempts when creating (zhz: rev 73b94d789969354bb9a6872d99976763ca8470d7) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/NameNodeProxiesClient.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSUtilClient.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSClientFailover.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk #2531 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2531/)
          HDFS-9364. Unnecessary DNS resolution attempts when creating (zhz: rev 73b94d789969354bb9a6872d99976763ca8470d7)

          • hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/NameNodeProxiesClient.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSClientFailover.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSUtilClient.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #2531 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2531/ ) HDFS-9364 . Unnecessary DNS resolution attempts when creating (zhz: rev 73b94d789969354bb9a6872d99976763ca8470d7) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/NameNodeProxiesClient.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSClientFailover.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSUtilClient.java

            People

            • Assignee:
              xiaochen Xiao Chen
              Reporter:
              xiaochen Xiao Chen
            • Votes:
              0 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development