Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-9333

Some tests using MiniDFSCluster errored complaining port in use

    Details

    • Type: Test
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.8.0, 2.7.4, 3.0.0-alpha2
    • Component/s: test
    • Labels:
      None

      Description

      Ref. the following:

      Tests run: 4, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 30.483 sec <<< FAILURE! - in org.apache.hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped
      testRead(org.apache.hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped)  Time elapsed: 11.021 sec  <<< ERROR!
      java.net.BindException: Port in use: localhost:49333
      	at sun.nio.ch.Net.bind0(Native Method)
      	at sun.nio.ch.Net.bind(Net.java:433)
      	at sun.nio.ch.Net.bind(Net.java:425)
      	at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)
      	at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
      	at org.mortbay.jetty.nio.SelectChannelConnector.open(SelectChannelConnector.java:216)
      	at org.apache.hadoop.http.HttpServer2.openListeners(HttpServer2.java:884)
      	at org.apache.hadoop.http.HttpServer2.start(HttpServer2.java:826)
      	at org.apache.hadoop.hdfs.server.namenode.NameNodeHttpServer.start(NameNodeHttpServer.java:142)
      	at org.apache.hadoop.hdfs.server.namenode.NameNode.startHttpServer(NameNode.java:821)
      	at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:675)
      	at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:883)
      	at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:862)
      	at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1555)
      	at org.apache.hadoop.hdfs.MiniDFSCluster.restartNameNode(MiniDFSCluster.java:2015)
      	at org.apache.hadoop.hdfs.MiniDFSCluster.restartNameNode(MiniDFSCluster.java:1996)
      	at org.apache.hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFS.doTestRead(TestBlockTokenWithDFS.java:539)
      	at org.apache.hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped.testRead(TestBlockTokenWithDFSStriped.java:62)
      

      Another one:

      Tests run: 5, Failures: 0, Errors: 2, Skipped: 0, Time elapsed: 9.859 sec <<< FAILURE! - in org.apache.hadoop.hdfs.tools.TestDFSZKFailoverController
      testFailoverAndBackOnNNShutdown(org.apache.hadoop.hdfs.tools.TestDFSZKFailoverController)  Time elapsed: 0.41 sec  <<< ERROR!
      java.net.BindException: Problem binding to [localhost:10021] java.net.BindException: Address already in use; For more details see:  http://wiki.apache.org/hadoop/BindException
      	at sun.nio.ch.Net.bind0(Native Method)
      	at sun.nio.ch.Net.bind(Net.java:433)
      	at sun.nio.ch.Net.bind(Net.java:425)
      	at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)
      	at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
      	at org.apache.hadoop.ipc.Server.bind(Server.java:469)
      	at org.apache.hadoop.ipc.Server$Listener.<init>(Server.java:695)
      	at org.apache.hadoop.ipc.Server.<init>(Server.java:2464)
      	at org.apache.hadoop.ipc.RPC$Server.<init>(RPC.java:945)
      	at org.apache.hadoop.ipc.ProtobufRpcEngine$Server.<init>(ProtobufRpcEngine.java:535)
      	at org.apache.hadoop.ipc.ProtobufRpcEngine.getServer(ProtobufRpcEngine.java:510)
      	at org.apache.hadoop.ipc.RPC$Builder.build(RPC.java:787)
      	at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.<init>(NameNodeRpcServer.java:399)
      	at org.apache.hadoop.hdfs.server.namenode.NameNode.createRpcServer(NameNode.java:742)
      	at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:680)
      	at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:883)
      	at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:862)
      	at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1555)
      	at org.apache.hadoop.hdfs.MiniDFSCluster.createNameNode(MiniDFSCluster.java:1245)
      	at org.apache.hadoop.hdfs.MiniDFSCluster.configureNameService(MiniDFSCluster.java:1014)
      	at org.apache.hadoop.hdfs.MiniDFSCluster.createNameNodesAndSetConf(MiniDFSCluster.java:889)
      	at org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:821)
      	at org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:480)
      	at org.apache.hadoop.hdfs.MiniDFSCluster$Builder.build(MiniDFSCluster.java:439)
      	at org.apache.hadoop.hdfs.tools.TestDFSZKFailoverController.setup(TestDFSZKFailoverController.java:90)
      
      testFailoverAndBackOnNNShutdown(org.apache.hadoop.hdfs.tools.TestDFSZKFailoverController)  Time elapsed: 0.41 sec  <<< ERROR!
      java.lang.NullPointerException: null
      	at org.apache.hadoop.hdfs.tools.TestDFSZKFailoverController.shutdown(TestDFSZKFailoverController.java:114)
      
      1. HDFS-9333.001.patch
        4 kB
        Masatake Iwasaki
      2. HDFS-9333.002.patch
        2 kB
        Masatake Iwasaki
      3. HDFS-9333.003.patch
        6 kB
        Masatake Iwasaki

        Activity

        Hide
        shahrs87 Rushabh S Shah added a comment -

        Still seeing this issue.
        Build logs from HADOOP-14104

        Show
        shahrs87 Rushabh S Shah added a comment - Still seeing this issue. Build logs from HADOOP-14104
        Hide
        zhz Zhe Zhang added a comment -

        Thanks for the work Masatake. This affects branch-2.7 as well so I just did the backport.

        Show
        zhz Zhe Zhang added a comment - Thanks for the work Masatake. This affects branch-2.7 as well so I just did the backport.
        Hide
        iwasakims Masatake Iwasaki added a comment -

        Thanks for the reviewing, Andrew Wang.

        Show
        iwasakims Masatake Iwasaki added a comment - Thanks for the reviewing, Andrew Wang .
        Hide
        andrew.wang Andrew Wang added a comment -

        Resolving since it looks like this was committed to trunk, branch-2, branch-2.8. Thanks for working on this Masatake Iwasaki!

        Show
        andrew.wang Andrew Wang added a comment - Resolving since it looks like this was committed to trunk, branch-2, branch-2.8. Thanks for working on this Masatake Iwasaki !
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10471 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10471/)
        HDFS-9333. Some tests using MiniDFSCluster errored complaining port in (iwasakims: rev 964e546ab1dba5f5d53b209ec6c9a70a85654765)

        • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/tools/TestDFSZKFailoverController.java
        • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockTokenWithDFS.java
        • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockTokenWithDFSStriped.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10471 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10471/ ) HDFS-9333 . Some tests using MiniDFSCluster errored complaining port in (iwasakims: rev 964e546ab1dba5f5d53b209ec6c9a70a85654765) (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/tools/TestDFSZKFailoverController.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockTokenWithDFS.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockTokenWithDFSStriped.java
        Hide
        andrew.wang Andrew Wang added a comment -

        LGTM +1, test failures look unrelated. Thanks for the working on this Masatake Iwasaki!

        Show
        andrew.wang Andrew Wang added a comment - LGTM +1, test failures look unrelated. Thanks for the working on this Masatake Iwasaki !
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 12s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 3 new or modified test files.
        +1 mvninstall 6m 45s trunk passed
        +1 compile 0m 44s trunk passed
        +1 checkstyle 0m 25s trunk passed
        +1 mvnsite 0m 51s trunk passed
        +1 mvneclipse 0m 12s trunk passed
        +1 findbugs 1m 40s trunk passed
        +1 javadoc 0m 55s trunk passed
        +1 mvninstall 0m 46s the patch passed
        +1 compile 0m 43s the patch passed
        +1 javac 0m 43s the patch passed
        +1 checkstyle 0m 23s hadoop-hdfs-project/hadoop-hdfs: The patch generated 0 new + 54 unchanged - 1 fixed = 54 total (was 55)
        +1 mvnsite 0m 49s the patch passed
        +1 mvneclipse 0m 10s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 1m 45s the patch passed
        +1 javadoc 0m 54s the patch passed
        -1 unit 57m 26s hadoop-hdfs in the patch failed.
        +1 asflicense 0m 17s The patch does not generate ASF License warnings.
        76m 13s



        Reason Tests
        Failed junit tests hadoop.hdfs.server.blockmanagement.TestComputeInvalidateWork
          hadoop.metrics2.sink.TestRollingFileSystemSinkWithHdfs



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:9560f25
        JIRA Issue HDFS-9333
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12829172/HDFS-9333.003.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux d509a5bb718f 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / b8a30f2
        Default Java 1.8.0_101
        findbugs v3.0.0
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/16792/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16792/testReport/
        modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16792/console
        Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 12s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 3 new or modified test files. +1 mvninstall 6m 45s trunk passed +1 compile 0m 44s trunk passed +1 checkstyle 0m 25s trunk passed +1 mvnsite 0m 51s trunk passed +1 mvneclipse 0m 12s trunk passed +1 findbugs 1m 40s trunk passed +1 javadoc 0m 55s trunk passed +1 mvninstall 0m 46s the patch passed +1 compile 0m 43s the patch passed +1 javac 0m 43s the patch passed +1 checkstyle 0m 23s hadoop-hdfs-project/hadoop-hdfs: The patch generated 0 new + 54 unchanged - 1 fixed = 54 total (was 55) +1 mvnsite 0m 49s the patch passed +1 mvneclipse 0m 10s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 45s the patch passed +1 javadoc 0m 54s the patch passed -1 unit 57m 26s hadoop-hdfs in the patch failed. +1 asflicense 0m 17s The patch does not generate ASF License warnings. 76m 13s Reason Tests Failed junit tests hadoop.hdfs.server.blockmanagement.TestComputeInvalidateWork   hadoop.metrics2.sink.TestRollingFileSystemSinkWithHdfs Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Issue HDFS-9333 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12829172/HDFS-9333.003.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux d509a5bb718f 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / b8a30f2 Default Java 1.8.0_101 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HDFS-Build/16792/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16792/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16792/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        iwasakims Masatake Iwasaki added a comment -

        attaching updated patch. I fixed tests using restartNameNode and restartDataNodes(true) to try non-ephemeral ports first in order to reduce the possibility of port collision with numerous tests using ephemeral ports. ServerSocketUtil#getPort fallbacks to random non-ephemeral and ephemeral if the given port is not available.

        Show
        iwasakims Masatake Iwasaki added a comment - attaching updated patch. I fixed tests using restartNameNode and restartDataNodes(true) to try non-ephemeral ports first in order to reduce the possibility of port collision with numerous tests using ephemeral ports. ServerSocketUtil#getPort fallbacks to random non-ephemeral and ephemeral if the given port is not available.
        Hide
        andrew.wang Andrew Wang added a comment -

        I think even if we leave it as restartDataNodes(true), just using ServerSocketUtil#getPort instead will reduce the frequency of port conflicts by getting us out of the ephemeral port range.

        I'm happy to +1 any patch so we can give it a try.

        Show
        andrew.wang Andrew Wang added a comment - I think even if we leave it as restartDataNodes(true) , just using ServerSocketUtil#getPort instead will reduce the frequency of port conflicts by getting us out of the ephemeral port range. I'm happy to +1 any patch so we can give it a try.
        Hide
        iwasakims Masatake Iwasaki added a comment -

        Thanks for the comment, Andrew Wang. I'm looking for the way to make the test work when restarting datanodes on random ports by MiniDFSCluster#restartDataNodes(false). Since the test retrieves LocatedBlocks and use it even after DN restart, it might not be trivial fix.

        Show
        iwasakims Masatake Iwasaki added a comment - Thanks for the comment, Andrew Wang . I'm looking for the way to make the test work when restarting datanodes on random ports by MiniDFSCluster#restartDataNodes(false) . Since the test retrieves LocatedBlocks and use it even after DN restart, it might not be trivial fix.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 0s Docker mode activated.
        -1 patch 0m 4s HDFS-9333 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help.



        Subsystem Report/Notes
        JIRA Issue HDFS-9333
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12772133/HDFS-9333.002.patch
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16702/console
        Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. -1 patch 0m 4s HDFS-9333 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. Subsystem Report/Notes JIRA Issue HDFS-9333 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12772133/HDFS-9333.002.patch Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16702/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        andrew.wang Andrew Wang added a comment -

        This looks good to me. Do you mind also fixing TestBlockTokenWithDFS? I've been seeing a number of port conflicts with the subclass TestBlockTokenWithDFSStriped, probably because it needs to start many more DNs than the parent class.

        Show
        andrew.wang Andrew Wang added a comment - This looks good to me. Do you mind also fixing TestBlockTokenWithDFS? I've been seeing a number of port conflicts with the subclass TestBlockTokenWithDFSStriped, probably because it needs to start many more DNs than the parent class.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 6s docker + precommit patch detected.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        +1 mvninstall 3m 18s trunk passed
        +1 compile 0m 36s trunk passed with JDK v1.8.0_60
        +1 compile 0m 34s trunk passed with JDK v1.7.0_79
        +1 checkstyle 0m 18s trunk passed
        +1 mvnsite 0m 45s trunk passed
        +1 mvneclipse 0m 15s trunk passed
        +1 findbugs 1m 59s trunk passed
        +1 javadoc 1m 15s trunk passed with JDK v1.8.0_60
        +1 javadoc 1m 57s trunk passed with JDK v1.7.0_79
        +1 mvninstall 0m 42s the patch passed
        +1 compile 0m 36s the patch passed with JDK v1.8.0_60
        +1 javac 0m 36s the patch passed
        +1 compile 0m 34s the patch passed with JDK v1.7.0_79
        +1 javac 0m 34s the patch passed
        +1 checkstyle 0m 17s the patch passed
        +1 mvnsite 0m 45s the patch passed
        +1 mvneclipse 0m 14s the patch passed
        +1 whitespace 0m 0s Patch has no whitespace issues.
        +1 findbugs 2m 13s the patch passed
        +1 javadoc 1m 14s the patch passed with JDK v1.8.0_60
        +1 javadoc 1m 58s the patch passed with JDK v1.7.0_79
        -1 unit 59m 9s hadoop-hdfs in the patch failed with JDK v1.8.0_60.
        -1 unit 63m 28s hadoop-hdfs in the patch failed with JDK v1.7.0_79.
        -1 asflicense 0m 25s Patch generated 58 ASF License warnings.
        145m 33s



        Reason Tests
        JDK v1.8.0_60 Failed junit tests hadoop.hdfs.server.blockmanagement.TestNodeCount
          hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes
          hadoop.hdfs.server.namenode.snapshot.TestSnapshotMetrics
          hadoop.hdfs.server.namenode.TestAddStripedBlocks
          hadoop.hdfs.server.namenode.TestCacheDirectives
        JDK v1.7.0_79 Failed junit tests hadoop.hdfs.TestReplaceDatanodeOnFailure
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure210
          hadoop.hdfs.server.namenode.ha.TestEditLogTailer
          hadoop.hdfs.TestDFSClientRetries



        Subsystem Report/Notes
        Docker Client=1.7.1 Server=1.7.1 Image:test-patch-base-hadoop-date2015-11-13
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12772133/HDFS-9333.002.patch
        JIRA Issue HDFS-9333
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 84c8983226c5 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /home/jenkins/jenkins-slave/workspace/PreCommit-HDFS-Build/patchprocess/apache-yetus-fa12328/precommit/personality/hadoop.sh
        git revision trunk / cccf884
        findbugs v3.0.0
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/13500/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_60.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/13500/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_79.txt
        unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/13500/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_60.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13500/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_79.txt
        JDK v1.7.0_79 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13500/testReport/
        asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/13500/artifact/patchprocess/patch-asflicense-problems.txt
        modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
        Max memory used 227MB
        Powered by Apache Yetus http://yetus.apache.org
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13500/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 6s docker + precommit patch detected. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 3m 18s trunk passed +1 compile 0m 36s trunk passed with JDK v1.8.0_60 +1 compile 0m 34s trunk passed with JDK v1.7.0_79 +1 checkstyle 0m 18s trunk passed +1 mvnsite 0m 45s trunk passed +1 mvneclipse 0m 15s trunk passed +1 findbugs 1m 59s trunk passed +1 javadoc 1m 15s trunk passed with JDK v1.8.0_60 +1 javadoc 1m 57s trunk passed with JDK v1.7.0_79 +1 mvninstall 0m 42s the patch passed +1 compile 0m 36s the patch passed with JDK v1.8.0_60 +1 javac 0m 36s the patch passed +1 compile 0m 34s the patch passed with JDK v1.7.0_79 +1 javac 0m 34s the patch passed +1 checkstyle 0m 17s the patch passed +1 mvnsite 0m 45s the patch passed +1 mvneclipse 0m 14s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 2m 13s the patch passed +1 javadoc 1m 14s the patch passed with JDK v1.8.0_60 +1 javadoc 1m 58s the patch passed with JDK v1.7.0_79 -1 unit 59m 9s hadoop-hdfs in the patch failed with JDK v1.8.0_60. -1 unit 63m 28s hadoop-hdfs in the patch failed with JDK v1.7.0_79. -1 asflicense 0m 25s Patch generated 58 ASF License warnings. 145m 33s Reason Tests JDK v1.8.0_60 Failed junit tests hadoop.hdfs.server.blockmanagement.TestNodeCount   hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes   hadoop.hdfs.server.namenode.snapshot.TestSnapshotMetrics   hadoop.hdfs.server.namenode.TestAddStripedBlocks   hadoop.hdfs.server.namenode.TestCacheDirectives JDK v1.7.0_79 Failed junit tests hadoop.hdfs.TestReplaceDatanodeOnFailure   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure210   hadoop.hdfs.server.namenode.ha.TestEditLogTailer   hadoop.hdfs.TestDFSClientRetries Subsystem Report/Notes Docker Client=1.7.1 Server=1.7.1 Image:test-patch-base-hadoop-date2015-11-13 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12772133/HDFS-9333.002.patch JIRA Issue HDFS-9333 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 84c8983226c5 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /home/jenkins/jenkins-slave/workspace/PreCommit-HDFS-Build/patchprocess/apache-yetus-fa12328/precommit/personality/hadoop.sh git revision trunk / cccf884 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HDFS-Build/13500/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_60.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/13500/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_79.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/13500/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_60.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13500/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_79.txt JDK v1.7.0_79 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13500/testReport/ asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/13500/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Max memory used 227MB Powered by Apache Yetus http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13500/console This message was automatically generated.
        Hide
        iwasakims Masatake Iwasaki added a comment -

        I attached 002 which uses ServerSocketUtil#getPort to get random port number. Though there is small chance that given port number is used before binding it, this approach is simpler and avoid adding fix to real class.

        Show
        iwasakims Masatake Iwasaki added a comment - I attached 002 which uses ServerSocketUtil#getPort to get random port number. Though there is small chance that given port number is used before binding it, this approach is simpler and avoid adding fix to real class.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 6s docker + precommit patch detected.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
        +1 mvninstall 3m 5s trunk passed
        +1 compile 4m 28s trunk passed with JDK v1.8.0_60
        +1 compile 4m 17s trunk passed with JDK v1.7.0_79
        +1 checkstyle 0m 59s trunk passed
        +1 mvneclipse 0m 28s trunk passed
        +1 findbugs 3m 33s trunk passed
        +1 javadoc 1m 54s trunk passed with JDK v1.8.0_60
        +1 javadoc 2m 50s trunk passed with JDK v1.7.0_79
        +1 mvninstall 2m 12s the patch passed
        +1 compile 4m 21s the patch passed with JDK v1.8.0_60
        +1 javac 4m 21s the patch passed
        +1 compile 4m 16s the patch passed with JDK v1.7.0_79
        +1 javac 4m 17s the patch passed
        +1 checkstyle 0m 56s the patch passed
        +1 mvneclipse 0m 29s the patch passed
        +1 whitespace 0m 0s Patch has no whitespace issues.
        +1 findbugs 3m 49s the patch passed
        +1 javadoc 2m 2s the patch passed with JDK v1.8.0_60
        +1 javadoc 2m 49s the patch passed with JDK v1.7.0_79
        -1 unit 16m 29s hadoop-common in the patch failed with JDK v1.8.0_60.
        -1 unit 53m 23s hadoop-hdfs in the patch failed with JDK v1.8.0_60.
        -1 unit 6m 38s hadoop-common in the patch failed with JDK v1.7.0_79.
        -1 unit 52m 52s hadoop-hdfs in the patch failed with JDK v1.7.0_79.
        -1 asflicense 0m 19s Patch generated 58 ASF License warnings.
        173m 34s



        Reason Tests
        JDK v1.8.0_60 Failed junit tests hadoop.fs.TestLocalFsFCStatistics
          hadoop.hdfs.server.blockmanagement.TestNodeCount
          hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes
        JDK v1.8.0_60 Timed out junit tests org.apache.hadoop.http.TestHttpServerLifecycle
        JDK v1.7.0_79 Failed junit tests hadoop.fs.TestLocalFsFCStatistics
          hadoop.hdfs.TestDFSClientRetries
          hadoop.hdfs.server.namenode.ha.TestDFSUpgradeWithHA
          hadoop.hdfs.server.balancer.TestBalancer
          hadoop.hdfs.TestRecoverStripedFile



        Subsystem Report/Notes
        Docker Client=1.7.1 Server=1.7.1 Image:test-patch-base-hadoop-date2015-11-11
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12771755/HDFS-9333.001.patch
        JIRA Issue HDFS-9333
        Optional Tests asflicense javac javadoc mvninstall unit findbugs checkstyle compile
        uname Linux a69839af5c54 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /home/jenkins/jenkins-slave/workspace/PreCommit-HDFS-Build/patchprocess/apache-yetus-ee5baeb/precommit/personality/hadoop.sh
        git revision trunk / d907697
        Default Java 1.7.0_79
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_60 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_79
        findbugs v3.0.0
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/13475/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_60.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/13475/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_60.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/13475/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_79.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/13475/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_79.txt
        unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/13475/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_60.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13475/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_60.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13475/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_79.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13475/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_79.txt
        JDK v1.7.0_79 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13475/testReport/
        asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/13475/artifact/patchprocess/patch-asflicense-problems.txt
        modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: .
        Max memory used 228MB
        Powered by Apache Yetus http://yetus.apache.org
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13475/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 6s docker + precommit patch detected. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. +1 mvninstall 3m 5s trunk passed +1 compile 4m 28s trunk passed with JDK v1.8.0_60 +1 compile 4m 17s trunk passed with JDK v1.7.0_79 +1 checkstyle 0m 59s trunk passed +1 mvneclipse 0m 28s trunk passed +1 findbugs 3m 33s trunk passed +1 javadoc 1m 54s trunk passed with JDK v1.8.0_60 +1 javadoc 2m 50s trunk passed with JDK v1.7.0_79 +1 mvninstall 2m 12s the patch passed +1 compile 4m 21s the patch passed with JDK v1.8.0_60 +1 javac 4m 21s the patch passed +1 compile 4m 16s the patch passed with JDK v1.7.0_79 +1 javac 4m 17s the patch passed +1 checkstyle 0m 56s the patch passed +1 mvneclipse 0m 29s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 3m 49s the patch passed +1 javadoc 2m 2s the patch passed with JDK v1.8.0_60 +1 javadoc 2m 49s the patch passed with JDK v1.7.0_79 -1 unit 16m 29s hadoop-common in the patch failed with JDK v1.8.0_60. -1 unit 53m 23s hadoop-hdfs in the patch failed with JDK v1.8.0_60. -1 unit 6m 38s hadoop-common in the patch failed with JDK v1.7.0_79. -1 unit 52m 52s hadoop-hdfs in the patch failed with JDK v1.7.0_79. -1 asflicense 0m 19s Patch generated 58 ASF License warnings. 173m 34s Reason Tests JDK v1.8.0_60 Failed junit tests hadoop.fs.TestLocalFsFCStatistics   hadoop.hdfs.server.blockmanagement.TestNodeCount   hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes JDK v1.8.0_60 Timed out junit tests org.apache.hadoop.http.TestHttpServerLifecycle JDK v1.7.0_79 Failed junit tests hadoop.fs.TestLocalFsFCStatistics   hadoop.hdfs.TestDFSClientRetries   hadoop.hdfs.server.namenode.ha.TestDFSUpgradeWithHA   hadoop.hdfs.server.balancer.TestBalancer   hadoop.hdfs.TestRecoverStripedFile Subsystem Report/Notes Docker Client=1.7.1 Server=1.7.1 Image:test-patch-base-hadoop-date2015-11-11 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12771755/HDFS-9333.001.patch JIRA Issue HDFS-9333 Optional Tests asflicense javac javadoc mvninstall unit findbugs checkstyle compile uname Linux a69839af5c54 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /home/jenkins/jenkins-slave/workspace/PreCommit-HDFS-Build/patchprocess/apache-yetus-ee5baeb/precommit/personality/hadoop.sh git revision trunk / d907697 Default Java 1.7.0_79 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_60 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_79 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HDFS-Build/13475/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_60.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/13475/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_60.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/13475/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_79.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/13475/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_79.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/13475/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_60.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13475/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_60.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13475/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_79.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13475/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_79.txt JDK v1.7.0_79 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13475/testReport/ asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/13475/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: . Max memory used 228MB Powered by Apache Yetus http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13475/console This message was automatically generated.
        Hide
        drankye Kai Zheng added a comment -

        Sounds good to me. Thank you for looking at this!

        Show
        drankye Kai Zheng added a comment - Sounds good to me. Thank you for looking at this!
        Hide
        iwasakims Masatake Iwasaki added a comment -

        Thanks for reporting this, Kai Zheng.

        TestBlockTokenWithDFSStriped already uses MiniDFSCluster with random port settings and 49333 is the randomly chosen port. It could be race with another process using ephemeral port.

        TestDFSZKFailoverController seems to use fixed port settings because zkfc part does not allow random port setting. Let me see this if you do not yet started the work.

        Show
        iwasakims Masatake Iwasaki added a comment - Thanks for reporting this, Kai Zheng . TestBlockTokenWithDFSStriped already uses MiniDFSCluster with random port settings and 49333 is the randomly chosen port. It could be race with another process using ephemeral port. TestDFSZKFailoverController seems to use fixed port settings because zkfc part does not allow random port setting. Let me see this if you do not yet started the work.
        Hide
        drankye Kai Zheng added a comment -

        Just a quick thought, maybe we could use random server port every time to avoid such conflicts?

        Show
        drankye Kai Zheng added a comment - Just a quick thought, maybe we could use random server port every time to avoid such conflicts?

          People

          • Assignee:
            iwasakims Masatake Iwasaki
            Reporter:
            drankye Kai Zheng
          • Votes:
            1 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development