Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 3.0.0-alpha1
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: test
    • Labels:
      None

      Description

      It fails in recent builds, see:
      https://builds.apache.org/job/PreCommit-HDFS-Build/14177/testReport/org.apache.hadoop.hdfs/
      https://builds.apache.org/job/PreCommit-HDFS-Build/14147/testReport/org.apache.hadoop.hdfs/

      Failing test methods include:

      • org.apache.hadoop.hdfs.TestLeaseRecovery2.testHardLeaseRecoveryWithRenameAfterNameNodeRestart
      • org.apache.hadoop.hdfs.TestLeaseRecovery2.testLeaseRecoverByAnotherUser
      • org.apache.hadoop.hdfs.TestLeaseRecovery2.testHardLeaseRecovery
      • org.apache.hadoop.hdfs.TestLeaseRecovery2.org.apache.hadoop.hdfs.TestLeaseRecovery2
      1. HDFS-9672.001.patch
        5 kB
        Mingliang Liu
      2. HDFS-9672.000.patch
        4 kB
        Mingliang Liu

        Activity

        Hide
        liuml07 Mingliang Liu added a comment -

        Thanks for the discussion, review and commit, Jitendra Nath Pandey!

        Show
        liuml07 Mingliang Liu added a comment - Thanks for the discussion, review and commit, Jitendra Nath Pandey !
        Hide
        jnp Jitendra Nath Pandey added a comment -

        I have committed this to trunk, branch-2 and branch-2.8. Thanks to Mingliang Liu.

        Show
        jnp Jitendra Nath Pandey added a comment - I have committed this to trunk, branch-2 and branch-2.8. Thanks to Mingliang Liu.
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-trunk-Commit #9183 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9183/)
        HDFS-9672. o.a.h.hdfs.TestLeaseRecovery2 fails intermittently. (jitendra: rev e8650fea1f0837026cbb36ae8bf51c6133259809)

        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestLeaseRecovery2.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #9183 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9183/ ) HDFS-9672 . o.a.h.hdfs.TestLeaseRecovery2 fails intermittently. (jitendra: rev e8650fea1f0837026cbb36ae8bf51c6133259809) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestLeaseRecovery2.java
        Hide
        jnp Jitendra Nath Pandey added a comment -

        +1

        Show
        jnp Jitendra Nath Pandey added a comment - +1
        Hide
        liuml07 Mingliang Liu added a comment -

        Failing tests are not related. Specially, hadoop.hdfs.TestRenameWhileOpen is tracked by HDFS-9689, and hadoop.hdfs.TestDFSClientRetries by HDFS-9690.

        Show
        liuml07 Mingliang Liu added a comment - Failing tests are not related. Specially, hadoop.hdfs.TestRenameWhileOpen is tracked by HDFS-9689 , and hadoop.hdfs.TestDFSClientRetries by HDFS-9690 .
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 0s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        +1 mvninstall 7m 49s trunk passed
        +1 compile 0m 41s trunk passed with JDK v1.8.0_66
        +1 compile 0m 41s trunk passed with JDK v1.7.0_91
        +1 checkstyle 0m 15s trunk passed
        +1 mvnsite 0m 52s trunk passed
        +1 mvneclipse 0m 13s trunk passed
        +1 findbugs 1m 54s trunk passed
        +1 javadoc 1m 5s trunk passed with JDK v1.8.0_66
        +1 javadoc 1m 43s trunk passed with JDK v1.7.0_91
        +1 mvninstall 0m 44s the patch passed
        +1 compile 0m 38s the patch passed with JDK v1.8.0_66
        +1 javac 0m 38s the patch passed
        +1 compile 0m 37s the patch passed with JDK v1.7.0_91
        +1 javac 0m 37s the patch passed
        +1 checkstyle 0m 15s the patch passed
        +1 mvnsite 0m 49s the patch passed
        +1 mvneclipse 0m 11s the patch passed
        +1 whitespace 0m 0s Patch has no whitespace issues.
        +1 findbugs 2m 3s the patch passed
        +1 javadoc 1m 4s the patch passed with JDK v1.8.0_66
        +1 javadoc 1m 42s the patch passed with JDK v1.7.0_91
        -1 unit 52m 37s hadoop-hdfs in the patch failed with JDK v1.8.0_66.
        -1 unit 49m 53s hadoop-hdfs in the patch failed with JDK v1.7.0_91.
        +1 asflicense 0m 21s Patch does not generate ASF License warnings.
        128m 21s



        Reason Tests
        JDK v1.8.0_66 Failed junit tests hadoop.hdfs.TestDFSClientRetries
          hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped
          hadoop.hdfs.TestDFSUpgradeFromImage
        JDK v1.7.0_91 Failed junit tests hadoop.hdfs.TestRenameWhileOpen



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:0ca8df7
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12783934/HDFS-9672.001.patch
        JIRA Issue HDFS-9672
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 65a23b32063b 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 46e5ea8
        Default Java 1.7.0_91
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91
        findbugs v3.0.0
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/14212/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/14212/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt
        unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14212/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14212/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt
        JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14212/testReport/
        modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
        Max memory used 76MB
        Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14212/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 7m 49s trunk passed +1 compile 0m 41s trunk passed with JDK v1.8.0_66 +1 compile 0m 41s trunk passed with JDK v1.7.0_91 +1 checkstyle 0m 15s trunk passed +1 mvnsite 0m 52s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 1m 54s trunk passed +1 javadoc 1m 5s trunk passed with JDK v1.8.0_66 +1 javadoc 1m 43s trunk passed with JDK v1.7.0_91 +1 mvninstall 0m 44s the patch passed +1 compile 0m 38s the patch passed with JDK v1.8.0_66 +1 javac 0m 38s the patch passed +1 compile 0m 37s the patch passed with JDK v1.7.0_91 +1 javac 0m 37s the patch passed +1 checkstyle 0m 15s the patch passed +1 mvnsite 0m 49s the patch passed +1 mvneclipse 0m 11s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 2m 3s the patch passed +1 javadoc 1m 4s the patch passed with JDK v1.8.0_66 +1 javadoc 1m 42s the patch passed with JDK v1.7.0_91 -1 unit 52m 37s hadoop-hdfs in the patch failed with JDK v1.8.0_66. -1 unit 49m 53s hadoop-hdfs in the patch failed with JDK v1.7.0_91. +1 asflicense 0m 21s Patch does not generate ASF License warnings. 128m 21s Reason Tests JDK v1.8.0_66 Failed junit tests hadoop.hdfs.TestDFSClientRetries   hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped   hadoop.hdfs.TestDFSUpgradeFromImage JDK v1.7.0_91 Failed junit tests hadoop.hdfs.TestRenameWhileOpen Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12783934/HDFS-9672.001.patch JIRA Issue HDFS-9672 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 65a23b32063b 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 46e5ea8 Default Java 1.7.0_91 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HDFS-Build/14212/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/14212/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14212/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14212/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14212/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Max memory used 76MB Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14212/console This message was automatically generated.
        Hide
        liuml07 Mingliang Liu added a comment -

        Suppose the LeaseManager$Monitor is doing blocking I/O via: run() -> FSEditLog#logSync() -> EditLogOutputStream#flush() -> JournalSet#flushAndSync() -> JournalSet#mapJournalsAndReportErrors(), if there is an InterruptException, the channel will be closed and the journals will be disabled via disableAndReportErrorOnJournals(). The test main thread then closes the edit log because of MiniDFSCluster#restartNameNode(). During the closing process, it will call FSEditLog#endCurrentLogSegment() which calls FSEditLog#logSync() internally. At the point, the logSync() will fail because all the journals are disabled; the System#exit() will be called immediately. This may be a potential bug which happens when the NN is stopping all active services.

        Normally, the in-progress edit log would be finalized by FSEditLog#endCurrentLogSegment. For testing purposes, we can disable that here.

        Show
        liuml07 Mingliang Liu added a comment - Suppose the LeaseManager$Monitor is doing blocking I/O via: run() -> FSEditLog#logSync() -> EditLogOutputStream#flush() -> JournalSet#flushAndSync() -> JournalSet#mapJournalsAndReportErrors() , if there is an InterruptException, the channel will be closed and the journals will be disabled via disableAndReportErrorOnJournals() . The test main thread then closes the edit log because of MiniDFSCluster#restartNameNode() . During the closing process, it will call FSEditLog#endCurrentLogSegment() which calls FSEditLog#logSync() internally. At the point, the logSync() will fail because all the journals are disabled; the System#exit() will be called immediately. This may be a potential bug which happens when the NN is stopping all active services. Normally, the in-progress edit log would be finalized by FSEditLog#endCurrentLogSegment . For testing purposes, we can disable that here.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 0s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        +1 mvninstall 7m 31s trunk passed
        +1 compile 0m 46s trunk passed with JDK v1.8.0_66
        +1 compile 0m 41s trunk passed with JDK v1.7.0_91
        +1 checkstyle 0m 15s trunk passed
        +1 mvnsite 0m 50s trunk passed
        +1 mvneclipse 0m 12s trunk passed
        +1 findbugs 1m 52s trunk passed
        +1 javadoc 1m 7s trunk passed with JDK v1.8.0_66
        +1 javadoc 1m 52s trunk passed with JDK v1.7.0_91
        +1 mvninstall 0m 47s the patch passed
        +1 compile 0m 44s the patch passed with JDK v1.8.0_66
        +1 javac 0m 44s the patch passed
        +1 compile 0m 39s the patch passed with JDK v1.7.0_91
        +1 javac 0m 39s the patch passed
        +1 checkstyle 0m 15s the patch passed
        +1 mvnsite 0m 48s the patch passed
        +1 mvneclipse 0m 12s the patch passed
        +1 whitespace 0m 0s Patch has no whitespace issues.
        +1 findbugs 2m 2s the patch passed
        +1 javadoc 1m 6s the patch passed with JDK v1.8.0_66
        +1 javadoc 1m 49s the patch passed with JDK v1.7.0_91
        -1 unit 61m 8s hadoop-hdfs in the patch failed with JDK v1.8.0_66.
        -1 unit 55m 26s hadoop-hdfs in the patch failed with JDK v1.7.0_91.
        +1 asflicense 0m 20s Patch does not generate ASF License warnings.
        142m 42s



        Reason Tests
        JDK v1.8.0_66 Failed junit tests hadoop.hdfs.TestReadStripedFileWithDecoding
          hadoop.hdfs.server.datanode.TestFsDatasetCache
          hadoop.hdfs.server.namenode.TestDecommissioningStatus
          hadoop.hdfs.server.balancer.TestBalancer
          hadoop.hdfs.server.namenode.TestNNThroughputBenchmark
          hadoop.hdfs.server.datanode.TestBlockScanner
        JDK v1.7.0_91 Failed junit tests hadoop.hdfs.TestLeaseRecovery2
          hadoop.hdfs.server.namenode.TestNNThroughputBenchmark
          hadoop.hdfs.TestDFSClientRetries



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:0ca8df7
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12783742/HDFS-9672.000.patch
        JIRA Issue HDFS-9672
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 51e6e623beef 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / f3427d3
        Default Java 1.7.0_91
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91
        findbugs v3.0.0
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/14201/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/14201/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt
        unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14201/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14201/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt
        JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14201/testReport/
        modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
        Max memory used 76MB
        Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14201/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 7m 31s trunk passed +1 compile 0m 46s trunk passed with JDK v1.8.0_66 +1 compile 0m 41s trunk passed with JDK v1.7.0_91 +1 checkstyle 0m 15s trunk passed +1 mvnsite 0m 50s trunk passed +1 mvneclipse 0m 12s trunk passed +1 findbugs 1m 52s trunk passed +1 javadoc 1m 7s trunk passed with JDK v1.8.0_66 +1 javadoc 1m 52s trunk passed with JDK v1.7.0_91 +1 mvninstall 0m 47s the patch passed +1 compile 0m 44s the patch passed with JDK v1.8.0_66 +1 javac 0m 44s the patch passed +1 compile 0m 39s the patch passed with JDK v1.7.0_91 +1 javac 0m 39s the patch passed +1 checkstyle 0m 15s the patch passed +1 mvnsite 0m 48s the patch passed +1 mvneclipse 0m 12s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 2m 2s the patch passed +1 javadoc 1m 6s the patch passed with JDK v1.8.0_66 +1 javadoc 1m 49s the patch passed with JDK v1.7.0_91 -1 unit 61m 8s hadoop-hdfs in the patch failed with JDK v1.8.0_66. -1 unit 55m 26s hadoop-hdfs in the patch failed with JDK v1.7.0_91. +1 asflicense 0m 20s Patch does not generate ASF License warnings. 142m 42s Reason Tests JDK v1.8.0_66 Failed junit tests hadoop.hdfs.TestReadStripedFileWithDecoding   hadoop.hdfs.server.datanode.TestFsDatasetCache   hadoop.hdfs.server.namenode.TestDecommissioningStatus   hadoop.hdfs.server.balancer.TestBalancer   hadoop.hdfs.server.namenode.TestNNThroughputBenchmark   hadoop.hdfs.server.datanode.TestBlockScanner JDK v1.7.0_91 Failed junit tests hadoop.hdfs.TestLeaseRecovery2   hadoop.hdfs.server.namenode.TestNNThroughputBenchmark   hadoop.hdfs.TestDFSClientRetries Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12783742/HDFS-9672.000.patch JIRA Issue HDFS-9672 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 51e6e623beef 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / f3427d3 Default Java 1.7.0_91 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HDFS-Build/14201/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/14201/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14201/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14201/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14201/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Max memory used 76MB Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14201/console This message was automatically generated.
        Hide
        liuml07 Mingliang Liu added a comment -

        The mini cluster is reused for all the tests. If any of the test fails to restart the cluster, all the following tests will fail. That's why we see fail message like:
        Tests run: 8, Failures: 1, Errors: 3, Skipped: 0, Time elapsed: 44.253 sec <<< FAILURE! - in org.apache.hadoop.hdfs.TestLeaseRecovery2

        As the mini cluster is changed (e.g. lease period) and restarted differently in the tests, we'd better build a mini cluster for each test.

        Show
        liuml07 Mingliang Liu added a comment - The mini cluster is reused for all the tests. If any of the test fails to restart the cluster, all the following tests will fail. That's why we see fail message like: Tests run: 8, Failures: 1, Errors: 3, Skipped: 0, Time elapsed: 44.253 sec <<< FAILURE! - in org.apache.hadoop.hdfs.TestLeaseRecovery2 As the mini cluster is changed (e.g. lease period) and restarted differently in the tests, we'd better build a mini cluster for each test.
        Hide
        liuml07 Mingliang Liu added a comment -

        The test sets the lease period (which should not happen in real case as the lease expired period is immutable), and triggers the Monitor in LeaseManager for lease recovery (via LeaseManager#checkLeases) and edit log sync (via FSEditLog#logSync()). The tests then sleep for 2 seconds before checking that the lease holder should now be the NN. After this, the test simply restarts the NN, which will close the edit log. See the following code:

        org.apache.hadoop.hdfs.TestLeaseRecovery2.hardLeaseRecoveryRestartHelper.java
        489    // set the hard limit to be 1 second 
        490    cluster.setLeasePeriod(LONG_LEASE_PERIOD, SHORT_LEASE_PERIOD);
        491    
        492    // Make sure lease recovery begins.
        493    Thread.sleep(HdfsServerConstants.NAMENODE_LEASE_RECHECK_INTERVAL * 2);
        494    
        495    checkLease(fileStr, size);
        496   
        497    cluster.restartNameNode(false);
        

        There are two problems here.
        1. If the lease recovery has not begun after main thread's 2s sleep, the checkLease(fileStr, size); in LOC 495 will fail. It happens intermittently.
        2. There is a data race between stopping NN and LeaseManager$Monitor flushing edit logs. If the flush fails, the journal set will be disabled. When NN closes the edit log and calls FSEditLog#logSync(), there will not be enough journals to persistent storage, 'cause all of them were disabled.

        Show
        liuml07 Mingliang Liu added a comment - The test sets the lease period (which should not happen in real case as the lease expired period is immutable), and triggers the Monitor in LeaseManager for lease recovery (via LeaseManager#checkLeases ) and edit log sync (via FSEditLog#logSync() ). The tests then sleep for 2 seconds before checking that the lease holder should now be the NN. After this, the test simply restarts the NN, which will close the edit log. See the following code: org.apache.hadoop.hdfs.TestLeaseRecovery2.hardLeaseRecoveryRestartHelper.java 489 // set the hard limit to be 1 second 490 cluster.setLeasePeriod(LONG_LEASE_PERIOD, SHORT_LEASE_PERIOD); 491 492 // Make sure lease recovery begins. 493 Thread .sleep(HdfsServerConstants.NAMENODE_LEASE_RECHECK_INTERVAL * 2); 494 495 checkLease(fileStr, size); 496 497 cluster.restartNameNode( false ); There are two problems here. 1. If the lease recovery has not begun after main thread's 2s sleep, the checkLease(fileStr, size); in LOC 495 will fail. It happens intermittently. 2. There is a data race between stopping NN and LeaseManager$Monitor flushing edit logs. If the flush fails, the journal set will be disabled. When NN closes the edit log and calls FSEditLog#logSync() , there will not be enough journals to persistent storage, 'cause all of them were disabled.
        Hide
        liuml07 Mingliang Liu added a comment -

        The detailed exception message is like:

        Running org.apache.hadoop.hdfs.TestLeaseRecovery2
        Tests run: 8, Failures: 1, Errors: 4, Skipped: 0, Time elapsed: 43.594 sec <<< FAILURE! - in org.apache.hadoop.hdfs.TestLeaseRecovery2
        testHardLeaseRecoveryAfterNameNodeRestart2(org.apache.hadoop.hdfs.TestLeaseRecovery2)  Time elapsed: 4.076 sec  <<< ERROR!
        org.apache.hadoop.util.ExitUtil$ExitException: org.apache.hadoop.util.ExitUtil$ExitException: Could not sync enough journals to persistent storage due to No journals available to flush. Unsynced transactions: 1
        	at org.apache.hadoop.util.ExitUtil.terminate(ExitUtil.java:126)
        	at org.apache.hadoop.hdfs.server.namenode.FSEditLog.logSync(FSEditLog.java:632)
        	at org.apache.hadoop.hdfs.server.namenode.FSEditLog.endCurrentLogSegment(FSEditLog.java:1265)
        	at org.apache.hadoop.hdfs.server.namenode.FSEditLog.close(FSEditLog.java:363)
        	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.stopActiveServices(FSNamesystem.java:1226)
        	at org.apache.hadoop.hdfs.server.namenode.NameNode$NameNodeHAContext.stopActiveServices(NameNode.java:1879)
        	at org.apache.hadoop.hdfs.server.namenode.ha.ActiveState.exitState(ActiveState.java:70)
        	at org.apache.hadoop.hdfs.server.namenode.NameNode.stop(NameNode.java:1005)
        	at org.apache.hadoop.hdfs.MiniDFSCluster.shutdownNameNode(MiniDFSCluster.java:1797)
        	at org.apache.hadoop.hdfs.MiniDFSCluster.restartNameNode(MiniDFSCluster.java:1848)
        	at org.apache.hadoop.hdfs.MiniDFSCluster.restartNameNode(MiniDFSCluster.java:1828)
        	at org.apache.hadoop.hdfs.TestLeaseRecovery2.hardLeaseRecoveryRestartHelper(TestLeaseRecovery2.java:494)
        	at org.apache.hadoop.hdfs.TestLeaseRecovery2.testHardLeaseRecoveryAfterNameNodeRestart2(TestLeaseRecovery2.java:427)
        	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        	at java.lang.reflect.Method.invoke(Method.java:606)
        	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
        	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
        	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
        	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
        	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
        	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
        	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
        	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
        	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
        	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
        	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
        	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
        	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
        	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
        	at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
        	at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:264)
        	at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:153)
        	at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:124)
        	at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:200)
        	at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:153)
        	at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103)
        
        	at org.apache.hadoop.util.ExitUtil.terminate(ExitUtil.java:126)
        	at org.apache.hadoop.util.ExitUtil.terminate(ExitUtil.java:170)
        	at org.apache.hadoop.hdfs.server.namenode.NameNode.doImmediateShutdown(NameNode.java:1848)
        	at org.apache.hadoop.hdfs.server.namenode.NameNode$NameNodeHAContext.stopActiveServices(NameNode.java:1883)
        	at org.apache.hadoop.hdfs.server.namenode.ha.ActiveState.exitState(ActiveState.java:70)
        	at org.apache.hadoop.hdfs.server.namenode.NameNode.stop(NameNode.java:1005)
        	at org.apache.hadoop.hdfs.MiniDFSCluster.shutdownNameNode(MiniDFSCluster.java:1797)
        	at org.apache.hadoop.hdfs.MiniDFSCluster.restartNameNode(MiniDFSCluster.java:1848)
        	at org.apache.hadoop.hdfs.MiniDFSCluster.restartNameNode(MiniDFSCluster.java:1828)
        	at org.apache.hadoop.hdfs.TestLeaseRecovery2.hardLeaseRecoveryRestartHelper(TestLeaseRecovery2.java:494)
        	at org.apache.hadoop.hdfs.TestLeaseRecovery2.testHardLeaseRecoveryAfterNameNodeRestart2(TestLeaseRecovery2.java:427)
        
        testHardLeaseRecoveryWithRenameAfterNameNodeRestart(org.apache.hadoop.hdfs.TestLeaseRecovery2)  Time elapsed: 0.009 sec  <<< ERROR!
        java.io.EOFException: End of File Exception between local host is: "hwx-mliu.local/10.22.25.160"; destination host is: "localhost":57711; : java.io.EOFException; For more details see:  http://wiki.apache.org/hadoop/EOFException
        	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
        	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        	at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
        	at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:792)
        	at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:765)
        	at org.apache.hadoop.ipc.Client.call(Client.java:1431)
        	at org.apache.hadoop.ipc.Client.call(Client.java:1358)
        	at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)
        	at com.sun.proxy.$Proxy19.create(Unknown Source)
        	at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.create(ClientNamenodeProtocolTranslatorPB.java:296)
        	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        	at java.lang.reflect.Method.invoke(Method.java:606)
        	at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:252)
        	at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:104)
        	at com.sun.proxy.$Proxy21.create(Unknown Source)
        	at org.apache.hadoop.hdfs.DFSOutputStream.newStreamForCreate(DFSOutputStream.java:1720)
        	at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1703)
        	at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1638)
        	at org.apache.hadoop.hdfs.DistributedFileSystem$7.doCall(DistributedFileSystem.java:448)
        	at org.apache.hadoop.hdfs.DistributedFileSystem$7.doCall(DistributedFileSystem.java:444)
        	at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
        	at org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:444)
        	at org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:387)
        	at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:909)
        	at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:890)
        	at org.apache.hadoop.hdfs.TestLeaseRecovery2.hardLeaseRecoveryRestartHelper(TestLeaseRecovery2.java:446)
        	at org.apache.hadoop.hdfs.TestLeaseRecovery2.testHardLeaseRecoveryWithRenameAfterNameNodeRestart(TestLeaseRecovery2.java:433)
        Caused by: java.io.EOFException: null
        	at java.io.DataInputStream.readInt(DataInputStream.java:392)
        	at org.apache.hadoop.ipc.Client$Connection.receiveRpcResponse(Client.java:1082)
        	at org.apache.hadoop.ipc.Client$Connection.run(Client.java:977)
        
        testLeaseRecoverByAnotherUser(org.apache.hadoop.hdfs.TestLeaseRecovery2)  Time elapsed: 0 sec  <<< ERROR!
        java.lang.IllegalStateException: Lease monitor is not running
        	at com.google.common.base.Preconditions.checkState(Preconditions.java:145)
        	at org.apache.hadoop.hdfs.server.namenode.LeaseManager.triggerMonitorCheckNow(LeaseManager.java:563)
        	at org.apache.hadoop.hdfs.server.namenode.NameNodeAdapter.setLeasePeriod(NameNodeAdapter.java:135)
        	at org.apache.hadoop.hdfs.MiniDFSCluster.setLeasePeriod(MiniDFSCluster.java:2453)
        	at org.apache.hadoop.hdfs.TestLeaseRecovery2.testLeaseRecoverByAnotherUser(TestLeaseRecovery2.java:159)
        
        testHardLeaseRecovery(org.apache.hadoop.hdfs.TestLeaseRecovery2)  Time elapsed: 0.005 sec  <<< ERROR!
        java.net.ConnectException: Call From hwx-mliu.local/10.22.25.160 to localhost:57711 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
        	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
        	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:744)
        	at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
        	at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531)
        	at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:495)
        	at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:612)
        	at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:710)
        	at org.apache.hadoop.ipc.Client$Connection.access$2800(Client.java:373)
        	at org.apache.hadoop.ipc.Client.getConnection(Client.java:1493)
        	at org.apache.hadoop.ipc.Client.call(Client.java:1397)
        	at org.apache.hadoop.ipc.Client.call(Client.java:1358)
        	at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)
        	at com.sun.proxy.$Proxy19.create(Unknown Source)
        	at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.create(ClientNamenodeProtocolTranslatorPB.java:296)
        	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        	at java.lang.reflect.Method.invoke(Method.java:606)
        	at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:252)
        	at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:104)
        	at com.sun.proxy.$Proxy21.create(Unknown Source)
        	at org.apache.hadoop.hdfs.DFSOutputStream.newStreamForCreate(DFSOutputStream.java:1720)
        	at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1703)
        	at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1638)
        	at org.apache.hadoop.hdfs.DistributedFileSystem$7.doCall(DistributedFileSystem.java:448)
        	at org.apache.hadoop.hdfs.DistributedFileSystem$7.doCall(DistributedFileSystem.java:444)
        	at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
        	at org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:444)
        	at org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:387)
        	at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:909)
        	at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:890)
        	at org.apache.hadoop.hdfs.TestLeaseRecovery2.testHardLeaseRecovery(TestLeaseRecovery2.java:276)
        
        org.apache.hadoop.hdfs.TestLeaseRecovery2  Time elapsed: 0.016 sec  <<< FAILURE!
        java.lang.AssertionError: Test resulted in an unexpected exit
        	at org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:1748)
        	at org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:1735)
        	at org.apache.hadoop.hdfs.TestLeaseRecovery2.tearDown(TestLeaseRecovery2.java:105)
        
        
        Results :
        
        Failed tests:
          TestLeaseRecovery2.tearDown:105 Test resulted in an unexpected exit
        
        Tests in error:
          TestLeaseRecovery2.testHardLeaseRecoveryAfterNameNodeRestart2:427->hardLeaseRecoveryRestartHelper:494 » Exit
          TestLeaseRecovery2.testHardLeaseRecoveryWithRenameAfterNameNodeRestart:433->hardLeaseRecoveryRestartHelper:446 » EOF
          TestLeaseRecovery2.testLeaseRecoverByAnotherUser:159 » IllegalState Lease moni...
          TestLeaseRecovery2.testHardLeaseRecovery:276 » Connect Call From hwx-mliu.loca...
        
        Tests run: 8, Failures: 1, Errors: 4, Skipped: 0
        
        [ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.17:test (default-test) on project hadoop-hdfs: There are test failures.
        [ERROR]
        [ERROR] Please refer to /Users/mliu/Workspace/hortonworks/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/surefire-reports for the individual test results.
        [ERROR] -> [Help 1]
        [ERROR]
        [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
        [ERROR] Re-run Maven using the -X switch to enable full debug logging.
        [ERROR]
        [ERROR] For more information about the errors and possible solutions, please read the following articles:
        [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
        [ERROR]
        [ERROR] After correcting the problems, you can resume the build with the command
        [ERROR]   mvn <goals> -rf :hadoop-hdfs
        
        Show
        liuml07 Mingliang Liu added a comment - The detailed exception message is like: Running org.apache.hadoop.hdfs.TestLeaseRecovery2 Tests run: 8, Failures: 1, Errors: 4, Skipped: 0, Time elapsed: 43.594 sec <<< FAILURE! - in org.apache.hadoop.hdfs.TestLeaseRecovery2 testHardLeaseRecoveryAfterNameNodeRestart2(org.apache.hadoop.hdfs.TestLeaseRecovery2) Time elapsed: 4.076 sec <<< ERROR! org.apache.hadoop.util.ExitUtil$ExitException: org.apache.hadoop.util.ExitUtil$ExitException: Could not sync enough journals to persistent storage due to No journals available to flush. Unsynced transactions: 1 at org.apache.hadoop.util.ExitUtil.terminate(ExitUtil.java:126) at org.apache.hadoop.hdfs.server.namenode.FSEditLog.logSync(FSEditLog.java:632) at org.apache.hadoop.hdfs.server.namenode.FSEditLog.endCurrentLogSegment(FSEditLog.java:1265) at org.apache.hadoop.hdfs.server.namenode.FSEditLog.close(FSEditLog.java:363) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.stopActiveServices(FSNamesystem.java:1226) at org.apache.hadoop.hdfs.server.namenode.NameNode$NameNodeHAContext.stopActiveServices(NameNode.java:1879) at org.apache.hadoop.hdfs.server.namenode.ha.ActiveState.exitState(ActiveState.java:70) at org.apache.hadoop.hdfs.server.namenode.NameNode.stop(NameNode.java:1005) at org.apache.hadoop.hdfs.MiniDFSCluster.shutdownNameNode(MiniDFSCluster.java:1797) at org.apache.hadoop.hdfs.MiniDFSCluster.restartNameNode(MiniDFSCluster.java:1848) at org.apache.hadoop.hdfs.MiniDFSCluster.restartNameNode(MiniDFSCluster.java:1828) at org.apache.hadoop.hdfs.TestLeaseRecovery2.hardLeaseRecoveryRestartHelper(TestLeaseRecovery2.java:494) at org.apache.hadoop.hdfs.TestLeaseRecovery2.testHardLeaseRecoveryAfterNameNodeRestart2(TestLeaseRecovery2.java:427) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.runners.ParentRunner.run(ParentRunner.java:309) at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:264) at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:153) at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:124) at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:200) at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:153) at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103) at org.apache.hadoop.util.ExitUtil.terminate(ExitUtil.java:126) at org.apache.hadoop.util.ExitUtil.terminate(ExitUtil.java:170) at org.apache.hadoop.hdfs.server.namenode.NameNode.doImmediateShutdown(NameNode.java:1848) at org.apache.hadoop.hdfs.server.namenode.NameNode$NameNodeHAContext.stopActiveServices(NameNode.java:1883) at org.apache.hadoop.hdfs.server.namenode.ha.ActiveState.exitState(ActiveState.java:70) at org.apache.hadoop.hdfs.server.namenode.NameNode.stop(NameNode.java:1005) at org.apache.hadoop.hdfs.MiniDFSCluster.shutdownNameNode(MiniDFSCluster.java:1797) at org.apache.hadoop.hdfs.MiniDFSCluster.restartNameNode(MiniDFSCluster.java:1848) at org.apache.hadoop.hdfs.MiniDFSCluster.restartNameNode(MiniDFSCluster.java:1828) at org.apache.hadoop.hdfs.TestLeaseRecovery2.hardLeaseRecoveryRestartHelper(TestLeaseRecovery2.java:494) at org.apache.hadoop.hdfs.TestLeaseRecovery2.testHardLeaseRecoveryAfterNameNodeRestart2(TestLeaseRecovery2.java:427) testHardLeaseRecoveryWithRenameAfterNameNodeRestart(org.apache.hadoop.hdfs.TestLeaseRecovery2) Time elapsed: 0.009 sec <<< ERROR! java.io.EOFException: End of File Exception between local host is: "hwx-mliu.local/10.22.25.160" ; destination host is: "localhost" :57711; : java.io.EOFException; For more details see: http: //wiki.apache.org/hadoop/EOFException at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:526) at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:792) at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:765) at org.apache.hadoop.ipc.Client.call(Client.java:1431) at org.apache.hadoop.ipc.Client.call(Client.java:1358) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229) at com.sun.proxy.$Proxy19.create(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.create(ClientNamenodeProtocolTranslatorPB.java:296) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:252) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:104) at com.sun.proxy.$Proxy21.create(Unknown Source) at org.apache.hadoop.hdfs.DFSOutputStream.newStreamForCreate(DFSOutputStream.java:1720) at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1703) at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1638) at org.apache.hadoop.hdfs.DistributedFileSystem$7.doCall(DistributedFileSystem.java:448) at org.apache.hadoop.hdfs.DistributedFileSystem$7.doCall(DistributedFileSystem.java:444) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:444) at org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:387) at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:909) at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:890) at org.apache.hadoop.hdfs.TestLeaseRecovery2.hardLeaseRecoveryRestartHelper(TestLeaseRecovery2.java:446) at org.apache.hadoop.hdfs.TestLeaseRecovery2.testHardLeaseRecoveryWithRenameAfterNameNodeRestart(TestLeaseRecovery2.java:433) Caused by: java.io.EOFException: null at java.io.DataInputStream.readInt(DataInputStream.java:392) at org.apache.hadoop.ipc.Client$Connection.receiveRpcResponse(Client.java:1082) at org.apache.hadoop.ipc.Client$Connection.run(Client.java:977) testLeaseRecoverByAnotherUser(org.apache.hadoop.hdfs.TestLeaseRecovery2) Time elapsed: 0 sec <<< ERROR! java.lang.IllegalStateException: Lease monitor is not running at com.google.common.base.Preconditions.checkState(Preconditions.java:145) at org.apache.hadoop.hdfs.server.namenode.LeaseManager.triggerMonitorCheckNow(LeaseManager.java:563) at org.apache.hadoop.hdfs.server.namenode.NameNodeAdapter.setLeasePeriod(NameNodeAdapter.java:135) at org.apache.hadoop.hdfs.MiniDFSCluster.setLeasePeriod(MiniDFSCluster.java:2453) at org.apache.hadoop.hdfs.TestLeaseRecovery2.testLeaseRecoverByAnotherUser(TestLeaseRecovery2.java:159) testHardLeaseRecovery(org.apache.hadoop.hdfs.TestLeaseRecovery2) Time elapsed: 0.005 sec <<< ERROR! java.net.ConnectException: Call From hwx-mliu.local/10.22.25.160 to localhost:57711 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http: //wiki.apache.org/hadoop/ConnectionRefused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:744) at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:495) at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:612) at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:710) at org.apache.hadoop.ipc.Client$Connection.access$2800(Client.java:373) at org.apache.hadoop.ipc.Client.getConnection(Client.java:1493) at org.apache.hadoop.ipc.Client.call(Client.java:1397) at org.apache.hadoop.ipc.Client.call(Client.java:1358) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229) at com.sun.proxy.$Proxy19.create(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.create(ClientNamenodeProtocolTranslatorPB.java:296) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:252) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:104) at com.sun.proxy.$Proxy21.create(Unknown Source) at org.apache.hadoop.hdfs.DFSOutputStream.newStreamForCreate(DFSOutputStream.java:1720) at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1703) at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1638) at org.apache.hadoop.hdfs.DistributedFileSystem$7.doCall(DistributedFileSystem.java:448) at org.apache.hadoop.hdfs.DistributedFileSystem$7.doCall(DistributedFileSystem.java:444) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:444) at org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:387) at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:909) at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:890) at org.apache.hadoop.hdfs.TestLeaseRecovery2.testHardLeaseRecovery(TestLeaseRecovery2.java:276) org.apache.hadoop.hdfs.TestLeaseRecovery2 Time elapsed: 0.016 sec <<< FAILURE! java.lang.AssertionError: Test resulted in an unexpected exit at org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:1748) at org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:1735) at org.apache.hadoop.hdfs.TestLeaseRecovery2.tearDown(TestLeaseRecovery2.java:105) Results : Failed tests: TestLeaseRecovery2.tearDown:105 Test resulted in an unexpected exit Tests in error: TestLeaseRecovery2.testHardLeaseRecoveryAfterNameNodeRestart2:427->hardLeaseRecoveryRestartHelper:494 » Exit TestLeaseRecovery2.testHardLeaseRecoveryWithRenameAfterNameNodeRestart:433->hardLeaseRecoveryRestartHelper:446 » EOF TestLeaseRecovery2.testLeaseRecoverByAnotherUser:159 » IllegalState Lease moni... TestLeaseRecovery2.testHardLeaseRecovery:276 » Connect Call From hwx-mliu.loca... Tests run: 8, Failures: 1, Errors: 4, Skipped: 0 [ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.17:test ( default -test) on project hadoop-hdfs: There are test failures. [ERROR] [ERROR] Please refer to /Users/mliu/Workspace/hortonworks/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/surefire-reports for the individual test results. [ERROR] -> [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch . [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http: //cwiki.apache.org/confluence/display/MAVEN/MojoFailureException [ERROR] [ERROR] After correcting the problems, you can resume the build with the command [ERROR] mvn <goals> -rf :hadoop-hdfs

          People

          • Assignee:
            liuml07 Mingliang Liu
            Reporter:
            liuml07 Mingliang Liu
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development