Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.8.0, 2.7.3, 3.0.0-alpha1
    • Component/s: test
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      I can't reproduce this failure reliably, but it seems like TestHFlush has been failing intermittently, with the frequency increasing of late.

      Note the following two pre-commit test runs from different JIRAs where TestHFlush seems to have failed spuriously:

      https://builds.apache.org/job/PreCommit-HDFS-Build/734//testReport/
      https://builds.apache.org/job/PreCommit-HDFS-Build/680//testReport/

      1. HDFS.001.patch
        2 kB
        Yiqun Lin
      2. HDFS-2043.002.patch
        5 kB
        Yiqun Lin
      3. HDFS-2043.003.patch
        5 kB
        Yiqun Lin
      4. HDFS-2043.004.patch
        0.9 kB
        Yiqun Lin
      5. HDFS-2043.005.patch
        2 kB
        Yiqun Lin
      6. HDFS-2043.006.patch
        1 kB
        John Zhuge

        Issue Links

          Activity

          Hide
          kihwal Kihwal Lee added a comment -

          This is how it fails nowadays in precommit.

          testHFlushInterrupted(org.apache.hadoop.hdfs.TestHFlush)  Time elapsed: 2.259 sec  <<< ERROR!
          java.nio.channels.ClosedByInterruptException: null
          	at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202)
          	at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:501)
          	at org.apache.hadoop.net.SocketOutputStream$Writer.performIO(SocketOutputStream.java:63)
          	at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:142)
          	at org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:159)
          	at org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:117)
          	at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
          	at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
          	at java.io.DataOutputStream.flush(DataOutputStream.java:123)
          	at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:653)
          
          Show
          kihwal Kihwal Lee added a comment - This is how it fails nowadays in precommit. testHFlushInterrupted(org.apache.hadoop.hdfs.TestHFlush) Time elapsed: 2.259 sec <<< ERROR! java.nio.channels.ClosedByInterruptException: null at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202) at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:501) at org.apache.hadoop.net.SocketOutputStream$Writer.performIO(SocketOutputStream.java:63) at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:142) at org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:159) at org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:117) at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82) at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140) at java.io.DataOutputStream.flush(DataOutputStream.java:123) at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:653)
          Hide
          kihwal Kihwal Lee added a comment -

          This seems to be an actual race in the code.

          Show
          kihwal Kihwal Lee added a comment - This seems to be an actual race in the code.
          Hide
          linyiqun Yiqun Lin added a comment -

          I also found this test failed frequently recently. I created a duplicate jira HDFS-10181 and the detail analysation from me can see this jira, upload a patch from me.

          Show
          linyiqun Yiqun Lin added a comment - I also found this test failed frequently recently. I created a duplicate jira HDFS-10181 and the detail analysation from me can see this jira, upload a patch from me.
          Hide
          linyiqun Yiqun Lin added a comment -

          As Kihwal Lee pointed out, it seems to be an actual race. The two exception logs info in HDFS-10181 also indicated that the stream has been executed other operations when the DataStreamer.run was did. Update a patch to catch the potential exceptions in stream writing operations and add some chances to retry. Assigned this jira to me, pending jenkins.

          Show
          linyiqun Yiqun Lin added a comment - As Kihwal Lee pointed out, it seems to be an actual race. The two exception logs info in HDFS-10181 also indicated that the stream has been executed other operations when the DataStreamer.run was did. Update a patch to catch the potential exceptions in stream writing operations and add some chances to retry. Assigned this jira to me, pending jenkins.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 21s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 12m 16s trunk passed
          +1 compile 1m 11s trunk passed with JDK v1.8.0_74
          +1 compile 0m 54s trunk passed with JDK v1.7.0_95
          +1 checkstyle 0m 24s trunk passed
          +1 mvnsite 1m 0s trunk passed
          +1 mvneclipse 0m 14s trunk passed
          +1 findbugs 2m 7s trunk passed
          +1 javadoc 1m 20s trunk passed with JDK v1.8.0_74
          +1 javadoc 1m 59s trunk passed with JDK v1.7.0_95
          +1 mvninstall 0m 56s the patch passed
          +1 compile 0m 54s the patch passed with JDK v1.8.0_74
          +1 javac 0m 54s the patch passed
          +1 compile 0m 46s the patch passed with JDK v1.7.0_95
          +1 javac 0m 46s the patch passed
          +1 checkstyle 0m 20s the patch passed
          +1 mvnsite 0m 52s the patch passed
          +1 mvneclipse 0m 11s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 2m 19s the patch passed
          +1 javadoc 1m 25s the patch passed with JDK v1.8.0_74
          +1 javadoc 2m 18s the patch passed with JDK v1.7.0_95
          -1 unit 86m 52s hadoop-hdfs in the patch failed with JDK v1.8.0_74.
          -1 unit 77m 46s hadoop-hdfs in the patch failed with JDK v1.7.0_95.
          +1 asflicense 0m 22s Patch does not generate ASF License warnings.
          199m 40s



          Reason Tests
          JDK v1.8.0_74 Failed junit tests hadoop.hdfs.security.TestDelegationTokenForProxyUser
            hadoop.hdfs.server.namenode.TestEditLog
            hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistReplicaRecovery
            hadoop.hdfs.server.datanode.TestDirectoryScanner
          JDK v1.7.0_95 Failed junit tests hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl
            hadoop.metrics2.sink.TestRollingFileSystemSinkWithSecureHdfs
            hadoop.hdfs.server.namenode.TestFileTruncate
            hadoop.hdfs.TestEncryptionZones



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:fbe3e86
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12794501/HDFS-2043.002.patch
          JIRA Issue HDFS-2043
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 55b4a3628bbb 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / ed1e23f
          Default Java 1.7.0_95
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_74 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/14880/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_74.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/14880/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt
          unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14880/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_74.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14880/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt
          JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14880/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14880/console
          Powered by Apache Yetus 0.2.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 21s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 12m 16s trunk passed +1 compile 1m 11s trunk passed with JDK v1.8.0_74 +1 compile 0m 54s trunk passed with JDK v1.7.0_95 +1 checkstyle 0m 24s trunk passed +1 mvnsite 1m 0s trunk passed +1 mvneclipse 0m 14s trunk passed +1 findbugs 2m 7s trunk passed +1 javadoc 1m 20s trunk passed with JDK v1.8.0_74 +1 javadoc 1m 59s trunk passed with JDK v1.7.0_95 +1 mvninstall 0m 56s the patch passed +1 compile 0m 54s the patch passed with JDK v1.8.0_74 +1 javac 0m 54s the patch passed +1 compile 0m 46s the patch passed with JDK v1.7.0_95 +1 javac 0m 46s the patch passed +1 checkstyle 0m 20s the patch passed +1 mvnsite 0m 52s the patch passed +1 mvneclipse 0m 11s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 2m 19s the patch passed +1 javadoc 1m 25s the patch passed with JDK v1.8.0_74 +1 javadoc 2m 18s the patch passed with JDK v1.7.0_95 -1 unit 86m 52s hadoop-hdfs in the patch failed with JDK v1.8.0_74. -1 unit 77m 46s hadoop-hdfs in the patch failed with JDK v1.7.0_95. +1 asflicense 0m 22s Patch does not generate ASF License warnings. 199m 40s Reason Tests JDK v1.8.0_74 Failed junit tests hadoop.hdfs.security.TestDelegationTokenForProxyUser   hadoop.hdfs.server.namenode.TestEditLog   hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistReplicaRecovery   hadoop.hdfs.server.datanode.TestDirectoryScanner JDK v1.7.0_95 Failed junit tests hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl   hadoop.metrics2.sink.TestRollingFileSystemSinkWithSecureHdfs   hadoop.hdfs.server.namenode.TestFileTruncate   hadoop.hdfs.TestEncryptionZones Subsystem Report/Notes Docker Image:yetus/hadoop:fbe3e86 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12794501/HDFS-2043.002.patch JIRA Issue HDFS-2043 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 55b4a3628bbb 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / ed1e23f Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_74 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HDFS-Build/14880/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_74.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/14880/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14880/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_74.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14880/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14880/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14880/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
          Hide
          linyiqun Yiqun Lin added a comment -

          Update the patch and adding the sleep time in each retry chance. Testing TestHFlush again, this patch will looks effective if the result is passed again. Thanks for reviewing the patch.

          Show
          linyiqun Yiqun Lin added a comment - Update the patch and adding the sleep time in each retry chance. Testing TestHFlush again, this patch will looks effective if the result is passed again. Thanks for reviewing the patch.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 11s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 7m 22s trunk passed
          +1 compile 0m 42s trunk passed with JDK v1.8.0_74
          +1 compile 0m 42s trunk passed with JDK v1.7.0_95
          +1 checkstyle 0m 22s trunk passed
          +1 mvnsite 0m 57s trunk passed
          +1 mvneclipse 0m 14s trunk passed
          +1 findbugs 2m 11s trunk passed
          +1 javadoc 1m 6s trunk passed with JDK v1.8.0_74
          +1 javadoc 1m 53s trunk passed with JDK v1.7.0_95
          +1 mvninstall 0m 47s the patch passed
          +1 compile 0m 42s the patch passed with JDK v1.8.0_74
          +1 javac 0m 42s the patch passed
          +1 compile 0m 38s the patch passed with JDK v1.7.0_95
          +1 javac 0m 38s the patch passed
          +1 checkstyle 0m 19s the patch passed
          +1 mvnsite 0m 49s the patch passed
          +1 mvneclipse 0m 11s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 2m 9s the patch passed
          +1 javadoc 1m 6s the patch passed with JDK v1.8.0_74
          +1 javadoc 1m 53s the patch passed with JDK v1.7.0_95
          +1 unit 59m 25s hadoop-hdfs in the patch passed with JDK v1.8.0_74.
          -1 unit 55m 15s hadoop-hdfs in the patch failed with JDK v1.7.0_95.
          +1 asflicense 0m 21s Patch does not generate ASF License warnings.
          141m 20s



          Reason Tests
          JDK v1.7.0_95 Failed junit tests hadoop.hdfs.server.namenode.TestEditLog
            hadoop.hdfs.server.datanode.TestFsDatasetCache



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:fbe3e86
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12794673/HDFS-2043.003.patch
          JIRA Issue HDFS-2043
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 28e1a3c7bfa7 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / e7ed05e
          Default Java 1.7.0_95
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_74 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/14889/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt
          unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14889/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt
          JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14889/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14889/console
          Powered by Apache Yetus 0.2.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 11s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 7m 22s trunk passed +1 compile 0m 42s trunk passed with JDK v1.8.0_74 +1 compile 0m 42s trunk passed with JDK v1.7.0_95 +1 checkstyle 0m 22s trunk passed +1 mvnsite 0m 57s trunk passed +1 mvneclipse 0m 14s trunk passed +1 findbugs 2m 11s trunk passed +1 javadoc 1m 6s trunk passed with JDK v1.8.0_74 +1 javadoc 1m 53s trunk passed with JDK v1.7.0_95 +1 mvninstall 0m 47s the patch passed +1 compile 0m 42s the patch passed with JDK v1.8.0_74 +1 javac 0m 42s the patch passed +1 compile 0m 38s the patch passed with JDK v1.7.0_95 +1 javac 0m 38s the patch passed +1 checkstyle 0m 19s the patch passed +1 mvnsite 0m 49s the patch passed +1 mvneclipse 0m 11s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 2m 9s the patch passed +1 javadoc 1m 6s the patch passed with JDK v1.8.0_74 +1 javadoc 1m 53s the patch passed with JDK v1.7.0_95 +1 unit 59m 25s hadoop-hdfs in the patch passed with JDK v1.8.0_74. -1 unit 55m 15s hadoop-hdfs in the patch failed with JDK v1.7.0_95. +1 asflicense 0m 21s Patch does not generate ASF License warnings. 141m 20s Reason Tests JDK v1.7.0_95 Failed junit tests hadoop.hdfs.server.namenode.TestEditLog   hadoop.hdfs.server.datanode.TestFsDatasetCache Subsystem Report/Notes Docker Image:yetus/hadoop:fbe3e86 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12794673/HDFS-2043.003.patch JIRA Issue HDFS-2043 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 28e1a3c7bfa7 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / e7ed05e Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_74 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HDFS-Build/14889/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14889/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14889/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14889/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
          Hide
          iwasakims Masatake Iwasaki added a comment -

          In both case of IOException and ClosedByInterruptException, I can see the message "Got expected exception during close" in the test logs. The exception was thrown on the second stm.close() in the catch block below.

                try {
                  stm.close();
                  // If we made it past the close(), then that means that the ack made it back
                  // from the pipeline before we got to the wait() call. In that case we should
                  // still have interrupted status.
                  assertTrue(Thread.interrupted());
                } catch (InterruptedIOException ioe) {
                  System.out.println("Got expected exception during close");
                  // If we got the exception, we shouldn't have interrupted status anymore.
                  assertFalse(Thread.currentThread().isInterrupted());
          
                  // Now do a successful close.
                  stm.close();
                }
          

          The catched ioe points to DFSOutputStream#closeImpl. (The stack trace is logged by fixing TestHFlush in my local environment.)

          java.io.InterruptedIOException: Interrupted while waiting for data to be acknowledged by pipeline
                  at org.apache.hadoop.hdfs.DataStreamer.waitForAckedSeqno(DataStreamer.java:771)
                  at org.apache.hadoop.hdfs.DFSOutputStream.flushInternal(DFSOutputStream.java:697)
                  at org.apache.hadoop.hdfs.DFSOutputStream.closeImpl(DFSOutputStream.java:778)
                  at org.apache.hadoop.hdfs.DFSOutputStream.close(DFSOutputStream.java:755)
                  at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:72)
                  at org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:101)
                  at org.apache.hadoop.hdfs.TestHFlush.testHFlushInterrupted(TestHFlush.java:480)
          

          The testHFlushInterrupted expects that the second stm.close() succeeds but it is not true. Underlying streamer thread is closed since closeThreads(true) is called in the finally block of DFSOutputStream#closeImpl.

              } finally {
                // Failures may happen when flushing data.
                // Streamers may keep waiting for the new block information.
                // Thus need to force closing these threads.
                // Don't need to call setClosed() because closeThreads(true)
                // calls setClosed() in the finally block.
                closeThreads(true);
              }
          

          I think we should just catch IOException on the second stm.close() and ignore it. The final check in the test should fail if there is a problem.

                // verify that entire file is good
                AppendTestUtil.checkFullFile(fs, p, 4, fileContents,
                    "Failed to deal with thread interruptions", false);
          
          Show
          iwasakims Masatake Iwasaki added a comment - In both case of IOException and ClosedByInterruptException, I can see the message "Got expected exception during close" in the test logs. The exception was thrown on the second stm.close() in the catch block below. try { stm.close(); // If we made it past the close(), then that means that the ack made it back // from the pipeline before we got to the wait() call. In that case we should // still have interrupted status. assertTrue( Thread .interrupted()); } catch (InterruptedIOException ioe) { System .out.println( "Got expected exception during close" ); // If we got the exception, we shouldn't have interrupted status anymore. assertFalse( Thread .currentThread().isInterrupted()); // Now do a successful close. stm.close(); } The catched ioe points to DFSOutputStream#closeImpl . (The stack trace is logged by fixing TestHFlush in my local environment.) java.io.InterruptedIOException: Interrupted while waiting for data to be acknowledged by pipeline at org.apache.hadoop.hdfs.DataStreamer.waitForAckedSeqno(DataStreamer.java:771) at org.apache.hadoop.hdfs.DFSOutputStream.flushInternal(DFSOutputStream.java:697) at org.apache.hadoop.hdfs.DFSOutputStream.closeImpl(DFSOutputStream.java:778) at org.apache.hadoop.hdfs.DFSOutputStream.close(DFSOutputStream.java:755) at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:72) at org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:101) at org.apache.hadoop.hdfs.TestHFlush.testHFlushInterrupted(TestHFlush.java:480) The testHFlushInterrupted expects that the second stm.close() succeeds but it is not true. Underlying streamer thread is closed since closeThreads(true) is called in the finally block of DFSOutputStream#closeImpl . } finally { // Failures may happen when flushing data. // Streamers may keep waiting for the new block information. // Thus need to force closing these threads. // Don't need to call setClosed() because closeThreads( true ) // calls setClosed() in the finally block. closeThreads( true ); } I think we should just catch IOException on the second stm.close() and ignore it. The final check in the test should fail if there is a problem. // verify that entire file is good AppendTestUtil.checkFullFile(fs, p, 4, fileContents, "Failed to deal with thread interruptions" , false );
          Hide
          linyiqun Yiqun Lin added a comment -

          Thanks Masatake Iwasaki for great analysis!

          The testHFlushInterrupted expects that the second stm.close() succeeds but it is not true. Underlying streamer thread is closed since closeThreads(true) is called in the finally block of DFSOutputStream#closeImpl.

          This operation was added in HDFS-9812, that issue fixed the problem of streamer threads leaking if failure happens when closing DFSOutputStream. And after HDFS-9812 fixed, the second stm.close() will failed more frequently.

          Thanks again for the comment. Post a new patch for this.

          Show
          linyiqun Yiqun Lin added a comment - Thanks Masatake Iwasaki for great analysis! The testHFlushInterrupted expects that the second stm.close() succeeds but it is not true. Underlying streamer thread is closed since closeThreads(true) is called in the finally block of DFSOutputStream#closeImpl. This operation was added in HDFS-9812 , that issue fixed the problem of streamer threads leaking if failure happens when closing DFSOutputStream. And after HDFS-9812 fixed, the second stm.close() will failed more frequently. Thanks again for the comment. Post a new patch for this.
          Hide
          jzhuge John Zhuge added a comment -

          Yiqun Lin and Masatake Iwasaki, great work on patch 004. It fixes my testHFlushInterrupted failure (ClosedByInterruptException) on my branch (latest 2.6.0-based CDH + HDFS-9812).

          Show
          jzhuge John Zhuge added a comment - Yiqun Lin and Masatake Iwasaki , great work on patch 004. It fixes my testHFlushInterrupted failure (ClosedByInterruptException) on my branch (latest 2.6.0-based CDH + HDFS-9812 ).
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 1m 1s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 9m 15s trunk passed
          +1 compile 1m 12s trunk passed with JDK v1.8.0_91
          +1 compile 0m 56s trunk passed with JDK v1.7.0_95
          +1 checkstyle 0m 24s trunk passed
          +1 mvnsite 1m 3s trunk passed
          +1 mvneclipse 0m 15s trunk passed
          +1 findbugs 2m 20s trunk passed
          +1 javadoc 1m 34s trunk passed with JDK v1.8.0_91
          +1 javadoc 2m 13s trunk passed with JDK v1.7.0_95
          +1 mvninstall 1m 1s the patch passed
          +1 compile 1m 1s the patch passed with JDK v1.8.0_91
          +1 javac 1m 1s the patch passed
          +1 compile 0m 50s the patch passed with JDK v1.7.0_95
          +1 javac 0m 50s the patch passed
          +1 checkstyle 0m 23s the patch passed
          +1 mvnsite 1m 5s the patch passed
          +1 mvneclipse 0m 14s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 2m 38s the patch passed
          +1 javadoc 1m 29s the patch passed with JDK v1.8.0_91
          +1 javadoc 2m 16s the patch passed with JDK v1.7.0_95
          -1 unit 98m 0s hadoop-hdfs in the patch failed with JDK v1.8.0_91.
          -1 unit 96m 20s hadoop-hdfs in the patch failed with JDK v1.7.0_95.
          +1 asflicense 0m 24s Patch does not generate ASF License warnings.
          228m 39s



          Reason Tests
          JDK v1.8.0_91 Failed junit tests hadoop.hdfs.security.TestDelegationTokenForProxyUser
            hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure
            hadoop.metrics2.sink.TestRollingFileSystemSinkWithHdfs
            hadoop.hdfs.server.datanode.TestDirectoryScanner
            hadoop.hdfs.TestAsyncDFSRename
          JDK v1.7.0_95 Failed junit tests hadoop.hdfs.server.namenode.ha.TestEditLogTailer
            hadoop.hdfs.shortcircuit.TestShortCircuitCache
            hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl
            hadoop.hdfs.server.namenode.TestEditLog
            hadoop.hdfs.server.balancer.TestBalancerWithMultipleNameNodes
            hadoop.hdfs.server.namenode.ha.TestHAAppend
            hadoop.hdfs.server.namenode.TestFileTruncate
            hadoop.hdfs.server.namenode.snapshot.TestOpenFilesWithSnapshot
            hadoop.hdfs.TestAsyncDFSRename



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:cf2ee45
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12802378/HDFS-2043.004.patch
          JIRA Issue HDFS-2043
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux bb2f89dd9657 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 72b0477
          Default Java 1.7.0_95
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_91 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/15369/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_91.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/15369/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt
          unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/15369/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_91.txt https://builds.apache.org/job/PreCommit-HDFS-Build/15369/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt
          JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/15369/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15369/console
          Powered by Apache Yetus 0.2.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 1m 1s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 9m 15s trunk passed +1 compile 1m 12s trunk passed with JDK v1.8.0_91 +1 compile 0m 56s trunk passed with JDK v1.7.0_95 +1 checkstyle 0m 24s trunk passed +1 mvnsite 1m 3s trunk passed +1 mvneclipse 0m 15s trunk passed +1 findbugs 2m 20s trunk passed +1 javadoc 1m 34s trunk passed with JDK v1.8.0_91 +1 javadoc 2m 13s trunk passed with JDK v1.7.0_95 +1 mvninstall 1m 1s the patch passed +1 compile 1m 1s the patch passed with JDK v1.8.0_91 +1 javac 1m 1s the patch passed +1 compile 0m 50s the patch passed with JDK v1.7.0_95 +1 javac 0m 50s the patch passed +1 checkstyle 0m 23s the patch passed +1 mvnsite 1m 5s the patch passed +1 mvneclipse 0m 14s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 2m 38s the patch passed +1 javadoc 1m 29s the patch passed with JDK v1.8.0_91 +1 javadoc 2m 16s the patch passed with JDK v1.7.0_95 -1 unit 98m 0s hadoop-hdfs in the patch failed with JDK v1.8.0_91. -1 unit 96m 20s hadoop-hdfs in the patch failed with JDK v1.7.0_95. +1 asflicense 0m 24s Patch does not generate ASF License warnings. 228m 39s Reason Tests JDK v1.8.0_91 Failed junit tests hadoop.hdfs.security.TestDelegationTokenForProxyUser   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure   hadoop.metrics2.sink.TestRollingFileSystemSinkWithHdfs   hadoop.hdfs.server.datanode.TestDirectoryScanner   hadoop.hdfs.TestAsyncDFSRename JDK v1.7.0_95 Failed junit tests hadoop.hdfs.server.namenode.ha.TestEditLogTailer   hadoop.hdfs.shortcircuit.TestShortCircuitCache   hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl   hadoop.hdfs.server.namenode.TestEditLog   hadoop.hdfs.server.balancer.TestBalancerWithMultipleNameNodes   hadoop.hdfs.server.namenode.ha.TestHAAppend   hadoop.hdfs.server.namenode.TestFileTruncate   hadoop.hdfs.server.namenode.snapshot.TestOpenFilesWithSnapshot   hadoop.hdfs.TestAsyncDFSRename Subsystem Report/Notes Docker Image:yetus/hadoop:cf2ee45 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12802378/HDFS-2043.004.patch JIRA Issue HDFS-2043 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux bb2f89dd9657 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 72b0477 Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_91 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HDFS-Build/15369/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_91.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/15369/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/15369/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_91.txt https://builds.apache.org/job/PreCommit-HDFS-Build/15369/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/15369/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15369/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
          Hide
          jzhuge John Zhuge added a comment -

          Yiqun Lin, could you please add a INFO log message with stack trace at line 494 when ignoring the exception? Otherwise LGTM.

          Show
          jzhuge John Zhuge added a comment - Yiqun Lin , could you please add a INFO log message with stack trace at line 494 when ignoring the exception? Otherwise LGTM.
          Hide
          linyiqun Yiqun Lin added a comment -

          Post the new patch for addressing the comment.

          Show
          linyiqun Yiqun Lin added a comment - Post the new patch for addressing the comment.
          Hide
          jzhuge John Zhuge added a comment -

          My bad Yiqun Lin, this file uses System.out instead of log. I uploaded a simple 006.patch to use System.out.

          +1 LGTM.

          Show
          jzhuge John Zhuge added a comment - My bad Yiqun Lin , this file uses System.out instead of log. I uploaded a simple 006.patch to use System.out. +1 LGTM.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 48s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 7m 17s trunk passed
          +1 compile 0m 56s trunk passed with JDK v1.8.0_91
          +1 compile 0m 44s trunk passed with JDK v1.7.0_95
          +1 checkstyle 0m 19s trunk passed
          +1 mvnsite 1m 0s trunk passed
          +1 mvneclipse 0m 15s trunk passed
          +1 findbugs 2m 10s trunk passed
          +1 javadoc 1m 22s trunk passed with JDK v1.8.0_91
          +1 javadoc 2m 2s trunk passed with JDK v1.7.0_95
          +1 mvninstall 0m 51s the patch passed
          +1 compile 0m 52s the patch passed with JDK v1.8.0_91
          +1 javac 0m 52s the patch passed
          +1 compile 0m 45s the patch passed with JDK v1.7.0_95
          +1 javac 0m 45s the patch passed
          +1 checkstyle 0m 19s the patch passed
          +1 mvnsite 0m 53s the patch passed
          +1 mvneclipse 0m 12s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 2m 26s the patch passed
          +1 javadoc 1m 26s the patch passed with JDK v1.8.0_91
          +1 javadoc 2m 8s the patch passed with JDK v1.7.0_95
          -1 unit 92m 13s hadoop-hdfs in the patch failed with JDK v1.8.0_91.
          -1 unit 78m 47s hadoop-hdfs in the patch failed with JDK v1.7.0_95.
          +1 asflicense 0m 22s Patch does not generate ASF License warnings.
          200m 42s



          Reason Tests
          JDK v1.8.0_91 Failed junit tests hadoop.hdfs.server.namenode.ha.TestEditLogTailer
            hadoop.hdfs.shortcircuit.TestShortCircuitCache
            hadoop.hdfs.security.TestDelegationTokenForProxyUser
            hadoop.hdfs.TestFileAppend
            hadoop.hdfs.server.namenode.TestNameNodeMXBean
            hadoop.hdfs.TestRollingUpgrade
            hadoop.hdfs.server.datanode.TestDirectoryScanner
            hadoop.hdfs.TestAsyncDFSRename
          JDK v1.7.0_95 Failed junit tests hadoop.hdfs.shortcircuit.TestShortCircuitCache
            hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl
            hadoop.metrics2.sink.TestRollingFileSystemSinkWithSecureHdfs
            hadoop.hdfs.TestAsyncDFSRename



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:cf2ee45
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12802580/HDFS-2043.005.patch
          JIRA Issue HDFS-2043
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 218a5aa1b535 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 8d48266
          Default Java 1.7.0_95
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_91 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/15378/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_91.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/15378/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt
          unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/15378/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_91.txt https://builds.apache.org/job/PreCommit-HDFS-Build/15378/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt
          JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/15378/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15378/console
          Powered by Apache Yetus 0.2.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 48s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 7m 17s trunk passed +1 compile 0m 56s trunk passed with JDK v1.8.0_91 +1 compile 0m 44s trunk passed with JDK v1.7.0_95 +1 checkstyle 0m 19s trunk passed +1 mvnsite 1m 0s trunk passed +1 mvneclipse 0m 15s trunk passed +1 findbugs 2m 10s trunk passed +1 javadoc 1m 22s trunk passed with JDK v1.8.0_91 +1 javadoc 2m 2s trunk passed with JDK v1.7.0_95 +1 mvninstall 0m 51s the patch passed +1 compile 0m 52s the patch passed with JDK v1.8.0_91 +1 javac 0m 52s the patch passed +1 compile 0m 45s the patch passed with JDK v1.7.0_95 +1 javac 0m 45s the patch passed +1 checkstyle 0m 19s the patch passed +1 mvnsite 0m 53s the patch passed +1 mvneclipse 0m 12s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 2m 26s the patch passed +1 javadoc 1m 26s the patch passed with JDK v1.8.0_91 +1 javadoc 2m 8s the patch passed with JDK v1.7.0_95 -1 unit 92m 13s hadoop-hdfs in the patch failed with JDK v1.8.0_91. -1 unit 78m 47s hadoop-hdfs in the patch failed with JDK v1.7.0_95. +1 asflicense 0m 22s Patch does not generate ASF License warnings. 200m 42s Reason Tests JDK v1.8.0_91 Failed junit tests hadoop.hdfs.server.namenode.ha.TestEditLogTailer   hadoop.hdfs.shortcircuit.TestShortCircuitCache   hadoop.hdfs.security.TestDelegationTokenForProxyUser   hadoop.hdfs.TestFileAppend   hadoop.hdfs.server.namenode.TestNameNodeMXBean   hadoop.hdfs.TestRollingUpgrade   hadoop.hdfs.server.datanode.TestDirectoryScanner   hadoop.hdfs.TestAsyncDFSRename JDK v1.7.0_95 Failed junit tests hadoop.hdfs.shortcircuit.TestShortCircuitCache   hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl   hadoop.metrics2.sink.TestRollingFileSystemSinkWithSecureHdfs   hadoop.hdfs.TestAsyncDFSRename Subsystem Report/Notes Docker Image:yetus/hadoop:cf2ee45 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12802580/HDFS-2043.005.patch JIRA Issue HDFS-2043 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 218a5aa1b535 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 8d48266 Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_91 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HDFS-Build/15378/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_91.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/15378/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/15378/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_91.txt https://builds.apache.org/job/PreCommit-HDFS-Build/15378/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/15378/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15378/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 12s Docker mode activated.
          +1 @author 0m 1s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 6m 50s trunk passed
          +1 compile 0m 40s trunk passed with JDK v1.8.0_91
          +1 compile 0m 42s trunk passed with JDK v1.7.0_95
          +1 checkstyle 0m 22s trunk passed
          +1 mvnsite 0m 53s trunk passed
          +1 mvneclipse 0m 13s trunk passed
          +1 findbugs 2m 0s trunk passed
          +1 javadoc 1m 6s trunk passed with JDK v1.8.0_91
          +1 javadoc 1m 47s trunk passed with JDK v1.7.0_95
          +1 mvninstall 0m 47s the patch passed
          +1 compile 0m 39s the patch passed with JDK v1.8.0_91
          +1 javac 0m 39s the patch passed
          +1 compile 0m 39s the patch passed with JDK v1.7.0_95
          +1 javac 0m 39s the patch passed
          +1 checkstyle 0m 18s the patch passed
          +1 mvnsite 0m 50s the patch passed
          +1 mvneclipse 0m 12s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 2m 10s the patch passed
          +1 javadoc 1m 1s the patch passed with JDK v1.8.0_91
          +1 javadoc 1m 47s the patch passed with JDK v1.7.0_95
          -1 unit 59m 52s hadoop-hdfs in the patch failed with JDK v1.8.0_91.
          +1 unit 54m 17s hadoop-hdfs in the patch passed with JDK v1.7.0_95.
          +1 asflicense 0m 21s Patch does not generate ASF License warnings.
          139m 42s



          Reason Tests
          JDK v1.8.0_91 Failed junit tests hadoop.hdfs.TestFileAppend
            hadoop.hdfs.server.namenode.ha.TestDFSUpgradeWithHA
            hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:cf2ee45
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12802594/HDFS-2043.006.patch
          JIRA Issue HDFS-2043
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 671912e69434 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 8d48266
          Default Java 1.7.0_95
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_91 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/15381/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_91.txt
          unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/15381/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_91.txt
          JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/15381/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15381/console
          Powered by Apache Yetus 0.2.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 12s Docker mode activated. +1 @author 0m 1s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 6m 50s trunk passed +1 compile 0m 40s trunk passed with JDK v1.8.0_91 +1 compile 0m 42s trunk passed with JDK v1.7.0_95 +1 checkstyle 0m 22s trunk passed +1 mvnsite 0m 53s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 2m 0s trunk passed +1 javadoc 1m 6s trunk passed with JDK v1.8.0_91 +1 javadoc 1m 47s trunk passed with JDK v1.7.0_95 +1 mvninstall 0m 47s the patch passed +1 compile 0m 39s the patch passed with JDK v1.8.0_91 +1 javac 0m 39s the patch passed +1 compile 0m 39s the patch passed with JDK v1.7.0_95 +1 javac 0m 39s the patch passed +1 checkstyle 0m 18s the patch passed +1 mvnsite 0m 50s the patch passed +1 mvneclipse 0m 12s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 2m 10s the patch passed +1 javadoc 1m 1s the patch passed with JDK v1.8.0_91 +1 javadoc 1m 47s the patch passed with JDK v1.7.0_95 -1 unit 59m 52s hadoop-hdfs in the patch failed with JDK v1.8.0_91. +1 unit 54m 17s hadoop-hdfs in the patch passed with JDK v1.7.0_95. +1 asflicense 0m 21s Patch does not generate ASF License warnings. 139m 42s Reason Tests JDK v1.8.0_91 Failed junit tests hadoop.hdfs.TestFileAppend   hadoop.hdfs.server.namenode.ha.TestDFSUpgradeWithHA   hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl Subsystem Report/Notes Docker Image:yetus/hadoop:cf2ee45 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12802594/HDFS-2043.006.patch JIRA Issue HDFS-2043 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 671912e69434 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 8d48266 Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_91 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HDFS-Build/15381/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_91.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/15381/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_91.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/15381/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15381/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
          Hide
          iwasakims Masatake Iwasaki added a comment -

          +1. I will commit this shortly.

          Show
          iwasakims Masatake Iwasaki added a comment - +1. I will commit this shortly.
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #9729 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9729/)
          HDFS-2043. TestHFlush failing intermittently. Contributed by Lin Yiqun. (iwasakims: rev 2835f146b551f911ea13307e7762759813898757)

          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestHFlush.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #9729 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9729/ ) HDFS-2043 . TestHFlush failing intermittently. Contributed by Lin Yiqun. (iwasakims: rev 2835f146b551f911ea13307e7762759813898757) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestHFlush.java
          Hide
          iwasakims Masatake Iwasaki added a comment -

          Committed. Thanks, Yiqun Lin and John Zhuge.

          Show
          iwasakims Masatake Iwasaki added a comment - Committed. Thanks, Yiqun Lin and John Zhuge .
          Hide
          linyiqun Yiqun Lin added a comment -

          Thanks Masatake Iwasaki for commit!

          Show
          linyiqun Yiqun Lin added a comment - Thanks Masatake Iwasaki for commit!
          Hide
          vinodkv Vinod Kumar Vavilapalli added a comment -

          Closing the JIRA as part of 2.7.3 release.

          Show
          vinodkv Vinod Kumar Vavilapalli added a comment - Closing the JIRA as part of 2.7.3 release.

            People

            • Assignee:
              linyiqun Yiqun Lin
              Reporter:
              atm Aaron T. Myers
            • Votes:
              0 Vote for this issue
              Watchers:
              15 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development