Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-10267

Extra "synchronized" on FsDatasetImpl#recoverAppend and FsDatasetImpl#recoverClose

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.8.0
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: datanode
    • Labels:
      None
    • Target Version/s:

      Description

      There is an extra "synchronized" on FsDatasetImpl#recoverAppend and FsDatasetImpl#recoverClose that prevents the HDFS-8496 fix from working as intended. This should be removed.

      1. HDFS-10267.001.patch
        8 kB
        Colin P. McCabe
      2. HDFS-10267.002.patch
        8 kB
        Colin P. McCabe
      3. HDFS-10267.003.patch
        13 kB
        Colin P. McCabe
      4. HDFS-10267.004.patch
        13 kB
        Colin P. McCabe

        Issue Links

          Activity

          Hide
          cmccabe Colin P. McCabe added a comment -

          Remove the extra synchronization, and add unit tests to ensure that it doesn't come back. I verified that the unit tests fail with the extra synchronization.

          Show
          cmccabe Colin P. McCabe added a comment - Remove the extra synchronization, and add unit tests to ensure that it doesn't come back. I verified that the unit tests fail with the extra synchronization.
          Hide
          andrew.wang Andrew Wang added a comment -

          Nit: I see SOPT where I think you meant STOP.

          Show
          andrew.wang Andrew Wang added a comment - Nit: I see SOPT where I think you meant STOP.
          Hide
          cmccabe Colin P. McCabe added a comment -

          v2

          • Correct misspelling of TEST_SPOT_WORKER_XCEIVER_STOP_TIMEOUT_MILLIS
          • Fix bug where testRecoverAppendDoesNotHoldLock was invoking recoverClose rather than recoverAppend
          Show
          cmccabe Colin P. McCabe added a comment - v2 Correct misspelling of TEST_SPOT_WORKER_XCEIVER_STOP_TIMEOUT_MILLIS Fix bug where testRecoverAppendDoesNotHoldLock was invoking recoverClose rather than recoverAppend
          Hide
          eddyxu Lei (Eddy) Xu added a comment -

          One small nit, in rr2Thread.run()

                LOG.debug("initiating recoverClose");
          

          should be LOG.debug("initiating {}", op), so as LOG.debug("finished recoverClose");

          The rest LGTM. +1 pending after addressing the log message.

          Thanks Colin P. McCabe!

          Show
          eddyxu Lei (Eddy) Xu added a comment - One small nit, in rr2Thread.run() LOG.debug( "initiating recoverClose" ); should be LOG.debug("initiating {}", op) , so as LOG.debug("finished recoverClose"); The rest LGTM. +1 pending after addressing the log message. Thanks Colin P. McCabe !
          Hide
          jojochuang Wei-Chiu Chuang added a comment -

          One nitt:
          Instead of

          LOG.debug("initiating recoverClose");
          

          You might change that to

          LOG.debug("initiating " + op);
          
          Show
          jojochuang Wei-Chiu Chuang added a comment - One nitt: Instead of LOG.debug( "initiating recoverClose" ); You might change that to LOG.debug( "initiating " + op);
          Hide
          jojochuang Wei-Chiu Chuang added a comment -

          Thanks for the fix!
          Since the test catches InterruptedException, the timeout

          @Test(timeout=60000)
            public void testRecoverAppendDoesNotHoldLock() throws Exception {
          

          for the test will not work. It would be great if this can be addressed, otherwise the rest looks good to me!

          Show
          jojochuang Wei-Chiu Chuang added a comment - Thanks for the fix! Since the test catches InterruptedException , the timeout @Test(timeout=60000) public void testRecoverAppendDoesNotHoldLock() throws Exception { for the test will not work. It would be great if this can be addressed, otherwise the rest looks good to me!
          Hide
          cmccabe Colin P. McCabe added a comment -

          v3

          • Refactor testStopWorker so that it accepts (essentially) a function pointer as an argument. This avoids the need to have clunky if statements in testStopWorker.
          • There are now three tests: testInitReplicaRecoveryDoesNotHoldLock, testRecoverAppendDoesNotHoldLock, and testRecoverCloseDoesNotHoldLock. Previously we were combining the test of initReplicaRecovery with testing other functions, which was awkward.
          • Check that the stopWriter thread has attempted to stop the writer before proceeding in the main thread. This makes test failures reproducible when the bad synchronization is present. Previously, the test could randomly succeed even when there were bugs, due to a race condition.
          • Add a timeout to our calls to Semaphore#acquire, so that junit's test timeout logic works as intended.
          Show
          cmccabe Colin P. McCabe added a comment - v3 Refactor testStopWorker so that it accepts (essentially) a function pointer as an argument. This avoids the need to have clunky if statements in testStopWorker . There are now three tests: testInitReplicaRecoveryDoesNotHoldLock , testRecoverAppendDoesNotHoldLock , and testRecoverCloseDoesNotHoldLock . Previously we were combining the test of initReplicaRecovery with testing other functions, which was awkward. Check that the stopWriter thread has attempted to stop the writer before proceeding in the main thread. This makes test failures reproducible when the bad synchronization is present. Previously, the test could randomly succeed even when there were bugs, due to a race condition. Add a timeout to our calls to Semaphore#acquire , so that junit's test timeout logic works as intended.
          Hide
          eddyxu Lei (Eddy) Xu added a comment -

          Colin P. McCabe I have a question for the following code

           while (!terminateSlowWriter.gotInterruption.get()) {
              // Wait until stopWriterThread attempts to stop our slow writer by sending
              // it an InterruptedException.
             Thread.sleep(1);
          }
          

          Could you help me to understand where does stopWriterThread interrupt slowWriterThread?

          try {
                    if (sem.tryAcquire(1, remTime, TimeUnit.MILLISECONDS)) {
                      return;
                    }
                  } catch (InterruptedException e) {
                    gotInterruption.set(true);
                  }
          

          Should we break the while loop if the Thread holding this semaphore being interrupted?

          Thanks.

          Show
          eddyxu Lei (Eddy) Xu added a comment - Colin P. McCabe I have a question for the following code while (!terminateSlowWriter.gotInterruption.get()) { // Wait until stopWriterThread attempts to stop our slow writer by sending // it an InterruptedException. Thread .sleep(1); } Could you help me to understand where does stopWriterThread interrupt slowWriterThread ? try { if (sem.tryAcquire(1, remTime, TimeUnit.MILLISECONDS)) { return ; } } catch (InterruptedException e) { gotInterruption.set( true ); } Should we break the while loop if the Thread holding this semaphore being interrupted? Thanks.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 18s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 6m 42s trunk passed
          +1 compile 0m 40s trunk passed with JDK v1.8.0_77
          +1 compile 0m 40s trunk passed with JDK v1.7.0_95
          +1 checkstyle 0m 23s trunk passed
          +1 mvnsite 0m 49s trunk passed
          +1 mvneclipse 0m 14s trunk passed
          +1 findbugs 1m 59s trunk passed
          +1 javadoc 1m 6s trunk passed with JDK v1.8.0_77
          +1 javadoc 1m 45s trunk passed with JDK v1.7.0_95
          +1 mvninstall 0m 46s the patch passed
          +1 compile 0m 36s the patch passed with JDK v1.8.0_77
          +1 javac 0m 36s the patch passed
          +1 compile 0m 38s the patch passed with JDK v1.7.0_95
          +1 javac 0m 38s the patch passed
          +1 checkstyle 0m 20s the patch passed
          +1 mvnsite 0m 48s the patch passed
          +1 mvneclipse 0m 11s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 2m 7s the patch passed
          +1 javadoc 1m 4s the patch passed with JDK v1.8.0_77
          +1 javadoc 1m 42s the patch passed with JDK v1.7.0_95
          -1 unit 62m 10s hadoop-hdfs in the patch failed with JDK v1.8.0_77.
          -1 unit 59m 42s hadoop-hdfs in the patch failed with JDK v1.7.0_95.
          +1 asflicense 0m 23s Patch does not generate ASF License warnings.
          147m 12s



          Reason Tests
          JDK v1.8.0_77 Failed junit tests hadoop.hdfs.server.namenode.TestEditLog
            hadoop.hdfs.TestHFlush
          JDK v1.8.0_77 Timed out junit tests org.apache.hadoop.hdfs.TestLeaseRecovery2
          JDK v1.7.0_95 Failed junit tests hadoop.hdfs.TestFileCreationDelete
            hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:fbe3e86
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12797377/HDFS-10267.002.patch
          JIRA Issue HDFS-10267
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux b78369e63554 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / aede8c1
          Default Java 1.7.0_95
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_77 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/15095/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_77.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/15095/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt
          unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/15095/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_77.txt https://builds.apache.org/job/PreCommit-HDFS-Build/15095/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt
          JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/15095/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15095/console
          Powered by Apache Yetus 0.2.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 18s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 6m 42s trunk passed +1 compile 0m 40s trunk passed with JDK v1.8.0_77 +1 compile 0m 40s trunk passed with JDK v1.7.0_95 +1 checkstyle 0m 23s trunk passed +1 mvnsite 0m 49s trunk passed +1 mvneclipse 0m 14s trunk passed +1 findbugs 1m 59s trunk passed +1 javadoc 1m 6s trunk passed with JDK v1.8.0_77 +1 javadoc 1m 45s trunk passed with JDK v1.7.0_95 +1 mvninstall 0m 46s the patch passed +1 compile 0m 36s the patch passed with JDK v1.8.0_77 +1 javac 0m 36s the patch passed +1 compile 0m 38s the patch passed with JDK v1.7.0_95 +1 javac 0m 38s the patch passed +1 checkstyle 0m 20s the patch passed +1 mvnsite 0m 48s the patch passed +1 mvneclipse 0m 11s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 2m 7s the patch passed +1 javadoc 1m 4s the patch passed with JDK v1.8.0_77 +1 javadoc 1m 42s the patch passed with JDK v1.7.0_95 -1 unit 62m 10s hadoop-hdfs in the patch failed with JDK v1.8.0_77. -1 unit 59m 42s hadoop-hdfs in the patch failed with JDK v1.7.0_95. +1 asflicense 0m 23s Patch does not generate ASF License warnings. 147m 12s Reason Tests JDK v1.8.0_77 Failed junit tests hadoop.hdfs.server.namenode.TestEditLog   hadoop.hdfs.TestHFlush JDK v1.8.0_77 Timed out junit tests org.apache.hadoop.hdfs.TestLeaseRecovery2 JDK v1.7.0_95 Failed junit tests hadoop.hdfs.TestFileCreationDelete   hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped Subsystem Report/Notes Docker Image:yetus/hadoop:fbe3e86 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12797377/HDFS-10267.002.patch JIRA Issue HDFS-10267 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux b78369e63554 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / aede8c1 Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_77 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HDFS-Build/15095/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_77.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/15095/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/15095/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_77.txt https://builds.apache.org/job/PreCommit-HDFS-Build/15095/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/15095/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15095/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
          Hide
          xiaochen Xiao Chen added a comment -

          Thanks Colin P. McCabe for working on this, patch looks great! Also thanks all for reviewing.

          Some nitty comments on Patch 3:

          • javadoc of uninterruptiblyAcquire has redundant parameter:
                 * @param sem         The semaphore.
          • we could use GenericTestUtils#assertExceptionContains to check exception message
          • javadoc on testStopWorker should be updated (replace initReplicaRecovery)
               * Test that initReplicaRecovery does ...
            

          Checkstyle is probably gonna catch these:

          • extra line on line 858-859, before TestStopWorkerRunnable
          • extra space in the end of line 1000:
            LOG.error("stopWriterThread got unexpected exception for " + 

          +1 (non-binding) once the above addressed.

          Show
          xiaochen Xiao Chen added a comment - Thanks Colin P. McCabe for working on this, patch looks great! Also thanks all for reviewing. Some nitty comments on Patch 3: javadoc of uninterruptiblyAcquire has redundant parameter: * @param sem The semaphore. we could use GenericTestUtils#assertExceptionContains to check exception message javadoc on testStopWorker should be updated (replace initReplicaRecovery) * Test that initReplicaRecovery does ... Checkstyle is probably gonna catch these: extra line on line 858-859, before TestStopWorkerRunnable extra space in the end of line 1000: LOG.error( "stopWriterThread got unexpected exception for " + +1 (non-binding) once the above addressed.
          Hide
          cmccabe Colin P. McCabe added a comment -

          Thanks for the reviews, Lei (Eddy) Xu and Xiao Chen.

          Could you help me to understand where does stopWriterThread interrupt slowWriterThread?

          stopWriterThread will call an operation that eventually leads to:

           e.getReplica().stopWriter(datanode.getDnConf().getXceiverStopTimeout());
          

          ReplicaInPipeline#stopWriter interrupts the existing writer thread and then tries to join it.

          Should we break the while loop if the Thread holding this semaphore being interrupted?

          Hmm. No one thread "holds" a semaphore. Are you proposing breaking on INE in uninterruptiblyAcquire? We cannot do that because it would lead to the slowWriter thread terminating immediately when ReplicaInPipeline#stopWriter was called. This defeats the point of the test, which is to ensure that nobody is holding the FsDatasetImpl lock when calling ReplicaInPipeline#stopWriter.

          javadoc of uninterruptiblyAcquire has redundant parameter:

          Fixed

          We could use GenericTestUtils#assertExceptionContains

          Good question. I did not want to use this method, because on failure throws an exception which is different than the exception which it is checking. I want to see the original exception if there is a failure.

          javadoc on testStopWorker should be updated (replace initReplicaRecovery)

          fixed

          removed extra spaces.

          Show
          cmccabe Colin P. McCabe added a comment - Thanks for the reviews, Lei (Eddy) Xu and Xiao Chen . Could you help me to understand where does stopWriterThread interrupt slowWriterThread? stopWriterThread will call an operation that eventually leads to: e.getReplica().stopWriter(datanode.getDnConf().getXceiverStopTimeout()); ReplicaInPipeline#stopWriter interrupts the existing writer thread and then tries to join it. Should we break the while loop if the Thread holding this semaphore being interrupted? Hmm. No one thread "holds" a semaphore. Are you proposing breaking on INE in uninterruptiblyAcquire ? We cannot do that because it would lead to the slowWriter thread terminating immediately when ReplicaInPipeline#stopWriter was called. This defeats the point of the test, which is to ensure that nobody is holding the FsDatasetImpl lock when calling ReplicaInPipeline#stopWriter . javadoc of uninterruptiblyAcquire has redundant parameter: Fixed We could use GenericTestUtils#assertExceptionContains Good question. I did not want to use this method, because on failure throws an exception which is different than the exception which it is checking. I want to see the original exception if there is a failure. javadoc on testStopWorker should be updated (replace initReplicaRecovery) fixed removed extra spaces.
          Hide
          cmccabe Colin P. McCabe added a comment -

          Basically the test, at the high level, is something like this:

          1. create the slowWriterThread thread and make it the Writer for recoveringBlock, by calling FsDatasetImpl#createRbw. Basically FsDatasetImpl grabs the Thread object and stores it in ReplicaInPipeline.
          2. create the stopWriterThread thread and have it call some operation that will call ReplicaInPipe#stopWriter on recovingBlock. This sends an INE (InterruptedException) to SlowWriterThread
          3. slowWriterThread receives the InterruptedException, and sets an AtomicBoolean. But it doesn't exit, meaning that stopWriterThread will hang.
          4. meanwhile, the main thread waits to see the AtomicBoolean set by step #3
          5. the main thread calls some operation on FsDatasetImpl that needs to take the lock. If stopWriterThread failed to drop the lock when calling stopWriter, the test will deadlock here and we will get our timeout. Otherwise, the test succeeds.
          6. main thread tells slowWriterThread to exit, then joins all threads. main thread ensures that no thread exited in a dirty way

          Show
          cmccabe Colin P. McCabe added a comment - Basically the test, at the high level, is something like this: 1. create the slowWriterThread thread and make it the Writer for recoveringBlock , by calling FsDatasetImpl#createRbw . Basically FsDatasetImpl grabs the Thread object and stores it in ReplicaInPipeline . 2. create the stopWriterThread thread and have it call some operation that will call ReplicaInPipe#stopWriter on recovingBlock . This sends an INE (InterruptedException) to SlowWriterThread 3. slowWriterThread receives the InterruptedException , and sets an AtomicBoolean . But it doesn't exit, meaning that stopWriterThread will hang. 4. meanwhile, the main thread waits to see the AtomicBoolean set by step #3 5. the main thread calls some operation on FsDatasetImpl that needs to take the lock. If stopWriterThread failed to drop the lock when calling stopWriter , the test will deadlock here and we will get our timeout. Otherwise, the test succeeds. 6. main thread tells slowWriterThread to exit, then joins all threads. main thread ensures that no thread exited in a dirty way
          Hide
          cmccabe Colin P. McCabe added a comment -

          posted v4 incorporating review comments

          Show
          cmccabe Colin P. McCabe added a comment - posted v4 incorporating review comments
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 16s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 6m 36s trunk passed
          +1 compile 0m 41s trunk passed with JDK v1.8.0_77
          +1 compile 0m 41s trunk passed with JDK v1.7.0_95
          +1 checkstyle 0m 22s trunk passed
          +1 mvnsite 0m 52s trunk passed
          +1 mvneclipse 0m 13s trunk passed
          +1 findbugs 1m 58s trunk passed
          +1 javadoc 1m 8s trunk passed with JDK v1.8.0_77
          +1 javadoc 1m 52s trunk passed with JDK v1.7.0_95
          +1 mvninstall 0m 47s the patch passed
          +1 compile 0m 38s the patch passed with JDK v1.8.0_77
          +1 javac 0m 38s the patch passed
          +1 compile 0m 39s the patch passed with JDK v1.7.0_95
          +1 javac 0m 39s the patch passed
          +1 checkstyle 0m 19s the patch passed
          +1 mvnsite 0m 50s the patch passed
          +1 mvneclipse 0m 12s the patch passed
          -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix.
          +1 findbugs 2m 8s the patch passed
          +1 javadoc 1m 7s the patch passed with JDK v1.8.0_77
          +1 javadoc 1m 48s the patch passed with JDK v1.7.0_95
          -1 unit 70m 35s hadoop-hdfs in the patch failed with JDK v1.8.0_77.
          -1 unit 69m 59s hadoop-hdfs in the patch failed with JDK v1.7.0_95.
          +1 asflicense 0m 21s Patch does not generate ASF License warnings.
          166m 15s



          Reason Tests
          JDK v1.8.0_77 Failed junit tests hadoop.hdfs.server.namenode.TestFileTruncate
            hadoop.hdfs.TestRollingUpgrade
          JDK v1.7.0_95 Failed junit tests hadoop.hdfs.TestHFlush



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:fbe3e86
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12797397/HDFS-10267.003.patch
          JIRA Issue HDFS-10267
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 022b31359c70 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / aede8c1
          Default Java 1.7.0_95
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_77 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
          findbugs v3.0.0
          whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/15096/artifact/patchprocess/whitespace-eol.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/15096/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_77.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/15096/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt
          unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/15096/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_77.txt https://builds.apache.org/job/PreCommit-HDFS-Build/15096/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt
          JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/15096/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15096/console
          Powered by Apache Yetus 0.2.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 16s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 6m 36s trunk passed +1 compile 0m 41s trunk passed with JDK v1.8.0_77 +1 compile 0m 41s trunk passed with JDK v1.7.0_95 +1 checkstyle 0m 22s trunk passed +1 mvnsite 0m 52s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 1m 58s trunk passed +1 javadoc 1m 8s trunk passed with JDK v1.8.0_77 +1 javadoc 1m 52s trunk passed with JDK v1.7.0_95 +1 mvninstall 0m 47s the patch passed +1 compile 0m 38s the patch passed with JDK v1.8.0_77 +1 javac 0m 38s the patch passed +1 compile 0m 39s the patch passed with JDK v1.7.0_95 +1 javac 0m 39s the patch passed +1 checkstyle 0m 19s the patch passed +1 mvnsite 0m 50s the patch passed +1 mvneclipse 0m 12s the patch passed -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 findbugs 2m 8s the patch passed +1 javadoc 1m 7s the patch passed with JDK v1.8.0_77 +1 javadoc 1m 48s the patch passed with JDK v1.7.0_95 -1 unit 70m 35s hadoop-hdfs in the patch failed with JDK v1.8.0_77. -1 unit 69m 59s hadoop-hdfs in the patch failed with JDK v1.7.0_95. +1 asflicense 0m 21s Patch does not generate ASF License warnings. 166m 15s Reason Tests JDK v1.8.0_77 Failed junit tests hadoop.hdfs.server.namenode.TestFileTruncate   hadoop.hdfs.TestRollingUpgrade JDK v1.7.0_95 Failed junit tests hadoop.hdfs.TestHFlush Subsystem Report/Notes Docker Image:yetus/hadoop:fbe3e86 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12797397/HDFS-10267.003.patch JIRA Issue HDFS-10267 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 022b31359c70 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / aede8c1 Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_77 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/15096/artifact/patchprocess/whitespace-eol.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/15096/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_77.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/15096/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/15096/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_77.txt https://builds.apache.org/job/PreCommit-HDFS-Build/15096/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/15096/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15096/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
          Hide
          jojochuang Wei-Chiu Chuang added a comment -

          Thanks, Colin.
          The last patch (v04) looks good to me. The test failures are not related.

          Show
          jojochuang Wei-Chiu Chuang added a comment - Thanks, Colin. The last patch (v04) looks good to me. The test failures are not related.
          Hide
          xiaochen Xiao Chen added a comment -

          Thanks for explaining Colin, that's very helpful. And I agree that current way of checking the exception message is better.
          Patch 4 LGTM.

          Show
          xiaochen Xiao Chen added a comment - Thanks for explaining Colin, that's very helpful. And I agree that current way of checking the exception message is better. Patch 4 LGTM.
          Hide
          eddyxu Lei (Eddy) Xu added a comment -

          +1. Thanks for the explanations, Colin P. McCabe.

          Show
          eddyxu Lei (Eddy) Xu added a comment - +1. Thanks for the explanations, Colin P. McCabe .
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 11m 47s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 6m 49s trunk passed
          +1 compile 0m 38s trunk passed with JDK v1.8.0_77
          +1 compile 0m 40s trunk passed with JDK v1.7.0_95
          +1 checkstyle 0m 21s trunk passed
          +1 mvnsite 0m 50s trunk passed
          +1 mvneclipse 0m 13s trunk passed
          +1 findbugs 1m 59s trunk passed
          +1 javadoc 1m 7s trunk passed with JDK v1.8.0_77
          +1 javadoc 1m 47s trunk passed with JDK v1.7.0_95
          +1 mvninstall 0m 45s the patch passed
          +1 compile 0m 41s the patch passed with JDK v1.8.0_77
          +1 javac 0m 41s the patch passed
          +1 compile 0m 38s the patch passed with JDK v1.7.0_95
          +1 javac 0m 38s the patch passed
          +1 checkstyle 0m 19s the patch passed
          +1 mvnsite 0m 48s the patch passed
          +1 mvneclipse 0m 11s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 2m 8s the patch passed
          +1 javadoc 1m 3s the patch passed with JDK v1.8.0_77
          +1 javadoc 1m 44s the patch passed with JDK v1.7.0_95
          +1 unit 58m 12s hadoop-hdfs in the patch passed with JDK v1.8.0_77.
          -1 unit 54m 45s hadoop-hdfs in the patch failed with JDK v1.7.0_95.
          +1 asflicense 0m 23s Patch does not generate ASF License warnings.
          149m 46s



          Reason Tests
          JDK v1.7.0_95 Failed junit tests hadoop.hdfs.TestHFlush



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:fbe3e86
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12797422/HDFS-10267.004.patch
          JIRA Issue HDFS-10267
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux e994f66696eb 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 3be1ab4
          Default Java 1.7.0_95
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_77 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/15098/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt
          unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/15098/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt
          JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/15098/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15098/console
          Powered by Apache Yetus 0.2.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 11m 47s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 6m 49s trunk passed +1 compile 0m 38s trunk passed with JDK v1.8.0_77 +1 compile 0m 40s trunk passed with JDK v1.7.0_95 +1 checkstyle 0m 21s trunk passed +1 mvnsite 0m 50s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 1m 59s trunk passed +1 javadoc 1m 7s trunk passed with JDK v1.8.0_77 +1 javadoc 1m 47s trunk passed with JDK v1.7.0_95 +1 mvninstall 0m 45s the patch passed +1 compile 0m 41s the patch passed with JDK v1.8.0_77 +1 javac 0m 41s the patch passed +1 compile 0m 38s the patch passed with JDK v1.7.0_95 +1 javac 0m 38s the patch passed +1 checkstyle 0m 19s the patch passed +1 mvnsite 0m 48s the patch passed +1 mvneclipse 0m 11s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 2m 8s the patch passed +1 javadoc 1m 3s the patch passed with JDK v1.8.0_77 +1 javadoc 1m 44s the patch passed with JDK v1.7.0_95 +1 unit 58m 12s hadoop-hdfs in the patch passed with JDK v1.8.0_77. -1 unit 54m 45s hadoop-hdfs in the patch failed with JDK v1.7.0_95. +1 asflicense 0m 23s Patch does not generate ASF License warnings. 149m 46s Reason Tests JDK v1.7.0_95 Failed junit tests hadoop.hdfs.TestHFlush Subsystem Report/Notes Docker Image:yetus/hadoop:fbe3e86 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12797422/HDFS-10267.004.patch JIRA Issue HDFS-10267 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux e994f66696eb 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 3be1ab4 Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_77 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HDFS-Build/15098/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/15098/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/15098/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15098/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
          Hide
          cmccabe Colin P. McCabe added a comment -

          Committed to 2.8, thanks!

          Show
          cmccabe Colin P. McCabe added a comment - Committed to 2.8, thanks!
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #9573 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9573/)
          HDFS-10267. Extra "synchronized" on FsDatasetImpl#recoverAppend and (cmccabe: rev 4bd7cbc29d142fc56324156333b9a8a7d7b68042)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBlockRecovery.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #9573 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9573/ ) HDFS-10267 . Extra "synchronized" on FsDatasetImpl#recoverAppend and (cmccabe: rev 4bd7cbc29d142fc56324156333b9a8a7d7b68042) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBlockRecovery.java

            People

            • Assignee:
              cmccabe Colin P. McCabe
              Reporter:
              cmccabe Colin P. McCabe
            • Votes:
              0 Vote for this issue
              Watchers:
              11 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development