Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-10426

TestPendingInvalidateBlock failed in trunk

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.8.0, 3.0.0-alpha2
    • Component/s: test
    • Labels:
      None

      Description

      The test TestPendingInvalidateBlock failed sometimes. The stack info:

      org.apache.hadoop.hdfs.server.blockmanagement.TestPendingInvalidateBlock
      testPendingDeletion(org.apache.hadoop.hdfs.server.blockmanagement.TestPendingInvalidateBlock)  Time elapsed: 7.703 sec  <<< FAILURE!
      java.lang.AssertionError: expected:<2> but was:<1>
      	at org.junit.Assert.fail(Assert.java:88)
      	at org.junit.Assert.failNotEquals(Assert.java:743)
      	at org.junit.Assert.assertEquals(Assert.java:118)
      	at org.junit.Assert.assertEquals(Assert.java:555)
      	at org.junit.Assert.assertEquals(Assert.java:542)
      	at org.apache.hadoop.hdfs.server.blockmanagement.TestPendingInvalidateBlock.testPendingDeletion(TestPendingInvalidateBlock.java:92)
      

      It looks that the invalidateBlock has been removed before we do the check

          // restart NN
          cluster.restartNameNode(true);
          dfs.delete(foo, true);
          Assert.assertEquals(0, cluster.getNamesystem().getBlocksTotal());
          Assert.assertEquals(REPLICATION, cluster.getNamesystem()
              .getPendingDeletionBlocks());
          Assert.assertEquals(REPLICATION,
              dfs.getPendingDeletionBlocksCount());
      

      And I look into the related configurations. I found the property dfs.namenode.replication.interval was just set as 1 second in this test. And after the delay time of dfs.namenode.startup.delay.block.deletion.sec and the delete operation was slowly, it will cause this case. We can see the stack info before, the failed test costs 7.7s more than 5+1 second.

      One way can improve this.

      • Increase the time of dfs.namenode.replication.interval
      1. HDFS-10426.001.patch
        2 kB
        Yiqun Lin
      2. HDFS-10426.002.patch
        3 kB
        Yiqun Lin
      3. HDFS-10426.003.patch
        4 kB
        Yiqun Lin
      4. HDFS-10426.004.patch
        5 kB
        Yiqun Lin
      5. HDFS-10426.005.patch
        3 kB
        Yiqun Lin
      6. HDFS-10426.006.patch
        3 kB
        Yiqun Lin

        Issue Links

          Activity

          Hide
          linyiqun Yiqun Lin added a comment -

          Attach a simple patch for this.

          Show
          linyiqun Yiqun Lin added a comment - Attach a simple patch for this.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 14s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 7m 23s trunk passed
          +1 compile 0m 48s trunk passed
          +1 checkstyle 0m 26s trunk passed
          +1 mvnsite 1m 4s trunk passed
          +1 mvneclipse 0m 13s trunk passed
          +1 findbugs 1m 50s trunk passed
          +1 javadoc 1m 15s trunk passed
          +1 mvninstall 0m 54s the patch passed
          +1 compile 0m 50s the patch passed
          +1 javac 0m 50s the patch passed
          +1 checkstyle 0m 28s the patch passed
          +1 mvnsite 0m 49s the patch passed
          +1 mvneclipse 0m 10s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 1m 44s the patch passed
          +1 javadoc 1m 2s the patch passed
          -1 unit 75m 32s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 22s Patch does not generate ASF License warnings.
          96m 14s



          Reason Tests
          Failed junit tests hadoop.hdfs.server.blockmanagement.TestPendingInvalidateBlock
            hadoop.hdfs.TestCrcCorruption



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:2c91fd8
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12804641/HDFS-10426.001.patch
          JIRA Issue HDFS-10426
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 871e0a0fd7f7 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 8a9ecb7
          Default Java 1.8.0_91
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/15482/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/15482/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/15482/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15482/console
          Powered by Apache Yetus 0.2.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 14s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 7m 23s trunk passed +1 compile 0m 48s trunk passed +1 checkstyle 0m 26s trunk passed +1 mvnsite 1m 4s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 1m 50s trunk passed +1 javadoc 1m 15s trunk passed +1 mvninstall 0m 54s the patch passed +1 compile 0m 50s the patch passed +1 javac 0m 50s the patch passed +1 checkstyle 0m 28s the patch passed +1 mvnsite 0m 49s the patch passed +1 mvneclipse 0m 10s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 1m 44s the patch passed +1 javadoc 1m 2s the patch passed -1 unit 75m 32s hadoop-hdfs in the patch failed. +1 asflicense 0m 22s Patch does not generate ASF License warnings. 96m 14s Reason Tests Failed junit tests hadoop.hdfs.server.blockmanagement.TestPendingInvalidateBlock   hadoop.hdfs.TestCrcCorruption Subsystem Report/Notes Docker Image:yetus/hadoop:2c91fd8 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12804641/HDFS-10426.001.patch JIRA Issue HDFS-10426 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 871e0a0fd7f7 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 8a9ecb7 Default Java 1.8.0_91 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HDFS-Build/15482/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/15482/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/15482/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15482/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
          Hide
          iwasakims Masatake Iwasaki added a comment -

          Test failures are relevant to the patch. invalidateBlocks can not be updated before checking if you inreasing replication interval.

          java.lang.AssertionError: expected:<2> but was:<1>
                  ...
          	at org.apache.hadoop.hdfs.server.blockmanagement.TestPendingInvalidateBlock.testPendingDeletion(TestPendingInvalidateBlock.java:92)
          

          This means that pendingDeletionBlocks count had been decreased by processing DN's heartbeat before the checking. How about disabling heartbeat of DataNodes by DataNodeTestUtils#setHeartbeatsDisabledForTests before dfs.delete(foo, true) and enable it again after checking pendingDeletionBlocks?

          Show
          iwasakims Masatake Iwasaki added a comment - Test failures are relevant to the patch. invalidateBlocks can not be updated before checking if you inreasing replication interval. java.lang.AssertionError: expected:<2> but was:<1> ... at org.apache.hadoop.hdfs.server.blockmanagement.TestPendingInvalidateBlock.testPendingDeletion(TestPendingInvalidateBlock.java:92) This means that pendingDeletionBlocks count had been decreased by processing DN's heartbeat before the checking. How about disabling heartbeat of DataNodes by DataNodeTestUtils#setHeartbeatsDisabledForTests before dfs.delete(foo, true) and enable it again after checking pendingDeletionBlocks?
          Hide
          linyiqun Yiqun Lin added a comment -

          Thanks Masatake Iwasaki for the comments. And I have tested the way as you comment. But I found it still not affect the cluster.getNamesystem().getPendingDeletionBlocks(), and the invalidateBlocks will still be removed. The DataNodeTestUtils#setHeartbeatsDisabledForTests is controlling the blocks scheduled to the datanodes, so it will make sense to dfs.getPendingDeletionBlocksCount().

          So I think a better way is to dynamicly adjusting the pendingPeriodInMs time since the NameNode startup. Before checking, we can set a MAX_VALUE and not let the invalidateBlocks to be processed. After that we can reset to the -1.

          Finally, I will post a patch for this and using this way.

          Show
          linyiqun Yiqun Lin added a comment - Thanks Masatake Iwasaki for the comments. And I have tested the way as you comment. But I found it still not affect the cluster.getNamesystem().getPendingDeletionBlocks() , and the invalidateBlocks will still be removed. The DataNodeTestUtils#setHeartbeatsDisabledForTests is controlling the blocks scheduled to the datanodes, so it will make sense to dfs.getPendingDeletionBlocksCount() . So I think a better way is to dynamicly adjusting the pendingPeriodInMs time since the NameNode startup. Before checking, we can set a MAX_VALUE and not let the invalidateBlocks to be processed. After that we can reset to the -1. Finally, I will post a patch for this and using this way.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 18s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 6m 48s trunk passed
          +1 compile 0m 45s trunk passed
          +1 checkstyle 0m 31s trunk passed
          +1 mvnsite 0m 54s trunk passed
          +1 mvneclipse 0m 12s trunk passed
          +1 findbugs 1m 38s trunk passed
          +1 javadoc 1m 10s trunk passed
          +1 mvninstall 0m 48s the patch passed
          +1 compile 0m 44s the patch passed
          +1 javac 0m 44s the patch passed
          +1 checkstyle 0m 26s the patch passed
          +1 mvnsite 0m 47s the patch passed
          +1 mvneclipse 0m 10s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 1m 48s the patch passed
          +1 javadoc 1m 1s the patch passed
          -1 unit 74m 0s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 20s Patch does not generate ASF License warnings.
          93m 37s



          Reason Tests
          Failed junit tests hadoop.hdfs.server.blockmanagement.TestBlockManager



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:2c91fd8
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12805565/HDFS-10426.002.patch
          JIRA Issue HDFS-10426
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 8eccefbbee4a 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 6161d9b
          Default Java 1.8.0_91
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/15522/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/15522/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/15522/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15522/console
          Powered by Apache Yetus 0.2.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 18s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 6m 48s trunk passed +1 compile 0m 45s trunk passed +1 checkstyle 0m 31s trunk passed +1 mvnsite 0m 54s trunk passed +1 mvneclipse 0m 12s trunk passed +1 findbugs 1m 38s trunk passed +1 javadoc 1m 10s trunk passed +1 mvninstall 0m 48s the patch passed +1 compile 0m 44s the patch passed +1 javac 0m 44s the patch passed +1 checkstyle 0m 26s the patch passed +1 mvnsite 0m 47s the patch passed +1 mvneclipse 0m 10s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 1m 48s the patch passed +1 javadoc 1m 1s the patch passed -1 unit 74m 0s hadoop-hdfs in the patch failed. +1 asflicense 0m 20s Patch does not generate ASF License warnings. 93m 37s Reason Tests Failed junit tests hadoop.hdfs.server.blockmanagement.TestBlockManager Subsystem Report/Notes Docker Image:yetus/hadoop:2c91fd8 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12805565/HDFS-10426.002.patch JIRA Issue HDFS-10426 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 8eccefbbee4a 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 6161d9b Default Java 1.8.0_91 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HDFS-Build/15522/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/15522/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/15522/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15522/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
          Hide
          iwasakims Masatake Iwasaki added a comment -

          And I have tested the way as you comment. But I found it still not affect the cluster.getNamesystem().getPendingDeletionBlocks(), and the invalidateBlocks will still be removed.

          Sorry, my assumption was wrong.

              dfs.delete(foo, true);
              Assert.assertEquals(0, cluster.getNamesystem().getBlocksTotal());
              Assert.assertEquals(REPLICATION, cluster.getNamesystem()
                  .getPendingDeletionBlocks());
          

          I could see 2 causes for getting AssertionError: expected:<2> but was:<1> on the above.

          1. ReplicationMonitor kicks in between delete and checking pending deletion blocks count.
          2. delete is called before the NN processes block report from 1 of the DNs.

          Your 002 patch is addressing #1 but I think it would be better to avoid adding code for testing only to BlockManager.

          In order to fix #2, we should wait for replication count of the block reaches to 2 before delete.

          Show
          iwasakims Masatake Iwasaki added a comment - And I have tested the way as you comment. But I found it still not affect the cluster.getNamesystem().getPendingDeletionBlocks(), and the invalidateBlocks will still be removed. Sorry, my assumption was wrong. dfs.delete(foo, true ); Assert.assertEquals(0, cluster.getNamesystem().getBlocksTotal()); Assert.assertEquals(REPLICATION, cluster.getNamesystem() .getPendingDeletionBlocks()); I could see 2 causes for getting AssertionError: expected:<2> but was:<1> on the above. ReplicationMonitor kicks in between delete and checking pending deletion blocks count. delete is called before the NN processes block report from 1 of the DNs. Your 002 patch is addressing #1 but I think it would be better to avoid adding code for testing only to BlockManager. In order to fix #2, we should wait for replication count of the block reaches to 2 before delete .
          Hide
          iwasakims Masatake Iwasaki added a comment -

          I found another intermittent failures in TestPendingInvalidateBlock. It would be nice if those are fixed too here.

          Tests run: 2, Failures: 2, Errors: 0, Skipped: 0, Time elapsed: 31.54 sec <<< FAILURE! - in org.apache.hadoop.hdfs.server.blockmanagement.TestPendingInvalidateBlock
          testPendingDeletion(org.apache.hadoop.hdfs.server.blockmanagement.TestPendingInvalidateBlock)  Time elapsed: 12.213 sec  <<< FAILURE!
          java.lang.AssertionError: expected:<0> but was:<1>
          	at org.junit.Assert.fail(Assert.java:88)
          	at org.junit.Assert.failNotEquals(Assert.java:743)
          	at org.junit.Assert.assertEquals(Assert.java:118)
          	at org.junit.Assert.assertEquals(Assert.java:555)
          	at org.junit.Assert.assertEquals(Assert.java:542)
          	at org.apache.hadoop.hdfs.server.blockmanagement.TestPendingInvalidateBlock.testPendingDeletion(TestPendingInvalidateBlock.java:100)
          
          testPendingDeleteUnknownBlocks(org.apache.hadoop.hdfs.server.blockmanagement.TestPendingInvalidateBlock)  Time elapsed: 18.993 sec  <<< FAILURE!
          java.lang.AssertionError: expected:<0> but was:<2>
          	at org.junit.Assert.fail(Assert.java:88)
          	at org.junit.Assert.failNotEquals(Assert.java:743)
          	at org.junit.Assert.assertEquals(Assert.java:118)
          	at org.junit.Assert.assertEquals(Assert.java:555)
          	at org.junit.Assert.assertEquals(Assert.java:542)
          	at org.apache.hadoop.hdfs.server.blockmanagement.TestPendingInvalidateBlock.testPendingDeleteUnknownBlocks(TestPendingInvalidateBlock.java:177)
          
          Show
          iwasakims Masatake Iwasaki added a comment - I found another intermittent failures in TestPendingInvalidateBlock . It would be nice if those are fixed too here. Tests run: 2, Failures: 2, Errors: 0, Skipped: 0, Time elapsed: 31.54 sec <<< FAILURE! - in org.apache.hadoop.hdfs.server.blockmanagement.TestPendingInvalidateBlock testPendingDeletion(org.apache.hadoop.hdfs.server.blockmanagement.TestPendingInvalidateBlock) Time elapsed: 12.213 sec <<< FAILURE! java.lang.AssertionError: expected:<0> but was:<1> at org.junit.Assert.fail(Assert.java:88) at org.junit.Assert.failNotEquals(Assert.java:743) at org.junit.Assert.assertEquals(Assert.java:118) at org.junit.Assert.assertEquals(Assert.java:555) at org.junit.Assert.assertEquals(Assert.java:542) at org.apache.hadoop.hdfs.server.blockmanagement.TestPendingInvalidateBlock.testPendingDeletion(TestPendingInvalidateBlock.java:100) testPendingDeleteUnknownBlocks(org.apache.hadoop.hdfs.server.blockmanagement.TestPendingInvalidateBlock) Time elapsed: 18.993 sec <<< FAILURE! java.lang.AssertionError: expected:<0> but was:<2> at org.junit.Assert.fail(Assert.java:88) at org.junit.Assert.failNotEquals(Assert.java:743) at org.junit.Assert.assertEquals(Assert.java:118) at org.junit.Assert.assertEquals(Assert.java:555) at org.junit.Assert.assertEquals(Assert.java:542) at org.apache.hadoop.hdfs.server.blockmanagement.TestPendingInvalidateBlock.testPendingDeleteUnknownBlocks(TestPendingInvalidateBlock.java:177)
          Hide
          linyiqun Yiqun Lin added a comment -

          Thanks Masatake Iwasaki for comments.

          In order to fix #2, we should wait for replication count of the block reaches to 2 before delete.

          I think the method DataNodeTestUtils.setHeartbeatsDisabledForTests will make sense for this. We can disable herartbeats for test here.

          ReplicationMonitor kicks in between delete and checking pending deletion blocks count.

          I'd like to add a flag for test in BlockManager#ReplicationMonitor and this will make few change. But I haven't found the way that we just don't add any code to control this. I am glad if you have some good comments for me.

          The other intermittent failures seem that the pengdingDeleteBlocks was not deleted completely when we did the check. We can add some retry chances for that. Attach a new patch.

          Show
          linyiqun Yiqun Lin added a comment - Thanks Masatake Iwasaki for comments. In order to fix #2, we should wait for replication count of the block reaches to 2 before delete. I think the method DataNodeTestUtils.setHeartbeatsDisabledForTests will make sense for this. We can disable herartbeats for test here. ReplicationMonitor kicks in between delete and checking pending deletion blocks count. I'd like to add a flag for test in BlockManager#ReplicationMonitor and this will make few change. But I haven't found the way that we just don't add any code to control this. I am glad if you have some good comments for me. The other intermittent failures seem that the pengdingDeleteBlocks was not deleted completely when we did the check. We can add some retry chances for that. Attach a new patch.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 16s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 6m 0s trunk passed
          +1 compile 0m 43s trunk passed
          +1 checkstyle 0m 26s trunk passed
          +1 mvnsite 0m 50s trunk passed
          +1 mvneclipse 0m 10s trunk passed
          +1 findbugs 1m 38s trunk passed
          +1 javadoc 1m 5s trunk passed
          +1 mvninstall 0m 45s the patch passed
          +1 compile 0m 41s the patch passed
          +1 javac 0m 41s the patch passed
          +1 checkstyle 0m 24s the patch passed
          +1 mvnsite 0m 47s the patch passed
          +1 mvneclipse 0m 8s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 1m 44s the patch passed
          +1 javadoc 1m 3s the patch passed
          -1 unit 88m 14s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 22s The patch does not generate ASF License warnings.
          106m 26s



          Reason Tests
          Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeMultipleRegistrations
            hadoop.hdfs.TestDataTransferKeepalive
            hadoop.hdfs.server.namenode.ha.TestDFSUpgradeWithHA
            hadoop.hdfs.qjournal.TestNNWithQJM
            hadoop.hdfs.TestEncryptionZonesWithKMS
          Timed out junit tests org.apache.hadoop.hdfs.TestLeaseRecovery2



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:2c91fd8
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12806921/HDFS-10426.003.patch
          JIRA Issue HDFS-10426
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 266517ede364 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 4e1f56e
          Default Java 1.8.0_91
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/15603/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/15603/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/15603/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15603/console
          Powered by Apache Yetus 0.3.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 16s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 6m 0s trunk passed +1 compile 0m 43s trunk passed +1 checkstyle 0m 26s trunk passed +1 mvnsite 0m 50s trunk passed +1 mvneclipse 0m 10s trunk passed +1 findbugs 1m 38s trunk passed +1 javadoc 1m 5s trunk passed +1 mvninstall 0m 45s the patch passed +1 compile 0m 41s the patch passed +1 javac 0m 41s the patch passed +1 checkstyle 0m 24s the patch passed +1 mvnsite 0m 47s the patch passed +1 mvneclipse 0m 8s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 44s the patch passed +1 javadoc 1m 3s the patch passed -1 unit 88m 14s hadoop-hdfs in the patch failed. +1 asflicense 0m 22s The patch does not generate ASF License warnings. 106m 26s Reason Tests Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeMultipleRegistrations   hadoop.hdfs.TestDataTransferKeepalive   hadoop.hdfs.server.namenode.ha.TestDFSUpgradeWithHA   hadoop.hdfs.qjournal.TestNNWithQJM   hadoop.hdfs.TestEncryptionZonesWithKMS Timed out junit tests org.apache.hadoop.hdfs.TestLeaseRecovery2 Subsystem Report/Notes Docker Image:yetus/hadoop:2c91fd8 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12806921/HDFS-10426.003.patch JIRA Issue HDFS-10426 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 266517ede364 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 4e1f56e Default Java 1.8.0_91 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HDFS-Build/15603/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/15603/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/15603/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15603/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
          Hide
          linyiqun Yiqun Lin added a comment -

          I make a improvement based on my last patch. I construct a new method for setting the existed variable blocksInvalidateWorkPct, so we can control the blocks deletion in BlockManager. And we can reuse that in the BlockManager test in the future. Hi, Masatake Iwasaki, can do a quick look for this? Thanks.

          Show
          linyiqun Yiqun Lin added a comment - I make a improvement based on my last patch. I construct a new method for setting the existed variable blocksInvalidateWorkPct , so we can control the blocks deletion in BlockManager. And we can reuse that in the BlockManager test in the future. Hi, Masatake Iwasaki , can do a quick look for this? Thanks.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 17s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
          +1 mvninstall 6m 59s trunk passed
          +1 compile 0m 47s trunk passed
          +1 checkstyle 0m 27s trunk passed
          +1 mvnsite 0m 51s trunk passed
          +1 mvneclipse 0m 13s trunk passed
          +1 findbugs 1m 41s trunk passed
          +1 javadoc 0m 55s trunk passed
          +1 mvninstall 0m 49s the patch passed
          +1 compile 0m 42s the patch passed
          +1 javac 0m 42s the patch passed
          -0 checkstyle 0m 25s hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 153 unchanged - 1 fixed = 154 total (was 154)
          +1 mvnsite 0m 52s the patch passed
          +1 mvneclipse 0m 10s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 1m 49s the patch passed
          +1 javadoc 0m 54s the patch passed
          -1 unit 83m 17s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 25s The patch does not generate ASF License warnings.
          102m 51s



          Reason Tests
          Timed out junit tests org.apache.hadoop.hdfs.TestLeaseRecovery2



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Issue HDFS-10426
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12826310/HDFS-10426.004.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux e812097f42f6 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 20ae1fa
          Default Java 1.8.0_101
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/16588/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/16588/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16588/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16588/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 17s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. +1 mvninstall 6m 59s trunk passed +1 compile 0m 47s trunk passed +1 checkstyle 0m 27s trunk passed +1 mvnsite 0m 51s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 1m 41s trunk passed +1 javadoc 0m 55s trunk passed +1 mvninstall 0m 49s the patch passed +1 compile 0m 42s the patch passed +1 javac 0m 42s the patch passed -0 checkstyle 0m 25s hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 153 unchanged - 1 fixed = 154 total (was 154) +1 mvnsite 0m 52s the patch passed +1 mvneclipse 0m 10s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 49s the patch passed +1 javadoc 0m 54s the patch passed -1 unit 83m 17s hadoop-hdfs in the patch failed. +1 asflicense 0m 25s The patch does not generate ASF License warnings. 102m 51s Reason Tests Timed out junit tests org.apache.hadoop.hdfs.TestLeaseRecovery2 Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Issue HDFS-10426 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12826310/HDFS-10426.004.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux e812097f42f6 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 20ae1fa Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/16588/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/16588/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16588/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16588/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          iwasakims Masatake Iwasaki added a comment -

          Sorry for delayed reply, Yiqun Lin.

          I think it would be better to avoid adding code for testing only to BlockManager.

          How about using Mockito as testPendingDeleteUnknownBlocks already do? GenericTestUtils.DelayAnswer could be used to hold method invocation in BlockManager or InvalidateBlocks.

          198	  private void waitForBlocksToDelete() throws Exception {
          199	    for (int tries = 0; tries < 10; tries++) {
          200	      Thread.sleep(6000);
          201	
          202	      for (int i = 0; i < REPLICATION; i++) {
          203	        DataNodeTestUtils.triggerBlockReport(cluster.getDataNodes().get(i));
          204	      }
          205	
          206	      if (cluster.getNamesystem().getPendingDeletionBlocks() == 0) {
          207	        break;
          208	      }
          209	    }
          210	  }
          

          You should use MiniDFSCluster#triggerBlockReports for kicking block reports of all datanodes and GenericTestUtils#waitFor for waiting condition.

          Show
          iwasakims Masatake Iwasaki added a comment - Sorry for delayed reply, Yiqun Lin . I think it would be better to avoid adding code for testing only to BlockManager. How about using Mockito as testPendingDeleteUnknownBlocks already do? GenericTestUtils.DelayAnswer could be used to hold method invocation in BlockManager or InvalidateBlocks. 198 private void waitForBlocksToDelete() throws Exception { 199 for (int tries = 0; tries < 10; tries++) { 200 Thread.sleep(6000); 201 202 for (int i = 0; i < REPLICATION; i++) { 203 DataNodeTestUtils.triggerBlockReport(cluster.getDataNodes().get(i)); 204 } 205 206 if (cluster.getNamesystem().getPendingDeletionBlocks() == 0) { 207 break; 208 } 209 } 210 } You should use MiniDFSCluster#triggerBlockReports for kicking block reports of all datanodes and GenericTestUtils#waitFor for waiting condition.
          Hide
          linyiqun Yiqun Lin added a comment -

          Thanks Masatake Iwasaki for the comments. The comments looks good to me. I think we can use the line Mockito.doReturn(1L).when(mockIb).getInvalidationDelay() for here to delay the block's deletion as that the test testPendingDeleteUnknownBlocks has already did.

          Attach a new patch to address the comments.

          Show
          linyiqun Yiqun Lin added a comment - Thanks Masatake Iwasaki for the comments. The comments looks good to me. I think we can use the line Mockito.doReturn(1L).when(mockIb).getInvalidationDelay() for here to delay the block's deletion as that the test testPendingDeleteUnknownBlocks has already did. Attach a new patch to address the comments.
          Hide
          linyiqun Yiqun Lin added a comment - - edited

          Reupload the same patch to kick jenkins.

          Show
          linyiqun Yiqun Lin added a comment - - edited Reupload the same patch to kick jenkins.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 18s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 8m 36s trunk passed
          +1 compile 1m 0s trunk passed
          +1 checkstyle 0m 31s trunk passed
          +1 mvnsite 1m 18s trunk passed
          +1 mvneclipse 0m 16s trunk passed
          +1 findbugs 2m 8s trunk passed
          +1 javadoc 1m 1s trunk passed
          +1 mvninstall 0m 49s the patch passed
          +1 compile 0m 43s the patch passed
          +1 javac 0m 43s the patch passed
          +1 checkstyle 0m 24s hadoop-hdfs-project/hadoop-hdfs: The patch generated 0 new + 1 unchanged - 1 fixed = 1 total (was 2)
          +1 mvnsite 0m 53s the patch passed
          +1 mvneclipse 0m 12s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 1m 50s the patch passed
          +1 javadoc 1m 1s the patch passed
          -1 unit 88m 2s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 20s The patch does not generate ASF License warnings.
          110m 44s



          Reason Tests
          Timed out junit tests org.apache.hadoop.hdfs.TestLeaseRecovery2



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Issue HDFS-10426
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12827163/HDFS-10426.005.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 6c75e2d2279a 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 62a9667
          Default Java 1.8.0_101
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/16644/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16644/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16644/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 18s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 8m 36s trunk passed +1 compile 1m 0s trunk passed +1 checkstyle 0m 31s trunk passed +1 mvnsite 1m 18s trunk passed +1 mvneclipse 0m 16s trunk passed +1 findbugs 2m 8s trunk passed +1 javadoc 1m 1s trunk passed +1 mvninstall 0m 49s the patch passed +1 compile 0m 43s the patch passed +1 javac 0m 43s the patch passed +1 checkstyle 0m 24s hadoop-hdfs-project/hadoop-hdfs: The patch generated 0 new + 1 unchanged - 1 fixed = 1 total (was 2) +1 mvnsite 0m 53s the patch passed +1 mvneclipse 0m 12s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 50s the patch passed +1 javadoc 1m 1s the patch passed -1 unit 88m 2s hadoop-hdfs in the patch failed. +1 asflicense 0m 20s The patch does not generate ASF License warnings. 110m 44s Reason Tests Timed out junit tests org.apache.hadoop.hdfs.TestLeaseRecovery2 Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Issue HDFS-10426 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12827163/HDFS-10426.005.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 6c75e2d2279a 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 62a9667 Default Java 1.8.0_101 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HDFS-Build/16644/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16644/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16644/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          iwasakims Masatake Iwasaki added a comment -

          Thanks for the update again, Yiqun Lin. The patch almost looks good.

          100	    // for the following operations.
          101	    Thread.sleep(2000);
          102	    dfs.delete(foo, true);
          ...
          109	    Mockito.doReturn(0L).when(mockIb).getInvalidationDelay();
          110	    Thread.sleep(2000);
          

          You do not need these sleeps. If there is synchronization issue between threads, sleeping is not right solution.

          206	  private void waitForBlocksToDelete() throws Exception {
          207	    GenericTestUtils.waitFor(new Supplier<Boolean>() {
          208	
          209	      @Override
          210	      public Boolean get() {
          211	        try {
          212	          cluster.triggerBlockReports();
          213	          Thread.sleep(2000);
          ...
          223	    }, 6000, 60000);
          224	  }
          

          Since GenericTestUtils#waitFor sleeps 6000ms before retry calling get, get should not call sleep by itself. If you think 2000ms is the right interval, you should pass it as 2nd argument of GenericTestUtils#waitFor.

          Show
          iwasakims Masatake Iwasaki added a comment - Thanks for the update again, Yiqun Lin . The patch almost looks good. 100 // for the following operations. 101 Thread.sleep(2000); 102 dfs.delete(foo, true); ... 109 Mockito.doReturn(0L).when(mockIb).getInvalidationDelay(); 110 Thread.sleep(2000); You do not need these sleeps. If there is synchronization issue between threads, sleeping is not right solution. 206 private void waitForBlocksToDelete() throws Exception { 207 GenericTestUtils.waitFor(new Supplier<Boolean>() { 208 209 @Override 210 public Boolean get() { 211 try { 212 cluster.triggerBlockReports(); 213 Thread.sleep(2000); ... 223 }, 6000, 60000); 224 } Since GenericTestUtils#waitFor sleeps 6000ms before retry calling get , get should not call sleep by itself. If you think 2000ms is the right interval, you should pass it as 2nd argument of GenericTestUtils#waitFor .
          Hide
          linyiqun Yiqun Lin added a comment -

          Thanks Masatake Iwasaki for the comments, attach a new patch to address your comments. Hope my latest patch can satisfy with you.

          Show
          linyiqun Yiqun Lin added a comment - Thanks Masatake Iwasaki for the comments, attach a new patch to address your comments. Hope my latest patch can satisfy with you.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 25s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 7m 5s trunk passed
          +1 compile 0m 46s trunk passed
          +1 checkstyle 0m 27s trunk passed
          +1 mvnsite 0m 54s trunk passed
          +1 mvneclipse 0m 12s trunk passed
          +1 findbugs 1m 44s trunk passed
          +1 javadoc 0m 56s trunk passed
          +1 mvninstall 0m 46s the patch passed
          +1 compile 0m 48s the patch passed
          +1 javac 0m 48s the patch passed
          +1 checkstyle 0m 23s hadoop-hdfs-project/hadoop-hdfs: The patch generated 0 new + 1 unchanged - 1 fixed = 1 total (was 2)
          +1 mvnsite 0m 51s the patch passed
          +1 mvneclipse 0m 10s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 1m 53s the patch passed
          +1 javadoc 0m 52s the patch passed
          -1 unit 59m 19s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 19s The patch does not generate ASF License warnings.
          79m 7s



          Reason Tests
          Failed junit tests hadoop.hdfs.TestCrcCorruption



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Issue HDFS-10426
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12829396/HDFS-10426.006.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 0ea7aecdb61b 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / c6d1d74
          Default Java 1.8.0_101
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/16809/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16809/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16809/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 25s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 7m 5s trunk passed +1 compile 0m 46s trunk passed +1 checkstyle 0m 27s trunk passed +1 mvnsite 0m 54s trunk passed +1 mvneclipse 0m 12s trunk passed +1 findbugs 1m 44s trunk passed +1 javadoc 0m 56s trunk passed +1 mvninstall 0m 46s the patch passed +1 compile 0m 48s the patch passed +1 javac 0m 48s the patch passed +1 checkstyle 0m 23s hadoop-hdfs-project/hadoop-hdfs: The patch generated 0 new + 1 unchanged - 1 fixed = 1 total (was 2) +1 mvnsite 0m 51s the patch passed +1 mvneclipse 0m 10s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 53s the patch passed +1 javadoc 0m 52s the patch passed -1 unit 59m 19s hadoop-hdfs in the patch failed. +1 asflicense 0m 19s The patch does not generate ASF License warnings. 79m 7s Reason Tests Failed junit tests hadoop.hdfs.TestCrcCorruption Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Issue HDFS-10426 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12829396/HDFS-10426.006.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 0ea7aecdb61b 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / c6d1d74 Default Java 1.8.0_101 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HDFS-Build/16809/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16809/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16809/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          iwasakims Masatake Iwasaki added a comment -

          +1, checking this in.

          Show
          iwasakims Masatake Iwasaki added a comment - +1, checking this in.
          Hide
          iwasakims Masatake Iwasaki added a comment -

          Committed. Thanks, Yiqun Lin.

          Show
          iwasakims Masatake Iwasaki added a comment - Committed. Thanks, Yiqun Lin .
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10497 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10497/)
          HDFS-10426. TestPendingInvalidateBlock failed in trunk. Contributed by (iwasakims: rev f55eb981dd408ca61d49488f83c99c1fc1f83134)

          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestPendingInvalidateBlock.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10497 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10497/ ) HDFS-10426 . TestPendingInvalidateBlock failed in trunk. Contributed by (iwasakims: rev f55eb981dd408ca61d49488f83c99c1fc1f83134) (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestPendingInvalidateBlock.java
          Show
          liuml07 Mingliang Liu added a comment - Is https://builds.apache.org/job/PreCommit-HDFS-Build/16885/testReport/org.apache.hadoop.hdfs.server.blockmanagement/TestPendingInvalidateBlock/testPendingDeletion/ a related build that still fails with this patch? Thanks.
          Hide
          linyiqun Yiqun Lin added a comment -

          Thanks Mingliang Liu for pointing this. It seem that TestPending InvalidateBlock#testPendingDeletion still failed sometimes(In HDFS-10915, it also appeared). It seems that blockManager still schedules the invalidate blocks even though we have already made the method getInvalidationDelay return 1 indicates that we don't want to delete blocks right now. I'm not sure if there is some race here. Can we delay the deletion operation, and skip the current loop in ReplicationMonitor. In the next loop, I think the mockito method will make sense. Ping Masatake Iwasaki for the comments.

          Show
          linyiqun Yiqun Lin added a comment - Thanks Mingliang Liu for pointing this. It seem that TestPending InvalidateBlock#testPendingDeletion still failed sometimes(In HDFS-10915 , it also appeared). It seems that blockManager still schedules the invalidate blocks even though we have already made the method getInvalidationDelay return 1 indicates that we don't want to delete blocks right now. I'm not sure if there is some race here. Can we delay the deletion operation, and skip the current loop in ReplicationMonitor. In the next loop, I think the mockito method will make sense. Ping Masatake Iwasaki for the comments.
          Hide
          iwasakims Masatake Iwasaki added a comment - - edited

          Thanks for the comments, Mingliang Liu and Yiqun Lin.

          I think the cause is that the delete and assertion was called before replication count reaches to 2. close after writing file can retrurns after at least 1 IBR from datanode reached to namenode. I had mentioned about the scenario but missed to check it before committing.. We can use DFSTestUtil#waitForReplication to wait IBRs. I will upload addendum patch later.

          Show
          iwasakims Masatake Iwasaki added a comment - - edited Thanks for the comments, Mingliang Liu and Yiqun Lin . I think the cause is that the delete and assertion was called before replication count reaches to 2. close after writing file can retrurns after at least 1 IBR from datanode reached to namenode. I had mentioned about the scenario but missed to check it before committing.. We can use DFSTestUtil#waitForReplication to wait IBRs. I will upload addendum patch later.
          Hide
          linyiqun Yiqun Lin added a comment -

          Good analyse! Your comment looks good to me, Masatake Iwasaki. I have filed a new JIRA HDFS-10990 to make a quick fix, I hope you will not mind that. I will very appreciate for that if you can make a quick review. Feel free to post your patch if you think my patch isn't good enough.

          Show
          linyiqun Yiqun Lin added a comment - Good analyse! Your comment looks good to me, Masatake Iwasaki . I have filed a new JIRA HDFS-10990 to make a quick fix, I hope you will not mind that. I will very appreciate for that if you can make a quick review. Feel free to post your patch if you think my patch isn't good enough.
          Hide
          iwasakims Masatake Iwasaki added a comment -

          Thanks, Yiqun Lin.

          Show
          iwasakims Masatake Iwasaki added a comment - Thanks, Yiqun Lin .

            People

            • Assignee:
              linyiqun Yiqun Lin
              Reporter:
              linyiqun Yiqun Lin
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development