Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-9865

TestBlockReplacement fails intermittently in trunk

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.7.1
    • Fix Version/s: 2.8.0, 2.7.3, 3.0.0-alpha1
    • Component/s: test
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      I found the testcase TestBlockReplacement will be failed sometimes in testing. And I looked the unit log, always I will found these infos:

      org.apache.hadoop.hdfs.server.datanode.TestBlockReplacement
      testDeletedBlockWhenAddBlockIsInEdit(org.apache.hadoop.hdfs.server.datanode.TestBlockReplacement)  Time elapsed: 8.764 sec  <<< FAILURE!
      java.lang.AssertionError: The block should be only on 1 datanode  expected:<1> but was:<2>
      	at org.junit.Assert.fail(Assert.java:88)
      	at org.junit.Assert.failNotEquals(Assert.java:743)
      	at org.junit.Assert.assertEquals(Assert.java:118)
      	at org.junit.Assert.assertEquals(Assert.java:555)
      	at org.apache.hadoop.hdfs.server.datanode.TestBlockReplacement.testDeletedBlockWhenAddBlockIsInEdit(TestBlockReplacement.java:436)
      

      Finally I found the reason is that not deleting block completely in testDeletedBlockWhenAddBlockIsInEdit cause the datanode's num not correct. And the time to wait FsDatasetAsyncDsikService to delete the block is not a accurate value.

      LOG.info("replaceBlock:  " + replaceBlock(block,
                (DatanodeInfo)sourceDnDesc, (DatanodeInfo)sourceDnDesc,
                (DatanodeInfo)destDnDesc));
      // Waiting for the FsDatasetAsyncDsikService to delete the block
      Thread.sleep(3000);
      

      When I adjust this time to 1 seconds, it will be always failed. Also the 3 seconds in test is not a accurate value too. We should adjust these code's logic to a better way such as waiting for the block to be replicated in testDecommision.

        Attachments

        1. HDFS-9865.002.patch
          2 kB
          Yiqun Lin
        2. HDFS-9865.001.patch
          2 kB
          Yiqun Lin

          Activity

            People

            • Assignee:
              linyiqun Yiqun Lin
              Reporter:
              linyiqun Yiqun Lin
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: