[HDFS-9865] TestBlockReplacement fails intermittently in trunk

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.7.1
• Fix Version/s: 2.8.0, 2.7.3, 3.0.0-alpha1
• Component/s: test
• Target Version/s: 2.7.3
• Hadoop Flags: Reviewed
• Assignee: Yiqun Lin
• Reporter: Yiqun Lin
• Created: 26/Feb/16 12:01
• Updated: 30/Aug/16 01:18
• Resolved: 07/Mar/16 17:15

Description

I found the testcase TestBlockReplacement will be failed sometimes in testing. And I looked the unit log, always I will found these infos:

org.apache.hadoop.hdfs.server.datanode.TestBlockReplacement
testDeletedBlockWhenAddBlockIsInEdit(org.apache.hadoop.hdfs.server.datanode.TestBlockReplacement)  Time elapsed: 8.764 sec  <<< FAILURE!
java.lang.AssertionError: The block should be only on 1 datanode  expected:<1> but was:<2>
	at org.junit.Assert.fail(Assert.java:88)
	at org.junit.Assert.failNotEquals(Assert.java:743)
	at org.junit.Assert.assertEquals(Assert.java:118)
	at org.junit.Assert.assertEquals(Assert.java:555)
	at org.apache.hadoop.hdfs.server.datanode.TestBlockReplacement.testDeletedBlockWhenAddBlockIsInEdit(TestBlockReplacement.java:436)

Finally I found the reason is that not deleting block completely in testDeletedBlockWhenAddBlockIsInEdit cause the datanode's num not correct. And the time to wait FsDatasetAsyncDsikService to delete the block is not a accurate value.

LOG.info("replaceBlock:  " + replaceBlock(block,
          (DatanodeInfo)sourceDnDesc, (DatanodeInfo)sourceDnDesc,
          (DatanodeInfo)destDnDesc));
// Waiting for the FsDatasetAsyncDsikService to delete the block
Thread.sleep(3000);

When I adjust this time to 1 seconds, it will be always failed. Also the 3 seconds in test is not a accurate value too. We should adjust these code's logic to a better way such as waiting for the block to be replicated in testDecommision.