[HDFS-15144] TestBlockStatsMXBean#testStorageTypeStatsWhenStorageFailed is incorrect

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Invalid
• Component/s: datanode
• Assignee: Ahmed Hussein
• Reporter: Ahmed Hussein
• Created: 24/Jan/20 15:55
• Updated: 27/Jan/20 23:20
• Resolved: 27/Jan/20 19:43

Description

TestBlockStatsMXBean#testStorageTypeStatsWhenStorageFailed loops three times to restart Datanodes. However, the code restart the DN-0 three times.
As a result, the JUnit does not really execute the scenario it was supposed to.

    DataNodeTestUtils.restoreDataDirFromFailure(dn1ArcVol1);
    DataNodeTestUtils.restoreDataDirFromFailure(dn2ArcVol1);
    DataNodeTestUtils.restoreDataDirFromFailure(dn3ArcVol1);
    for (int i = 0; i < 3; i++) {
      cluster.restartDataNode(0, true);
    }
    // wait for heartbeat
    Thread.sleep(6000);
    storageTypeStatsMap = cluster.getNamesystem().getBlockManager()
        .getStorageTypeStats();
    storageTypeStats = storageTypeStatsMap.get(StorageType.RAM_DISK);
    assertEquals(6, storageTypeStats.getNodesInService());

When I changed the loop inner block to cluster.restartDataNode(i, true), the test did not pass with the stack trace below. I suspect that one of the datanodes does not start properly after calling restart.

[INFO] Running org.apache.hadoop.hdfs.server.blockmanagement.TestBlockStatsMXBean
[ERROR] Tests run: 3, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 28.805 s <<< FAILURE! - in org.apache.hadoop.hdfs.server.blockmanagement.TestBlockStatsMXBean
[ERROR] testStorageTypeStatsWhenStorageFailed(org.apache.hadoop.hdfs.server.blockmanagement.TestBlockStatsMXBean)  Time elapsed: 17.682 s  <<< FAILURE!
java.lang.AssertionError: expected:<6> but was:<5>
	at org.junit.Assert.fail(Assert.java:88)
	at org.junit.Assert.failNotEquals(Assert.java:834)
	at org.junit.Assert.assertEquals(Assert.java:645)
	at org.junit.Assert.assertEquals(Assert.java:631)
	at org.apache.hadoop.hdfs.server.blockmanagement.TestBlockStatsMXBean.testStorageTypeStatsWhenStorageFailed(TestBlockStatsMXBean.java:213)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
	at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
	at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.lang.Thread.run(Thread.java:748)