[HDFS-15146] TestBalancerRPCDelay. testBalancerRPCDelayQpsDefault fails intermittently

• Type: Sub-task
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 3.3.0, 3.2.2, 2.10.1, 3.4.0
• Fix Version/s: 3.3.0, 3.2.2, 2.10.1, 3.4.0
• Component/s: balancer & mover, test
• Target Version/s: 3.3.0, 3.2.2, 2.10.1, 3.4.0
• Hadoop Flags: Reviewed
• Assignee: Ahmed Hussein
• Reporter: Ahmed Hussein
• Created: 27/Jan/20 18:51
• Updated: 16/Oct/24 18:30
• Resolved: 29/Jan/20 17:10

Description

TestBalancerRPCDelay.testBalancerRPCDelayQpsDefault fails intermittently when the number of blocks does not match the expected. In testBalancerRPCDelay, it seems like some datanodes will not be up by the time we fetch the block locations.

I see the following stack trace:

[ERROR] Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 39.969 s <<< FAILURE! - in org.apache.hadoop.hdfs.server.balancer.TestBalancerRPCDelay
[ERROR] testBalancerRPCDelayQpsDefault(org.apache.hadoop.hdfs.server.balancer.TestBalancerRPCDelay)  Time elapsed: 12.035 s  <<< FAILURE!
java.lang.AssertionError: Number of getBlocks should be not less than 20
	at org.junit.Assert.fail(Assert.java:88)
	at org.junit.Assert.assertTrue(Assert.java:41)
	at org.apache.hadoop.hdfs.server.balancer.TestBalancer.testBalancerRPCDelay(TestBalancer.java:2197)
	at org.apache.hadoop.hdfs.server.balancer.TestBalancerRPCDelay.testBalancerRPCDelayQpsDefault(TestBalancerRPCDelay.java:53)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
	at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
	at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.lang.Thread.run(Thread.java:748)