[HDDS-2535] TestOzoneManagerDoubleBufferWithOMResponse is flaky

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.5.0
• Component/s: Ozone Manager
• Labels: pull-request-available
• Target Version/s: 0.5.0
• Assignee: Bharat Viswanadham
• Reporter: Marton Elek
• Created: 18/Nov/19 13:28
• Updated: 28/Feb/20 12:11
• Resolved: 20/Nov/19 05:27

Description

Flakiness can be reproduced locally. Usually it passes, but when I started to run it 100 times parallel with high cpu load it failed with the 3rd attempt (timed out)

-------------------------------------------------------------------------------
Test set: org.apache.hadoop.ozone.om.ratis.TestOzoneManagerDoubleBufferWithOMResponse
-------------------------------------------------------------------------------
Tests run: 3, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 503.297 s <<< FAILURE! - in org.apache.hadoop.ozone.om.ratis.TestOzoneManagerDoubleBufferWithOMResponse
testDoubleBuffer(org.apache.hadoop.ozone.om.ratis.TestOzoneManagerDoubleBufferWithOMResponse)  Time elapsed: 500.122 s  <<< ERROR!
java.lang.Exception: test timed out after 500000 milliseconds
        at java.lang.Thread.sleep(Native Method)
        at org.apache.hadoop.test.GenericTestUtils.waitFor(GenericTestUtils.java:382)
        at org.apache.hadoop.ozone.om.ratis.TestOzoneManagerDoubleBufferWithOMResponse.testDoubleBuffer(TestOzoneManagerDoubleBufferWithOMResponse.java:385)
        at org.apache.hadoop.ozone.om.ratis.TestOzoneManagerDoubleBufferWithOMResponse.testDoubleBuffer(TestOzoneManagerDoubleBufferWithOMResponse.java:129)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
        at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
        at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
        at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
        at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
 

Independent from the flakiness I think a test where the timeout is 8 minutes and starts 1000 threads to insert 500 buckets (500_000 buckets all together) it's more like an integration test and would be better to move the slowest part to the integration-test project.

Issue Links

• is related to: HDDS-3106 Intermittent timeout in TestOzoneManagerDoubleBufferWithOMResponse#testDoubleBuffer
• links to: GitHub Pull Request #216