[HDDS-3330] TestDeleteWithSlowFollower is still flaky

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.0.0
• Component/s: test
• Labels: TriagePending ozone-flaky-test pull-request-available
• Target Version/s: 1.0.0
• Assignee: Shashikant Banerjee
• Reporter: Marton Elek
• Created: 02/Apr/20 12:36
• Updated: 09/Jun/20 16:00
• Resolved: 09/Jun/20 03:22

Description

Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 666.209 s <<< FAILURE! - in org.apache.hadoop.ozone.client.rpc.TestDeleteWithSlowFollower
testDeleteKeyWithSlowFollower(org.apache.hadoop.ozone.client.rpc.TestDeleteWithSlowFollower)  Time elapsed: 640.745 s  <<< ERROR!
java.io.IOException: INTERNAL_ERROR org.apache.hadoop.ozone.om.exceptions.OMException: Allocated 0 blocks. Requested 1 blocks
        at org.apache.hadoop.ozone.client.io.KeyOutputStream.handleWrite(KeyOutputStream.java:229)
        at org.apache.hadoop.ozone.client.io.KeyOutputStream.handleRetry(KeyOutputStream.java:402)
        at org.apache.hadoop.ozone.client.io.KeyOutputStream.handleException(KeyOutputStream.java:347)
        at org.apache.hadoop.ozone.client.io.KeyOutputStream.handleFlushOrClose(KeyOutputStream.java:458)
        at org.apache.hadoop.ozone.client.io.KeyOutputStream.close(KeyOutputStream.java:509)
        at org.apache.hadoop.ozone.client.io.OzoneOutputStream.close(OzoneOutputStream.java:60)
        at org.apache.hadoop.ozone.client.rpc.TestDeleteWithSlowFollower.testDeleteKeyWithSlowFollower(TestDeleteWithSlowFollower.java:225)

I learned this from shashikant

we kill a datanode after some IO, SCM is out of safe mode by then . SCM takes time to destroy a pipeline and form a new one

With only minimal set of dn in cluster, if we want to write again, we need to wait for a new pipeline to open up before writing again

Will turn off this test until the fix.

Issue Links

• links to: GitHub Pull Request #1026