[HDDS-5312] Intermittent failure in SCM Ratis integration test

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.2.0
• Fix Version/s: 1.2.0
• Component/s: SCM HA, test
• Labels: pull-request-available
• Target Version/s: 1.2.0
• Assignee: Attila Doroszlai
• Reporter: Attila Doroszlai
• Created: 07/Jun/21 12:54
• Updated: 09/Jun/21 15:21
• Resolved: 09/Jun/21 15:21

Description

Some integration tests intermittently fail due to mini cluster not existing safe mode within 2 minutes timeout. The problem is that pipeline creation interval is also 2 minutes. It may happen that pipeline is created only while the cluster is being shut down due to timeout.

https://github.com/elek/ozone-build-results/blob/master/2021/06/02/8191/it-filesystem-hdds/hadoop-ozone/integration-test/org.apache.hadoop.hdds.scm.TestSCMInstallSnapshot-output.txt

2021-06-02 03:21:03,005 [RatisPipelineUtilsThread - 0] WARN  pipeline.PipelinePlacementPolicy (PipelinePlacementPolicy.java:filterViableNodes(151)) - Pipeline creation failed due to no sufficient healthy datanodes. Required 3. Found 2.
...
2021-06-02 03:21:04,007 [Listener at 127.0.0.1/40677] INFO  ozone.MiniOzoneClusterImpl (MiniOzoneClusterImpl.java:lambda$waitForClusterToBeReady$0(224)) - Nodes are ready. Got 3 of 3 DN Heartbeats.
...
2021-06-02 03:22:59,107 [Listener at 127.0.0.1/40677] INFO  ozone.MiniOzoneClusterImpl (MiniOzoneClusterImpl.java:shutdown(443)) - Shutting down the Mini Ozone Cluster
...
2021-06-02 03:23:03,031 [6d4e3dd1-e161-4c07-861b-817db46a0427@group-0D81E0660BF9-StateMachineUpdater] INFO  pipeline.PipelineStateManager (PipelineStateManagerV2Impl.java:addPipeline(101)) - Created pipeline Pipeline ... RATIS/THREE ...

https://github.com/elek/ozone-build-results/blob/master/2021/06/02/8191/it-filesystem-hdds/hadoop-ozone/integration-test/org.apache.hadoop.hdds.scm.TestSCMInstallSnapshot.txt

org.apache.hadoop.hdds.scm.TestSCMInstallSnapshot  Time elapsed: 146.994 s  <<< ERROR!
java.util.concurrent.TimeoutException: 
  ...
  at org.apache.hadoop.ozone.MiniOzoneClusterImpl.waitForClusterToBeReady(MiniOzoneClusterImpl.java:217)

Related test failures:

2021/05/26/8113/it-client/hadoop-ozone/integration-test/org.apache.hadoop.ozone.client.rpc.TestOzoneRpcClientWithRatis.txt
2021/05/26/8118/it-ozone/hadoop-ozone/integration-test/org.apache.hadoop.ozone.scm.TestStorageContainerManagerHA.txt
2021/05/27/8142/it-filesystem-hdds/hadoop-ozone/integration-test/org.apache.hadoop.hdds.scm.TestSCMInstallSnapshot.txt
2021/05/30/8164/it-ozone/hadoop-ozone/integration-test/org.apache.hadoop.ozone.om.TestOzoneManagerRestInterface.txt
2021/05/31/8166/it-filesystem-hdds/hadoop-ozone/integration-test/org.apache.hadoop.hdds.scm.TestSCMSnapshot.txt
2021/05/31/8177/it-filesystem-hdds/hadoop-ozone/integration-test/org.apache.hadoop.hdds.scm.pipeline.TestPipelineClose.txt
2021/06/02/8191/it-filesystem-hdds/hadoop-ozone/integration-test/org.apache.hadoop.hdds.scm.TestSCMInstallSnapshot.txt
2021/06/02/8193/it-ozone/hadoop-ozone/integration-test/org.apache.hadoop.ozone.scm.TestStorageContainerManagerHA.txt
2021/06/02/8211/it-filesystem-hdds/hadoop-ozone/integration-test/org.apache.hadoop.hdds.scm.TestSCMSnapshot.txt
2021/06/02/8217/it-filesystem-hdds/hadoop-ozone/integration-test/org.apache.hadoop.hdds.scm.TestSCMInstallSnapshot.txt
2021/06/07/8299/it-filesystem-hdds/hadoop-ozone/integration-test/org.apache.hadoop.hdds.scm.TestSCMInstallSnapshot.txt

Issue Links

• links to: GitHub Pull Request #2308