[HDDS-8184] Intermittent timeout in TestContainerReplication

• Type: Sub-task
• Status: Resolved
• Priority: Minor
• Resolution: Cannot Reproduce
• Affects Version/s: 1.4.0
• Assignee: Raju Balpande
• Reporter: Attila Doroszlai
• Created: 16/Mar/23 08:22
• Updated: 24/Apr/24 17:56
• Resolved: 24/Apr/24 17:56

Description

https://github.com/adoroszlai/ozone-build-results/blob/master/2023/03/15/20796/it-ozone/hadoop-ozone/integration-test/org.apache.hadoop.ozone.container.replication.TestContainerReplication.txt

org.apache.hadoop.ozone.container.replication.TestContainerReplication.testPush(CopyContainerCompression)[5]  Time elapsed: 30.108 s  <<< ERROR!
java.util.concurrent.TimeoutException: 
...
  at 
org.apache.hadoop.ozone.container.replication.TestContainerReplication.queueAndWaitForCompletion(TestContainerReplication.java:187)
  at org.apache.hadoop.ozone.container.replication.TestContainerReplication.testPush(TestContainerReplication.java:111)

https://github.com/adoroszlai/ozone-build-results/blob/master/2023/03/15/20796/it-ozone/hadoop-ozone/integration-test/org.apache.hadoop.ozone.container.replication.TestContainerReplication-output.txt

2023-03-15 18:13:33,572 [ContainerReplicationThread-0] INFO  replication.PushReplicator (PushReplicator.java:replicate(58)) - Starting replication of container 1 to 2dde47c4-9545-41ac-a72b-8075b684fab1(fv-az196-962.ad143mqho3xu3jekw201s0oc5a.jx.internal.cloudapp.net/10.1.0.30) using NO_COMPRESSION
2023-03-15 18:13:33,583 [ContainerReplicationThread-0] WARN  replication.PushReplicator (PushReplicator.java:replicate(73)) - Container 1 replication was unsuccessful.
org.apache.hadoop.hdds.scm.container.common.helpers.StorageContainerException: Container 1 is not found.
	at org.apache.hadoop.ozone.container.replication.OnDemandContainerReplicationSource.copyData(OnDemandContainerReplicationSource.java:58)
	at org.apache.hadoop.ozone.container.replication.PushReplicator.replicate(PushReplicator.java:67)
	at org.apache.hadoop.ozone.container.replication.MeasuredReplicator.replicate(MeasuredReplicator.java:83)
	at org.apache.hadoop.ozone.container.replication.ReplicationTask.runTask(ReplicationTask.java:122)
	at org.apache.hadoop.ozone.container.replication.ReplicationSupervisor$TaskRunner.run(ReplicationSupervisor.java:215)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:750)
2023-03-15 18:13:33,589 [ContainerReplicationThread-0] INFO  replication.GrpcOutputStream (GrpcOutputStream.java:close(111)) - Sent 0 bytes for container 1
2023-03-15 18:13:33,590 [grpc-default-executor-4] WARN  replication.SendContainerRequestHandler (SendContainerRequestHandler.java:onCompleted(104)) - Received container without any parts
...
2023-03-15 18:13:38,590 [ContainerReplicationThread-0] WARN  replication.ReplicationSupervisor (ReplicationSupervisor.java:run(217)) - Failed FAILED replicateContainerCommand: containerId=1, replicaIndex=0, targetNode=2dde47c4-9545-41ac-a72b-8075b684fab1(fv-az196-962.ad143mqho3xu3jekw201s0oc5a.jx.internal.cloudapp.net/10.1.0.30), priority=NORMAL