[HDDS-8603] Push replication may fail with StatusRuntimeException: UNAVAILABLE: Channel shutdown invoked

• Type: Sub-task
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 1.4.0
• Component/s: Ozone Datanode
• Assignee: Unassigned
• Reporter: Attila Doroszlai
• Created: 12/May/23 09:00
• Updated: 15/Dec/23 09:46

Description

SCM

2023-05-12 06:37:25,691 [Under Replicated Processor] INFO  replication.ReplicationManager (ReplicationManager.java:sendDatanodeCommand(684)) - Sending command [replicateContainerCommand: containerId=11, replicaIndex=1, targetNode=dacb7084-9cbe-4e43-829c-fba4bce0d9b7(fv-az173-42.h4pcb1qgckmubfchnqyq5btvmd.gx.internal.cloudapp.net/10.1.0.47), priority=NORMAL] for container ContainerInfo{id=#11, state=CLOSED, pipelineID=PipelineID=1fdad600-ca58-422a-8cbc-f3f0ea92deb5, stateEnterTime=2023-05-12T06:37:20.327Z, owner=om1} to 1aeadef1-6c90-4225-addb-3772c43da32e(fv-az173-42.h4pcb1qgckmubfchnqyq5btvmd.gx.internal.cloudapp.net/10.1.0.47) with datanode deadline 1683875215691 and scm deadline 1683875245691
2023-05-12 06:37:25,691 [Under Replicated Processor] INFO  replication.ReplicationManager (ReplicationManager.java:sendDatanodeCommand(684)) - Sending command [replicateContainerCommand: containerId=10, replicaIndex=4, targetNode=dacb7084-9cbe-4e43-829c-fba4bce0d9b7(fv-az173-42.h4pcb1qgckmubfchnqyq5btvmd.gx.internal.cloudapp.net/10.1.0.47), priority=NORMAL] for container ContainerInfo{id=#10, state=CLOSED, pipelineID=PipelineID=f8d093fe-a02d-4415-b965-bb0272b84054, stateEnterTime=2023-05-12T06:37:19.680Z, owner=om1} to 1aeadef1-6c90-4225-addb-3772c43da32e(fv-az173-42.h4pcb1qgckmubfchnqyq5btvmd.gx.internal.cloudapp.net/10.1.0.47) with datanode deadline 1683875215691 and scm deadline 1683875245691

source datanode

2023-05-12 06:37:27,364 [ContainerReplicationThread-1] INFO  replication.PushReplicator (PushReplicator.java:replicate(58)) - Starting replication of container 11 to dacb7084-9cbe-4e43-829c-fba4bce0d9b7(fv-az173-42.h4pcb1qgckmubfchnqyq5btvmd.gx.internal.cloudapp.net/10.1.0.47) using NO_COMPRESSION
2023-05-12 06:37:27,367 [ContainerReplicationThread-2] INFO  replication.PushReplicator (PushReplicator.java:replicate(58)) - Starting replication of container 10 to dacb7084-9cbe-4e43-829c-fba4bce0d9b7(fv-az173-42.h4pcb1qgckmubfchnqyq5btvmd.gx.internal.cloudapp.net/10.1.0.47) using NO_COMPRESSION
2023-05-12 06:37:27,384 [ContainerReplicationThread-1] INFO  replication.GrpcOutputStream (GrpcOutputStream.java:close(111)) - Sent 13824 bytes for container 11
2023-05-12 06:37:27,407 [grpc-default-executor-7] WARN  replication.GrpcContainerUploader (GrpcContainerUploader.java:onError(113)) - Failed to upload container 11 to dacb7084-9cbe-4e43-829c-fba4bce0d9b7(fv-az173-42.h4pcb1qgckmubfchnqyq5btvmd.gx.internal.cloudapp.net/10.1.0.47)
org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: UNAVAILABLE: Channel shutdown invoked
...
2023-05-12 06:37:27,407 [ContainerReplicationThread-1] WARN  replication.PushReplicator (PushReplicator.java:replicate(73)) - Container 11 replication was unsuccessful.

2023-05-12 06:37:27,408 [ContainerReplicationThread-1] WARN  replication.ReplicationSupervisor (ReplicationSupervisor.java:run(350)) - Failed FAILED replicateContainerCommand: containerId=11, replicaIndex=1, targetNode=dacb7084-9cbe-4e43-829c-fba4bce0d9b7(fv-az173-42.h4pcb1qgckmubfchnqyq5btvmd.gx.internal.cloudapp.net/10.1.0.47), priority=NORMAL, transferred 13824 bytes
...
2023-05-12 06:37:27,411 [ContainerReplicationThread-2] INFO  replication.GrpcOutputStream (GrpcOutputStream.java:close(111)) - Sent 12800 bytes for container 10

2023-05-12 06:37:27,439 [ContainerReplicationThread-2] INFO  replication.ReplicationSupervisor (ReplicationSupervisor.java:run(353)) - Successful DONE replicateContainerCommand: containerId=10, replicaIndex=4, targetNode=dacb7084-9cbe-4e43-829c-fba4bce0d9b7(fv-az173-42.h4pcb1qgckmubfchnqyq5btvmd.gx.internal.cloudapp.net/10.1.0.47), priority=NORMAL, transferred 12800 bytes