[HDDS-3853] Container marked as missing on datanode while container directory do exist

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Cannot Reproduce
• Fix Version/s: 1.4.0
• Component/s: Ozone Datanode
• Target Version/s: 1.4.0
• Assignee: runzhiwang
• Reporter: Sammi Chen
• Created: 23/Jun/20 07:26
• Updated: 21/Sep/23 13:48
• Resolved: 21/Sep/23 13:46

Description

INFO org.apache.hadoop.ozone.container.common.impl.HddsDispatcher: Operation: PutBlock , Trace ID: 487c959563e884b9:509a3386ba37abc6:487c959563e884b9:0 , Message: ContainerID 1744 has been lost and and cannot be recreated on this DataNode , Result: CONTAINER_MISSING , StorageContainerException Occurred.
org.apache.hadoop.hdds.scm.container.common.helpers.StorageContainerException: ContainerID 1744 has been lost and and cannot be recreated on this DataNode
        at org.apache.hadoop.ozone.container.common.impl.HddsDispatcher.dispatchRequest(HddsDispatcher.java:238)
        at org.apache.hadoop.ozone.container.common.impl.HddsDispatcher.dispatch(HddsDispatcher.java:166)
        at org.apache.hadoop.ozone.container.common.transport.server.ratis.ContainerStateMachine.dispatchCommand(ContainerStateMachine.java:395)
        at org.apache.hadoop.ozone.container.common.transport.server.ratis.ContainerStateMachine.runCommand(ContainerStateMachine.java:405)
        at org.apache.hadoop.ozone.container.common.transport.server.ratis.ContainerStateMachine.lambda$applyTransaction$6(ContainerStateMachine.java:749)
        at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1590)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

 ERROR org.apache.hadoop.ozone.container.common.transport.server.ratis.ContainerStateMachine: gid group-1376E41FD581 : ApplyTransaction failed. cmd PutBlock logIndex 40079 msg : ContainerID 1744 has been lost and and cannot be recreated on this DataNode Container Result: CONTAINER_MISSING

 ERROR org.apache.hadoop.ozone.container.common.transport.server.ratis.XceiverServerRatis: pipeline Action CLOSE on pipeline PipelineID=de21dfcf-415c-4901-84ca-1376e41fd581.Reason : Ratis Transaction failure in datanode 33b49c34-caa2-4b4f-894e-dce7db4f97b9 with role FOLLOWER .Triggering pipeline close action