Uploaded image for project: 'Hadoop Distributed Data Store'
  1. Hadoop Distributed Data Store
  2. HDDS-3853

Container marked as missing on datanode while container directory do exist

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: Ozone Datanode
    • Labels:
      None
    • Target Version/s:

      Description

      INFO org.apache.hadoop.ozone.container.common.impl.HddsDispatcher: Operation: PutBlock , Trace ID: 487c959563e884b9:509a3386ba37abc6:487c959563e884b9:0 , Message: ContainerID 1744 has been lost and and cannot be recreated on this DataNode , Result: CONTAINER_MISSING , StorageContainerException Occurred.
      org.apache.hadoop.hdds.scm.container.common.helpers.StorageContainerException: ContainerID 1744 has been lost and and cannot be recreated on this DataNode
              at org.apache.hadoop.ozone.container.common.impl.HddsDispatcher.dispatchRequest(HddsDispatcher.java:238)
              at org.apache.hadoop.ozone.container.common.impl.HddsDispatcher.dispatch(HddsDispatcher.java:166)
              at org.apache.hadoop.ozone.container.common.transport.server.ratis.ContainerStateMachine.dispatchCommand(ContainerStateMachine.java:395)
              at org.apache.hadoop.ozone.container.common.transport.server.ratis.ContainerStateMachine.runCommand(ContainerStateMachine.java:405)
              at org.apache.hadoop.ozone.container.common.transport.server.ratis.ContainerStateMachine.lambda$applyTransaction$6(ContainerStateMachine.java:749)
              at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1590)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
              at java.lang.Thread.run(Thread.java:748)
      
       ERROR org.apache.hadoop.ozone.container.common.transport.server.ratis.ContainerStateMachine: gid group-1376E41FD581 : ApplyTransaction failed. cmd PutBlock logIndex 40079 msg : ContainerID 1744 has been lost and and cannot be recreated on this DataNode Container Result: CONTAINER_MISSING
      
       ERROR org.apache.hadoop.ozone.container.common.transport.server.ratis.XceiverServerRatis: pipeline Action CLOSE on pipeline PipelineID=de21dfcf-415c-4901-84ca-1376e41fd581.Reason : Ratis Transaction failure in datanode 33b49c34-caa2-4b4f-894e-dce7db4f97b9 with role FOLLOWER .Triggering pipeline close action
       

        Attachments

          Activity

            People

            • Assignee:
              yjxxtd runzhiwang
              Reporter:
              Sammi Sammi Chen
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated: