Uploaded image for project: 'Apache Ozone'
  1. Apache Ozone
  2. HDDS-460

Replication manager failed to import container data

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.2.1
    • Fix Version/s: 0.2.1
    • Component/s: SCM
    • Labels:
      None
    • Target Version/s:

      Description

      replication manager is not able to import downloaded container data on datanode failure. 

       

      snippet of ozone.log

      -----------------------------

      2018-09-14 09:34:05,249 [grpc-default-executor-139] INFO (GrpcReplicationClient.java:161) - Container is downloaded to /tmp/container-copy/container-14.tar.gz
       2018-09-14 09:34:05,389 [grpc-default-executor-131] ERROR (ReplicateContainerCommandHandler.java:164) - Can't import the downloaded container data id=8
       java.io.EOFException
       at org.apache.commons.compress.compressors.gzip.GzipCompressorInputStream.read(GzipCompressorInputStream.java:241)
       at org.apache.commons.compress.archivers.tar.TarBuffer.readBlock(TarBuffer.java:224)
       at org.apache.commons.compress.archivers.tar.TarBuffer.readRecord(TarBuffer.java:195)
       at org.apache.commons.compress.archivers.tar.TarArchiveInputStream.read(TarArchiveInputStream.java:486)
       at org.apache.commons.compress.archivers.tar.TarArchiveInputStream.skip(TarArchiveInputStream.java:182)
       at org.apache.commons.compress.archivers.tar.TarArchiveInputStream.getNextTarEntry(TarArchiveInputStream.java:220)
       at org.apache.hadoop.ozone.container.keyvalue.TarContainerPacker.unpackContainerDescriptor(TarContainerPacker.java:200)
       at org.apache.hadoop.ozone.container.common.statemachine.commandhandler.ReplicateContainerCommandHandler.importContainer(ReplicateContainerCommandHandler.java:144)
       at org.apache.hadoop.ozone.container.common.statemachine.commandhandler.ReplicateContainerCommandHandler.lambda$handle$0(ReplicateContainerCommandHandler.java:121)
       at java.util.concurrent.CompletableFuture.uniAccept(CompletableFuture.java:656)
       at java.util.concurrent.CompletableFuture$UniAccept.tryFire(CompletableFuture.java:632)
       at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
       at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962)
       at org.apache.hadoop.ozone.container.replication.GrpcReplicationClient$StreamDownloader.onCompleted(GrpcReplicationClient.java:160)
       at org.apache.ratis.shaded.io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:418)
       at org.apache.ratis.shaded.io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
       at org.apache.ratis.shaded.io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
       at org.apache.ratis.shaded.io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
       at org.apache.ratis.shaded.io.grpc.internal.CensusStatsModule$StatsClientInterceptor$1$1.onClose(CensusStatsModule.java:684)
       at org.apache.ratis.shaded.io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
       at org.apache.ratis.shaded.io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
       at org.apache.ratis.shaded.io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
       at org.apache.ratis.shaded.io.grpc.internal.CensusTracingModule$TracingClientInterceptor$1$1.onClose(CensusTracingModule.java:403)
       at org.apache.ratis.shaded.io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:459)
       at org.apache.ratis.shaded.io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:63)
       at org.apache.ratis.shaded.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.close(ClientCallImpl.java:546)
       at org.apache.ratis.shaded.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.access$600(ClientCallImpl.java:467)
       at org.apache.ratis.shaded.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:584)
       at org.apache.ratis.shaded.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
       at org.apache.ratis.shaded.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
       at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
       at java.lang.Thread.run(Thread.java:745)

        Attachments

        1. HDDS-460-ozone-0.2.001.patch
          38 kB
          Marton Elek
        2. all-node-ozone-logs-1536920345.tar.gz
          2.37 MB
          Nilotpal Nandi

          Activity

            People

            • Assignee:
              elek Marton Elek
              Reporter:
              nilotpalnandi Nilotpal Nandi
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: