[HDDS-1898] GrpcReplicationService#download cannot replicate the container

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.4.0
• Fix Version/s: 0.5.0
• Component/s: Ozone Datanode
• Labels: MiniOzoneChaosCluster pull-request-available
• Target Version/s: 0.5.0
• Assignee: Nandakumar
• Reporter: Mukul Kumar Singh
• Created: 02/Aug/19 08:41
• Updated: 24/Feb/20 20:56
• Resolved: 05/Sep/19 12:43

Description

Replication of container is failing because of rocksdb is unable to find the underlying files.

2019-08-02 14:07:26,670 INFO  replication.GrpcReplicationService (GrpcReplicationService.java:close(124)) - 663284 bytes written to th
e rpc stream from container 12
2019-08-02 14:07:26,670 ERROR replication.GrpcReplicationService (GrpcReplicationService.java:download(65)) - Can't stream the contain
er data
java.io.FileNotFoundException: /Users/msingh/code/apache/ozone/github/chaos_runs/hadoop-ozone/integration-test/target/test/data/MiniOzoneClusterImpl-403d87c2-5cbe-4511-8e14-dce727f10cf9/datanode-7/data/containers/hdds/9f2a75dc-3243-462a-a90e-c83f63ad0d55/current/containerDir0/12/metadata/12-dn-container.db/002084.log (No such file or directory)
        at java.io.FileInputStream.open0(Native Method)
        at java.io.FileInputStream.open(FileInputStream.java:195)
        at java.io.FileInputStream.<init>(FileInputStream.java:138)
        at org.apache.hadoop.ozone.container.keyvalue.TarContainerPacker.includeFile(TarContainerPacker.java:243)
        at org.apache.hadoop.ozone.container.keyvalue.TarContainerPacker.includePath(TarContainerPacker.java:233)
        at org.apache.hadoop.ozone.container.keyvalue.TarContainerPacker.pack(TarContainerPacker.java:164)
        at org.apache.hadoop.ozone.container.replication.OnDemandContainerReplicationSource.copyData(OnDemandContainerReplicationSource.java:67)
        at org.apache.hadoop.ozone.container.replication.GrpcReplicationService.download(GrpcReplicationService.java:63)
        at org.apache.hadoop.hdds.protocol.datanode.proto.IntraDatanodeProtocolServiceGrpc$MethodHandlers.invoke(IntraDatanodeProtocolServiceGrpc.java:217)
        at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171)
        at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283)
        at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710)
        at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
        at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
        Suppressed: java.io.IOException: This archives contains unclosed entries.
                at org.apache.commons.compress.archivers.tar.TarArchiveOutputStream.finish(TarArchiveOutputStream.java:214)
                at org.apache.commons.compress.archivers.tar.TarArchiveOutputStream.close(TarArchiveOutputStream.java:229)
                at org.apache.hadoop.ozone.container.keyvalue.TarContainerPacker.pack(TarContainerPacker.java:173)
                ... 11 more

Issue Links

• links to: GitHub Pull Request #1326