[HDDS-5149] when source datanode download container tar from target datanode,but the target datanode container file missing,import error

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Labels: pull-request-available
• Assignee: cchenaxchen
• Reporter: cchenaxchen
• Created: 27/Apr/21 03:24
• Updated: 11/Mar/22 20:52
• Resolved: 11/Mar/22 20:52

Description

2021-03-25 11:18:21,703 [grpc-default-executor-1879] INFO org.apache.hadoop.ozone.container.replication.GrpcReplicationService: Streaming container data (50383) to other datanode2021-03-25 11:18:21,703 [grpc-default-executor-1879] INFO org.apache.hadoop.ozone.container.replication.GrpcReplicationService: Streaming container data (50383) to other datanode2021-03-25 11:18:21,812 [grpc-default-executor-1879] INFO org.apache.hadoop.ozone.container.replication.GrpcOutputStream: Sent 119153 bytes for container 503832021-03-25 11:18:21,812 [grpc-default-executor-1879] ERROR org.apache.hadoop.ozone.container.replication.GrpcReplicationService: Error streaming container 50383java.nio.file.NoSuchFileException: /data3/hdds/hdds/326a5fe1-e63c-44b6-a57e-2f858fe4eaa7/current/containerDir98/50383/chunks at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86) at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107) at sun.nio.fs.UnixFileSystemProvider.newDirectoryStream(UnixFileSystemProvider.java:427) at java.nio.file.Files.newDirectoryStream(Files.java:457) at java.nio.file.Files.list(Files.java:3451) at 

 

 

2021-03-25 11:18:21,695 [ContainerReplicationThread-4] INFO org.apache.hadoop.ozone.container.replication.DownloadAndImportReplicator: Starting replication of container 50383 from [d481556f-0451-4669-9aba-74360e0b26fd\{ip: 9.179.149.196, host: tdw-9-179-149-196, networkLocation: /rack2, certSerialId: null, persistedOpState: IN_SERVICE, persistedOpStateExpiryEpochSec: 0}, bcdf3bd5-7b8e-435d-b3fa-b3e29f0eb307\{ip: 9.180.5.41, host: tdw-9-180-5-41, networkLocation: /rack10, certSerialId: null, persistedOpState: IN_SERVICE, persistedOpStateExpiryEpochSec: 0}]2021-03-25 11:18:21,695 [ContainerReplicationThread-4] INFO org.apache.hadoop.ozone.container.replication.DownloadAndImportReplicator: Starting replication of container 50383 from [d481556f-0451-4669-9aba-74360e0b26fd\{ip: 9.179.149.196, host: tdw-9-179-149-196, networkLocation: /rack2, certSerialId: null, persistedOpState: IN_SERVICE, persistedOpStateExpiryEpochSec: 0}, bcdf3bd5-7b8e-435d-b3fa-b3e29f0eb307\{ip: 9.180.5.41, host: tdw-9-180-5-41, networkLocation: /rack10, certSerialId: null, persistedOpState: IN_SERVICE, persistedOpStateExpiryEpochSec: 0}]2021-03-25 11:18:21,723 [EndpointStateMachine task thread for /10.51.87.181:9891 - 0 ] INFO org.apache.hadoop.ipc.Client: Retrying connect to server: tdw-10-51-87-181/10.51.87.181:9891. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=15, sleepTime=1000 MILLISECONDS)2021-03-25 11:18:21,807 [grpc-default-executor-1390] INFO org.apache.hadoop.ozone.container.replication.GrpcReplicationClient: Container 50383 is downloaded to /tmp/container-copy/container-50383.tar.gz2021-03-25 11:18:21,807 [ContainerReplicationThread-4] INFO org.apache.hadoop.ozone.container.replication.DownloadAndImportReplicator: Container 50383 is downloaded, starting to import.2021-03-25 11:18:21,810 [ContainerReplicationThread-4] ERROR org.apache.hadoop.ozone.container.replication.DownloadAndImportReplicator: Container 50383 replication was unsuccessful.java.io.IOException: Container descriptor is missing from the container archive. at org.apache.hadoop.ozone.container.keyvalue.TarContainerPacker.unpackContainerDescriptor(TarContainerPacker.java:190) at org.apache.hadoop.ozone.container.replication.DownloadAndImportReplicator.importContainer(DownloadAndImportReplicator.java:76)

Issue Links

• duplicates: HDDS-6235 Empty KeyValueContainers are replicated without chunks directory
• links to: GitHub Pull Request #2196
• links to: GitHub Pull Request #2376