[HDDS-3293] read operation failing when two container replicas are corrupted

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.0.0
• Component/s: Ozone Datanode
• Labels: fault_injection
• Assignee: Shashikant Banerjee
• Reporter: Nilotpal Nandi
• Created: 27/Mar/20 18:37
• Updated: 28/May/20 09:41
• Resolved: 28/May/20 09:41

Description

steps taken :

1) Mounted noise injection FUSE on all datanodes.

2) Write a key ( multi blocks)

3) Select one of the container ids ,  inject error on 2 container replicas for that container id.

4) Run GET key operation.

GET key operation fails intermittenly.

Error seen :

-------------

 

20/03/27 18:30:40 WARN impl.MetricsConfig: Cannot locate configuration: tried hadoop-metrics2-xceiverclientmetrics.properties,hadoop-metrics2.properties
E 20/03/27 18:30:40 INFO impl.MetricsSystemImpl: Scheduled Metric snapshot period at 10 second(s).
E 20/03/27 18:30:40 INFO impl.MetricsSystemImpl: XceiverClientMetrics metrics system started
E 20/03/27 18:31:12 ERROR scm.XceiverClientGrpc: Failed to execute command cmdType: ReadChunk
E traceID: "f80a51eaec481a1c:cbb8e92869015a53:f80a51eaec481a1c:0"
E containerID: 67
E datanodeUuid: "96101390-2446-40e6-a54e-36e170497e57"
E readChunk {
E blockID {
E containerID: 67
E localID: 103896435892617248
E blockCommitSequenceId: 1010
E }
E chunkData {
E chunkName: "103896435892617248_chunk_28"
E offset: 113246208
E len: 4194304
E checksumData {
E type: CRC32
E bytesPerChecksum: 1048576
E checksums: "\034\376\313\031"
E checksums: ";U\225\037"
E checksums: "\327m\332."
E checksums: "|\307\004E"
E }
E }
E }
E on the pipeline Pipeline[ Id: bce6316c-9690-452b-80e3-0f3590533444, Nodes: 96101390-2446-40e6-a54e-36e170497e57{ip: 172.27.111.129, host: quasar-olrywk-3.quasar-olrywk.root.hwx.site, networkLocation: /default-rack, certSerialId: null}3e85204d-2399-43b5-952a-55b837eb4c1d{ip: 172.27.100.0, host: quasar-olrywk-1.quasar-olrywk.root.hwx.site, networkLocation: /default-rack, certSerialId: null}5af0340a-6fee-4ce8-9f68-37fa35566a5a{ip: 172.27.73.0, host: quasar-olrywk-9.quasar-olrywk.root.hwx.site, networkLocation: /default-rack, certSerialId: null}, Type:STAND_ALONE, Factor:THREE, State:OPEN, leaderId:96101390-2446-40e6-a54e-36e170497e57, CreationTimestamp2020-03-27T03:36:51.880Z].
E Unexpected OzoneException: java.io.IOException: java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: DEADLINE_EXCEEDED: deadline exceeded after 84603913ns. [remote_addr=/172.27.73.0:9859]]

 

 

 

Issue Links

• Is contained by: HDDS-3064 Get Key is hung when READ delay is injected in chunk file path