[HDDS-9292] [disabled] Intermittent failure in S3 multipart copy with HA proxy - ASF JIRA

XML

Word

Printable

JSON

Details

Type: Bug
Status: Open
Priority: Major
Resolution: Unresolved
Affects Version/s: None
Fix Version/s: None
Component/s: Ozone Datanode, S3
Labels:
None

Description

Download of object created via multipart upload in S3 HA environment is failing intermittently.

Executing test ozone/test-s3-haproxy.sh
...
Test Multipart Upload with the simplified aws s3 cp API               | FAIL |
...
ERROR: Test execution of ozone/test-s3-haproxy.sh is FAILED!!!!

S3 Gateway log:

2023-09-15 05:06:57,143 [qtp2112233878-18] ERROR scm.XceiverClientGrpc: Failed to execute command GetBlock on the pipeline Pipeline[ Id: 3e70c06d-5798-4025-8791-78b8b2500f11, Nodes: 9416081d-f44a-4263-bc28-1838037e3ca1(ozone_datanode_1.ozone_default/172.19.0.7)82a30eb8-2baa-482f-a23a-ad010ae2684d(ozone_datanode_3.ozone_default/172.19.0.10)c229b2ba-c364-4c38-879e-952c8da62282(ozone_datanode_2.ozone_default/172.19.0.3), ReplicationConfig: STANDALONE/THREE, State:OPEN, leaderId:c229b2ba-c364-4c38-879e-952c8da62282, CreationTimestamp2023-09-15T05:05:12.106Z[UTC]].
2023-09-15 05:06:57,144 [qtp2112233878-18] WARN storage.ContainerProtocolCalls: Failed to get block #111677748019200096 in container #1 from 9416081d-f44a-4263-bc28-1838037e3ca1(ozone_datanode_1.ozone_default/172.19.0.7); will try another datanode.
org.apache.hadoop.hdds.scm.container.common.helpers.StorageContainerException: bcsId 388 mismatches with existing block Id 387 for block conID: 1 locID: 111677748019200096 bcsId: 388.
	at org.apache.hadoop.hdds.scm.storage.ContainerProtocolCalls.validateContainerResponse(ContainerProtocolCalls.java:674)
	at org.apache.hadoop.hdds.scm.storage.ContainerProtocolCalls.lambda$createValidators$4(ContainerProtocolCalls.java:685)
	at org.apache.hadoop.hdds.scm.XceiverClientGrpc.sendCommandWithRetry(XceiverClientGrpc.java:407)
	at org.apache.hadoop.hdds.scm.XceiverClientGrpc.lambda$sendCommandWithTraceIDAndRetry$0(XceiverClientGrpc.java:347)
	at org.apache.hadoop.hdds.tracing.TracingUtil.executeInSpan(TracingUtil.java:169)
	at org.apache.hadoop.hdds.tracing.TracingUtil.executeInNewSpan(TracingUtil.java:149)
	at org.apache.hadoop.hdds.scm.XceiverClientGrpc.sendCommandWithTraceIDAndRetry(XceiverClientGrpc.java:342)
	at org.apache.hadoop.hdds.scm.XceiverClientGrpc.sendCommand(XceiverClientGrpc.java:323)
	at org.apache.hadoop.hdds.scm.storage.ContainerProtocolCalls.getBlock(ContainerProtocolCalls.java:208)
	at org.apache.hadoop.hdds.scm.storage.ContainerProtocolCalls.lambda$getBlock$0(ContainerProtocolCalls.java:186)
	at org.apache.hadoop.hdds.scm.storage.ContainerProtocolCalls.tryEachDatanode(ContainerProtocolCalls.java:146)
	at org.apache.hadoop.hdds.scm.storage.ContainerProtocolCalls.getBlock(ContainerProtocolCalls.java:185)
	at org.apache.hadoop.hdds.scm.storage.BlockInputStream.getChunkInfos(BlockInputStream.java:255)
	at org.apache.hadoop.hdds.scm.storage.BlockInputStream.initialize(BlockInputStream.java:146)
	at org.apache.hadoop.hdds.scm.storage.BlockInputStream.readWithStrategy(BlockInputStream.java:308)
	at org.apache.hadoop.hdds.scm.storage.ExtendedInputStream.read(ExtendedInputStream.java:56)
	at org.apache.hadoop.hdds.scm.storage.ByteArrayReader.readFromBlock(ByteArrayReader.java:57)
	at org.apache.hadoop.hdds.scm.storage.MultipartInputStream.readWithStrategy(MultipartInputStream.java:96)
	at org.apache.hadoop.hdds.scm.storage.ExtendedInputStream.read(ExtendedInputStream.java:56)
	at org.apache.hadoop.ozone.client.io.OzoneInputStream.read(OzoneInputStream.java:56)
	at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:1384)
	at org.apache.hadoop.ozone.s3.endpoint.ObjectEndpoint.lambda$get$1(ObjectEndpoint.java:407)

Message about bcdId mismatch appears in all 3 datanode logs, too.

Attachments

Activity

People

Assignee:: Ashish Kumar

Reporter:: Attila Doroszlai

Votes:: 0 Vote for this issue

Watchers:: 2 Start watching this issue

Dates

Created:: 15/Sep/23 07:57

Updated:: 09/Oct/23 12:46