[HDDS-8947] Intermittent failure in TestOzoneAtRestEncryption streaming

• Type: Sub-task
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Assignee: GuoHao
• Reporter: Attila Doroszlai
• Created: 28/Jun/23 06:04
• Updated: 20/Nov/23 08:46

Description

Intermittent failure in TestOzoneAtRestEncryption#testMPUwithOneStreamPart.

TestOzoneAtRestEncryption.testMPUwithOneStreamPart  Time elapsed: 1.927 s  <<< ERROR!
org.apache.hadoop.hdds.scm.container.common.helpers.StorageContainerException: Unable to find the block with bcsID 7 .Container 18 bcsId is 5.
	at org.apache.hadoop.hdds.scm.storage.ContainerProtocolCalls.validateContainerResponse(ContainerProtocolCalls.java:707)
	at org.apache.hadoop.hdds.scm.storage.ContainerProtocolCalls.lambda$createValidators$4(ContainerProtocolCalls.java:718)
	at org.apache.hadoop.hdds.scm.XceiverClientGrpc.sendCommandWithRetry(XceiverClientGrpc.java:407)
	at org.apache.hadoop.hdds.scm.XceiverClientGrpc.lambda$sendCommandWithTraceIDAndRetry$0(XceiverClientGrpc.java:347)
	at org.apache.hadoop.hdds.tracing.TracingUtil.executeInSpan(TracingUtil.java:169)
	at org.apache.hadoop.hdds.tracing.TracingUtil.executeInNewSpan(TracingUtil.java:149)
	at org.apache.hadoop.hdds.scm.XceiverClientGrpc.sendCommandWithTraceIDAndRetry(XceiverClientGrpc.java:342)
	at org.apache.hadoop.hdds.scm.XceiverClientGrpc.sendCommand(XceiverClientGrpc.java:323)
	at org.apache.hadoop.hdds.scm.storage.ContainerProtocolCalls.getBlock(ContainerProtocolCalls.java:208)
	at org.apache.hadoop.hdds.scm.storage.ContainerProtocolCalls.lambda$getBlock$0(ContainerProtocolCalls.java:186)
	at org.apache.hadoop.hdds.scm.storage.ContainerProtocolCalls.tryEachDatanode(ContainerProtocolCalls.java:146)
	at org.apache.hadoop.hdds.scm.storage.ContainerProtocolCalls.getBlock(ContainerProtocolCalls.java:185)
	at org.apache.hadoop.hdds.scm.storage.BlockInputStream.getChunkInfos(BlockInputStream.java:255)
	at org.apache.hadoop.hdds.scm.storage.BlockInputStream.initialize(BlockInputStream.java:146)
	at org.apache.hadoop.hdds.scm.storage.BlockInputStream.readWithStrategy(BlockInputStream.java:308)
	at org.apache.hadoop.hdds.scm.storage.ExtendedInputStream.read(ExtendedInputStream.java:66)
	at org.apache.hadoop.hdds.scm.storage.ByteBufferReader.readFromBlock(ByteBufferReader.java:56)
	at org.apache.hadoop.hdds.scm.storage.MultipartInputStream.readWithStrategy(MultipartInputStream.java:96)
	at org.apache.hadoop.hdds.scm.storage.ExtendedInputStream.read(ExtendedInputStream.java:66)
	at org.apache.hadoop.crypto.CryptoInputStream.read(CryptoInputStream.java:203)
	at org.apache.hadoop.ozone.client.io.OzoneCryptoInputStream.read(OzoneCryptoInputStream.java:130)
	at org.apache.hadoop.hdds.scm.storage.ByteArrayReader.readFromBlock(ByteArrayReader.java:57)
	at org.apache.hadoop.hdds.scm.storage.MultipartInputStream.readWithStrategy(MultipartInputStream.java:96)
	at org.apache.hadoop.hdds.scm.storage.ExtendedInputStream.read(ExtendedInputStream.java:56)
	at org.apache.hadoop.ozone.client.io.OzoneInputStream.read(OzoneInputStream.java:56)
	at org.apache.hadoop.ozone.client.rpc.TestOzoneAtRestEncryption.testMultipartUploadWithEncryption(TestOzoneAtRestEncryption.java:567)
	at org.apache.hadoop.ozone.client.rpc.TestOzoneAtRestEncryption.testMPUwithOneStreamPart(TestOzoneAtRestEncryption.java:474)

and

TestOzoneAtRestEncryption.testMPUwithOneStreamPart  Time elapsed: 1.263 s  <<< ERROR!
org.apache.hadoop.hdds.scm.container.common.helpers.StorageContainerException: bcsId 22 mismatches with existing block Id 20 for block conID: 9 locID: 111677748019200024 bcsId: 22.
	at org.apache.hadoop.hdds.scm.storage.ContainerProtocolCalls.validateContainerResponse(ContainerProtocolCalls.java:707)
	at org.apache.hadoop.hdds.scm.storage.ContainerProtocolCalls.lambda$createValidators$4(ContainerProtocolCalls.java:718)
	at org.apache.hadoop.hdds.scm.XceiverClientGrpc.sendCommandWithRetry(XceiverClientGrpc.java:407)
	at org.apache.hadoop.hdds.scm.XceiverClientGrpc.lambda$sendCommandWithTraceIDAndRetry$0(XceiverClientGrpc.java:347)
	at org.apache.hadoop.hdds.tracing.TracingUtil.executeInSpan(TracingUtil.java:169)
	at org.apache.hadoop.hdds.tracing.TracingUtil.executeInNewSpan(TracingUtil.java:149)
	at org.apache.hadoop.hdds.scm.XceiverClientGrpc.sendCommandWithTraceIDAndRetry(XceiverClientGrpc.java:342)
	at org.apache.hadoop.hdds.scm.XceiverClientGrpc.sendCommand(XceiverClientGrpc.java:323)
	at org.apache.hadoop.hdds.scm.storage.ContainerProtocolCalls.getBlock(ContainerProtocolCalls.java:208)
	at org.apache.hadoop.hdds.scm.storage.ContainerProtocolCalls.lambda$getBlock$0(ContainerProtocolCalls.java:186)
	at org.apache.hadoop.hdds.scm.storage.ContainerProtocolCalls.tryEachDatanode(ContainerProtocolCalls.java:146)
	at org.apache.hadoop.hdds.scm.storage.ContainerProtocolCalls.getBlock(ContainerProtocolCalls.java:185)
	at org.apache.hadoop.hdds.scm.storage.BlockInputStream.getChunkInfos(BlockInputStream.java:255)
	at org.apache.hadoop.hdds.scm.storage.BlockInputStream.initialize(BlockInputStream.java:146)
	at org.apache.hadoop.hdds.scm.storage.BlockInputStream.readWithStrategy(BlockInputStream.java:308)
	at org.apache.hadoop.hdds.scm.storage.ExtendedInputStream.read(ExtendedInputStream.java:66)
	at org.apache.hadoop.hdds.scm.storage.ByteBufferReader.readFromBlock(ByteBufferReader.java:56)
	at org.apache.hadoop.hdds.scm.storage.MultipartInputStream.readWithStrategy(MultipartInputStream.java:96)
	at org.apache.hadoop.hdds.scm.storage.ExtendedInputStream.read(ExtendedInputStream.java:66)
	at org.apache.hadoop.crypto.CryptoInputStream.read(CryptoInputStream.java:189)
	at org.apache.hadoop.ozone.client.io.OzoneCryptoInputStream.read(OzoneCryptoInputStream.java:130)
	at org.apache.hadoop.hdds.scm.storage.ByteArrayReader.readFromBlock(ByteArrayReader.java:57)
	at org.apache.hadoop.hdds.scm.storage.MultipartInputStream.readWithStrategy(MultipartInputStream.java:96)
	at org.apache.hadoop.hdds.scm.storage.ExtendedInputStream.read(ExtendedInputStream.java:56)
	at org.apache.hadoop.ozone.client.io.OzoneInputStream.read(OzoneInputStream.java:56)
	at org.apache.hadoop.ozone.client.rpc.TestOzoneAtRestEncryption.testMultipartUploadWithEncryption(TestOzoneAtRestEncryption.java:567)
	at org.apache.hadoop.ozone.client.rpc.TestOzoneAtRestEncryption.testMPUwithOneStreamPart(TestOzoneAtRestEncryption.java:474)

• https://github.com/adoroszlai/ozone-build-results/blob/master/2023/06/23/23670/it-client/hadoop-ozone/integration-test/org.apache.hadoop.ozone.client.rpc.TestOzoneAtRestEncryption.txt
• https://github.com/adoroszlai/ozone-build-results/blob/master/2023/06/23/23700/it-client/hadoop-ozone/integration-test/org.apache.hadoop.ozone.client.rpc.TestOzoneAtRestEncryption.txt
• https://github.com/adoroszlai/ozone-build-results/blob/master/2023/06/28/23827/it-client/hadoop-ozone/integration-test/org.apache.hadoop.ozone.client.rpc.TestOzoneAtRestEncryption.txt
• https://github.com/adoroszlai/ozone-build-results/blob/master/2023/07/13/24201/it-client/hadoop-ozone/integration-test/org.apache.hadoop.ozone.client.rpc.TestOzoneAtRestEncryption.txt

Also in testMPUwithTwoStreamParts

• https://github.com/adoroszlai/ozone-build-results/blob/master/2023/06/30/23960/it-client/hadoop-ozone/integration-test/org.apache.hadoop.ozone.client.rpc.TestOzoneAtRestEncryption.txt
• https://github.com/adoroszlai/ozone-build-results/blob/master/2023/07/13/24212/it-client/hadoop-ozone/integration-test/org.apache.hadoop.ozone.client.rpc.TestOzoneAtRestEncryption.txt
• https://github.com/adoroszlai/ozone-build-results/blob/master/2023/07/16/24280/it-client/hadoop-ozone/integration-test/org.apache.hadoop.ozone.client.rpc.TestOzoneAtRestEncryption.txt

Issue Links

• duplicates: HDDS-3425 Enable TestOzoneAtRestEncryption test cases
• is related to: HDDS-5892 RpcClient#createMultipartStreamKey method, need to support file encrypt