[HDDS-10547] Failure to write a file if checksum validation is on

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Fix Version/s: 1.5.0, 1.4.1
• Component/s: Ozone Datanode
• Labels: pull-request-available
• Target Version/s: 1.4.1
• Assignee: Kirill Sizov
• Reporter: Kirill Sizov
• Created: 18/Mar/24 09:50
• Updated: 10/Apr/24 08:13
• Resolved: 25/Mar/24 19:08

Description

If hdds.datanode.chunk.data.validation.check is set to true, any attempt to put a file into Ozone fails.

2024-03-18 10:42:45 2024-03-18 07:42:45 DEBUG FilePerBlockStrategy:155 - Writing chunk ChunkInfo{chunkName='113750153625600001_chunk_1, offset=0, len=174} (overwrite: false) in stage COMBINED to file /data/hdds/hdds/CID-690724fb-e793-4016-ad36-427c2721634a/current/containerDir0/1/chunks/113750153625600001.block
2024-03-18 10:42:45 2024-03-18 07:42:45 ERROR ChunkUtils:424 - Unexpected buffer size. expected: 174, actual: 0
2024-03-18 10:42:45 2024-03-18 07:42:45 WARN  KeyValueHandler:97 - Operation: WriteChunk , Trace ID:  , Message: Unexpected buffer size. expected: 174, actual: 0 , Result: INVALID_WRITE_SIZE , StorageContainerException Occurred.
2024-03-18 10:42:45 org.apache.hadoop.hdds.scm.container.common.helpers.StorageContainerException: Unexpected buffer size. expected: 174, actual: 0
2024-03-18 10:42:45     at org.apache.hadoop.ozone.container.keyvalue.helpers.ChunkUtils.checkSize(ChunkUtils.java:425)
2024-03-18 10:42:45     at org.apache.hadoop.ozone.container.keyvalue.helpers.ChunkUtils.validateBufferSize(ChunkUtils.java:415)
2024-03-18 10:42:45     at org.apache.hadoop.ozone.container.keyvalue.helpers.ChunkUtils.writeData(ChunkUtils.java:125)
2024-03-18 10:42:45     at org.apache.hadoop.ozone.container.keyvalue.helpers.ChunkUtils.writeData(ChunkUtils.java:117)
2024-03-18 10:42:45     at org.apache.hadoop.ozone.container.keyvalue.impl.FilePerBlockStrategy.writeChunk(FilePerBlockStrategy.java:165)
2024-03-18 10:42:45     at org.apache.hadoop.ozone.container.keyvalue.impl.ChunkManagerDispatcher.writeChunk(ChunkManagerDispatcher.java:75)
2024-03-18 10:42:45     at org.apache.hadoop.ozone.container.keyvalue.KeyValueHandler.handleWriteChunk(KeyValueHandler.java:822)
2024-03-18 10:42:45     at org.apache.hadoop.ozone.container.keyvalue.KeyValueHandler.dispatchRequest(KeyValueHandler.java:263)
2024-03-18 10:42:45     at org.apache.hadoop.ozone.container.keyvalue.KeyValueHandler.handle(KeyValueHandler.java:222)
2024-03-18 10:42:45     at org.apache.hadoop.ozone.container.common.impl.HddsDispatcher.dispatchRequest(HddsDispatcher.java:329)
2024-03-18 10:42:45     at org.apache.hadoop.ozone.container.common.impl.HddsDispatcher.lambda$dispatch$0(HddsDispatcher.java:179)
2024-03-18 10:42:45     at org.apache.hadoop.hdds.server.OzoneProtocolMessageDispatcher.processRequest(OzoneProtocolMessageDispatcher.java:89)
2024-03-18 10:42:45     at org.apache.hadoop.ozone.container.common.impl.HddsDispatcher.dispatch(HddsDispatcher.java:178)
2024-03-18 10:42:45     at org.apache.hadoop.ozone.container.common.transport.server.GrpcXceiverService$1.onNext(GrpcXceiverService.java:112)
2024-03-18 10:42:45     at org.apache.hadoop.ozone.container.common.transport.server.GrpcXceiverService$1.onNext(GrpcXceiverService.java:105)
2024-03-18 10:42:45     at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$StreamingServerCallHandler$StreamingServerCallListener.onMessage(ServerCalls.java:262)
2024-03-18 10:42:45     at org.apache.ratis.thirdparty.io.grpc.ForwardingServerCallListener.onMessage(ForwardingServerCallListener.java:33)
2024-03-18 10:42:45     at org.apache.hadoop.hdds.tracing.GrpcServerInterceptor$1.onMessage(GrpcServerInterceptor.java:49)
2024-03-18 10:42:45     at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.messagesAvailableInternal(ServerCallImpl.java:329)
2024-03-18 10:42:45     at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.messagesAvailable(ServerCallImpl.java:314)
2024-03-18 10:42:45     at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1MessagesAvailable.runInContext(ServerImpl.java:833)
2024-03-18 10:42:45     at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
2024-03-18 10:42:45     at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:133)
2024-03-18 10:42:45     at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
2024-03-18 10:42:45     at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
2024-03-18 10:42:45     at java.base/java.lang.Thread.run(Thread.java:829)

This issue was caused by HDDS-9611.
The fix is to restore the buffer posiiton after computing the checksum as the code is further reading from the same buffer but apparently the remaining length is 0.

Issue Links

• is caused by: HDDS-9611 Avoid unnecessary buffer conversion
• links to: GitHub Pull Request #6402