[HDDS-11308] ClassCastException during simultaneous putBlock operations on an EC container

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: EC, Ozone Datanode
• Assignee: Alex Juncevich
• Reporter: Ethan Rose
• Created: 12/Aug/24 17:37
• Updated: 05/Nov/24 13:57

Description

Issue

While writing EC data the following exception was observed:

2024-07-30 10:42:22,547 WARN [ChunkReader-8]-org.apache.hadoop.ozone.container.keyvalue.KeyValueHandler: Operation: PutBlock , Trace ID:  , Message: java.lang.ClassCastException: java.util.HashMap$Node cannot be cast to java.util.HashMap$TreeNode , Result: CONTAINER_INTERNAL_ERROR , StorageContainerException Occurred.
org.apache.hadoop.hdds.scm.container.common.helpers.StorageContainerException: java.lang.ClassCastException: java.util.HashMap$Node cannot be cast to java.util.HashMap$TreeNode
	at org.apache.hadoop.ozone.container.keyvalue.KeyValueHandler.handle(KeyValueHandler.java:228)
	at org.apache.hadoop.ozone.container.common.impl.HddsDispatcher.dispatchRequest(HddsDispatcher.java:328)
	at org.apache.hadoop.ozone.container.common.impl.HddsDispatcher.lambda$dispatch$0(HddsDispatcher.java:176)
	at org.apache.hadoop.hdds.server.OzoneProtocolMessageDispatcher.processRequest(OzoneProtocolMessageDispatcher.java:87)
	at org.apache.hadoop.ozone.container.common.impl.HddsDispatcher.dispatch(HddsDispatcher.java:175)
	at org.apache.hadoop.ozone.container.common.transport.server.GrpcXceiverService$1.onNext(GrpcXceiverService.java:57)
	at org.apache.hadoop.ozone.container.common.transport.server.GrpcXceiverService$1.onNext(GrpcXceiverService.java:50)
	at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$StreamingServerCallHandler$StreamingServerCallListener.onMessage(ServerCalls.java:262)
	at org.apache.ratis.thirdparty.io.grpc.ForwardingServerCallListener.onMessage(ForwardingServerCallListener.java:33)
	at org.apache.hadoop.hdds.tracing.GrpcServerInterceptor$1.onMessage(GrpcServerInterceptor.java:49)
	at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.messagesAvailableInternal(ServerCallImpl.java:333)
	at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.messagesAvailable(ServerCallImpl.java:316)
	at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1MessagesAvailable.runInContext(ServerImpl.java:835)
	at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
	at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:133)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:750)
Caused by: java.lang.ClassCastException: java.util.HashMap$Node cannot be cast to java.util.HashMap$TreeNode
	at java.util.HashMap$TreeNode.moveRootToFront(HashMap.java:1859)
	at java.util.HashMap$TreeNode.putTreeVal(HashMap.java:2038)
	at java.util.HashMap.putVal(HashMap.java:639)
	at java.util.HashMap.put(HashMap.java:613)
	at java.util.HashSet.add(HashSet.java:220)
	at org.apache.hadoop.ozone.container.keyvalue.KeyValueContainer.addToPendingPutBlockCache(KeyValueContainer.java:831)
	at org.apache.hadoop.ozone.container.keyvalue.impl.BlockManagerImpl.persistPutBlock(BlockManagerImpl.java:209)
	at org.apache.hadoop.ozone.container.keyvalue.impl.BlockManagerImpl.putBlock(BlockManagerImpl.java:103)
	at org.apache.hadoop.ozone.container.keyvalue.KeyValueHandler.handlePutBlock(KeyValueHandler.java:551)
	at org.apache.hadoop.ozone.container.keyvalue.KeyValueHandler.dispatchRequest(KeyValueHandler.java:254)
	at org.apache.hadoop.ozone.container.keyvalue.KeyValueHandler.handle(KeyValueHandler.java:225)
	... 17 more
2024-07-30 10:42:22,550 WARN [ChunkReader-8]-org.apache.hadoop.ozone.container.keyvalue.KeyValueContainer: Moving container /data/...current/containerDir189/227982 to state UNHEALTHY from state:OPEN

Container 227982 is an EC container, and Ozone was later able to restore all healthy replicas using offline reconstruction.

Cause

There is a putBlock cache for tracking updates to the block count metadata of an open container. The cache is not thread safe so when two threads access it at the same time we may get undefined behavior like the exception shown above.

Explanation

The cache is implemented as a non-concurrent HashSet. The code comments acknowledge this and assume only one instance of putBlock per container will run at a time. This assumption that only one putBlock per container will run at a time is also assumed by the metadata updates done by putBlock, which increment counters in RocksDB and can have incorrect values if two updates happen at the same time (time of check time of use, see HDDS-8129 causing issues like HDDS-5359)

These assumptions were probably in place before EC was added. For Ratis, the ContainerStateMachine enforces that only one putBlock per container will run at a time. For EC, however, there is no such enforcement. EC write requests go through XceiverServerGrpc to GrpcXceiverService where they are handled by a thread pool that assumes it is only handling reads. If two EC clients issue putBlock for different blocks in the same container, we may see:

1. Incorrect metadata counters.
2. Exceptions like the one above that mark the container unhealthy and fail the write.

Based on inspection of the HashSet code, I believe this specific exception will only occur if the two concurrent putBlock IDs hash to the same bin in the set. This explains why the issue has existed for about 2 years but have just seen it for the first time now.