[IGNITE-22280] Error "A critical thread is blocked" on restart

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.0
• Component/s: general
• Labels: ignite-3

Environment

2 nodes (with arguments "-Xms4096m", "-Xmx4096m" ) on 1 host
cpuCount=10
memorySizeMb=15360

• Ignite Flags: Docs Required, Release Notes Required
• Assignee: Unassigned
• Reporter: Igor
• Created: 17/May/24 16:02
• Updated: 04/Sep/24 16:22

Description

Steps to reproduce:

1. Start cluster of 2 nodes on single host.
2. Create 5 tables and insert 1000 rows into each.
3. Kill 1 server.
4. Start the killed server.
5. Check logs for errors.

Expected:

No errors in logs.

Actual:
Errors in logs

2024-05-17 04:26:37:808 +0000 [ERROR][%ClusterFailover2NodesTest_cluster_0%common-scheduler-0][CriticalWorkerWatchdog] A critical thread is blocked for 688 ms that is more than the allowed 500 ms, it is "ClusterFailover2NodesTest_cluster_0-srv-worker-3" prio=10 Id=41 RUNNABLE
    at app//org.apache.ignite.internal.network.message.InvokeResponseDeserializer.getMessage(InvokeResponseDeserializer.java:25)
    at app//org.apache.ignite.internal.network.message.InvokeResponseDeserializer.getMessage(InvokeResponseDeserializer.java:11)
    at app//org.apache.ignite.internal.network.netty.InboundDecoder.decode(InboundDecoder.java:136)
    at app//io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:529)
    at app//io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:468)
    at app//io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290)
    at app//io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
    at app//io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
    at app//io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
    at app//io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
    at app//io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
    at app//io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
    at app//io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
    at app//io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
    at app//io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)
    at app//io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
    at app//io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
    at app//io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
    at app//io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
    at app//io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
    at app//io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    at java.base@17.0.6/java.lang.Thread.run(Thread.java:833) 

GC calls of node ClusterFailover2NodesTest_cluster_0 (LOG: ignite3db-0.log)
GC calls of node ClusterFailover2NodesTest_cluster_1 (LOG: ignite3db-0.log)