WARNING: An illegal reflective access operation has occurred WARNING: Illegal reflective access by org.apache.ignite.internal.util.GridUnsafe$2 (file:/home/ibessonov/git/apache-ignite-3/modules/core/target/classes/) to field java.nio.Buffer.address WARNING: Please consider reporting this to the maintainers of org.apache.ignite.internal.util.GridUnsafe$2 WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations WARNING: All illegal access operations will be denied in a future release 2021-10-20 14:13:59:051 +0300 [INFO][main][ConnectionManager] Connection created [address=/0:0:0:0:0:0:0:0:10000] 2021-10-20 14:13:59:092 +0300 [INFO][main][Cluster] [null][doStart] Starting, config: ClusterConfig[metadata=null, metadataTimeout=1000, metadataCodec=io.scalecube.cluster.metadata.JdkMetadataCodec@895416d, memberAlias='ITDistributedConfigurationPropertiesTest_testFallingBehindDistributedStorageValue_10000', externalHost='null', externalPort=null, transportConfig=TransportConfig[port=0, isSecured=false, connectTimeout=1000, messageCodec=io.scalecube.cluster.transport.api.JdkMessageCodec@71a06021, maxFrameLength=2097152, transportFactory=org.apache.ignite.network.scalecube.DelegatingTransportFactory@80bfdc6], failureDetectorConfig=FailureDetectorConfig[pingInterval=500, pingTimeout=200, pingReqMembers=1], gossipConfig=GossipConfig[gossipFanout=3, gossipInterval=10, gossipRepeatMult=2, gossipSegmentationThreshold=1000], membershipConfig=MembershipConfig[seedMembers=[127.0.1.1:10000, 127.0.1.1:10001], syncInterval=1000, syncTimeout=3000, suspicionMult=1, namespace='default', removedMembersHistorySize=42]] 2021-10-20 14:13:59:124 +0300 [WARNING][main][MembershipProtocol] [default:ITDistributedConfigurationPropertiesTest_testFallingBehindDistributedStorageValue_10000:b9086737-f68d-474c-a393-759ee8d3f4bc@127.0.1.1:10000] Filtering out seed address: 127.0.1.1:10000 2021-10-20 14:13:59:133 +0300 [INFO][main][MembershipProtocol] [default:ITDistributedConfigurationPropertiesTest_testFallingBehindDistributedStorageValue_10000:b9086737-f68d-474c-a393-759ee8d3f4bc@127.0.1.1:10000] Making initial Sync to all seed members: [127.0.1.1:10001] 2021-10-20 14:13:59:227 +0300 [INFO][sc-cluster-10000-1][Cluster] [default:ITDistributedConfigurationPropertiesTest_testFallingBehindDistributedStorageValue_10000:b9086737-f68d-474c-a393-759ee8d3f4bc@127.0.1.1:10000][doStart] Started 2021-10-20 14:13:59:235 +0300 [INFO][main][ScaleCubeTopologyService] Node joined: ClusterNode [id=b9086737-f68d-474c-a393-759ee8d3f4bc, name=ITDistributedConfigurationPropertiesTest_testFallingBehindDistributedStorageValue_10000, address=127.0.1.1:10000] 2021-10-20 14:13:59:236 +0300 [INFO][main][ScaleCubeTopologyService] Topology snapshot [nodes=1] ^-- ClusterNode [id=b9086737-f68d-474c-a393-759ee8d3f4bc, name=ITDistributedConfigurationPropertiesTest_testFallingBehindDistributedStorageValue_10000, address=127.0.1.1:10000] 2021-10-20 14:13:59:295 +0300 [INFO][main][StripedDisruptor] Striped disruptor was started [name=%ITDistributedConfigurationPropertiesTest_testFallingBehindDistributedStorageValue_10000%JRaft-FSMCaller-Disruptor-] 2021-10-20 14:13:59:325 +0300 [INFO][main][StripedDisruptor] Striped disruptor was started [name=%ITDistributedConfigurationPropertiesTest_testFallingBehindDistributedStorageValue_10000%JRaft-NodeImpl-Disruptor-] 2021-10-20 14:13:59:338 +0300 [INFO][main][StripedDisruptor] Striped disruptor was started [name=%ITDistributedConfigurationPropertiesTest_testFallingBehindDistributedStorageValue_10000%JRaft-ReadOnlyService-Disruptor-] 2021-10-20 14:13:59:344 +0300 [INFO][main][StripedDisruptor] Striped disruptor was started [name=%ITDistributedConfigurationPropertiesTest_testFallingBehindDistributedStorageValue_10000%JRaft-LogManager-Disruptor-] 2021-10-20 14:13:59:364 +0300 [INFO][main][StripedDisruptor] Consumer subscribed [poolName=%ITDistributedConfigurationPropertiesTest_testFallingBehindDistributedStorageValue_10000%JRaft-NodeImpl-Disruptor-, group=metastorage_raft_group] 2021-10-20 14:13:59:571 +0300 [INFO][main][StripedDisruptor] Consumer subscribed [poolName=%ITDistributedConfigurationPropertiesTest_testFallingBehindDistributedStorageValue_10000%JRaft-LogManager-Disruptor-, group=metastorage_raft_group] 2021-10-20 14:13:59:587 +0300 [INFO][main][StripedDisruptor] Consumer subscribed [poolName=%ITDistributedConfigurationPropertiesTest_testFallingBehindDistributedStorageValue_10000%JRaft-FSMCaller-Disruptor-, group=metastorage_raft_group] 2021-10-20 14:13:59:589 +0300 [INFO][main][FSMCallerImpl] Starts FSMCaller successfully. 2021-10-20 14:13:59:601 +0300 [WARNING][main][LocalSnapshotStorage] No data for snapshot reader target/work/ITDistributedConfigurationPropertiesTest/testFallingBehindDistributedStorageValue_1634728438053/firstNode/metastorage_raft_group_127.0.1.1_10000/snapshot. 2021-10-20 14:13:59:607 +0300 [INFO][main][StripedDisruptor] Consumer subscribed [poolName=%ITDistributedConfigurationPropertiesTest_testFallingBehindDistributedStorageValue_10000%JRaft-ReadOnlyService-Disruptor-, group=metastorage_raft_group] 2021-10-20 14:13:59:611 +0300 [INFO][main][NodeImpl] Node init, term=0, lastLogId=LogId [index=0, term=0], conf=127.0.1.1:10000, oldConf=. 2021-10-20 14:13:59:613 +0300 [INFO][main][NodeImpl] Node start vote and grant vote self, term=0. 2021-10-20 14:13:59:623 +0300 [INFO][main][LocalRaftMetaStorage] Save raft meta, path=target/work/ITDistributedConfigurationPropertiesTest/testFallingBehindDistributedStorageValue_1634728438053/firstNode/metastorage_raft_group_127.0.1.1_10000/meta, term=1, votedFor=127.0.1.1:10000, cost time=9 ms 2021-10-20 14:13:59:624 +0300 [INFO][main][NodeImpl] Node become leader of group, term=1, conf=127.0.1.1:10000, oldConf=. 2021-10-20 14:13:59:627 +0300 [INFO][main][Recyclers] -Djraft.recyclers.maxCapacityPerThread: 4096. 2021-10-20 14:13:59:630 +0300 [INFO][main][RaftGroupService] Start the RaftGroupService successfully 2021-10-20 14:13:59:639 +0300 [INFO][%ITDistributedConfigurationPropertiesTest_testFallingBehindDistributedStorageValue_10000%JRaft-FSMCaller-Disruptor-_stripe_10-0][StateMachineAdapter] onLeaderStart: term=1. 2021-10-20 14:13:59:658 +0300 [INFO][main][ConnectionManager] Connection created [address=/0:0:0:0:0:0:0:0:10001] 2021-10-20 14:13:59:659 +0300 [INFO][main][Cluster] [null][doStart] Starting, config: ClusterConfig[metadata=null, metadataTimeout=1000, metadataCodec=io.scalecube.cluster.metadata.JdkMetadataCodec@895416d, memberAlias='ITDistributedConfigurationPropertiesTest_testFallingBehindDistributedStorageValue_10001', externalHost='null', externalPort=null, transportConfig=TransportConfig[port=0, isSecured=false, connectTimeout=1000, messageCodec=io.scalecube.cluster.transport.api.JdkMessageCodec@71a06021, maxFrameLength=2097152, transportFactory=org.apache.ignite.network.scalecube.DelegatingTransportFactory@2f7efd0b], failureDetectorConfig=FailureDetectorConfig[pingInterval=500, pingTimeout=200, pingReqMembers=1], gossipConfig=GossipConfig[gossipFanout=3, gossipInterval=10, gossipRepeatMult=2, gossipSegmentationThreshold=1000], membershipConfig=MembershipConfig[seedMembers=[127.0.1.1:10000, 127.0.1.1:10001], syncInterval=1000, syncTimeout=3000, suspicionMult=1, namespace='default', removedMembersHistorySize=42]] 2021-10-20 14:13:59:664 +0300 [WARNING][main][MembershipProtocol] [default:ITDistributedConfigurationPropertiesTest_testFallingBehindDistributedStorageValue_10001:33b837ef-efdf-402d-9551-b3eff77ae6ff@127.0.1.1:10001] Filtering out seed address: 127.0.1.1:10001 2021-10-20 14:13:59:666 +0300 [INFO][main][MembershipProtocol] [default:ITDistributedConfigurationPropertiesTest_testFallingBehindDistributedStorageValue_10001:33b837ef-efdf-402d-9551-b3eff77ae6ff@127.0.1.1:10001] Making initial Sync to all seed members: [127.0.1.1:10000] 2021-10-20 14:13:59:758 +0300 [INFO][sc-cluster-10000-1][MembershipProtocol] [default:ITDistributedConfigurationPropertiesTest_testFallingBehindDistributedStorageValue_10000:b9086737-f68d-474c-a393-759ee8d3f4bc@127.0.1.1:10000][publishEvent] MembershipEvent[type=ADDED, member=default:ITDistributedConfigurationPropertiesTest_testFallingBehindDistributedStorageValue_10001:33b837ef-efdf-402d-9551-b3eff77ae6ff@127.0.1.1:10001, oldMetadata=null, newMetadata=7e16449-5, timestamp=2021-10-20T11:13:59.758Z] 2021-10-20 14:13:59:758 +0300 [INFO][sc-cluster-10000-1][ScaleCubeTopologyService] Node joined: ClusterNode [id=33b837ef-efdf-402d-9551-b3eff77ae6ff, name=ITDistributedConfigurationPropertiesTest_testFallingBehindDistributedStorageValue_10001, address=127.0.1.1:10001] 2021-10-20 14:13:59:759 +0300 [INFO][sc-cluster-10000-1][ScaleCubeTopologyService] Topology snapshot [nodes=2] ^-- ClusterNode [id=b9086737-f68d-474c-a393-759ee8d3f4bc, name=ITDistributedConfigurationPropertiesTest_testFallingBehindDistributedStorageValue_10000, address=127.0.1.1:10000] ^-- ClusterNode [id=33b837ef-efdf-402d-9551-b3eff77ae6ff, name=ITDistributedConfigurationPropertiesTest_testFallingBehindDistributedStorageValue_10001, address=127.0.1.1:10001] 2021-10-20 14:13:59:773 +0300 [INFO][sc-cluster-10001-2][MembershipProtocol] [default:ITDistributedConfigurationPropertiesTest_testFallingBehindDistributedStorageValue_10001:33b837ef-efdf-402d-9551-b3eff77ae6ff@127.0.1.1:10001][publishEvent] MembershipEvent[type=ADDED, member=default:ITDistributedConfigurationPropertiesTest_testFallingBehindDistributedStorageValue_10000:b9086737-f68d-474c-a393-759ee8d3f4bc@127.0.1.1:10000, oldMetadata=null, newMetadata=7e16449-5, timestamp=2021-10-20T11:13:59.773Z] 2021-10-20 14:13:59:773 +0300 [INFO][sc-cluster-10001-2][ScaleCubeTopologyService] Node joined: ClusterNode [id=b9086737-f68d-474c-a393-759ee8d3f4bc, name=ITDistributedConfigurationPropertiesTest_testFallingBehindDistributedStorageValue_10000, address=127.0.1.1:10000] 2021-10-20 14:13:59:773 +0300 [INFO][sc-cluster-10001-2][ScaleCubeTopologyService] Topology snapshot [nodes=1] ^-- ClusterNode [id=b9086737-f68d-474c-a393-759ee8d3f4bc, name=ITDistributedConfigurationPropertiesTest_testFallingBehindDistributedStorageValue_10000, address=127.0.1.1:10000] 2021-10-20 14:13:59:774 +0300 [INFO][sc-cluster-10001-2][Cluster] [default:ITDistributedConfigurationPropertiesTest_testFallingBehindDistributedStorageValue_10001:33b837ef-efdf-402d-9551-b3eff77ae6ff@127.0.1.1:10001][doStart] Started 2021-10-20 14:13:59:774 +0300 [INFO][main][ScaleCubeTopologyService] Node joined: ClusterNode [id=33b837ef-efdf-402d-9551-b3eff77ae6ff, name=ITDistributedConfigurationPropertiesTest_testFallingBehindDistributedStorageValue_10001, address=127.0.1.1:10001] 2021-10-20 14:13:59:775 +0300 [INFO][main][ScaleCubeTopologyService] Topology snapshot [nodes=2] ^-- ClusterNode [id=b9086737-f68d-474c-a393-759ee8d3f4bc, name=ITDistributedConfigurationPropertiesTest_testFallingBehindDistributedStorageValue_10000, address=127.0.1.1:10000] ^-- ClusterNode [id=33b837ef-efdf-402d-9551-b3eff77ae6ff, name=ITDistributedConfigurationPropertiesTest_testFallingBehindDistributedStorageValue_10001, address=127.0.1.1:10001] 2021-10-20 14:13:59:827 +0300 [INFO][main][StripedDisruptor] Striped disruptor was started [name=%ITDistributedConfigurationPropertiesTest_testFallingBehindDistributedStorageValue_10001%JRaft-FSMCaller-Disruptor-] 2021-10-20 14:13:59:835 +0300 [INFO][main][StripedDisruptor] Striped disruptor was started [name=%ITDistributedConfigurationPropertiesTest_testFallingBehindDistributedStorageValue_10001%JRaft-NodeImpl-Disruptor-] 2021-10-20 14:13:59:842 +0300 [INFO][main][StripedDisruptor] Striped disruptor was started [name=%ITDistributedConfigurationPropertiesTest_testFallingBehindDistributedStorageValue_10001%JRaft-ReadOnlyService-Disruptor-] 2021-10-20 14:13:59:849 +0300 [INFO][main][StripedDisruptor] Striped disruptor was started [name=%ITDistributedConfigurationPropertiesTest_testFallingBehindDistributedStorageValue_10001%JRaft-LogManager-Disruptor-] 2021-10-20 14:14:00:107 +0300 [INFO][main][NodeImpl] Node shutdown, currTerm=1 state=STATE_LEADER. 2021-10-20 14:14:00:108 +0300 [INFO][main][ReplicatorGroupImpl] Fail to find the next candidate. 2021-10-20 14:14:00:119 +0300 [INFO][%ITDistributedConfigurationPropertiesTest_testFallingBehindDistributedStorageValue_10000%JRaft-FSMCaller-Disruptor-_stripe_10-0][StateMachineAdapter] onLeaderStop: status=Status[ESHUTDOWN<1007>: Raft node is going to quit.]. 2021-10-20 14:14:00:112 +0300 [ERROR][%ITDistributedConfigurationPropertiesTest_testFallingBehindDistributedStorageValue_10000%JRaft-Request-Processor-17][LogThreadPoolExecutor] Uncaught exception in pool: %ITDistributedConfigurationPropertiesTest_testFallingBehindDistributedStorageValue_10000%JRaft-Request-Processor-, org.apache.ignite.raft.jraft.util.MetricThreadPoolExecutor@4769ac34[Running, pool size = 48, active threads = 1, queued tasks = 0, completed tasks = 66]. java.lang.IllegalStateException: Service already shutdown. at org.apache.ignite.raft.jraft.core.ReadOnlyServiceImpl.addRequest(ReadOnlyServiceImpl.java:307) at org.apache.ignite.raft.jraft.core.NodeImpl.readIndex(NodeImpl.java:1413) at org.apache.ignite.raft.jraft.rpc.impl.ActionRequestProcessor.handleRequest(ActionRequestProcessor.java:85) at org.apache.ignite.raft.jraft.rpc.impl.ActionRequestProcessor.handleRequest(ActionRequestProcessor.java:49) at org.apache.ignite.raft.jraft.rpc.impl.IgniteRpcServer$RpcMessageHandler.lambda$onReceived$0(IgniteRpcServer.java:182) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) 2021-10-20 14:14:00:120 +0300 [ERROR][Thread-145][MetaStorageServiceImpl] Unexpected exception class org.apache.ignite.lang.IgniteInternalException: class org.apache.ignite.raft.jraft.rpc.impl.RaftException: EHOSTDOWN:Was stopped at org.apache.ignite.internal.metastorage.client.CursorImpl$InnerIterator.hasNext(CursorImpl.java:112) at org.apache.ignite.internal.metastorage.client.MetaStorageServiceImpl$WatchProcessor$Watcher.run(MetaStorageServiceImpl.java:461) Caused by: java.util.concurrent.ExecutionException: class org.apache.ignite.raft.jraft.rpc.impl.RaftException: EHOSTDOWN:Was stopped at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:395) at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1999) at org.apache.ignite.internal.metastorage.client.CursorImpl$InnerIterator.hasNext(CursorImpl.java:104) ... 1 more Caused by: class org.apache.ignite.raft.jraft.rpc.impl.RaftException: EHOSTDOWN:Was stopped at org.apache.ignite.raft.jraft.rpc.impl.RaftGroupServiceImpl$1.accept(RaftGroupServiceImpl.java:547) at org.apache.ignite.raft.jraft.rpc.impl.RaftGroupServiceImpl$1.accept(RaftGroupServiceImpl.java:494) at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859) at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837) at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2073) at reactor.core.publisher.MonoToCompletableFuture.onNext(MonoToCompletableFuture.java:63) at reactor.core.publisher.SerializedSubscriber.onNext(SerializedSubscriber.java:99) at reactor.core.publisher.FluxTimeout$TimeoutMainSubscriber.onNext(FluxTimeout.java:174) at reactor.core.publisher.MonoCreate$DefaultMonoSink.success(MonoCreate.java:156) at reactor.core.publisher.LambdaSubscriber.onNext(LambdaSubscriber.java:160) at reactor.core.publisher.FluxTake$TakeFuseableSubscriber.onNext(FluxTake.java:398) at reactor.core.publisher.FluxFilterFuseable$FilterFuseableSubscriber.onNext(FluxFilterFuseable.java:112) at reactor.core.publisher.FluxFilterFuseable$FilterFuseableConditionalSubscriber.onNext(FluxFilterFuseable.java:330) at reactor.core.publisher.FluxOnBackpressureBuffer$BackpressureBufferSubscriber.drainRegular(FluxOnBackpressureBuffer.java:261) at reactor.core.publisher.FluxOnBackpressureBuffer$BackpressureBufferSubscriber.drain(FluxOnBackpressureBuffer.java:225) at reactor.core.publisher.FluxOnBackpressureBuffer$BackpressureBufferSubscriber.onNext(FluxOnBackpressureBuffer.java:184) at reactor.core.publisher.DirectProcessor$DirectInner.onNext(DirectProcessor.java:333) at reactor.core.publisher.DirectProcessor.onNext(DirectProcessor.java:142) at reactor.core.publisher.FluxCreate$IgnoreSink.next(FluxCreate.java:618) at reactor.core.publisher.FluxCreate$SerializedSink.next(FluxCreate.java:153) at org.apache.ignite.network.scalecube.ScaleCubeDirectMarshallerTransport.onMessage(ScaleCubeDirectMarshallerTransport.java:190) at org.apache.ignite.internal.network.netty.ConnectionManager.lambda$onMessage$2(ConnectionManager.java:203) at java.base/java.util.concurrent.CopyOnWriteArrayList.forEach(CopyOnWriteArrayList.java:807) at org.apache.ignite.internal.network.netty.ConnectionManager.onMessage(ConnectionManager.java:203) at org.apache.ignite.internal.network.netty.MessageHandler.channelRead(MessageHandler.java:47) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:324) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:296) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:719) at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:655) at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:581) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Thread.java:829) 2021-10-20 14:14:00:122 +0300 [ERROR][%ITDistributedConfigurationPropertiesTest_testFallingBehindDistributedStorageValue_10000%JRaft-Request-Processor-17][NamedThreadFactory] Uncaught exception in thread Thread[%ITDistributedConfigurationPropertiesTest_testFallingBehindDistributedStorageValue_10000%JRaft-Request-Processor-17,5,main] java.lang.IllegalStateException: Service already shutdown. at org.apache.ignite.raft.jraft.core.ReadOnlyServiceImpl.addRequest(ReadOnlyServiceImpl.java:307) at org.apache.ignite.raft.jraft.core.NodeImpl.readIndex(NodeImpl.java:1413) at org.apache.ignite.raft.jraft.rpc.impl.ActionRequestProcessor.handleRequest(ActionRequestProcessor.java:85) at org.apache.ignite.raft.jraft.rpc.impl.ActionRequestProcessor.handleRequest(ActionRequestProcessor.java:49) at org.apache.ignite.raft.jraft.rpc.impl.IgniteRpcServer$RpcMessageHandler.lambda$onReceived$0(IgniteRpcServer.java:182) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) 2021-10-20 14:14:00:125 +0300 [ERROR][%ITDistributedConfigurationPropertiesTest_testFallingBehindDistributedStorageValue_10000%JRaft-Request-Processor-19][LogThreadPoolExecutor] Uncaught exception in pool: %ITDistributedConfigurationPropertiesTest_testFallingBehindDistributedStorageValue_10000%JRaft-Request-Processor-, org.apache.ignite.raft.jraft.util.MetricThreadPoolExecutor@4769ac34[Running, pool size = 48, active threads = 1, queued tasks = 0, completed tasks = 67]. java.lang.IllegalStateException: Service already shutdown. at org.apache.ignite.raft.jraft.core.ReadOnlyServiceImpl.addRequest(ReadOnlyServiceImpl.java:307) at org.apache.ignite.raft.jraft.core.NodeImpl.readIndex(NodeImpl.java:1413) at org.apache.ignite.raft.jraft.rpc.impl.ActionRequestProcessor.handleRequest(ActionRequestProcessor.java:85) at org.apache.ignite.raft.jraft.rpc.impl.ActionRequestProcessor.handleRequest(ActionRequestProcessor.java:49) at org.apache.ignite.raft.jraft.rpc.impl.IgniteRpcServer$RpcMessageHandler.lambda$onReceived$0(IgniteRpcServer.java:182) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) 2021-10-20 14:14:00:126 +0300 [ERROR][%ITDistributedConfigurationPropertiesTest_testFallingBehindDistributedStorageValue_10000%JRaft-Request-Processor-19][NamedThreadFactory] Uncaught exception in thread Thread[%ITDistributedConfigurationPropertiesTest_testFallingBehindDistributedStorageValue_10000%JRaft-Request-Processor-19,10,main] java.lang.IllegalStateException: Service already shutdown. at org.apache.ignite.raft.jraft.core.ReadOnlyServiceImpl.addRequest(ReadOnlyServiceImpl.java:307) at org.apache.ignite.raft.jraft.core.NodeImpl.readIndex(NodeImpl.java:1413) at org.apache.ignite.raft.jraft.rpc.impl.ActionRequestProcessor.handleRequest(ActionRequestProcessor.java:85) at org.apache.ignite.raft.jraft.rpc.impl.ActionRequestProcessor.handleRequest(ActionRequestProcessor.java:49) at org.apache.ignite.raft.jraft.rpc.impl.IgniteRpcServer$RpcMessageHandler.lambda$onReceived$0(IgniteRpcServer.java:182) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) 2021-10-20 14:14:00:126 +0300 [ERROR][Thread-145][MetaStorageServiceImpl] Unexpected exception class org.apache.ignite.lang.IgniteInternalException: class org.apache.ignite.raft.jraft.rpc.impl.RaftException: EHOSTDOWN:Was stopped at org.apache.ignite.internal.metastorage.client.CursorImpl$InnerIterator.hasNext(CursorImpl.java:112) at org.apache.ignite.internal.metastorage.client.MetaStorageServiceImpl$WatchProcessor$Watcher.run(MetaStorageServiceImpl.java:461) Caused by: java.util.concurrent.ExecutionException: class org.apache.ignite.raft.jraft.rpc.impl.RaftException: EHOSTDOWN:Was stopped at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:395) at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1999) at org.apache.ignite.internal.metastorage.client.CursorImpl$InnerIterator.hasNext(CursorImpl.java:104) ... 1 more Caused by: class org.apache.ignite.raft.jraft.rpc.impl.RaftException: EHOSTDOWN:Was stopped at org.apache.ignite.raft.jraft.rpc.impl.RaftGroupServiceImpl$1.accept(RaftGroupServiceImpl.java:547) at org.apache.ignite.raft.jraft.rpc.impl.RaftGroupServiceImpl$1.accept(RaftGroupServiceImpl.java:494) at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859) at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837) at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2073) at reactor.core.publisher.MonoToCompletableFuture.onNext(MonoToCompletableFuture.java:63) at reactor.core.publisher.SerializedSubscriber.onNext(SerializedSubscriber.java:99) at reactor.core.publisher.FluxTimeout$TimeoutMainSubscriber.onNext(FluxTimeout.java:174) at reactor.core.publisher.MonoCreate$DefaultMonoSink.success(MonoCreate.java:156) at reactor.core.publisher.LambdaSubscriber.onNext(LambdaSubscriber.java:160) at reactor.core.publisher.FluxTake$TakeFuseableSubscriber.onNext(FluxTake.java:398) at reactor.core.publisher.FluxFilterFuseable$FilterFuseableSubscriber.onNext(FluxFilterFuseable.java:112) at reactor.core.publisher.FluxFilterFuseable$FilterFuseableConditionalSubscriber.onNext(FluxFilterFuseable.java:330) at reactor.core.publisher.FluxOnBackpressureBuffer$BackpressureBufferSubscriber.drainRegular(FluxOnBackpressureBuffer.java:261) at reactor.core.publisher.FluxOnBackpressureBuffer$BackpressureBufferSubscriber.drain(FluxOnBackpressureBuffer.java:225) at reactor.core.publisher.FluxOnBackpressureBuffer$BackpressureBufferSubscriber.onNext(FluxOnBackpressureBuffer.java:184) at reactor.core.publisher.DirectProcessor$DirectInner.onNext(DirectProcessor.java:333) at reactor.core.publisher.DirectProcessor.onNext(DirectProcessor.java:142) at reactor.core.publisher.FluxCreate$IgnoreSink.next(FluxCreate.java:618) at reactor.core.publisher.FluxCreate$SerializedSink.next(FluxCreate.java:153) at org.apache.ignite.network.scalecube.ScaleCubeDirectMarshallerTransport.onMessage(ScaleCubeDirectMarshallerTransport.java:190) at org.apache.ignite.internal.network.netty.ConnectionManager.lambda$onMessage$2(ConnectionManager.java:203) at java.base/java.util.concurrent.CopyOnWriteArrayList.forEach(CopyOnWriteArrayList.java:807) at org.apache.ignite.internal.network.netty.ConnectionManager.onMessage(ConnectionManager.java:203) at org.apache.ignite.internal.network.netty.MessageHandler.channelRead(MessageHandler.java:47) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:324) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:296) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:719) at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:655) at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:581) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Thread.java:829) 2021-10-20 14:14:00:130 +0300 [INFO][main][LocalRaftMetaStorage] Save raft meta, path=target/work/ITDistributedConfigurationPropertiesTest/testFallingBehindDistributedStorageValue_1634728438053/firstNode/metastorage_raft_group_127.0.1.1_10000/meta, term=1, votedFor=127.0.1.1:10000, cost time=10 ms 2021-10-20 14:14:00:130 +0300 [INFO][main][FSMCallerImpl] Shutting down FSMCaller... 2021-10-20 14:14:00:132 +0300 [ERROR][%ITDistributedConfigurationPropertiesTest_testFallingBehindDistributedStorageValue_10000%JRaft-Request-Processor-20][LogThreadPoolExecutor] Uncaught exception in pool: %ITDistributedConfigurationPropertiesTest_testFallingBehindDistributedStorageValue_10000%JRaft-Request-Processor-, org.apache.ignite.raft.jraft.util.MetricThreadPoolExecutor@4769ac34[Running, pool size = 48, active threads = 1, queued tasks = 0, completed tasks = 68]. java.lang.IllegalStateException: Service already shutdown. at org.apache.ignite.raft.jraft.core.ReadOnlyServiceImpl.addRequest(ReadOnlyServiceImpl.java:307) at org.apache.ignite.raft.jraft.core.NodeImpl.readIndex(NodeImpl.java:1413) at org.apache.ignite.raft.jraft.rpc.impl.ActionRequestProcessor.handleRequest(ActionRequestProcessor.java:85) at org.apache.ignite.raft.jraft.rpc.impl.ActionRequestProcessor.handleRequest(ActionRequestProcessor.java:49) at org.apache.ignite.raft.jraft.rpc.impl.IgniteRpcServer$RpcMessageHandler.lambda$onReceived$0(IgniteRpcServer.java:182) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) 2021-10-20 14:14:00:133 +0300 [ERROR][%ITDistributedConfigurationPropertiesTest_testFallingBehindDistributedStorageValue_10000%JRaft-Request-Processor-20][NamedThreadFactory] Uncaught exception in thread Thread[%ITDistributedConfigurationPropertiesTest_testFallingBehindDistributedStorageValue_10000%JRaft-Request-Processor-20,10,main] java.lang.IllegalStateException: Service already shutdown. at org.apache.ignite.raft.jraft.core.ReadOnlyServiceImpl.addRequest(ReadOnlyServiceImpl.java:307) at org.apache.ignite.raft.jraft.core.NodeImpl.readIndex(NodeImpl.java:1413) at org.apache.ignite.raft.jraft.rpc.impl.ActionRequestProcessor.handleRequest(ActionRequestProcessor.java:85) at org.apache.ignite.raft.jraft.rpc.impl.ActionRequestProcessor.handleRequest(ActionRequestProcessor.java:49) at org.apache.ignite.raft.jraft.rpc.impl.IgniteRpcServer$RpcMessageHandler.lambda$onReceived$0(IgniteRpcServer.java:182) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) 2021-10-20 14:14:00:135 +0300 [ERROR][Thread-145][MetaStorageServiceImpl] Unexpected exception class org.apache.ignite.lang.IgniteInternalException: class org.apache.ignite.raft.jraft.rpc.impl.RaftException: EHOSTDOWN:Was stopped at org.apache.ignite.internal.metastorage.client.CursorImpl$InnerIterator.hasNext(CursorImpl.java:112) at org.apache.ignite.internal.metastorage.client.MetaStorageServiceImpl$WatchProcessor$Watcher.run(MetaStorageServiceImpl.java:461) Caused by: java.util.concurrent.ExecutionException: class org.apache.ignite.raft.jraft.rpc.impl.RaftException: EHOSTDOWN:Was stopped at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:395) at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1999) at org.apache.ignite.internal.metastorage.client.CursorImpl$InnerIterator.hasNext(CursorImpl.java:104) ... 1 more Caused by: class org.apache.ignite.raft.jraft.rpc.impl.RaftException: EHOSTDOWN:Was stopped at org.apache.ignite.raft.jraft.rpc.impl.RaftGroupServiceImpl$1.accept(RaftGroupServiceImpl.java:547) at org.apache.ignite.raft.jraft.rpc.impl.RaftGroupServiceImpl$1.accept(RaftGroupServiceImpl.java:494) at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859) at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837) at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2073) at reactor.core.publisher.MonoToCompletableFuture.onNext(MonoToCompletableFuture.java:63) at reactor.core.publisher.SerializedSubscriber.onNext(SerializedSubscriber.java:99) at reactor.core.publisher.FluxTimeout$TimeoutMainSubscriber.onNext(FluxTimeout.java:174) at reactor.core.publisher.MonoCreate$DefaultMonoSink.success(MonoCreate.java:156) at reactor.core.publisher.LambdaSubscriber.onNext(LambdaSubscriber.java:160) at reactor.core.publisher.FluxTake$TakeFuseableSubscriber.onNext(FluxTake.java:398) at reactor.core.publisher.FluxFilterFuseable$FilterFuseableSubscriber.onNext(FluxFilterFuseable.java:112) at reactor.core.publisher.FluxFilterFuseable$FilterFuseableConditionalSubscriber.onNext(FluxFilterFuseable.java:330) at reactor.core.publisher.FluxOnBackpressureBuffer$BackpressureBufferSubscriber.drainRegular(FluxOnBackpressureBuffer.java:261) at reactor.core.publisher.FluxOnBackpressureBuffer$BackpressureBufferSubscriber.drain(FluxOnBackpressureBuffer.java:225) at reactor.core.publisher.FluxOnBackpressureBuffer$BackpressureBufferSubscriber.onNext(FluxOnBackpressureBuffer.java:184) at reactor.core.publisher.DirectProcessor$DirectInner.onNext(DirectProcessor.java:333) at reactor.core.publisher.DirectProcessor.onNext(DirectProcessor.java:142) at reactor.core.publisher.FluxCreate$IgnoreSink.next(FluxCreate.java:618) at reactor.core.publisher.FluxCreate$SerializedSink.next(FluxCreate.java:153) at org.apache.ignite.network.scalecube.ScaleCubeDirectMarshallerTransport.onMessage(ScaleCubeDirectMarshallerTransport.java:190) at org.apache.ignite.internal.network.netty.ConnectionManager.lambda$onMessage$2(ConnectionManager.java:203) at java.base/java.util.concurrent.CopyOnWriteArrayList.forEach(CopyOnWriteArrayList.java:807) at org.apache.ignite.internal.network.netty.ConnectionManager.onMessage(ConnectionManager.java:203) at org.apache.ignite.internal.network.netty.MessageHandler.channelRead(MessageHandler.java:47) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:324) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:296) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:719) at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:655) at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:581) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Thread.java:829) 2021-10-20 14:14:00:138 +0300 [ERROR][%ITDistributedConfigurationPropertiesTest_testFallingBehindDistributedStorageValue_10000%JRaft-Request-Processor-21][LogThreadPoolExecutor] Uncaught exception in pool: %ITDistributedConfigurationPropertiesTest_testFallingBehindDistributedStorageValue_10000%JRaft-Request-Processor-, org.apache.ignite.raft.jraft.util.MetricThreadPoolExecutor@4769ac34[Running, pool size = 48, active threads = 1, queued tasks = 0, completed tasks = 69]. java.lang.IllegalStateException: Node is shutting down at org.apache.ignite.raft.jraft.core.NodeImpl.readIndex(NodeImpl.java:1410) at org.apache.ignite.raft.jraft.rpc.impl.ActionRequestProcessor.handleRequest(ActionRequestProcessor.java:85) at org.apache.ignite.raft.jraft.rpc.impl.ActionRequestProcessor.handleRequest(ActionRequestProcessor.java:49) at org.apache.ignite.raft.jraft.rpc.impl.IgniteRpcServer$RpcMessageHandler.lambda$onReceived$0(IgniteRpcServer.java:182) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) 2021-10-20 14:14:00:139 +0300 [ERROR][%ITDistributedConfigurationPropertiesTest_testFallingBehindDistributedStorageValue_10000%JRaft-Request-Processor-21][NamedThreadFactory] Uncaught exception in thread Thread[%ITDistributedConfigurationPropertiesTest_testFallingBehindDistributedStorageValue_10000%JRaft-Request-Processor-21,10,main] java.lang.IllegalStateException: Node is shutting down at org.apache.ignite.raft.jraft.core.NodeImpl.readIndex(NodeImpl.java:1410) at org.apache.ignite.raft.jraft.rpc.impl.ActionRequestProcessor.handleRequest(ActionRequestProcessor.java:85) at org.apache.ignite.raft.jraft.rpc.impl.ActionRequestProcessor.handleRequest(ActionRequestProcessor.java:49) at org.apache.ignite.raft.jraft.rpc.impl.IgniteRpcServer$RpcMessageHandler.lambda$onReceived$0(IgniteRpcServer.java:182) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) 2021-10-20 14:14:00:139 +0300 [ERROR][Thread-145][MetaStorageServiceImpl] Unexpected exception class org.apache.ignite.lang.IgniteInternalException: class org.apache.ignite.raft.jraft.rpc.impl.RaftException: ENODESHUTDOWN:Node is shutting down. at org.apache.ignite.internal.metastorage.client.CursorImpl$InnerIterator.hasNext(CursorImpl.java:112) at org.apache.ignite.internal.metastorage.client.MetaStorageServiceImpl$WatchProcessor$Watcher.run(MetaStorageServiceImpl.java:461) Caused by: java.util.concurrent.ExecutionException: class org.apache.ignite.raft.jraft.rpc.impl.RaftException: ENODESHUTDOWN:Node is shutting down. at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:395) at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1999) at org.apache.ignite.internal.metastorage.client.CursorImpl$InnerIterator.hasNext(CursorImpl.java:104) ... 1 more Caused by: class org.apache.ignite.raft.jraft.rpc.impl.RaftException: ENODESHUTDOWN:Node is shutting down. at org.apache.ignite.raft.jraft.rpc.impl.RaftGroupServiceImpl$1.accept(RaftGroupServiceImpl.java:547) at org.apache.ignite.raft.jraft.rpc.impl.RaftGroupServiceImpl$1.accept(RaftGroupServiceImpl.java:494) at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859) at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837) at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2073) at reactor.core.publisher.MonoToCompletableFuture.onNext(MonoToCompletableFuture.java:63) at reactor.core.publisher.SerializedSubscriber.onNext(SerializedSubscriber.java:99) at reactor.core.publisher.FluxTimeout$TimeoutMainSubscriber.onNext(FluxTimeout.java:174) at reactor.core.publisher.MonoCreate$DefaultMonoSink.success(MonoCreate.java:156) at reactor.core.publisher.LambdaSubscriber.onNext(LambdaSubscriber.java:160) at reactor.core.publisher.FluxTake$TakeFuseableSubscriber.onNext(FluxTake.java:398) at reactor.core.publisher.FluxFilterFuseable$FilterFuseableSubscriber.onNext(FluxFilterFuseable.java:112) at reactor.core.publisher.FluxFilterFuseable$FilterFuseableConditionalSubscriber.onNext(FluxFilterFuseable.java:330) at reactor.core.publisher.FluxOnBackpressureBuffer$BackpressureBufferSubscriber.drainRegular(FluxOnBackpressureBuffer.java:261) at reactor.core.publisher.FluxOnBackpressureBuffer$BackpressureBufferSubscriber.drain(FluxOnBackpressureBuffer.java:225) at reactor.core.publisher.FluxOnBackpressureBuffer$BackpressureBufferSubscriber.onNext(FluxOnBackpressureBuffer.java:184) at reactor.core.publisher.DirectProcessor$DirectInner.onNext(DirectProcessor.java:333) at reactor.core.publisher.DirectProcessor.onNext(DirectProcessor.java:142) at reactor.core.publisher.FluxCreate$IgnoreSink.next(FluxCreate.java:618) at reactor.core.publisher.FluxCreate$SerializedSink.next(FluxCreate.java:153) at org.apache.ignite.network.scalecube.ScaleCubeDirectMarshallerTransport.onMessage(ScaleCubeDirectMarshallerTransport.java:190) at org.apache.ignite.internal.network.netty.ConnectionManager.lambda$onMessage$2(ConnectionManager.java:203) at java.base/java.util.concurrent.CopyOnWriteArrayList.forEach(CopyOnWriteArrayList.java:807) at org.apache.ignite.internal.network.netty.ConnectionManager.onMessage(ConnectionManager.java:203) at org.apache.ignite.internal.network.netty.MessageHandler.channelRead(MessageHandler.java:47) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:324) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:296) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:719) at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:655) at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:581) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Thread.java:829) 2021-10-20 14:14:00:142 +0300 [ERROR][%ITDistributedConfigurationPropertiesTest_testFallingBehindDistributedStorageValue_10000%JRaft-Request-Processor-22][LogThreadPoolExecutor] Uncaught exception in pool: %ITDistributedConfigurationPropertiesTest_testFallingBehindDistributedStorageValue_10000%JRaft-Request-Processor-, org.apache.ignite.raft.jraft.util.MetricThreadPoolExecutor@4769ac34[Running, pool size = 48, active threads = 1, queued tasks = 0, completed tasks = 70]. java.lang.IllegalStateException: Node is shutting down at org.apache.ignite.raft.jraft.core.NodeImpl.readIndex(NodeImpl.java:1410) at org.apache.ignite.raft.jraft.rpc.impl.ActionRequestProcessor.handleRequest(ActionRequestProcessor.java:85) at org.apache.ignite.raft.jraft.rpc.impl.ActionRequestProcessor.handleRequest(ActionRequestProcessor.java:49) at org.apache.ignite.raft.jraft.rpc.impl.IgniteRpcServer$RpcMessageHandler.lambda$onReceived$0(IgniteRpcServer.java:182) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) 2021-10-20 14:14:00:143 +0300 [ERROR][Thread-145][MetaStorageServiceImpl] Unexpected exception class org.apache.ignite.lang.IgniteInternalException: class org.apache.ignite.raft.jraft.rpc.impl.RaftException: ENODESHUTDOWN:Node is shutting down. at org.apache.ignite.internal.metastorage.client.CursorImpl$InnerIterator.hasNext(CursorImpl.java:112) at org.apache.ignite.internal.metastorage.client.MetaStorageServiceImpl$WatchProcessor$Watcher.run(MetaStorageServiceImpl.java:461) Caused by: java.util.concurrent.ExecutionException: class org.apache.ignite.raft.jraft.rpc.impl.RaftException: ENODESHUTDOWN:Node is shutting down. at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:395) at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1999) at org.apache.ignite.internal.metastorage.client.CursorImpl$InnerIterator.hasNext(CursorImpl.java:104) ... 1 more Caused by: class org.apache.ignite.raft.jraft.rpc.impl.RaftException: ENODESHUTDOWN:Node is shutting down. at org.apache.ignite.raft.jraft.rpc.impl.RaftGroupServiceImpl$1.accept(RaftGroupServiceImpl.java:547) at org.apache.ignite.raft.jraft.rpc.impl.RaftGroupServiceImpl$1.accept(RaftGroupServiceImpl.java:494) at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859) at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837) at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2073) at reactor.core.publisher.MonoToCompletableFuture.onNext(MonoToCompletableFuture.java:63) at reactor.core.publisher.SerializedSubscriber.onNext(SerializedSubscriber.java:99) at reactor.core.publisher.FluxTimeout$TimeoutMainSubscriber.onNext(FluxTimeout.java:174) at reactor.core.publisher.MonoCreate$DefaultMonoSink.success(MonoCreate.java:156) at reactor.core.publisher.LambdaSubscriber.onNext(LambdaSubscriber.java:160) at reactor.core.publisher.FluxTake$TakeFuseableSubscriber.onNext(FluxTake.java:398) at reactor.core.publisher.FluxFilterFuseable$FilterFuseableSubscriber.onNext(FluxFilterFuseable.java:112) at reactor.core.publisher.FluxFilterFuseable$FilterFuseableConditionalSubscriber.onNext(FluxFilterFuseable.java:330) at reactor.core.publisher.FluxOnBackpressureBuffer$BackpressureBufferSubscriber.drainRegular(FluxOnBackpressureBuffer.java:261) at reactor.core.publisher.FluxOnBackpressureBuffer$BackpressureBufferSubscriber.drain(FluxOnBackpressureBuffer.java:225) at reactor.core.publisher.FluxOnBackpressureBuffer$BackpressureBufferSubscriber.onNext(FluxOnBackpressureBuffer.java:184) at reactor.core.publisher.DirectProcessor$DirectInner.onNext(DirectProcessor.java:333) at reactor.core.publisher.DirectProcessor.onNext(DirectProcessor.java:142) at reactor.core.publisher.FluxCreate$IgnoreSink.next(FluxCreate.java:618) at reactor.core.publisher.FluxCreate$SerializedSink.next(FluxCreate.java:153) at org.apache.ignite.network.scalecube.ScaleCubeDirectMarshallerTransport.onMessage(ScaleCubeDirectMarshallerTransport.java:190) at org.apache.ignite.internal.network.netty.ConnectionManager.lambda$onMessage$2(ConnectionManager.java:203) at java.base/java.util.concurrent.CopyOnWriteArrayList.forEach(CopyOnWriteArrayList.java:807) at org.apache.ignite.internal.network.netty.ConnectionManager.onMessage(ConnectionManager.java:203) at org.apache.ignite.internal.network.netty.MessageHandler.channelRead(MessageHandler.java:47) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:324) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:296) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:719) at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:655) at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:581) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Thread.java:829) 2021-10-20 14:14:00:144 +0300 [ERROR][%ITDistributedConfigurationPropertiesTest_testFallingBehindDistributedStorageValue_10000%JRaft-Request-Processor-22][NamedThreadFactory] Uncaught exception in thread Thread[%ITDistributedConfigurationPropertiesTest_testFallingBehindDistributedStorageValue_10000%JRaft-Request-Processor-22,10,main] java.lang.IllegalStateException: Node is shutting down at org.apache.ignite.raft.jraft.core.NodeImpl.readIndex(NodeImpl.java:1410) at org.apache.ignite.raft.jraft.rpc.impl.ActionRequestProcessor.handleRequest(ActionRequestProcessor.java:85) at org.apache.ignite.raft.jraft.rpc.impl.ActionRequestProcessor.handleRequest(ActionRequestProcessor.java:49) at org.apache.ignite.raft.jraft.rpc.impl.IgniteRpcServer$RpcMessageHandler.lambda$onReceived$0(IgniteRpcServer.java:182) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) 2021-10-20 14:14:00:146 +0300 [ERROR][%ITDistributedConfigurationPropertiesTest_testFallingBehindDistributedStorageValue_10000%JRaft-Request-Processor-23][LogThreadPoolExecutor] Uncaught exception in pool: %ITDistributedConfigurationPropertiesTest_testFallingBehindDistributedStorageValue_10000%JRaft-Request-Processor-, org.apache.ignite.raft.jraft.util.MetricThreadPoolExecutor@4769ac34[Running, pool size = 48, active threads = 1, queued tasks = 0, completed tasks = 71]. java.lang.IllegalStateException: Node is shutting down at org.apache.ignite.raft.jraft.core.NodeImpl.readIndex(NodeImpl.java:1410) at org.apache.ignite.raft.jraft.rpc.impl.ActionRequestProcessor.handleRequest(ActionRequestProcessor.java:85) at org.apache.ignite.raft.jraft.rpc.impl.ActionRequestProcessor.handleRequest(ActionRequestProcessor.java:49) at org.apache.ignite.raft.jraft.rpc.impl.IgniteRpcServer$RpcMessageHandler.lambda$onReceived$0(IgniteRpcServer.java:182) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) 2021-10-20 14:14:00:150 +0300 [ERROR][%ITDistributedConfigurationPropertiesTest_testFallingBehindDistributedStorageValue_10000%JRaft-Request-Processor-23][NamedThreadFactory] Uncaught exception in thread Thread[%ITDistributedConfigurationPropertiesTest_testFallingBehindDistributedStorageValue_10000%JRaft-Request-Processor-23,10,main] java.lang.IllegalStateException: Node is shutting down at org.apache.ignite.raft.jraft.core.NodeImpl.readIndex(NodeImpl.java:1410) at org.apache.ignite.raft.jraft.rpc.impl.ActionRequestProcessor.handleRequest(ActionRequestProcessor.java:85) at org.apache.ignite.raft.jraft.rpc.impl.ActionRequestProcessor.handleRequest(ActionRequestProcessor.java:49) at org.apache.ignite.raft.jraft.rpc.impl.IgniteRpcServer$RpcMessageHandler.lambda$onReceived$0(IgniteRpcServer.java:182) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) 2021-10-20 14:14:00:150 +0300 [INFO][main][RepeatedTimer] Destroy timer: RepeatedTimer{timeout=null, stopped=true, running=false, destroyed=true, invoking=false, timeoutMs=1000, name='JRaft-ElectionTimer-ITDistributedConfigurationPropertiesTest_testFallingBehindDistributedStorageValue_10000-'}. 2021-10-20 14:14:00:151 +0300 [ERROR][Thread-145][MetaStorageServiceImpl] Unexpected exception class org.apache.ignite.lang.IgniteInternalException: class org.apache.ignite.raft.jraft.rpc.impl.RaftException: ENODESHUTDOWN:Node is shutting down. at org.apache.ignite.internal.metastorage.client.CursorImpl$InnerIterator.hasNext(CursorImpl.java:112) at org.apache.ignite.internal.metastorage.client.MetaStorageServiceImpl$WatchProcessor$Watcher.run(MetaStorageServiceImpl.java:461) Caused by: java.util.concurrent.ExecutionException: class org.apache.ignite.raft.jraft.rpc.impl.RaftException: ENODESHUTDOWN:Node is shutting down. at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:395) at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1999) at org.apache.ignite.internal.metastorage.client.CursorImpl$InnerIterator.hasNext(CursorImpl.java:104) ... 1 more Caused by: class org.apache.ignite.raft.jraft.rpc.impl.RaftException: ENODESHUTDOWN:Node is shutting down. at org.apache.ignite.raft.jraft.rpc.impl.RaftGroupServiceImpl$1.accept(RaftGroupServiceImpl.java:547) at org.apache.ignite.raft.jraft.rpc.impl.RaftGroupServiceImpl$1.accept(RaftGroupServiceImpl.java:494) at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859) at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837) at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2073) at reactor.core.publisher.MonoToCompletableFuture.onNext(MonoToCompletableFuture.java:63) at reactor.core.publisher.SerializedSubscriber.onNext(SerializedSubscriber.java:99) at reactor.core.publisher.FluxTimeout$TimeoutMainSubscriber.onNext(FluxTimeout.java:174) at reactor.core.publisher.MonoCreate$DefaultMonoSink.success(MonoCreate.java:156) at reactor.core.publisher.LambdaSubscriber.onNext(LambdaSubscriber.java:160) at reactor.core.publisher.FluxTake$TakeFuseableSubscriber.onNext(FluxTake.java:398) at reactor.core.publisher.FluxFilterFuseable$FilterFuseableSubscriber.onNext(FluxFilterFuseable.java:112) at reactor.core.publisher.FluxFilterFuseable$FilterFuseableConditionalSubscriber.onNext(FluxFilterFuseable.java:330) at reactor.core.publisher.FluxOnBackpressureBuffer$BackpressureBufferSubscriber.drainRegular(FluxOnBackpressureBuffer.java:261) at reactor.core.publisher.FluxOnBackpressureBuffer$BackpressureBufferSubscriber.drain(FluxOnBackpressureBuffer.java:225) at reactor.core.publisher.FluxOnBackpressureBuffer$BackpressureBufferSubscriber.onNext(FluxOnBackpressureBuffer.java:184) at reactor.core.publisher.DirectProcessor$DirectInner.onNext(DirectProcessor.java:333) at reactor.core.publisher.DirectProcessor.onNext(DirectProcessor.java:142) at reactor.core.publisher.FluxCreate$IgnoreSink.next(FluxCreate.java:618) at reactor.core.publisher.FluxCreate$SerializedSink.next(FluxCreate.java:153) at org.apache.ignite.network.scalecube.ScaleCubeDirectMarshallerTransport.onMessage(ScaleCubeDirectMarshallerTransport.java:190) at org.apache.ignite.internal.network.netty.ConnectionManager.lambda$onMessage$2(ConnectionManager.java:203) at java.base/java.util.concurrent.CopyOnWriteArrayList.forEach(CopyOnWriteArrayList.java:807) at org.apache.ignite.internal.network.netty.ConnectionManager.onMessage(ConnectionManager.java:203) at org.apache.ignite.internal.network.netty.MessageHandler.channelRead(MessageHandler.java:47) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:324) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:296) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:719) at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:655) at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:581) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Thread.java:829) 2021-10-20 14:14:00:151 +0300 [INFO][main][RepeatedTimer] Destroy timer: RepeatedTimer{timeout=null, stopped=true, running=false, destroyed=true, invoking=false, timeoutMs=1000, name='JRaft-VoteTimer-ITDistributedConfigurationPropertiesTest_testFallingBehindDistributedStorageValue_10000-'}. 2021-10-20 14:14:00:152 +0300 [INFO][main][RepeatedTimer] Destroy timer: RepeatedTimer{timeout=null, stopped=true, running=false, destroyed=true, invoking=false, timeoutMs=500, name='JRaft-StepDownTimer-ITDistributedConfigurationPropertiesTest_testFallingBehindDistributedStorageValue_10000-'}. 2021-10-20 14:14:00:153 +0300 [INFO][main][RepeatedTimer] Destroy timer: RepeatedTimer{timeout=null, stopped=true, running=false, destroyed=true, invoking=false, timeoutMs=3600000, name='JRaft-SnapshotTimer-ITDistributedConfigurationPropertiesTest_testFallingBehindDistributedStorageValue_10000-'}. 2021-10-20 14:14:00:154 +0300 [ERROR][%ITDistributedConfigurationPropertiesTest_testFallingBehindDistributedStorageValue_10000%JRaft-Request-Processor-24][LogThreadPoolExecutor] Uncaught exception in pool: %ITDistributedConfigurationPropertiesTest_testFallingBehindDistributedStorageValue_10000%JRaft-Request-Processor-, org.apache.ignite.raft.jraft.util.MetricThreadPoolExecutor@4769ac34[Running, pool size = 48, active threads = 1, queued tasks = 0, completed tasks = 72]. java.lang.IllegalStateException: Node is shutting down at org.apache.ignite.raft.jraft.core.NodeImpl.readIndex(NodeImpl.java:1410) at org.apache.ignite.raft.jraft.rpc.impl.ActionRequestProcessor.handleRequest(ActionRequestProcessor.java:85) at org.apache.ignite.raft.jraft.rpc.impl.ActionRequestProcessor.handleRequest(ActionRequestProcessor.java:49) at org.apache.ignite.raft.jraft.rpc.impl.IgniteRpcServer$RpcMessageHandler.lambda$onReceived$0(IgniteRpcServer.java:182) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) 2021-10-20 14:14:00:154 +0300 [INFO][main][StripedDisruptor] Consumer unsubscribe [poolName=%ITDistributedConfigurationPropertiesTest_testFallingBehindDistributedStorageValue_10000%JRaft-ReadOnlyService-Disruptor-, group=metastorage_raft_group] 2021-10-20 14:14:00:154 +0300 [INFO][main][StripedDisruptor] Consumer unsubscribe [poolName=%ITDistributedConfigurationPropertiesTest_testFallingBehindDistributedStorageValue_10000%JRaft-LogManager-Disruptor-, group=metastorage_raft_group] 2021-10-20 14:14:00:155 +0300 [INFO][main][StripedDisruptor] Consumer unsubscribe [poolName=%ITDistributedConfigurationPropertiesTest_testFallingBehindDistributedStorageValue_10000%JRaft-NodeImpl-Disruptor-, group=metastorage_raft_group] 2021-10-20 14:14:00:155 +0300 [INFO][main][StripedDisruptor] Consumer unsubscribe [poolName=%ITDistributedConfigurationPropertiesTest_testFallingBehindDistributedStorageValue_10000%JRaft-FSMCaller-Disruptor-, group=metastorage_raft_group] 2021-10-20 14:14:00:155 +0300 [ERROR][%ITDistributedConfigurationPropertiesTest_testFallingBehindDistributedStorageValue_10000%JRaft-Request-Processor-24][NamedThreadFactory] Uncaught exception in thread Thread[%ITDistributedConfigurationPropertiesTest_testFallingBehindDistributedStorageValue_10000%JRaft-Request-Processor-24,10,main] java.lang.IllegalStateException: Node is shutting down at org.apache.ignite.raft.jraft.core.NodeImpl.readIndex(NodeImpl.java:1410) at org.apache.ignite.raft.jraft.rpc.impl.ActionRequestProcessor.handleRequest(ActionRequestProcessor.java:85) at org.apache.ignite.raft.jraft.rpc.impl.ActionRequestProcessor.handleRequest(ActionRequestProcessor.java:49) at org.apache.ignite.raft.jraft.rpc.impl.IgniteRpcServer$RpcMessageHandler.lambda$onReceived$0(IgniteRpcServer.java:182) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) 2021-10-20 14:14:00:155 +0300 [ERROR][Thread-145][MetaStorageServiceImpl] Unexpected exception class org.apache.ignite.lang.IgniteInternalException: class org.apache.ignite.raft.jraft.rpc.impl.RaftException: ENODESHUTDOWN:Node is shutting down. at org.apache.ignite.internal.metastorage.client.CursorImpl$InnerIterator.hasNext(CursorImpl.java:112) at org.apache.ignite.internal.metastorage.client.MetaStorageServiceImpl$WatchProcessor$Watcher.run(MetaStorageServiceImpl.java:461) Caused by: java.util.concurrent.ExecutionException: class org.apache.ignite.raft.jraft.rpc.impl.RaftException: ENODESHUTDOWN:Node is shutting down. at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:395) at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1999) at org.apache.ignite.internal.metastorage.client.CursorImpl$InnerIterator.hasNext(CursorImpl.java:104) ... 1 more Caused by: class org.apache.ignite.raft.jraft.rpc.impl.RaftException: ENODESHUTDOWN:Node is shutting down. at org.apache.ignite.raft.jraft.rpc.impl.RaftGroupServiceImpl$1.accept(RaftGroupServiceImpl.java:547) at org.apache.ignite.raft.jraft.rpc.impl.RaftGroupServiceImpl$1.accept(RaftGroupServiceImpl.java:494) at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859) at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837) at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2073) at reactor.core.publisher.MonoToCompletableFuture.onNext(MonoToCompletableFuture.java:63) at reactor.core.publisher.SerializedSubscriber.onNext(SerializedSubscriber.java:99) at reactor.core.publisher.FluxTimeout$TimeoutMainSubscriber.onNext(FluxTimeout.java:174) at reactor.core.publisher.MonoCreate$DefaultMonoSink.success(MonoCreate.java:156) at reactor.core.publisher.LambdaSubscriber.onNext(LambdaSubscriber.java:160) at reactor.core.publisher.FluxTake$TakeFuseableSubscriber.onNext(FluxTake.java:398) at reactor.core.publisher.FluxFilterFuseable$FilterFuseableSubscriber.onNext(FluxFilterFuseable.java:112) at reactor.core.publisher.FluxFilterFuseable$FilterFuseableConditionalSubscriber.onNext(FluxFilterFuseable.java:330) at reactor.core.publisher.FluxOnBackpressureBuffer$BackpressureBufferSubscriber.drainRegular(FluxOnBackpressureBuffer.java:261) at reactor.core.publisher.FluxOnBackpressureBuffer$BackpressureBufferSubscriber.drain(FluxOnBackpressureBuffer.java:225) at reactor.core.publisher.FluxOnBackpressureBuffer$BackpressureBufferSubscriber.onNext(FluxOnBackpressureBuffer.java:184) at reactor.core.publisher.DirectProcessor$DirectInner.onNext(DirectProcessor.java:333) at reactor.core.publisher.DirectProcessor.onNext(DirectProcessor.java:142) at reactor.core.publisher.FluxCreate$IgnoreSink.next(FluxCreate.java:618) at reactor.core.publisher.FluxCreate$SerializedSink.next(FluxCreate.java:153) at org.apache.ignite.network.scalecube.ScaleCubeDirectMarshallerTransport.onMessage(ScaleCubeDirectMarshallerTransport.java:190) at org.apache.ignite.internal.network.netty.ConnectionManager.lambda$onMessage$2(ConnectionManager.java:203) at java.base/java.util.concurrent.CopyOnWriteArrayList.forEach(CopyOnWriteArrayList.java:807) at org.apache.ignite.internal.network.netty.ConnectionManager.onMessage(ConnectionManager.java:203) at org.apache.ignite.internal.network.netty.MessageHandler.channelRead(MessageHandler.java:47) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:324) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:296) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:719) at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:655) at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:581) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Thread.java:829) 2021-10-20 14:14:00:160 +0300 [INFO][main][RocksDBLogStorage] DB destroyed, the db path is: target/work/ITDistributedConfigurationPropertiesTest/testFallingBehindDistributedStorageValue_1634728438053/firstNode/metastorage_raft_group_127.0.1.1_10000/logs. 2021-10-20 14:14:00:161 +0300 [INFO][main][RaftGroupService] Stop the RaftGroupService successfully. 2021-10-20 14:14:00:162 +0300 [INFO][main][StripedDisruptor] Striped disruptor stopped [name=%ITDistributedConfigurationPropertiesTest_testFallingBehindDistributedStorageValue_10000%JRaft-FSMCaller-Disruptor-] 2021-10-20 14:14:00:168 +0300 [INFO][main][StripedDisruptor] Striped disruptor stopped [name=%ITDistributedConfigurationPropertiesTest_testFallingBehindDistributedStorageValue_10000%JRaft-NodeImpl-Disruptor-] 2021-10-20 14:14:00:168 +0300 [INFO][main][StripedDisruptor] Striped disruptor stopped [name=%ITDistributedConfigurationPropertiesTest_testFallingBehindDistributedStorageValue_10000%JRaft-ReadOnlyService-Disruptor-] 2021-10-20 14:14:00:169 +0300 [INFO][main][StripedDisruptor] Striped disruptor stopped [name=%ITDistributedConfigurationPropertiesTest_testFallingBehindDistributedStorageValue_10000%JRaft-LogManager-Disruptor-] 2021-10-20 14:14:00:170 +0300 [INFO][%ITDistributedConfigurationPropertiesTest_testFallingBehindDistributedStorageValue_10000%JRaft-Common-Executor-4][LogThreadPoolExecutor] ThreadPool is terminated: %ITDistributedConfigurationPropertiesTest_testFallingBehindDistributedStorageValue_10000%JRaft-Common-Executor-, org.apache.ignite.raft.jraft.util.MetricThreadPoolExecutor@76e9485c[Shutting down, pool size = 3, active threads = 0, queued tasks = 0, completed tasks = 12]. 2021-10-20 14:14:00:192 +0300 [INFO][%ITDistributedConfigurationPropertiesTest_testFallingBehindDistributedStorageValue_10000%JRaft-Node-Scheduler-0][LogScheduledThreadPoolExecutor] ThreadPool is terminated: %ITDistributedConfigurationPropertiesTest_testFallingBehindDistributedStorageValue_10000%JRaft-Node-Scheduler-, org.apache.ignite.raft.jraft.util.MetricScheduledThreadPoolExecutor@666dffca[Shutting down, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0]. 2021-10-20 14:14:00:192 +0300 [INFO][main][LogThreadPoolExecutor] ThreadPool is terminated: %ITDistributedConfigurationPropertiesTest_testFallingBehindDistributedStorageValue_10000%JRaft-Response-Processor-, org.apache.ignite.raft.jraft.util.MetricThreadPoolExecutor@724aefc3[Shutting down, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0]. 2021-10-20 14:14:00:194 +0300 [INFO][%ITDistributedConfigurationPropertiesTest_testFallingBehindDistributedStorageValue_10000%JRaft-Request-Processor-16][LogThreadPoolExecutor] ThreadPool is terminated: %ITDistributedConfigurationPropertiesTest_testFallingBehindDistributedStorageValue_10000%JRaft-Request-Processor-, org.apache.ignite.raft.jraft.util.MetricThreadPoolExecutor@4769ac34[Shutting down, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 74]. 2021-10-20 14:14:00:196 +0300 [INFO][main][Cluster] [default:ITDistributedConfigurationPropertiesTest_testFallingBehindDistributedStorageValue_10000:b9086737-f68d-474c-a393-759ee8d3f4bc@127.0.1.1:10000][doShutdown] Shutting down 2021-10-20 14:14:00:200 +0300 [INFO][main][Cluster] [default:ITDistributedConfigurationPropertiesTest_testFallingBehindDistributedStorageValue_10000:b9086737-f68d-474c-a393-759ee8d3f4bc@127.0.1.1:10000][leaveCluster] Leaving cluster 2021-10-20 14:14:00:210 +0300 [WARNING][ForkJoinPool.commonPool-worker-3][MetaStorageServiceImpl] Cursor close command was rejected because raft executor has been already stopped. 2021-10-20 14:14:00:211 +0300 [INFO][sc-cluster-10001-2][MembershipProtocol] [default:ITDistributedConfigurationPropertiesTest_testFallingBehindDistributedStorageValue_10001:33b837ef-efdf-402d-9551-b3eff77ae6ff@127.0.1.1:10001][publishEvent] MembershipEvent[type=LEAVING, member=default:ITDistributedConfigurationPropertiesTest_testFallingBehindDistributedStorageValue_10000:b9086737-f68d-474c-a393-759ee8d3f4bc@127.0.1.1:10000, oldMetadata=null, newMetadata=7e16449-5, timestamp=2021-10-20T11:14:00.211Z] 2021-10-20 14:14:00:212 +0300 [INFO][sc-cluster-10001-2][ScaleCubeTopologyService] Topology snapshot [nodes=2] ^-- ClusterNode [id=b9086737-f68d-474c-a393-759ee8d3f4bc, name=ITDistributedConfigurationPropertiesTest_testFallingBehindDistributedStorageValue_10000, address=127.0.1.1:10000] ^-- ClusterNode [id=33b837ef-efdf-402d-9551-b3eff77ae6ff, name=ITDistributedConfigurationPropertiesTest_testFallingBehindDistributedStorageValue_10001, address=127.0.1.1:10001] 2021-10-20 14:14:00:260 +0300 [INFO][sc-cluster-10000-1][Cluster] [default:ITDistributedConfigurationPropertiesTest_testFallingBehindDistributedStorageValue_10000:b9086737-f68d-474c-a393-759ee8d3f4bc@127.0.1.1:10000][leaveCluster] Left cluster 2021-10-20 14:14:00:262 +0300 [INFO][sc-cluster-10000-1][Transport] Stopping 127.0.1.1:10000 2021-10-20 14:14:00:263 +0300 [INFO][sc-cluster-10000-1][Transport] Stopped 127.0.1.1:10000 2021-10-20 14:14:00:263 +0300 [INFO][sc-cluster-10000-1][Cluster] [default:ITDistributedConfigurationPropertiesTest_testFallingBehindDistributedStorageValue_10000:b9086737-f68d-474c-a393-759ee8d3f4bc@127.0.1.1:10000][doShutdown] Shutdown 2021-10-20 14:14:00:375 +0300 [INFO][main][StripedDisruptor] Striped disruptor stopped [name=%ITDistributedConfigurationPropertiesTest_testFallingBehindDistributedStorageValue_10001%JRaft-FSMCaller-Disruptor-] 2021-10-20 14:14:00:376 +0300 [INFO][main][StripedDisruptor] Striped disruptor stopped [name=%ITDistributedConfigurationPropertiesTest_testFallingBehindDistributedStorageValue_10001%JRaft-NodeImpl-Disruptor-] 2021-10-20 14:14:00:377 +0300 [INFO][main][StripedDisruptor] Striped disruptor stopped [name=%ITDistributedConfigurationPropertiesTest_testFallingBehindDistributedStorageValue_10001%JRaft-ReadOnlyService-Disruptor-] 2021-10-20 14:14:00:377 +0300 [INFO][main][StripedDisruptor] Striped disruptor stopped [name=%ITDistributedConfigurationPropertiesTest_testFallingBehindDistributedStorageValue_10001%JRaft-LogManager-Disruptor-] 2021-10-20 14:14:00:378 +0300 [INFO][main][LogThreadPoolExecutor] ThreadPool is terminated: %ITDistributedConfigurationPropertiesTest_testFallingBehindDistributedStorageValue_10001%JRaft-Common-Executor-, org.apache.ignite.raft.jraft.util.MetricThreadPoolExecutor@4fc5563d[Shutting down, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0]. 2021-10-20 14:14:00:382 +0300 [INFO][main][LogScheduledThreadPoolExecutor] ThreadPool is terminated: %ITDistributedConfigurationPropertiesTest_testFallingBehindDistributedStorageValue_10001%JRaft-Node-Scheduler-, org.apache.ignite.raft.jraft.util.MetricScheduledThreadPoolExecutor@286090c[Shutting down, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0]. 2021-10-20 14:14:00:383 +0300 [INFO][main][LogThreadPoolExecutor] ThreadPool is terminated: %ITDistributedConfigurationPropertiesTest_testFallingBehindDistributedStorageValue_10001%JRaft-Response-Processor-, org.apache.ignite.raft.jraft.util.MetricThreadPoolExecutor@31120021[Shutting down, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0]. 2021-10-20 14:14:00:383 +0300 [INFO][main][LogThreadPoolExecutor] ThreadPool is terminated: %ITDistributedConfigurationPropertiesTest_testFallingBehindDistributedStorageValue_10001%JRaft-Request-Processor-, org.apache.ignite.raft.jraft.util.MetricThreadPoolExecutor@3df1a1ac[Shutting down, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0]. 2021-10-20 14:14:00:383 +0300 [INFO][main][Cluster] [default:ITDistributedConfigurationPropertiesTest_testFallingBehindDistributedStorageValue_10001:33b837ef-efdf-402d-9551-b3eff77ae6ff@127.0.1.1:10001][doShutdown] Shutting down 2021-10-20 14:14:00:384 +0300 [INFO][main][Cluster] [default:ITDistributedConfigurationPropertiesTest_testFallingBehindDistributedStorageValue_10001:33b837ef-efdf-402d-9551-b3eff77ae6ff@127.0.1.1:10001][leaveCluster] Leaving cluster 2021-10-20 14:14:00:436 +0300 [INFO][sc-cluster-10001-2][Cluster] [default:ITDistributedConfigurationPropertiesTest_testFallingBehindDistributedStorageValue_10001:33b837ef-efdf-402d-9551-b3eff77ae6ff@127.0.1.1:10001][leaveCluster] Left cluster 2021-10-20 14:14:00:437 +0300 [INFO][sc-cluster-10001-2][Transport] Stopping 127.0.1.1:10001 2021-10-20 14:14:00:437 +0300 [INFO][sc-cluster-10001-2][Transport] Stopped 127.0.1.1:10001 2021-10-20 14:14:00:437 +0300 [INFO][sc-cluster-10001-2][Cluster] [default:ITDistributedConfigurationPropertiesTest_testFallingBehindDistributedStorageValue_10001:33b837ef-efdf-402d-9551-b3eff77ae6ff@127.0.1.1:10001][doShutdown] Shutdown 2021-10-20 14:14:00:664 +0300 [INFO][main][ConnectionManager] Connection created [address=/0:0:0:0:0:0:0:0:10000] 2021-10-20 14:14:00:666 +0300 [INFO][main][Cluster] [null][doStart] Starting, config: ClusterConfig[metadata=null, metadataTimeout=1000, metadataCodec=io.scalecube.cluster.metadata.JdkMetadataCodec@895416d, memberAlias='ITDistributedConfigurationPropertiesTest_testFallingBehindDistributedStorageValue_10000', externalHost='null', externalPort=null, transportConfig=TransportConfig[port=0, isSecured=false, connectTimeout=1000, messageCodec=io.scalecube.cluster.transport.api.JdkMessageCodec@71a06021, maxFrameLength=2097152, transportFactory=org.apache.ignite.network.scalecube.DelegatingTransportFactory@41a16eb3], failureDetectorConfig=FailureDetectorConfig[pingInterval=500, pingTimeout=200, pingReqMembers=1], gossipConfig=GossipConfig[gossipFanout=3, gossipInterval=10, gossipRepeatMult=2, gossipSegmentationThreshold=1000], membershipConfig=MembershipConfig[seedMembers=[127.0.1.1:10000, 127.0.1.1:10001], syncInterval=1000, syncTimeout=3000, suspicionMult=1, namespace='default', removedMembersHistorySize=42]] 2021-10-20 14:14:00:679 +0300 [WARNING][main][MembershipProtocol] [default:ITDistributedConfigurationPropertiesTest_testFallingBehindDistributedStorageValue_10000:ef8b5306-36ea-4f4b-a7c6-cad7c253addf@127.0.1.1:10000] Filtering out seed address: 127.0.1.1:10000 2021-10-20 14:14:00:689 +0300 [INFO][main][MembershipProtocol] [default:ITDistributedConfigurationPropertiesTest_testFallingBehindDistributedStorageValue_10000:ef8b5306-36ea-4f4b-a7c6-cad7c253addf@127.0.1.1:10000] Making initial Sync to all seed members: [127.0.1.1:10001] 2021-10-20 14:14:00:710 +0300 [INFO][sc-cluster-10000-3][Cluster] [default:ITDistributedConfigurationPropertiesTest_testFallingBehindDistributedStorageValue_10000:ef8b5306-36ea-4f4b-a7c6-cad7c253addf@127.0.1.1:10000][doStart] Started 2021-10-20 14:14:00:711 +0300 [INFO][main][ScaleCubeTopologyService] Node joined: ClusterNode [id=ef8b5306-36ea-4f4b-a7c6-cad7c253addf, name=ITDistributedConfigurationPropertiesTest_testFallingBehindDistributedStorageValue_10000, address=127.0.1.1:10000] 2021-10-20 14:14:00:711 +0300 [INFO][main][ScaleCubeTopologyService] Topology snapshot [nodes=1] ^-- ClusterNode [id=ef8b5306-36ea-4f4b-a7c6-cad7c253addf, name=ITDistributedConfigurationPropertiesTest_testFallingBehindDistributedStorageValue_10000, address=127.0.1.1:10000] 2021-10-20 14:14:00:748 +0300 [INFO][main][StripedDisruptor] Striped disruptor was started [name=%ITDistributedConfigurationPropertiesTest_testFallingBehindDistributedStorageValue_10000%JRaft-FSMCaller-Disruptor-] 2021-10-20 14:14:00:779 +0300 [INFO][main][StripedDisruptor] Striped disruptor was started [name=%ITDistributedConfigurationPropertiesTest_testFallingBehindDistributedStorageValue_10000%JRaft-NodeImpl-Disruptor-] 2021-10-20 14:14:00:802 +0300 [INFO][main][StripedDisruptor] Striped disruptor was started [name=%ITDistributedConfigurationPropertiesTest_testFallingBehindDistributedStorageValue_10000%JRaft-ReadOnlyService-Disruptor-] 2021-10-20 14:14:00:819 +0300 [INFO][main][StripedDisruptor] Striped disruptor was started [name=%ITDistributedConfigurationPropertiesTest_testFallingBehindDistributedStorageValue_10000%JRaft-LogManager-Disruptor-] 2021-10-20 14:14:00:820 +0300 [INFO][main][StripedDisruptor] Consumer subscribed [poolName=%ITDistributedConfigurationPropertiesTest_testFallingBehindDistributedStorageValue_10000%JRaft-NodeImpl-Disruptor-, group=metastorage_raft_group] 2021-10-20 14:14:00:885 +0300 [INFO][main][StripedDisruptor] Consumer subscribed [poolName=%ITDistributedConfigurationPropertiesTest_testFallingBehindDistributedStorageValue_10000%JRaft-LogManager-Disruptor-, group=metastorage_raft_group] 2021-10-20 14:14:00:885 +0300 [INFO][main][StripedDisruptor] Consumer subscribed [poolName=%ITDistributedConfigurationPropertiesTest_testFallingBehindDistributedStorageValue_10000%JRaft-FSMCaller-Disruptor-, group=metastorage_raft_group] 2021-10-20 14:14:00:885 +0300 [INFO][main][FSMCallerImpl] Starts FSMCaller successfully. 2021-10-20 14:14:00:885 +0300 [WARNING][main][LocalSnapshotStorage] No data for snapshot reader target/work/ITDistributedConfigurationPropertiesTest/testFallingBehindDistributedStorageValue_1634728440461/firstNode/metastorage_raft_group_127.0.1.1_10000/snapshot. 2021-10-20 14:14:00:885 +0300 [INFO][main][StripedDisruptor] Consumer subscribed [poolName=%ITDistributedConfigurationPropertiesTest_testFallingBehindDistributedStorageValue_10000%JRaft-ReadOnlyService-Disruptor-, group=metastorage_raft_group] 2021-10-20 14:14:00:886 +0300 [INFO][main][NodeImpl] Node init, term=0, lastLogId=LogId [index=0, term=0], conf=127.0.1.1:10000, oldConf=. 2021-10-20 14:14:00:887 +0300 [INFO][main][NodeImpl] Node start vote and grant vote self, term=0. 2021-10-20 14:14:00:893 +0300 [INFO][main][LocalRaftMetaStorage] Save raft meta, path=target/work/ITDistributedConfigurationPropertiesTest/testFallingBehindDistributedStorageValue_1634728440461/firstNode/metastorage_raft_group_127.0.1.1_10000/meta, term=1, votedFor=127.0.1.1:10000, cost time=6 ms 2021-10-20 14:14:00:894 +0300 [INFO][main][NodeImpl] Node become leader of group, term=1, conf=127.0.1.1:10000, oldConf=. 2021-10-20 14:14:00:894 +0300 [INFO][main][RaftGroupService] Start the RaftGroupService successfully 2021-10-20 14:14:00:901 +0300 [INFO][%ITDistributedConfigurationPropertiesTest_testFallingBehindDistributedStorageValue_10000%JRaft-FSMCaller-Disruptor-_stripe_10-0][StateMachineAdapter] onLeaderStart: term=1. 2021-10-20 14:14:00:912 +0300 [INFO][main][ConnectionManager] Connection created [address=/0:0:0:0:0:0:0:0:10001] 2021-10-20 14:14:00:913 +0300 [INFO][main][Cluster] [null][doStart] Starting, config: ClusterConfig[metadata=null, metadataTimeout=1000, metadataCodec=io.scalecube.cluster.metadata.JdkMetadataCodec@895416d, memberAlias='ITDistributedConfigurationPropertiesTest_testFallingBehindDistributedStorageValue_10001', externalHost='null', externalPort=null, transportConfig=TransportConfig[port=0, isSecured=false, connectTimeout=1000, messageCodec=io.scalecube.cluster.transport.api.JdkMessageCodec@71a06021, maxFrameLength=2097152, transportFactory=org.apache.ignite.network.scalecube.DelegatingTransportFactory@4ce267cc], failureDetectorConfig=FailureDetectorConfig[pingInterval=500, pingTimeout=200, pingReqMembers=1], gossipConfig=GossipConfig[gossipFanout=3, gossipInterval=10, gossipRepeatMult=2, gossipSegmentationThreshold=1000], membershipConfig=MembershipConfig[seedMembers=[127.0.1.1:10000, 127.0.1.1:10001], syncInterval=1000, syncTimeout=3000, suspicionMult=1, namespace='default', removedMembersHistorySize=42]] 2021-10-20 14:14:00:914 +0300 [WARNING][main][MembershipProtocol] [default:ITDistributedConfigurationPropertiesTest_testFallingBehindDistributedStorageValue_10001:32780ecd-a1b7-4f44-8dbe-41ec3ec4b04a@127.0.1.1:10001] Filtering out seed address: 127.0.1.1:10001 2021-10-20 14:14:00:915 +0300 [INFO][main][MembershipProtocol] [default:ITDistributedConfigurationPropertiesTest_testFallingBehindDistributedStorageValue_10001:32780ecd-a1b7-4f44-8dbe-41ec3ec4b04a@127.0.1.1:10001] Making initial Sync to all seed members: [127.0.1.1:10000] 2021-10-20 14:14:00:968 +0300 [INFO][sc-cluster-10000-3][MembershipProtocol] [default:ITDistributedConfigurationPropertiesTest_testFallingBehindDistributedStorageValue_10000:ef8b5306-36ea-4f4b-a7c6-cad7c253addf@127.0.1.1:10000][publishEvent] MembershipEvent[type=ADDED, member=default:ITDistributedConfigurationPropertiesTest_testFallingBehindDistributedStorageValue_10001:32780ecd-a1b7-4f44-8dbe-41ec3ec4b04a@127.0.1.1:10001, oldMetadata=null, newMetadata=7e16449-5, timestamp=2021-10-20T11:14:00.967Z] 2021-10-20 14:14:00:968 +0300 [INFO][sc-cluster-10000-3][ScaleCubeTopologyService] Node joined: ClusterNode [id=32780ecd-a1b7-4f44-8dbe-41ec3ec4b04a, name=ITDistributedConfigurationPropertiesTest_testFallingBehindDistributedStorageValue_10001, address=127.0.1.1:10001] 2021-10-20 14:14:00:968 +0300 [INFO][sc-cluster-10000-3][ScaleCubeTopologyService] Topology snapshot [nodes=2] ^-- ClusterNode [id=ef8b5306-36ea-4f4b-a7c6-cad7c253addf, name=ITDistributedConfigurationPropertiesTest_testFallingBehindDistributedStorageValue_10000, address=127.0.1.1:10000] ^-- ClusterNode [id=32780ecd-a1b7-4f44-8dbe-41ec3ec4b04a, name=ITDistributedConfigurationPropertiesTest_testFallingBehindDistributedStorageValue_10001, address=127.0.1.1:10001] 2021-10-20 14:14:00:972 +0300 [INFO][sc-cluster-10001-4][MembershipProtocol] [default:ITDistributedConfigurationPropertiesTest_testFallingBehindDistributedStorageValue_10001:32780ecd-a1b7-4f44-8dbe-41ec3ec4b04a@127.0.1.1:10001][publishEvent] MembershipEvent[type=ADDED, member=default:ITDistributedConfigurationPropertiesTest_testFallingBehindDistributedStorageValue_10000:ef8b5306-36ea-4f4b-a7c6-cad7c253addf@127.0.1.1:10000, oldMetadata=null, newMetadata=7e16449-5, timestamp=2021-10-20T11:14:00.971Z] 2021-10-20 14:14:00:972 +0300 [INFO][sc-cluster-10001-4][ScaleCubeTopologyService] Node joined: ClusterNode [id=ef8b5306-36ea-4f4b-a7c6-cad7c253addf, name=ITDistributedConfigurationPropertiesTest_testFallingBehindDistributedStorageValue_10000, address=127.0.1.1:10000] 2021-10-20 14:14:00:972 +0300 [INFO][sc-cluster-10001-4][ScaleCubeTopologyService] Topology snapshot [nodes=1] ^-- ClusterNode [id=ef8b5306-36ea-4f4b-a7c6-cad7c253addf, name=ITDistributedConfigurationPropertiesTest_testFallingBehindDistributedStorageValue_10000, address=127.0.1.1:10000] 2021-10-20 14:14:00:972 +0300 [INFO][sc-cluster-10001-4][Cluster] [default:ITDistributedConfigurationPropertiesTest_testFallingBehindDistributedStorageValue_10001:32780ecd-a1b7-4f44-8dbe-41ec3ec4b04a@127.0.1.1:10001][doStart] Started 2021-10-20 14:14:00:973 +0300 [INFO][main][ScaleCubeTopologyService] Node joined: ClusterNode [id=32780ecd-a1b7-4f44-8dbe-41ec3ec4b04a, name=ITDistributedConfigurationPropertiesTest_testFallingBehindDistributedStorageValue_10001, address=127.0.1.1:10001] 2021-10-20 14:14:00:973 +0300 [INFO][main][ScaleCubeTopologyService] Topology snapshot [nodes=2] ^-- ClusterNode [id=ef8b5306-36ea-4f4b-a7c6-cad7c253addf, name=ITDistributedConfigurationPropertiesTest_testFallingBehindDistributedStorageValue_10000, address=127.0.1.1:10000] ^-- ClusterNode [id=32780ecd-a1b7-4f44-8dbe-41ec3ec4b04a, name=ITDistributedConfigurationPropertiesTest_testFallingBehindDistributedStorageValue_10001, address=127.0.1.1:10001] 2021-10-20 14:14:00:984 +0300 [INFO][main][StripedDisruptor] Striped disruptor was started [name=%ITDistributedConfigurationPropertiesTest_testFallingBehindDistributedStorageValue_10001%JRaft-FSMCaller-Disruptor-] 2021-10-20 14:14:01:002 +0300 [INFO][main][StripedDisruptor] Striped disruptor was started [name=%ITDistributedConfigurationPropertiesTest_testFallingBehindDistributedStorageValue_10001%JRaft-NodeImpl-Disruptor-] 2021-10-20 14:14:01:034 +0300 [INFO][main][StripedDisruptor] Striped disruptor was started [name=%ITDistributedConfigurationPropertiesTest_testFallingBehindDistributedStorageValue_10001%JRaft-ReadOnlyService-Disruptor-] 2021-10-20 14:14:01:071 +0300 [INFO][main][StripedDisruptor] Striped disruptor was started [name=%ITDistributedConfigurationPropertiesTest_testFallingBehindDistributedStorageValue_10001%JRaft-LogManager-Disruptor-] 2021-10-20 14:14:02:383 +0300 [INFO][sc-cluster-10000-3][MembershipProtocol] [default:ITDistributedConfigurationPropertiesTest_testFallingBehindDistributedStorageValue_10000:ef8b5306-36ea-4f4b-a7c6-cad7c253addf@127.0.1.1:10000] Member leaved without notification: default:ITDistributedConfigurationPropertiesTest_testFallingBehindDistributedStorageValue_10001:32780ecd-a1b7-4f44-8dbe-41ec3ec4b04a@127.0.1.1:10001 2021-10-20 14:14:02:384 +0300 [INFO][sc-cluster-10000-3][MembershipProtocol] [default:ITDistributedConfigurationPropertiesTest_testFallingBehindDistributedStorageValue_10000:ef8b5306-36ea-4f4b-a7c6-cad7c253addf@127.0.1.1:10000][publishEvent] MembershipEvent[type=REMOVED, member=default:ITDistributedConfigurationPropertiesTest_testFallingBehindDistributedStorageValue_10001:32780ecd-a1b7-4f44-8dbe-41ec3ec4b04a@127.0.1.1:10001, oldMetadata=7e16449-5, newMetadata=null, timestamp=2021-10-20T11:14:02.384Z] 2021-10-20 14:14:02:385 +0300 [INFO][sc-cluster-10000-3][ScaleCubeTopologyService] Node left: ClusterNode [id=32780ecd-a1b7-4f44-8dbe-41ec3ec4b04a, name=ITDistributedConfigurationPropertiesTest_testFallingBehindDistributedStorageValue_10001, address=127.0.1.1:10001] 2021-10-20 14:14:02:386 +0300 [INFO][sc-cluster-10000-3][ScaleCubeTopologyService] Topology snapshot [nodes=1] ^-- ClusterNode [id=ef8b5306-36ea-4f4b-a7c6-cad7c253addf, name=ITDistributedConfigurationPropertiesTest_testFallingBehindDistributedStorageValue_10000, address=127.0.1.1:10000] 2021-10-20 14:14:02:616 +0300 [INFO][sc-cluster-10001-4][MembershipProtocol] [default:ITDistributedConfigurationPropertiesTest_testFallingBehindDistributedStorageValue_10001:32780ecd-a1b7-4f44-8dbe-41ec3ec4b04a@127.0.1.1:10001] Member leaved without notification: default:ITDistributedConfigurationPropertiesTest_testFallingBehindDistributedStorageValue_10000:ef8b5306-36ea-4f4b-a7c6-cad7c253addf@127.0.1.1:10000 2021-10-20 14:14:02:616 +0300 [INFO][sc-cluster-10001-4][MembershipProtocol] [default:ITDistributedConfigurationPropertiesTest_testFallingBehindDistributedStorageValue_10001:32780ecd-a1b7-4f44-8dbe-41ec3ec4b04a@127.0.1.1:10001][publishEvent] MembershipEvent[type=REMOVED, member=default:ITDistributedConfigurationPropertiesTest_testFallingBehindDistributedStorageValue_10000:ef8b5306-36ea-4f4b-a7c6-cad7c253addf@127.0.1.1:10000, oldMetadata=7e16449-5, newMetadata=null, timestamp=2021-10-20T11:14:02.616Z] 2021-10-20 14:14:02:616 +0300 [INFO][sc-cluster-10001-4][ScaleCubeTopologyService] Node left: ClusterNode [id=ef8b5306-36ea-4f4b-a7c6-cad7c253addf, name=ITDistributedConfigurationPropertiesTest_testFallingBehindDistributedStorageValue_10000, address=127.0.1.1:10000] 2021-10-20 14:14:02:617 +0300 [INFO][sc-cluster-10001-4][ScaleCubeTopologyService] Topology snapshot [nodes=1] ^-- ClusterNode [id=32780ecd-a1b7-4f44-8dbe-41ec3ec4b04a, name=ITDistributedConfigurationPropertiesTest_testFallingBehindDistributedStorageValue_10001, address=127.0.1.1:10001] 2021-10-20 14:14:03:974 +0300 [WARNING][sc-cluster-10000-3][MetadataStore] [default:ITDistributedConfigurationPropertiesTest_testFallingBehindDistributedStorageValue_10000:ef8b5306-36ea-4f4b-a7c6-cad7c253addf@127.0.1.1:10000][ef8b5306-36ea-4f4b-a7c6-cad7c253addf-1634728440673] Timeout getting GetMetadataResp from 127.0.1.1:10001 within 1000 ms, cause: java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 1000ms in 'source(MonoDefer)' (and no fallback has been configured) 2021-10-20 14:14:03:975 +0300 [WARNING][sc-cluster-10000-3][MembershipProtocol] [default:ITDistributedConfigurationPropertiesTest_testFallingBehindDistributedStorageValue_10000:ef8b5306-36ea-4f4b-a7c6-cad7c253addf@127.0.1.1:10000][updateMembership][SYNC] Skipping to add/update member: {m: default:ITDistributedConfigurationPropertiesTest_testFallingBehindDistributedStorageValue_10001:32780ecd-a1b7-4f44-8dbe-41ec3ec4b04a@127.0.1.1:10001, s: ALIVE, inc: 0}, due to failed fetchMetadata call (cause: java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 1000ms in 'source(MonoDefer)' (and no fallback has been configured)) 2021-10-20 14:14:04:091 +0300 [INFO][main][NodeImpl] Node shutdown, currTerm=1 state=STATE_LEADER. 2021-10-20 14:14:04:093 +0300 [INFO][main][ReplicatorGroupImpl] Fail to find the next candidate. 2021-10-20 14:14:04:093 +0300 [INFO][%ITDistributedConfigurationPropertiesTest_testFallingBehindDistributedStorageValue_10000%JRaft-FSMCaller-Disruptor-_stripe_10-0][StateMachineAdapter] onLeaderStop: status=Status[ESHUTDOWN<1007>: Raft node is going to quit.]. 2021-10-20 14:14:04:096 +0300 [INFO][sc-cluster-10001-4][MembershipProtocol] [default:ITDistributedConfigurationPropertiesTest_testFallingBehindDistributedStorageValue_10001:32780ecd-a1b7-4f44-8dbe-41ec3ec4b04a@127.0.1.1:10001][publishEvent] MembershipEvent[type=ADDED, member=default:ITDistributedConfigurationPropertiesTest_testFallingBehindDistributedStorageValue_10000:ef8b5306-36ea-4f4b-a7c6-cad7c253addf@127.0.1.1:10000, oldMetadata=null, newMetadata=7e16449-5, timestamp=2021-10-20T11:14:04.096Z] 2021-10-20 14:14:04:097 +0300 [INFO][sc-cluster-10000-3][MembershipProtocol] [default:ITDistributedConfigurationPropertiesTest_testFallingBehindDistributedStorageValue_10000:ef8b5306-36ea-4f4b-a7c6-cad7c253addf@127.0.1.1:10000][publishEvent] MembershipEvent[type=ADDED, member=default:ITDistributedConfigurationPropertiesTest_testFallingBehindDistributedStorageValue_10001:32780ecd-a1b7-4f44-8dbe-41ec3ec4b04a@127.0.1.1:10001, oldMetadata=null, newMetadata=7e16449-5, timestamp=2021-10-20T11:14:04.096Z] 2021-10-20 14:14:04:097 +0300 [INFO][sc-cluster-10001-4][ScaleCubeTopologyService] Node joined: ClusterNode [id=ef8b5306-36ea-4f4b-a7c6-cad7c253addf, name=ITDistributedConfigurationPropertiesTest_testFallingBehindDistributedStorageValue_10000, address=127.0.1.1:10000] 2021-10-20 14:14:04:097 +0300 [INFO][sc-cluster-10000-3][ScaleCubeTopologyService] Node joined: ClusterNode [id=32780ecd-a1b7-4f44-8dbe-41ec3ec4b04a, name=ITDistributedConfigurationPropertiesTest_testFallingBehindDistributedStorageValue_10001, address=127.0.1.1:10001] 2021-10-20 14:14:04:098 +0300 [INFO][sc-cluster-10001-4][ScaleCubeTopologyService] Topology snapshot [nodes=2] ^-- ClusterNode [id=ef8b5306-36ea-4f4b-a7c6-cad7c253addf, name=ITDistributedConfigurationPropertiesTest_testFallingBehindDistributedStorageValue_10000, address=127.0.1.1:10000] ^-- ClusterNode [id=32780ecd-a1b7-4f44-8dbe-41ec3ec4b04a, name=ITDistributedConfigurationPropertiesTest_testFallingBehindDistributedStorageValue_10001, address=127.0.1.1:10001] 2021-10-20 14:14:04:098 +0300 [INFO][sc-cluster-10000-3][ScaleCubeTopologyService] Topology snapshot [nodes=2] ^-- ClusterNode [id=ef8b5306-36ea-4f4b-a7c6-cad7c253addf, name=ITDistributedConfigurationPropertiesTest_testFallingBehindDistributedStorageValue_10000, address=127.0.1.1:10000] ^-- ClusterNode [id=32780ecd-a1b7-4f44-8dbe-41ec3ec4b04a, name=ITDistributedConfigurationPropertiesTest_testFallingBehindDistributedStorageValue_10001, address=127.0.1.1:10001] 2021-10-20 14:14:04:110 +0300 [INFO][main][LocalRaftMetaStorage] Save raft meta, path=target/work/ITDistributedConfigurationPropertiesTest/testFallingBehindDistributedStorageValue_1634728440461/firstNode/metastorage_raft_group_127.0.1.1_10000/meta, term=1, votedFor=127.0.1.1:10000, cost time=15 ms 2021-10-20 14:14:04:110 +0300 [INFO][main][FSMCallerImpl] Shutting down FSMCaller... 2021-10-20 14:14:04:112 +0300 [INFO][main][RepeatedTimer] Destroy timer: RepeatedTimer{timeout=null, stopped=true, running=false, destroyed=true, invoking=false, timeoutMs=1000, name='JRaft-ElectionTimer-ITDistributedConfigurationPropertiesTest_testFallingBehindDistributedStorageValue_10000-'}. 2021-10-20 14:14:04:113 +0300 [INFO][main][RepeatedTimer] Destroy timer: RepeatedTimer{timeout=null, stopped=true, running=false, destroyed=true, invoking=false, timeoutMs=1000, name='JRaft-VoteTimer-ITDistributedConfigurationPropertiesTest_testFallingBehindDistributedStorageValue_10000-'}. 2021-10-20 14:14:04:113 +0300 [INFO][main][RepeatedTimer] Destroy timer: RepeatedTimer{timeout=null, stopped=true, running=false, destroyed=true, invoking=false, timeoutMs=500, name='JRaft-StepDownTimer-ITDistributedConfigurationPropertiesTest_testFallingBehindDistributedStorageValue_10000-'}. 2021-10-20 14:14:04:113 +0300 [INFO][main][RepeatedTimer] Destroy timer: RepeatedTimer{timeout=null, stopped=true, running=false, destroyed=true, invoking=false, timeoutMs=3600000, name='JRaft-SnapshotTimer-ITDistributedConfigurationPropertiesTest_testFallingBehindDistributedStorageValue_10000-'}. 2021-10-20 14:14:04:114 +0300 [INFO][main][StripedDisruptor] Consumer unsubscribe [poolName=%ITDistributedConfigurationPropertiesTest_testFallingBehindDistributedStorageValue_10000%JRaft-ReadOnlyService-Disruptor-, group=metastorage_raft_group] 2021-10-20 14:14:04:114 +0300 [INFO][main][StripedDisruptor] Consumer unsubscribe [poolName=%ITDistributedConfigurationPropertiesTest_testFallingBehindDistributedStorageValue_10000%JRaft-LogManager-Disruptor-, group=metastorage_raft_group] 2021-10-20 14:14:04:114 +0300 [INFO][main][StripedDisruptor] Consumer unsubscribe [poolName=%ITDistributedConfigurationPropertiesTest_testFallingBehindDistributedStorageValue_10000%JRaft-NodeImpl-Disruptor-, group=metastorage_raft_group] 2021-10-20 14:14:04:114 +0300 [INFO][main][StripedDisruptor] Consumer unsubscribe [poolName=%ITDistributedConfigurationPropertiesTest_testFallingBehindDistributedStorageValue_10000%JRaft-FSMCaller-Disruptor-, group=metastorage_raft_group] 2021-10-20 14:14:04:115 +0300 [INFO][main][RocksDBLogStorage] DB destroyed, the db path is: target/work/ITDistributedConfigurationPropertiesTest/testFallingBehindDistributedStorageValue_1634728440461/firstNode/metastorage_raft_group_127.0.1.1_10000/logs. 2021-10-20 14:14:04:115 +0300 [INFO][main][RaftGroupService] Stop the RaftGroupService successfully. 2021-10-20 14:14:04:115 +0300 [INFO][main][StripedDisruptor] Striped disruptor stopped [name=%ITDistributedConfigurationPropertiesTest_testFallingBehindDistributedStorageValue_10000%JRaft-FSMCaller-Disruptor-] 2021-10-20 14:14:04:115 +0300 [INFO][main][StripedDisruptor] Striped disruptor stopped [name=%ITDistributedConfigurationPropertiesTest_testFallingBehindDistributedStorageValue_10000%JRaft-NodeImpl-Disruptor-] 2021-10-20 14:14:04:116 +0300 [INFO][main][StripedDisruptor] Striped disruptor stopped [name=%ITDistributedConfigurationPropertiesTest_testFallingBehindDistributedStorageValue_10000%JRaft-ReadOnlyService-Disruptor-] 2021-10-20 14:14:04:116 +0300 [INFO][main][StripedDisruptor] Striped disruptor stopped [name=%ITDistributedConfigurationPropertiesTest_testFallingBehindDistributedStorageValue_10000%JRaft-LogManager-Disruptor-] 2021-10-20 14:14:04:116 +0300 [INFO][%ITDistributedConfigurationPropertiesTest_testFallingBehindDistributedStorageValue_10000%JRaft-Common-Executor-1][LogThreadPoolExecutor] ThreadPool is terminated: %ITDistributedConfigurationPropertiesTest_testFallingBehindDistributedStorageValue_10000%JRaft-Common-Executor-, org.apache.ignite.raft.jraft.util.MetricThreadPoolExecutor@5ece9ccb[Shutting down, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 5]. 2021-10-20 14:14:04:127 +0300 [INFO][main][LogThreadPoolExecutor] ThreadPool is terminated: %ITDistributedConfigurationPropertiesTest_testFallingBehindDistributedStorageValue_10000%JRaft-Response-Processor-, org.apache.ignite.raft.jraft.util.MetricThreadPoolExecutor@cfd5cd2[Shutting down, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0]. 2021-10-20 14:14:04:127 +0300 [INFO][%ITDistributedConfigurationPropertiesTest_testFallingBehindDistributedStorageValue_10000%JRaft-Node-Scheduler-2][LogScheduledThreadPoolExecutor] ThreadPool is terminated: %ITDistributedConfigurationPropertiesTest_testFallingBehindDistributedStorageValue_10000%JRaft-Node-Scheduler-, org.apache.ignite.raft.jraft.util.MetricScheduledThreadPoolExecutor@5e0a179a[Shutting down, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 3]. 2021-10-20 14:14:04:129 +0300 [INFO][%ITDistributedConfigurationPropertiesTest_testFallingBehindDistributedStorageValue_10000%JRaft-Request-Processor-46][LogThreadPoolExecutor] ThreadPool is terminated: %ITDistributedConfigurationPropertiesTest_testFallingBehindDistributedStorageValue_10000%JRaft-Request-Processor-, org.apache.ignite.raft.jraft.util.MetricThreadPoolExecutor@614e6d8a[Shutting down, pool size = 5, active threads = 0, queued tasks = 0, completed tasks = 275]. 2021-10-20 14:14:04:130 +0300 [INFO][main][Cluster] [default:ITDistributedConfigurationPropertiesTest_testFallingBehindDistributedStorageValue_10000:ef8b5306-36ea-4f4b-a7c6-cad7c253addf@127.0.1.1:10000][doShutdown] Shutting down 2021-10-20 14:14:04:131 +0300 [INFO][main][Cluster] [default:ITDistributedConfigurationPropertiesTest_testFallingBehindDistributedStorageValue_10000:ef8b5306-36ea-4f4b-a7c6-cad7c253addf@127.0.1.1:10000][leaveCluster] Leaving cluster 2021-10-20 14:14:04:142 +0300 [INFO][sc-cluster-10001-4][MembershipProtocol] [default:ITDistributedConfigurationPropertiesTest_testFallingBehindDistributedStorageValue_10001:32780ecd-a1b7-4f44-8dbe-41ec3ec4b04a@127.0.1.1:10001][publishEvent] MembershipEvent[type=LEAVING, member=default:ITDistributedConfigurationPropertiesTest_testFallingBehindDistributedStorageValue_10000:ef8b5306-36ea-4f4b-a7c6-cad7c253addf@127.0.1.1:10000, oldMetadata=null, newMetadata=7e16449-5, timestamp=2021-10-20T11:14:04.142Z] 2021-10-20 14:14:04:143 +0300 [INFO][sc-cluster-10001-4][ScaleCubeTopologyService] Topology snapshot [nodes=2] ^-- ClusterNode [id=ef8b5306-36ea-4f4b-a7c6-cad7c253addf, name=ITDistributedConfigurationPropertiesTest_testFallingBehindDistributedStorageValue_10000, address=127.0.1.1:10000] ^-- ClusterNode [id=32780ecd-a1b7-4f44-8dbe-41ec3ec4b04a, name=ITDistributedConfigurationPropertiesTest_testFallingBehindDistributedStorageValue_10001, address=127.0.1.1:10001] 2021-10-20 14:14:04:191 +0300 [INFO][sc-cluster-10000-3][Cluster] [default:ITDistributedConfigurationPropertiesTest_testFallingBehindDistributedStorageValue_10000:ef8b5306-36ea-4f4b-a7c6-cad7c253addf@127.0.1.1:10000][leaveCluster] Left cluster 2021-10-20 14:14:04:192 +0300 [INFO][sc-cluster-10000-3][Transport] Stopping 127.0.1.1:10000 2021-10-20 14:14:04:193 +0300 [WARNING][ForkJoinPool.commonPool-worker-5][MetaStorageServiceImpl] Cursor close command was rejected because raft executor has been already stopped. 2021-10-20 14:14:04:193 +0300 [INFO][sc-cluster-10000-3][Transport] Stopped 127.0.1.1:10000 2021-10-20 14:14:04:193 +0300 [INFO][sc-cluster-10000-3][Cluster] [default:ITDistributedConfigurationPropertiesTest_testFallingBehindDistributedStorageValue_10000:ef8b5306-36ea-4f4b-a7c6-cad7c253addf@127.0.1.1:10000][doShutdown] Shutdown 2021-10-20 14:14:04:212 +0300 [ERROR][main][MetaStorageManager] Failed to get watch. org.apache.ignite.internal.configuration.storage.StorageException: Exception when closing a Meta Storage cursor at org.apache.ignite.internal.configuration.storage.DistributedConfigurationStorage.readAllLatest(DistributedConfigurationStorage.java:142) at org.apache.ignite.internal.configuration.ConfigurationChanger.getLatest(ConfigurationChanger.java:316) at org.apache.ignite.internal.configuration.DirectDynamicProperty.directValue(DirectDynamicProperty.java:58) at org.apache.ignite.internal.configuration.DirectDynamicProperty.directValue(DirectDynamicProperty.java:31) at org.apache.ignite.internal.configuration.util.ConfigurationUtil.directValue(ConfigurationUtil.java:759) at org.apache.ignite.internal.configuration.ITDistributedConfigurationPropertiesTest.testFallingBehindDistributedStorageValue(ITDistributedConfigurationPropertiesTest.java:287) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:725) at org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60) at org.junit.jupiter.engine.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChain.java:131) at org.junit.jupiter.engine.extension.TimeoutExtension.intercept(TimeoutExtension.java:149) at org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestableMethod(TimeoutExtension.java:140) at org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestMethod(TimeoutExtension.java:84) at org.junit.jupiter.engine.execution.ExecutableInvoker$ReflectiveInterceptorCall.lambda$ofVoidMethod$0(ExecutableInvoker.java:115) at org.junit.jupiter.engine.execution.ExecutableInvoker.lambda$invoke$0(ExecutableInvoker.java:105) at org.junit.jupiter.engine.execution.InvocationInterceptorChain$InterceptedInvocation.proceed(InvocationInterceptorChain.java:106) at org.junit.jupiter.engine.execution.InvocationInterceptorChain.proceed(InvocationInterceptorChain.java:64) at org.junit.jupiter.engine.execution.InvocationInterceptorChain.chainAndInvoke(InvocationInterceptorChain.java:45) at org.junit.jupiter.engine.execution.InvocationInterceptorChain.invoke(InvocationInterceptorChain.java:37) at org.junit.jupiter.engine.execution.ExecutableInvoker.invoke(ExecutableInvoker.java:104) at org.junit.jupiter.engine.execution.ExecutableInvoker.invoke(ExecutableInvoker.java:98) at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.lambda$invokeTestMethod$7(TestMethodTestDescriptor.java:214) at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73) at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.invokeTestMethod(TestMethodTestDescriptor.java:210) at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:135) at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:66) at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:151) at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73) at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141) at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137) at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139) at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73) at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138) at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95) at java.base/java.util.ArrayList.forEach(ArrayList.java:1541) at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:41) at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:155) at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73) at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141) at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137) at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139) at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73) at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138) at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95) at java.base/java.util.ArrayList.forEach(ArrayList.java:1541) at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:41) at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:155) at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73) at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141) at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137) at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139) at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73) at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138) at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95) at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.submit(SameThreadHierarchicalTestExecutorService.java:35) at org.junit.platform.engine.support.hierarchical.HierarchicalTestExecutor.execute(HierarchicalTestExecutor.java:57) at org.junit.platform.engine.support.hierarchical.HierarchicalTestEngine.execute(HierarchicalTestEngine.java:54) at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:107) at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:88) at org.junit.platform.launcher.core.EngineExecutionOrchestrator.lambda$execute$0(EngineExecutionOrchestrator.java:54) at org.junit.platform.launcher.core.EngineExecutionOrchestrator.withInterceptedStreams(EngineExecutionOrchestrator.java:67) at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:52) at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:114) at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:86) at org.junit.platform.launcher.core.DefaultLauncherSession$DelegatingLauncher.execute(DefaultLauncherSession.java:86) at org.junit.platform.launcher.core.SessionPerRequestLauncher.execute(SessionPerRequestLauncher.java:53) at com.intellij.junit5.JUnit5IdeaTestRunner.startRunnerWithArgs(JUnit5IdeaTestRunner.java:71) at com.intellij.rt.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:52) at com.intellij.rt.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:235) at com.intellij.rt.junit.JUnitStarter.main(JUnitStarter.java:54) Suppressed: class org.apache.ignite.lang.IgniteInternalException: java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 3000ms in 'source(MonoDefer)' (and no fallback has been configured) at org.apache.ignite.internal.metastorage.MetaStorageManager.stop(MetaStorageManager.java:252) at org.apache.ignite.internal.configuration.ITDistributedConfigurationPropertiesTest$Node.stop(ITDistributedConfigurationPropertiesTest.java:212) at org.apache.ignite.internal.util.IgniteUtils.lambda$closeAll$0(IgniteUtils.java:481) at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.base/java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:177) at java.base/java.util.AbstractList$RandomAccessSpliterator.forEachRemaining(AbstractList.java:720) at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484) at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474) at java.base/java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150) at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173) at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) at java.base/java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:497) at org.apache.ignite.internal.util.IgniteUtils.closeAll(IgniteUtils.java:479) at org.apache.ignite.internal.util.IgniteUtils.closeAll(IgniteUtils.java:502) at org.apache.ignite.internal.configuration.ITDistributedConfigurationPropertiesTest.tearDown(ITDistributedConfigurationPropertiesTest.java:266) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:725) at org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60) at org.junit.jupiter.engine.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChain.java:131) at org.junit.jupiter.engine.extension.TimeoutExtension.intercept(TimeoutExtension.java:149) at org.junit.jupiter.engine.extension.TimeoutExtension.interceptLifecycleMethod(TimeoutExtension.java:126) at org.junit.jupiter.engine.extension.TimeoutExtension.interceptAfterEachMethod(TimeoutExtension.java:108) at org.junit.jupiter.engine.execution.ExecutableInvoker$ReflectiveInterceptorCall.lambda$ofVoidMethod$0(ExecutableInvoker.java:115) at org.junit.jupiter.engine.execution.ExecutableInvoker.lambda$invoke$0(ExecutableInvoker.java:105) at org.junit.jupiter.engine.execution.InvocationInterceptorChain$InterceptedInvocation.proceed(InvocationInterceptorChain.java:106) at org.junit.jupiter.engine.execution.InvocationInterceptorChain.proceed(InvocationInterceptorChain.java:64) at org.junit.jupiter.engine.execution.InvocationInterceptorChain.chainAndInvoke(InvocationInterceptorChain.java:45) at org.junit.jupiter.engine.execution.InvocationInterceptorChain.invoke(InvocationInterceptorChain.java:37) at org.junit.jupiter.engine.execution.ExecutableInvoker.invoke(ExecutableInvoker.java:104) at org.junit.jupiter.engine.execution.ExecutableInvoker.invoke(ExecutableInvoker.java:98) at org.junit.jupiter.engine.descriptor.ClassBasedTestDescriptor.invokeMethodInExtensionContext(ClassBasedTestDescriptor.java:506) at org.junit.jupiter.engine.descriptor.ClassBasedTestDescriptor.lambda$synthesizeAfterEachMethodAdapter$22(ClassBasedTestDescriptor.java:496) at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.lambda$invokeAfterEachMethods$10(TestMethodTestDescriptor.java:240) at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.lambda$invokeAllAfterMethodsOrCallbacks$13(TestMethodTestDescriptor.java:273) at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73) at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.lambda$invokeAllAfterMethodsOrCallbacks$14(TestMethodTestDescriptor.java:273) at java.base/java.util.ArrayList.forEach(ArrayList.java:1541) at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.invokeAllAfterMethodsOrCallbacks(TestMethodTestDescriptor.java:272) at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.invokeAfterEachMethods(TestMethodTestDescriptor.java:238) at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:139) ... 45 more Caused by: java.util.concurrent.ExecutionException: java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 3000ms in 'source(MonoDefer)' (and no fallback has been configured) at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:395) at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1999) at org.apache.ignite.internal.metastorage.MetaStorageManager.stop(MetaStorageManager.java:237) ... 87 more Caused by: java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 3000ms in 'source(MonoDefer)' (and no fallback has been configured) at reactor.core.publisher.FluxTimeout$TimeoutMainSubscriber.handleTimeout(FluxTimeout.java:289) at reactor.core.publisher.FluxTimeout$TimeoutMainSubscriber.doTimeout(FluxTimeout.java:274) at reactor.core.publisher.FluxTimeout$TimeoutTimeoutSubscriber.onNext(FluxTimeout.java:396) at reactor.core.publisher.StrictSubscriber.onNext(StrictSubscriber.java:89) at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onNext(FluxOnErrorResume.java:73) at reactor.core.publisher.MonoDelay$MonoDelayRunnable.run(MonoDelay.java:117) at reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:68) at reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:28) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) Caused by: class org.apache.ignite.lang.IgniteInternalException: class org.apache.ignite.lang.IgniteInternalException: java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 3000ms in 'source(MonoDefer)' (and no fallback has been configured) at org.apache.ignite.internal.metastorage.MetaStorageManager$CursorWrapper$InnerIterator.hasNext(MetaStorageManager.java:968) at org.apache.ignite.internal.configuration.storage.DistributedConfigurationStorage.readAllLatest(DistributedConfigurationStorage.java:123) ... 72 more Suppressed: java.util.concurrent.ExecutionException: class org.apache.ignite.lang.IgniteInternalException: java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 3000ms in 'source(MonoDefer)' (and no fallback has been configured) at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:395) at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1999) at org.apache.ignite.internal.metastorage.MetaStorageManager$CursorWrapper.close(MetaStorageManager.java:935) at org.apache.ignite.internal.configuration.storage.DistributedConfigurationStorage.readAllLatest(DistributedConfigurationStorage.java:122) ... 72 more Caused by: class org.apache.ignite.lang.IgniteInternalException: java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 3000ms in 'source(MonoDefer)' (and no fallback has been configured) at org.apache.ignite.internal.metastorage.MetaStorageManager$CursorWrapper.lambda$close$0(MetaStorageManager.java:933) at java.base/java.util.concurrent.CompletableFuture.uniApplyNow(CompletableFuture.java:680) at java.base/java.util.concurrent.CompletableFuture.uniApplyStage(CompletableFuture.java:658) at java.base/java.util.concurrent.CompletableFuture.thenApply(CompletableFuture.java:2094) at org.apache.ignite.internal.metastorage.MetaStorageManager$CursorWrapper.close(MetaStorageManager.java:926) ... 73 more Caused by: class org.apache.ignite.lang.IgniteInternalException: java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 3000ms in 'source(MonoDefer)' (and no fallback has been configured) at org.apache.ignite.internal.metastorage.client.CursorImpl.close(CursorImpl.java:84) at org.apache.ignite.internal.metastorage.MetaStorageManager$CursorWrapper.lambda$close$0(MetaStorageManager.java:928) ... 77 more Caused by: java.util.concurrent.ExecutionException: java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 3000ms in 'source(MonoDefer)' (and no fallback has been configured) at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:395) at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1999) at org.apache.ignite.internal.metastorage.client.CursorImpl.close(CursorImpl.java:76) ... 78 more Caused by: java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 3000ms in 'source(MonoDefer)' (and no fallback has been configured) at reactor.core.publisher.FluxTimeout$TimeoutMainSubscriber.handleTimeout(FluxTimeout.java:289) at reactor.core.publisher.FluxTimeout$TimeoutMainSubscriber.doTimeout(FluxTimeout.java:274) at reactor.core.publisher.FluxTimeout$TimeoutTimeoutSubscriber.onNext(FluxTimeout.java:396) at reactor.core.publisher.StrictSubscriber.onNext(StrictSubscriber.java:89) at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onNext(FluxOnErrorResume.java:73) at reactor.core.publisher.MonoDelay$MonoDelayRunnable.run(MonoDelay.java:117) at reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:68) at reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:28) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) Caused by: java.util.concurrent.ExecutionException: class org.apache.ignite.lang.IgniteInternalException: java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 3000ms in 'source(MonoDefer)' (and no fallback has been configured) at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:395) at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1999) at org.apache.ignite.internal.metastorage.MetaStorageManager$CursorWrapper$InnerIterator.hasNext(MetaStorageManager.java:965) ... 73 more Caused by: class org.apache.ignite.lang.IgniteInternalException: java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 3000ms in 'source(MonoDefer)' (and no fallback has been configured) at org.apache.ignite.internal.metastorage.client.CursorImpl$InnerIterator.hasNext(CursorImpl.java:112) at java.base/java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:642) at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2073) at org.apache.ignite.raft.jraft.rpc.impl.RaftGroupServiceImpl$1.accept(RaftGroupServiceImpl.java:552) at org.apache.ignite.raft.jraft.rpc.impl.RaftGroupServiceImpl$1.accept(RaftGroupServiceImpl.java:494) at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859) at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837) at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2073) at reactor.core.publisher.MonoToCompletableFuture.onNext(MonoToCompletableFuture.java:63) at reactor.core.publisher.SerializedSubscriber.onNext(SerializedSubscriber.java:99) at reactor.core.publisher.FluxTimeout$TimeoutMainSubscriber.onNext(FluxTimeout.java:174) at reactor.core.publisher.MonoCreate$DefaultMonoSink.success(MonoCreate.java:156) at reactor.core.publisher.LambdaSubscriber.onNext(LambdaSubscriber.java:160) at reactor.core.publisher.FluxTake$TakeFuseableSubscriber.onNext(FluxTake.java:398) at reactor.core.publisher.FluxFilterFuseable$FilterFuseableSubscriber.onNext(FluxFilterFuseable.java:112) at reactor.core.publisher.FluxFilterFuseable$FilterFuseableConditionalSubscriber.onNext(FluxFilterFuseable.java:330) at reactor.core.publisher.FluxOnBackpressureBuffer$BackpressureBufferSubscriber.drainRegular(FluxOnBackpressureBuffer.java:261) at reactor.core.publisher.FluxOnBackpressureBuffer$BackpressureBufferSubscriber.drain(FluxOnBackpressureBuffer.java:225) at reactor.core.publisher.FluxOnBackpressureBuffer$BackpressureBufferSubscriber.onNext(FluxOnBackpressureBuffer.java:184) at reactor.core.publisher.DirectProcessor$DirectInner.onNext(DirectProcessor.java:333) at reactor.core.publisher.DirectProcessor.onNext(DirectProcessor.java:142) at reactor.core.publisher.FluxCreate$IgnoreSink.next(FluxCreate.java:618) at reactor.core.publisher.FluxCreate$SerializedSink.next(FluxCreate.java:153) at org.apache.ignite.network.scalecube.ScaleCubeDirectMarshallerTransport.onMessage(ScaleCubeDirectMarshallerTransport.java:190) at org.apache.ignite.internal.network.netty.ConnectionManager.lambda$onMessage$2(ConnectionManager.java:203) at java.base/java.util.concurrent.CopyOnWriteArrayList.forEach(CopyOnWriteArrayList.java:807) at org.apache.ignite.internal.network.netty.ConnectionManager.onMessage(ConnectionManager.java:203) at org.apache.ignite.internal.network.netty.MessageHandler.channelRead(MessageHandler.java:47) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:324) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:296) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:719) at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:655) at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:581) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Thread.java:829) Caused by: java.util.concurrent.ExecutionException: java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 3000ms in 'source(MonoDefer)' (and no fallback has been configured) at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:395) at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1999) at org.apache.ignite.internal.metastorage.client.CursorImpl$InnerIterator.hasNext(CursorImpl.java:104) ... 50 more Caused by: [CIRCULAR REFERENCE: java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 3000ms in 'source(MonoDefer)' (and no fallback has been configured)]