2023-08-07 13:15:09:201 +0300 [INFO][main][MultiActorPlacementDriverTest] >>> Starting test: MultiActorPlacementDriverTest#testLeaseProlong, displayName: testLeaseProlong(), workDir: E:\projects\gridgain\ignite-3\modules\placement-driver\build\work\MultiActorPlacementDriverTest\tlp_29285 2023-08-07 13:15:09:266 +0300 [INFO][main][ConnectionManager] Server started [address=/0:0:0:0:0:0:0:0:1234] 2023-08-07 13:15:11:291 +0300 [INFO][main][ScaleCubeTopologyService] Node joined [node=ClusterNode [id=24f00a11-c233-431e-bcc5-bde71d7270c8, name=mapdt_tlp_1234, address=192.168.2.14:1234, nodeMetadata=null]] 2023-08-07 13:15:11:291 +0300 [INFO][main][ScaleCubeTopologyService] Topology snapshot [nodes=[mapdt_tlp_1234]] 2023-08-07 13:15:11:333 +0300 [INFO][main][Loza] Start new raft node=RaftNodeId [groupId=metastorage_group, peer=Peer [consistentId=mapdt_tlp_1234, idx=0]] with initial configuration=PeersAndLearners [peers=SetN [Peer [consistentId=mapdt_tlp_1236, idx=0], Peer [consistentId=mapdt_tlp_1235, idx=0], Peer [consistentId=mapdt_tlp_1234, idx=0]], learners=SetN []] 2023-08-07 13:15:11:333 +0300 [INFO][main][FSMCallerImpl] Starts FSMCaller successfully. 2023-08-07 13:15:11:333 +0300 [WARNING][main][LocalSnapshotStorage] No data for snapshot reader build\work\MultiActorPlacementDriverTest\tlp_29285\mapdt_tlp_1234_loza\metastorage_group_mapdt_tlp_1234_0\snapshot. 2023-08-07 13:15:11:333 +0300 [INFO][main][NodeImpl] Node init, term=0, lastLogId=LogId [index=0, term=0], conf=mapdt_tlp_1236,mapdt_tlp_1235,mapdt_tlp_1234, oldConf=. 2023-08-07 13:15:11:333 +0300 [INFO][main][RaftGroupService] Start the RaftGroupService successfully 2023-08-07 13:15:11:334 +0300 [INFO][main][TopologyTracker] Logical topology initialized for placement driver [topologySnap=LogicalTopologySnapshot [version=1, nodes=Set12 [LogicalNode [nodeAttributes=EmptyMap {}, super=ClusterNode [id=24f00a11-c233-431e-bcc5-bde71d7270c8, name=mapdt_tlp_1234, address=192.168.2.14:1234, nodeMetadata=null]]]]] 2023-08-07 13:15:11:334 +0300 [INFO][main][AssignmentsTracker] Assignment cache initialized for placement driver [groupAssignments={}] 2023-08-07 13:15:11:334 +0300 [INFO][main][LeaseTracker] Leases cache recovered [leases=IgniteBiTuple [val1=UnmodifiableMap {}, val2=[B@542ac2b2]] 2023-08-07 13:15:11:348 +0300 [INFO][main][ConnectionManager] Server started [address=/0:0:0:0:0:0:0:0:1235] 2023-08-07 13:15:11:353 +0300 [INFO][mapdt_tlp_1234-client-5][RecoveryClientHandshakeManager] Failed to acquire recovery descriptor during handshake, it is held by: [id: 0x562b583e, L:/192.168.2.14:1234 - R:/192.168.2.14:55012] 2023-08-07 13:15:11:354 +0300 [INFO][sc-cluster-1234-13321][ScaleCubeTopologyService] Node joined [node=ClusterNode [id=3cee3aa0-35dc-4595-a69d-fc799255627a, name=mapdt_tlp_1235, address=192.168.2.14:1235, nodeMetadata=null]] 2023-08-07 13:15:11:354 +0300 [INFO][sc-cluster-1234-13321][ScaleCubeTopologyService] Topology snapshot [nodes=[mapdt_tlp_1234, mapdt_tlp_1235]] 2023-08-07 13:15:11:355 +0300 [INFO][sc-cluster-1235-13322][ScaleCubeTopologyService] Node joined [node=ClusterNode [id=24f00a11-c233-431e-bcc5-bde71d7270c8, name=mapdt_tlp_1234, address=192.168.2.14:1234, nodeMetadata=null]] 2023-08-07 13:15:11:355 +0300 [INFO][sc-cluster-1235-13322][ScaleCubeTopologyService] Topology snapshot [nodes=[mapdt_tlp_1234]] 2023-08-07 13:15:12:901 +0300 [INFO][%mapdt_tlp_1234%JRaft-ElectionTimer-0][NodeImpl] Node term 0 start preVote. 2023-08-07 13:15:13:376 +0300 [INFO][main][ScaleCubeTopologyService] Node joined [node=ClusterNode [id=3cee3aa0-35dc-4595-a69d-fc799255627a, name=mapdt_tlp_1235, address=192.168.2.14:1235, nodeMetadata=null]] 2023-08-07 13:15:13:376 +0300 [INFO][main][ScaleCubeTopologyService] Topology snapshot [nodes=[mapdt_tlp_1234, mapdt_tlp_1235]] 2023-08-07 13:15:13:442 +0300 [INFO][main][Loza] Start new raft node=RaftNodeId [groupId=metastorage_group, peer=Peer [consistentId=mapdt_tlp_1235, idx=0]] with initial configuration=PeersAndLearners [peers=SetN [Peer [consistentId=mapdt_tlp_1236, idx=0], Peer [consistentId=mapdt_tlp_1235, idx=0], Peer [consistentId=mapdt_tlp_1234, idx=0]], learners=SetN []] 2023-08-07 13:15:13:443 +0300 [INFO][main][FSMCallerImpl] Starts FSMCaller successfully. 2023-08-07 13:15:13:443 +0300 [WARNING][main][LocalSnapshotStorage] No data for snapshot reader build\work\MultiActorPlacementDriverTest\tlp_29285\mapdt_tlp_1235_loza\metastorage_group_mapdt_tlp_1235_0\snapshot. 2023-08-07 13:15:13:443 +0300 [INFO][main][NodeImpl] Node init, term=0, lastLogId=LogId [index=0, term=0], conf=mapdt_tlp_1236,mapdt_tlp_1235,mapdt_tlp_1234, oldConf=. 2023-08-07 13:15:13:443 +0300 [INFO][main][RaftGroupService] Start the RaftGroupService successfully 2023-08-07 13:15:13:443 +0300 [INFO][main][TopologyTracker] Logical topology initialized for placement driver [topologySnap=LogicalTopologySnapshot [version=1, nodes=Set12 [LogicalNode [nodeAttributes=EmptyMap {}, super=ClusterNode [id=24f00a11-c233-431e-bcc5-bde71d7270c8, name=mapdt_tlp_1234, address=192.168.2.14:1234, nodeMetadata=null]], LogicalNode [nodeAttributes=EmptyMap {}, super=ClusterNode [id=3cee3aa0-35dc-4595-a69d-fc799255627a, name=mapdt_tlp_1235, address=192.168.2.14:1235, nodeMetadata=null]]]]] 2023-08-07 13:15:13:443 +0300 [INFO][main][AssignmentsTracker] Assignment cache initialized for placement driver [groupAssignments={}] 2023-08-07 13:15:13:443 +0300 [INFO][main][LeaseTracker] Leases cache recovered [leases=IgniteBiTuple [val1=UnmodifiableMap {}, val2=[B@763ca4e6]] 2023-08-07 13:15:13:458 +0300 [INFO][main][ConnectionManager] Server started [address=/0:0:0:0:0:0:0:0:1236] 2023-08-07 13:15:13:474 +0300 [INFO][mapdt_tlp_1236-client-5][RecoveryClientHandshakeManager] Failed to acquire recovery descriptor during handshake, it is held by: [id: 0x42b550e8, L:/192.168.2.14:1236 - R:/192.168.2.14:55120] 2023-08-07 13:15:13:474 +0300 [INFO][sc-cluster-1235-13322][ScaleCubeTopologyService] Node joined [node=ClusterNode [id=e3819f58-dc89-4385-8fe4-9e2244a17dfe, name=mapdt_tlp_1236, address=192.168.2.14:1236, nodeMetadata=null]] 2023-08-07 13:15:13:474 +0300 [INFO][sc-cluster-1235-13322][ScaleCubeTopologyService] Topology snapshot [nodes=[mapdt_tlp_1236, mapdt_tlp_1234, mapdt_tlp_1235]] 2023-08-07 13:15:13:476 +0300 [INFO][mapdt_tlp_1236-client-6][RecoveryClientHandshakeManager] Failed to acquire recovery descriptor during handshake, it is held by: [id: 0x42b550e8, L:/192.168.2.14:1236 - R:/192.168.2.14:55120] 2023-08-07 13:15:13:476 +0300 [INFO][sc-cluster-1236-13323][ScaleCubeTopologyService] Node joined [node=ClusterNode [id=3cee3aa0-35dc-4595-a69d-fc799255627a, name=mapdt_tlp_1235, address=192.168.2.14:1235, nodeMetadata=null]] 2023-08-07 13:15:13:476 +0300 [INFO][sc-cluster-1236-13323][ScaleCubeTopologyService] Topology snapshot [nodes=[mapdt_tlp_1235]] 2023-08-07 13:15:13:821 +0300 [INFO][sc-cluster-1236-13323][ScaleCubeTopologyService] Node joined [node=ClusterNode [id=24f00a11-c233-431e-bcc5-bde71d7270c8, name=mapdt_tlp_1234, address=192.168.2.14:1234, nodeMetadata=null]] 2023-08-07 13:15:13:821 +0300 [INFO][sc-cluster-1236-13323][ScaleCubeTopologyService] Topology snapshot [nodes=[mapdt_tlp_1234, mapdt_tlp_1235]] 2023-08-07 13:15:13:822 +0300 [INFO][mapdt_tlp_1236-client-7][RecoveryClientHandshakeManager] Failed to acquire recovery descriptor during handshake, it is held by: [id: 0xe59906cd, L:/192.168.2.14:1236 - R:/192.168.2.14:55018] 2023-08-07 13:15:13:822 +0300 [INFO][sc-cluster-1234-13321][ScaleCubeTopologyService] Node joined [node=ClusterNode [id=e3819f58-dc89-4385-8fe4-9e2244a17dfe, name=mapdt_tlp_1236, address=192.168.2.14:1236, nodeMetadata=null]] 2023-08-07 13:15:13:822 +0300 [INFO][sc-cluster-1234-13321][ScaleCubeTopologyService] Topology snapshot [nodes=[mapdt_tlp_1236, mapdt_tlp_1234, mapdt_tlp_1235]] 2023-08-07 13:15:14:492 +0300 [INFO][%mapdt_tlp_1234%JRaft-ElectionTimer-0][NodeImpl] Unsuccessful election round number 2 2023-08-07 13:15:14:492 +0300 [INFO][%mapdt_tlp_1234%JRaft-ElectionTimer-0][NodeImpl] Node term 0 start preVote. 2023-08-07 13:15:14:493 +0300 [INFO][%mapdt_tlp_1235%JRaft-Request-Processor-1][NodeImpl] Node received PreVoteRequest from mapdt_tlp_1234, term=1, currTerm=0, granted=true, requestLastLogId=LogId [index=0, term=0], lastLogId=LogId [index=0, term=0]. 2023-08-07 13:15:14:493 +0300 [INFO][%mapdt_tlp_1234%JRaft-Response-Processor-3][NodeImpl] Node received PreVoteResponse from mapdt_tlp_1235, term=0, granted=true. 2023-08-07 13:15:14:494 +0300 [INFO][%mapdt_tlp_1234%JRaft-Response-Processor-3][NodeImpl] Node start vote and grant vote self, term=0. 2023-08-07 13:15:14:494 +0300 [INFO][%mapdt_tlp_1235%JRaft-Request-Processor-2][NodeImpl] Node received RequestVoteRequest from mapdt_tlp_1234, term=1, currTerm=0. 2023-08-07 13:15:14:502 +0300 [INFO][%mapdt_tlp_1234%JRaft-Response-Processor-3][LocalRaftMetaStorage] Save raft meta, path=build\work\MultiActorPlacementDriverTest\tlp_29285\mapdt_tlp_1234_loza\metastorage_group_mapdt_tlp_1234_0\meta, term=1, votedFor=mapdt_tlp_1234, cost time=9 ms 2023-08-07 13:15:14:503 +0300 [INFO][%mapdt_tlp_1235%JRaft-Request-Processor-2][LocalRaftMetaStorage] Save raft meta, path=build\work\MultiActorPlacementDriverTest\tlp_29285\mapdt_tlp_1235_loza\metastorage_group_mapdt_tlp_1235_0\meta, term=1, votedFor=, cost time=9 ms 2023-08-07 13:15:14:507 +0300 [INFO][%mapdt_tlp_1235%JRaft-Request-Processor-2][LocalRaftMetaStorage] Save raft meta, path=build\work\MultiActorPlacementDriverTest\tlp_29285\mapdt_tlp_1235_loza\metastorage_group_mapdt_tlp_1235_0\meta, term=1, votedFor=mapdt_tlp_1234, cost time=3 ms 2023-08-07 13:15:14:507 +0300 [INFO][%mapdt_tlp_1234%JRaft-Response-Processor-4][NodeImpl] Node become leader of group, term=1, conf=mapdt_tlp_1236,mapdt_tlp_1235,mapdt_tlp_1234, oldConf=. 2023-08-07 13:15:14:918 +0300 [ERROR][%mapdt_tlp_1234%JRaft-Response-Processor-4][AbstractClientService] Fail to connect mapdt_tlp_1236, exception: java.util.concurrent.TimeoutException. 2023-08-07 13:15:14:918 +0300 [ERROR][%mapdt_tlp_1234%JRaft-Response-Processor-4][ReplicatorGroupImpl] Fail to check replicator connection to peer=mapdt_tlp_1236, replicatorType=Follower. 2023-08-07 13:15:14:918 +0300 [ERROR][%mapdt_tlp_1234%JRaft-Response-Processor-4][NodeImpl] Fail to add a replicator, peer=mapdt_tlp_1236. 2023-08-07 13:15:14:918 +0300 [INFO][%mapdt_tlp_1234%JRaft-Response-Processor-4][Replicator] Replicator=Replicator [state=Created, statInfo=, peerId=mapdt_tlp_1235, type=Follower]@mapdt_tlp_1235 is started 2023-08-07 13:15:14:919 +0300 [INFO][%mapdt_tlp_1235%JRaft-FSMCaller-Disruptor-metastorage-_stripe_0-0][StateMachineAdapter] onStartFollowing: LeaderChangeContext [leaderId=mapdt_tlp_1234, term=1, status=Status[ENEWLEADER<10011>: Follower receives message from new leader with the same term.]]. 2023-08-07 13:15:14:925 +0300 [INFO][%mapdt_tlp_1234%JRaft-FSMCaller-Disruptor-metastorage-_stripe_0-0][StateMachineAdapter] onLeaderStart: term=1. 2023-08-07 13:15:14:925 +0300 [INFO][%mapdt_tlp_1234%JRaft-Request-Processor-8][MetaStorageLeaderElectionListener] Node has been elected as the leader, starting Idle Safe Time scheduler 2023-08-07 13:15:14:926 +0300 [INFO][%mapdt_tlp_1234%JRaft-Request-Processor-7][PlacementDriverManager] Placement driver active actor is starting. 2023-08-07 13:15:14:926 +0300 [INFO][%mapdt_tlp_1235%JRaft-Request-Processor-8][PlacementDriverManager] Placement driver active actor is stopping. 2023-08-07 13:15:15:126 +0300 [INFO][%mapdt_tlp_1235%MessagingService-inbound--0][MetaStorageManagerImpl] Performing MetaStorage recovery from revision 0 to 0 2023-08-07 13:15:15:127 +0300 [INFO][%mapdt_tlp_1235%MessagingService-inbound--0][MetaStorageManagerImpl] Finished MetaStorage recovery 2023-08-07 13:15:15:128 +0300 [INFO][%mapdt_tlp_1234%JRaft-AppendEntries-Processor-0][MetaStorageManagerImpl] Performing MetaStorage recovery from revision 0 to 0 2023-08-07 13:15:15:128 +0300 [INFO][%mapdt_tlp_1234%JRaft-AppendEntries-Processor-0][RaftGroupServiceImpl] Sending changePeersAsync request for group=metastorage_group to peers=[Peer [consistentId=mapdt_tlp_1236, idx=0], Peer [consistentId=mapdt_tlp_1235, idx=0], Peer [consistentId=mapdt_tlp_1234, idx=0]] and learners=[] with leader term=1 2023-08-07 13:15:15:130 +0300 [INFO][%mapdt_tlp_1234%JRaft-Request-Processor-14][BaseCliRequestProcessor] Receive ChangePeersAsyncRequest with term 1 to from 192.168.2.14:1234, new conf is mapdt_tlp_1236,mapdt_tlp_1235,mapdt_tlp_1234 2023-08-07 13:15:15:130 +0300 [INFO][%mapdt_tlp_1234%JRaft-Request-Processor-14][NodeImpl] Node change peers from mapdt_tlp_1236,mapdt_tlp_1235,mapdt_tlp_1234 to mapdt_tlp_1236,mapdt_tlp_1235,mapdt_tlp_1234. 2023-08-07 13:15:15:133 +0300 [INFO][%mapdt_tlp_1234%JRaft-AppendEntries-Processor-0][MetaStorageManagerImpl] Finished MetaStorage recovery 2023-08-07 13:15:15:483 +0300 [INFO][main][ScaleCubeTopologyService] Node joined [node=ClusterNode [id=e3819f58-dc89-4385-8fe4-9e2244a17dfe, name=mapdt_tlp_1236, address=192.168.2.14:1236, nodeMetadata=null]] 2023-08-07 13:15:15:483 +0300 [INFO][main][ScaleCubeTopologyService] Topology snapshot [nodes=[mapdt_tlp_1236, mapdt_tlp_1234, mapdt_tlp_1235]] 2023-08-07 13:15:15:529 +0300 [INFO][%mapdt_tlp_1234%JRaft-StepDownTimer-0][Replicator] Replicator=Replicator [state=Created, statInfo=, peerId=mapdt_tlp_1236, type=Follower]@mapdt_tlp_1236 is started 2023-08-07 13:15:15:538 +0300 [INFO][main][Loza] Start new raft node=RaftNodeId [groupId=metastorage_group, peer=Peer [consistentId=mapdt_tlp_1236, idx=0]] with initial configuration=PeersAndLearners [peers=SetN [Peer [consistentId=mapdt_tlp_1236, idx=0], Peer [consistentId=mapdt_tlp_1235, idx=0], Peer [consistentId=mapdt_tlp_1234, idx=0]], learners=SetN []] 2023-08-07 13:15:15:538 +0300 [INFO][main][FSMCallerImpl] Starts FSMCaller successfully. 2023-08-07 13:15:15:539 +0300 [WARNING][main][LocalSnapshotStorage] No data for snapshot reader build\work\MultiActorPlacementDriverTest\tlp_29285\mapdt_tlp_1236_loza\metastorage_group_mapdt_tlp_1236_0\snapshot. 2023-08-07 13:15:15:539 +0300 [INFO][main][NodeImpl] Node init, term=0, lastLogId=LogId [index=0, term=0], conf=mapdt_tlp_1236,mapdt_tlp_1235,mapdt_tlp_1234, oldConf=. 2023-08-07 13:15:15:539 +0300 [INFO][main][RaftGroupService] Start the RaftGroupService successfully 2023-08-07 13:15:15:539 +0300 [INFO][main][TopologyTracker] Logical topology initialized for placement driver [topologySnap=LogicalTopologySnapshot [version=1, nodes=SetN [LogicalNode [nodeAttributes=EmptyMap {}, super=ClusterNode [id=e3819f58-dc89-4385-8fe4-9e2244a17dfe, name=mapdt_tlp_1236, address=192.168.2.14:1236, nodeMetadata=null]], LogicalNode [nodeAttributes=EmptyMap {}, super=ClusterNode [id=3cee3aa0-35dc-4595-a69d-fc799255627a, name=mapdt_tlp_1235, address=192.168.2.14:1235, nodeMetadata=null]], LogicalNode [nodeAttributes=EmptyMap {}, super=ClusterNode [id=24f00a11-c233-431e-bcc5-bde71d7270c8, name=mapdt_tlp_1234, address=192.168.2.14:1234, nodeMetadata=null]]]]] 2023-08-07 13:15:15:539 +0300 [INFO][main][AssignmentsTracker] Assignment cache initialized for placement driver [groupAssignments={}] 2023-08-07 13:15:15:539 +0300 [INFO][main][LeaseTracker] Leases cache recovered [leases=IgniteBiTuple [val1=UnmodifiableMap {}, val2=[B@437c6f5b]] 2023-08-07 13:15:15:541 +0300 [INFO][%mapdt_tlp_1236%MessagingService-inbound--0][MetaStorageManagerImpl] Performing MetaStorage recovery from revision 0 to 0 2023-08-07 13:15:15:541 +0300 [INFO][%mapdt_tlp_1236%MessagingService-inbound--0][MetaStorageManagerImpl] Finished MetaStorage recovery 2023-08-07 13:15:15:550 +0300 [INFO][%mapdt_tlp_1236%JRaft-AppendEntries-Processor-0][LocalRaftMetaStorage] Save raft meta, path=build\work\MultiActorPlacementDriverTest\tlp_29285\mapdt_tlp_1236_loza\metastorage_group_mapdt_tlp_1236_0\meta, term=1, votedFor=, cost time=9 ms 2023-08-07 13:15:15:550 +0300 [WARNING][%mapdt_tlp_1236%JRaft-AppendEntries-Processor-0][NodeImpl] Node reject term_unmatched AppendEntriesRequest from mapdt_tlp_1234, term=1, prevLogIndex=1, prevLogTerm=1, localPrevLogTerm=0, lastLogIndex=0, entriesSize=0. 2023-08-07 13:15:15:550 +0300 [INFO][%mapdt_tlp_1236%JRaft-FSMCaller-Disruptor-metastorage-_stripe_0-0][StateMachineAdapter] onStartFollowing: LeaderChangeContext [leaderId=mapdt_tlp_1234, term=1, status=Status[ENEWLEADER<10011>: Raft node receives message from new leader with higher term.]]. 2023-08-07 13:15:15:551 +0300 [INFO][%mapdt_tlp_1236%Raft-Group-Client-4][PlacementDriverManager] Placement driver active actor is stopping. 2023-08-07 13:15:15:562 +0300 [INFO][main][ConnectionManager] Server started [address=/0:0:0:0:0:0:0:0:1237] 2023-08-07 13:15:15:567 +0300 [INFO][mapdt_tlp_1237-client-5][RecoveryClientHandshakeManager] Failed to acquire recovery descriptor during handshake, it is held by: [id: 0xc41cf56b, L:/192.168.2.14:1237 - R:/192.168.2.14:55228] 2023-08-07 13:15:15:567 +0300 [INFO][mapdt_tlp_1237-client-6][RecoveryClientHandshakeManager] Failed to acquire recovery descriptor during handshake, it is held by: [id: 0x9521ad44, L:/192.168.2.14:1237 - R:/192.168.2.14:55229] 2023-08-07 13:15:15:567 +0300 [INFO][mapdt_tlp_1237-client-7][RecoveryClientHandshakeManager] Failed to acquire recovery descriptor during handshake, it is held by: [id: 0x77a0a846, L:/192.168.2.14:1237 - R:/192.168.2.14:55230] 2023-08-07 13:15:15:567 +0300 [INFO][sc-cluster-1234-13321][ScaleCubeTopologyService] Node joined [node=ClusterNode [id=c016acd7-457a-47d1-bf28-8cbe0e760468, name=mapdt_tlp_1237, address=192.168.2.14:1237, nodeMetadata=null]] 2023-08-07 13:15:15:567 +0300 [INFO][sc-cluster-1235-13322][ScaleCubeTopologyService] Node joined [node=ClusterNode [id=c016acd7-457a-47d1-bf28-8cbe0e760468, name=mapdt_tlp_1237, address=192.168.2.14:1237, nodeMetadata=null]] 2023-08-07 13:15:15:567 +0300 [INFO][sc-cluster-1234-13321][ScaleCubeTopologyService] Topology snapshot [nodes=[mapdt_tlp_1236, mapdt_tlp_1237, mapdt_tlp_1234, mapdt_tlp_1235]] 2023-08-07 13:15:15:567 +0300 [INFO][sc-cluster-1235-13322][ScaleCubeTopologyService] Topology snapshot [nodes=[mapdt_tlp_1236, mapdt_tlp_1237, mapdt_tlp_1234, mapdt_tlp_1235]] 2023-08-07 13:15:15:567 +0300 [INFO][sc-cluster-1236-13323][ScaleCubeTopologyService] Node joined [node=ClusterNode [id=c016acd7-457a-47d1-bf28-8cbe0e760468, name=mapdt_tlp_1237, address=192.168.2.14:1237, nodeMetadata=null]] 2023-08-07 13:15:15:567 +0300 [INFO][sc-cluster-1236-13323][ScaleCubeTopologyService] Topology snapshot [nodes=[mapdt_tlp_1236, mapdt_tlp_1237, mapdt_tlp_1234, mapdt_tlp_1235]] 2023-08-07 13:15:15:568 +0300 [INFO][mapdt_tlp_1237-client-8][RecoveryClientHandshakeManager] Failed to acquire recovery descriptor during handshake, it is held by: [id: 0x77a0a846, L:/192.168.2.14:1237 - R:/192.168.2.14:55230] 2023-08-07 13:15:15:568 +0300 [INFO][mapdt_tlp_1237-client-9][RecoveryClientHandshakeManager] Failed to acquire recovery descriptor during handshake, it is held by: [id: 0xc41cf56b, L:/192.168.2.14:1237 - R:/192.168.2.14:55228] 2023-08-07 13:15:15:568 +0300 [INFO][mapdt_tlp_1237-client-10][RecoveryClientHandshakeManager] Failed to acquire recovery descriptor during handshake, it is held by: [id: 0x9521ad44, L:/192.168.2.14:1237 - R:/192.168.2.14:55229] 2023-08-07 13:15:15:569 +0300 [INFO][sc-cluster-1237-13324][ScaleCubeTopologyService] Node joined [node=ClusterNode [id=24f00a11-c233-431e-bcc5-bde71d7270c8, name=mapdt_tlp_1234, address=192.168.2.14:1234, nodeMetadata=null]] 2023-08-07 13:15:15:569 +0300 [INFO][sc-cluster-1237-13324][ScaleCubeTopologyService] Topology snapshot [nodes=[mapdt_tlp_1234]] 2023-08-07 13:15:15:569 +0300 [INFO][sc-cluster-1237-13324][ScaleCubeTopologyService] Node joined [node=ClusterNode [id=3cee3aa0-35dc-4595-a69d-fc799255627a, name=mapdt_tlp_1235, address=192.168.2.14:1235, nodeMetadata=null]] 2023-08-07 13:15:15:569 +0300 [INFO][sc-cluster-1237-13324][ScaleCubeTopologyService] Topology snapshot [nodes=[mapdt_tlp_1234, mapdt_tlp_1235]] 2023-08-07 13:15:15:576 +0300 [INFO][sc-cluster-1237-13324][ScaleCubeTopologyService] Node joined [node=ClusterNode [id=e3819f58-dc89-4385-8fe4-9e2244a17dfe, name=mapdt_tlp_1236, address=192.168.2.14:1236, nodeMetadata=null]] 2023-08-07 13:15:15:576 +0300 [INFO][sc-cluster-1237-13324][ScaleCubeTopologyService] Topology snapshot [nodes=[mapdt_tlp_1236, mapdt_tlp_1234, mapdt_tlp_1235]] 2023-08-07 13:15:15:655 +0300 [WARNING][%mapdt_tlp_1236%JRaft-AppendEntries-Processor-0][NodeImpl] Node reject term_unmatched AppendEntriesRequest from mapdt_tlp_1234, term=1, prevLogIndex=1, prevLogTerm=1, localPrevLogTerm=0, lastLogIndex=0, entriesSize=0. 2023-08-07 13:15:15:655 +0300 [WARNING][%mapdt_tlp_1236%JRaft-AppendEntries-Processor-0][NodeImpl] Node reject term_unmatched AppendEntriesRequest from mapdt_tlp_1234, term=1, prevLogIndex=1, prevLogTerm=1, localPrevLogTerm=0, lastLogIndex=0, entriesSize=0. 2023-08-07 13:15:17:598 +0300 [INFO][main][ScaleCubeTopologyService] Node joined [node=ClusterNode [id=c016acd7-457a-47d1-bf28-8cbe0e760468, name=mapdt_tlp_1237, address=192.168.2.14:1237, nodeMetadata=null]] 2023-08-07 13:15:17:598 +0300 [INFO][main][ScaleCubeTopologyService] Topology snapshot [nodes=[mapdt_tlp_1236, mapdt_tlp_1237, mapdt_tlp_1234, mapdt_tlp_1235]] 2023-08-07 13:15:17:617 +0300 [INFO][main][ConnectionManager] Server started [address=/0:0:0:0:0:0:0:0:1238] 2023-08-07 13:15:17:620 +0300 [INFO][mapdt_tlp_1237-client-11][RecoveryClientHandshakeManager] Failed to acquire recovery descriptor during handshake, it is held by: [id: 0xdb2d18aa, L:/192.168.2.14:1237 - R:/192.168.2.14:55339] 2023-08-07 13:15:17:621 +0300 [INFO][sc-cluster-1237-13324][ScaleCubeTopologyService] Node joined [node=ClusterNode [id=f63cb080-a0f0-48c7-8a5e-8f3299afbecf, name=mapdt_tlp_1238, address=192.168.2.14:1238, nodeMetadata=null]] 2023-08-07 13:15:17:621 +0300 [INFO][sc-cluster-1237-13324][ScaleCubeTopologyService] Topology snapshot [nodes=[mapdt_tlp_1238, mapdt_tlp_1236, mapdt_tlp_1237, mapdt_tlp_1234, mapdt_tlp_1235]] 2023-08-07 13:15:17:622 +0300 [INFO][sc-cluster-1238-13325][ScaleCubeTopologyService] Node joined [node=ClusterNode [id=c016acd7-457a-47d1-bf28-8cbe0e760468, name=mapdt_tlp_1237, address=192.168.2.14:1237, nodeMetadata=null]] 2023-08-07 13:15:17:622 +0300 [INFO][sc-cluster-1238-13325][ScaleCubeTopologyService] Topology snapshot [nodes=[mapdt_tlp_1237]] 2023-08-07 13:15:17:630 +0300 [INFO][mapdt_tlp_1236-client-9][RecoveryClientHandshakeManager] Failed to acquire recovery descriptor during handshake, it is held by: [id: 0x2b35b5f4, L:/192.168.2.14:1236 - R:/192.168.2.14:55338] 2023-08-07 13:15:17:630 +0300 [INFO][sc-cluster-1238-13325][ScaleCubeTopologyService] Node joined [node=ClusterNode [id=e3819f58-dc89-4385-8fe4-9e2244a17dfe, name=mapdt_tlp_1236, address=192.168.2.14:1236, nodeMetadata=null]] 2023-08-07 13:15:17:630 +0300 [INFO][sc-cluster-1238-13325][ScaleCubeTopologyService] Topology snapshot [nodes=[mapdt_tlp_1236, mapdt_tlp_1237]] 2023-08-07 13:15:17:630 +0300 [INFO][sc-cluster-1236-13323][ScaleCubeTopologyService] Node joined [node=ClusterNode [id=f63cb080-a0f0-48c7-8a5e-8f3299afbecf, name=mapdt_tlp_1238, address=192.168.2.14:1238, nodeMetadata=null]] 2023-08-07 13:15:17:631 +0300 [INFO][sc-cluster-1236-13323][ScaleCubeTopologyService] Topology snapshot [nodes=[mapdt_tlp_1238, mapdt_tlp_1236, mapdt_tlp_1237, mapdt_tlp_1234, mapdt_tlp_1235]] 2023-08-07 13:15:17:631 +0300 [INFO][sc-cluster-1238-13325][ScaleCubeTopologyService] Node joined [node=ClusterNode [id=3cee3aa0-35dc-4595-a69d-fc799255627a, name=mapdt_tlp_1235, address=192.168.2.14:1235, nodeMetadata=null]] 2023-08-07 13:15:17:631 +0300 [INFO][sc-cluster-1238-13325][ScaleCubeTopologyService] Topology snapshot [nodes=[mapdt_tlp_1236, mapdt_tlp_1237, mapdt_tlp_1235]] 2023-08-07 13:15:17:631 +0300 [INFO][sc-cluster-1238-13325][ScaleCubeTopologyService] Node joined [node=ClusterNode [id=24f00a11-c233-431e-bcc5-bde71d7270c8, name=mapdt_tlp_1234, address=192.168.2.14:1234, nodeMetadata=null]] 2023-08-07 13:15:17:631 +0300 [INFO][sc-cluster-1238-13325][ScaleCubeTopologyService] Topology snapshot [nodes=[mapdt_tlp_1236, mapdt_tlp_1237, mapdt_tlp_1234, mapdt_tlp_1235]] 2023-08-07 13:15:17:632 +0300 [INFO][mapdt_tlp_1238-client-5][RecoveryClientHandshakeManager] Failed to acquire recovery descriptor during handshake, it is held by: [id: 0x5c5c6ae8, L:/192.168.2.14:1238 - R:/192.168.2.14:55342] 2023-08-07 13:15:17:632 +0300 [INFO][mapdt_tlp_1238-client-6][RecoveryClientHandshakeManager] Failed to acquire recovery descriptor during handshake, it is held by: [id: 0x1b398a1e, L:/192.168.2.14:1238 - R:/192.168.2.14:55343] 2023-08-07 13:15:17:632 +0300 [INFO][sc-cluster-1235-13322][ScaleCubeTopologyService] Node joined [node=ClusterNode [id=f63cb080-a0f0-48c7-8a5e-8f3299afbecf, name=mapdt_tlp_1238, address=192.168.2.14:1238, nodeMetadata=null]] 2023-08-07 13:15:17:632 +0300 [INFO][sc-cluster-1235-13322][ScaleCubeTopologyService] Topology snapshot [nodes=[mapdt_tlp_1238, mapdt_tlp_1236, mapdt_tlp_1237, mapdt_tlp_1234, mapdt_tlp_1235]] 2023-08-07 13:15:17:632 +0300 [INFO][sc-cluster-1234-13321][ScaleCubeTopologyService] Node joined [node=ClusterNode [id=f63cb080-a0f0-48c7-8a5e-8f3299afbecf, name=mapdt_tlp_1238, address=192.168.2.14:1238, nodeMetadata=null]] 2023-08-07 13:15:17:632 +0300 [INFO][sc-cluster-1234-13321][ScaleCubeTopologyService] Topology snapshot [nodes=[mapdt_tlp_1238, mapdt_tlp_1236, mapdt_tlp_1237, mapdt_tlp_1234, mapdt_tlp_1235]] 2023-08-07 13:15:17:632 +0300 [INFO][main][ScaleCubeTopologyService] Node joined [node=ClusterNode [id=f63cb080-a0f0-48c7-8a5e-8f3299afbecf, name=mapdt_tlp_1238, address=192.168.2.14:1238, nodeMetadata=null]] 2023-08-07 13:15:17:632 +0300 [INFO][main][ScaleCubeTopologyService] Topology snapshot [nodes=[mapdt_tlp_1238, mapdt_tlp_1236, mapdt_tlp_1237, mapdt_tlp_1234, mapdt_tlp_1235]] 2023-08-07 13:15:17:632 +0300 [INFO][main][RaftGroupServiceImpl] Sending changePeersAsync request for group=metastorage_group to peers=[Peer [consistentId=mapdt_tlp_1236, idx=0], Peer [consistentId=mapdt_tlp_1235, idx=0], Peer [consistentId=mapdt_tlp_1234, idx=0]] and learners=[Peer [consistentId=mapdt_tlp_1238, idx=0], Peer [consistentId=mapdt_tlp_1237, idx=0]] with leader term=1 2023-08-07 13:15:17:633 +0300 [INFO][%mapdt_tlp_1234%JRaft-Request-Processor-26][BaseCliRequestProcessor] Receive ChangePeersAsyncRequest with term 1 to from 192.168.2.14:1234, new conf is mapdt_tlp_1236,mapdt_tlp_1235,mapdt_tlp_1234,mapdt_tlp_1238/learner,mapdt_tlp_1237/learner 2023-08-07 13:15:17:633 +0300 [INFO][%mapdt_tlp_1234%JRaft-Request-Processor-26][NodeImpl] Node change peers from mapdt_tlp_1236,mapdt_tlp_1235,mapdt_tlp_1234 to mapdt_tlp_1236,mapdt_tlp_1235,mapdt_tlp_1234,mapdt_tlp_1238/learner,mapdt_tlp_1237/learner. 2023-08-07 13:15:17:633 +0300 [INFO][%mapdt_tlp_1234%JRaft-Request-Processor-26][NodeImpl] Adding learners: [mapdt_tlp_1238, mapdt_tlp_1237]. 2023-08-07 13:15:17:930 +0300 [INFO][%mapdt_tlp_1235%Raft-Group-Client-10][TopologyAwareRaftGroupService] Could not subscribe to leader update from a specific node, because the node had left from the cluster [node=ClusterNode [id=e3819f58-dc89-4385-8fe4-9e2244a17dfe, name=mapdt_tlp_1236, address=192.168.2.14:1236, nodeMetadata=null]] 2023-08-07 13:15:17:930 +0300 [INFO][%mapdt_tlp_1235%Raft-Group-Client-11][TopologyAwareRaftGroupService] Could not subscribe to leader update from a specific node, because the node had left from the cluster [node=ClusterNode [id=e3819f58-dc89-4385-8fe4-9e2244a17dfe, name=mapdt_tlp_1236, address=192.168.2.14:1236, nodeMetadata=null]] 2023-08-07 13:15:17:930 +0300 [INFO][%mapdt_tlp_1234%Raft-Group-Client-4][TopologyAwareRaftGroupService] Could not subscribe to leader update from a specific node, because the node had left from the cluster [node=ClusterNode [id=e3819f58-dc89-4385-8fe4-9e2244a17dfe, name=mapdt_tlp_1236, address=192.168.2.14:1236, nodeMetadata=null]] 2023-08-07 13:15:17:930 +0300 [INFO][%mapdt_tlp_1234%Raft-Group-Client-19][TopologyAwareRaftGroupService] Could not subscribe to leader update from a specific node, because the node had left from the cluster [node=ClusterNode [id=e3819f58-dc89-4385-8fe4-9e2244a17dfe, name=mapdt_tlp_1236, address=192.168.2.14:1236, nodeMetadata=null]] 2023-08-07 13:15:18:039 +0300 [ERROR][%mapdt_tlp_1234%JRaft-Request-Processor-26][AbstractClientService] Fail to connect mapdt_tlp_1238, exception: java.util.concurrent.TimeoutException. 2023-08-07 13:15:18:039 +0300 [ERROR][%mapdt_tlp_1234%JRaft-Request-Processor-26][ReplicatorGroupImpl] Fail to check replicator connection to peer=mapdt_tlp_1238, replicatorType=Learner. 2023-08-07 13:15:18:039 +0300 [ERROR][%mapdt_tlp_1234%JRaft-Request-Processor-26][NodeImpl] Node start the learner replicator failed, peer=mapdt_tlp_1238. 2023-08-07 13:15:18:449 +0300 [ERROR][%mapdt_tlp_1234%JRaft-Request-Processor-26][AbstractClientService] Fail to connect mapdt_tlp_1237, exception: java.util.concurrent.TimeoutException. 2023-08-07 13:15:18:449 +0300 [ERROR][%mapdt_tlp_1234%JRaft-Request-Processor-26][ReplicatorGroupImpl] Fail to check replicator connection to peer=mapdt_tlp_1237, replicatorType=Learner. 2023-08-07 13:15:18:449 +0300 [ERROR][%mapdt_tlp_1234%JRaft-Request-Processor-26][NodeImpl] Node start the learner replicator failed, peer=mapdt_tlp_1237. 2023-08-07 13:15:18:449 +0300 [INFO][%mapdt_tlp_1234%JRaft-Request-Processor-26][NodeImpl] Catch up phase to change peers from=[mapdt_tlp_1236, mapdt_tlp_1235, mapdt_tlp_1234] to=[mapdt_tlp_1236, mapdt_tlp_1235, mapdt_tlp_1234] was successfully finished 2023-08-07 13:15:18:858 +0300 [ERROR][%mapdt_tlp_1234%JRaft-StepDownTimer-2][AbstractClientService] Fail to connect mapdt_tlp_1238, exception: java.util.concurrent.TimeoutException. 2023-08-07 13:15:18:858 +0300 [ERROR][%mapdt_tlp_1234%JRaft-StepDownTimer-2][ReplicatorGroupImpl] Fail to check replicator connection to peer=mapdt_tlp_1238, replicatorType=Learner. 2023-08-07 13:15:19:268 +0300 [ERROR][%mapdt_tlp_1234%JRaft-StepDownTimer-2][AbstractClientService] Fail to connect mapdt_tlp_1237, exception: java.util.concurrent.TimeoutException. 2023-08-07 13:15:19:268 +0300 [ERROR][%mapdt_tlp_1234%JRaft-StepDownTimer-2][ReplicatorGroupImpl] Fail to check replicator connection to peer=mapdt_tlp_1237, replicatorType=Learner. 2023-08-07 13:15:19:268 +0300 [WARNING][%mapdt_tlp_1234%JRaft-Common-Executor-2][RpcRequestClosure] A response: ChangePeersAsyncResponseImpl [newLearnersList=ArrayList [mapdt_tlp_1238, mapdt_tlp_1237], newPeersList=ArrayList [mapdt_tlp_1236, mapdt_tlp_1235, mapdt_tlp_1234], oldLearnersList=ArrayList [], oldPeersList=ArrayList [mapdt_tlp_1236, mapdt_tlp_1235, mapdt_tlp_1234]] sent repeatedly! 2023-08-07 13:15:19:268 +0300 [INFO][main][MultiActorPlacementDriverTest] Fake table created [id=2, repGrp=2_part_0] 2023-08-07 13:15:20:277 +0300 [ERROR][%mapdt_tlp_1234%JRaft-StepDownTimer-1][AbstractClientService] Fail to connect mapdt_tlp_1238, exception: java.util.concurrent.TimeoutException. 2023-08-07 13:15:20:277 +0300 [ERROR][%mapdt_tlp_1234%JRaft-StepDownTimer-1][ReplicatorGroupImpl] Fail to check replicator connection to peer=mapdt_tlp_1238, replicatorType=Learner. 2023-08-07 13:15:20:689 +0300 [ERROR][%mapdt_tlp_1234%JRaft-StepDownTimer-1][AbstractClientService] Fail to connect mapdt_tlp_1237, exception: java.util.concurrent.TimeoutException. 2023-08-07 13:15:20:689 +0300 [ERROR][%mapdt_tlp_1234%JRaft-StepDownTimer-1][ReplicatorGroupImpl] Fail to check replicator connection to peer=mapdt_tlp_1237, replicatorType=Learner. 2023-08-07 13:15:21:698 +0300 [ERROR][%mapdt_tlp_1234%JRaft-StepDownTimer-3][AbstractClientService] Fail to connect mapdt_tlp_1238, exception: java.util.concurrent.TimeoutException. 2023-08-07 13:15:21:698 +0300 [ERROR][%mapdt_tlp_1234%JRaft-StepDownTimer-3][ReplicatorGroupImpl] Fail to check replicator connection to peer=mapdt_tlp_1238, replicatorType=Learner. 2023-08-07 13:15:22:109 +0300 [ERROR][%mapdt_tlp_1234%JRaft-StepDownTimer-3][AbstractClientService] Fail to connect mapdt_tlp_1237, exception: java.util.concurrent.TimeoutException. 2023-08-07 13:15:22:109 +0300 [ERROR][%mapdt_tlp_1234%JRaft-StepDownTimer-3][ReplicatorGroupImpl] Fail to check replicator connection to peer=mapdt_tlp_1237, replicatorType=Learner. 2023-08-07 13:15:23:131 +0300 [ERROR][%mapdt_tlp_1234%JRaft-StepDownTimer-0][AbstractClientService] Fail to connect mapdt_tlp_1238, exception: java.util.concurrent.TimeoutException. 2023-08-07 13:15:23:131 +0300 [ERROR][%mapdt_tlp_1234%JRaft-StepDownTimer-0][ReplicatorGroupImpl] Fail to check replicator connection to peer=mapdt_tlp_1238, replicatorType=Learner. 2023-08-07 13:15:23:541 +0300 [ERROR][%mapdt_tlp_1234%JRaft-StepDownTimer-0][AbstractClientService] Fail to connect mapdt_tlp_1237, exception: java.util.concurrent.TimeoutException. 2023-08-07 13:15:23:541 +0300 [ERROR][%mapdt_tlp_1234%JRaft-StepDownTimer-0][ReplicatorGroupImpl] Fail to check replicator connection to peer=mapdt_tlp_1237, replicatorType=Learner. 2023-08-07 13:15:24:562 +0300 [ERROR][%mapdt_tlp_1234%JRaft-StepDownTimer-4][AbstractClientService] Fail to connect mapdt_tlp_1238, exception: java.util.concurrent.TimeoutException. 2023-08-07 13:15:24:562 +0300 [ERROR][%mapdt_tlp_1234%JRaft-StepDownTimer-4][ReplicatorGroupImpl] Fail to check replicator connection to peer=mapdt_tlp_1238, replicatorType=Learner. 2023-08-07 13:15:24:973 +0300 [ERROR][%mapdt_tlp_1234%JRaft-StepDownTimer-4][AbstractClientService] Fail to connect mapdt_tlp_1237, exception: java.util.concurrent.TimeoutException. 2023-08-07 13:15:24:973 +0300 [ERROR][%mapdt_tlp_1234%JRaft-StepDownTimer-4][ReplicatorGroupImpl] Fail to check replicator connection to peer=mapdt_tlp_1237, replicatorType=Learner. 2023-08-07 13:15:25:991 +0300 [ERROR][%mapdt_tlp_1234%JRaft-StepDownTimer-2][AbstractClientService] Fail to connect mapdt_tlp_1238, exception: java.util.concurrent.TimeoutException. 2023-08-07 13:15:25:991 +0300 [ERROR][%mapdt_tlp_1234%JRaft-StepDownTimer-2][ReplicatorGroupImpl] Fail to check replicator connection to peer=mapdt_tlp_1238, replicatorType=Learner. 2023-08-07 13:15:26:399 +0300 [ERROR][%mapdt_tlp_1234%JRaft-StepDownTimer-2][AbstractClientService] Fail to connect mapdt_tlp_1237, exception: java.util.concurrent.TimeoutException. 2023-08-07 13:15:26:399 +0300 [ERROR][%mapdt_tlp_1234%JRaft-StepDownTimer-2][ReplicatorGroupImpl] Fail to check replicator connection to peer=mapdt_tlp_1237, replicatorType=Learner. 2023-08-07 13:15:27:425 +0300 [ERROR][%mapdt_tlp_1234%JRaft-StepDownTimer-5][AbstractClientService] Fail to connect mapdt_tlp_1238, exception: java.util.concurrent.TimeoutException. 2023-08-07 13:15:27:425 +0300 [ERROR][%mapdt_tlp_1234%JRaft-StepDownTimer-5][ReplicatorGroupImpl] Fail to check replicator connection to peer=mapdt_tlp_1238, replicatorType=Learner. 2023-08-07 13:15:27:834 +0300 [ERROR][%mapdt_tlp_1234%JRaft-StepDownTimer-5][AbstractClientService] Fail to connect mapdt_tlp_1237, exception: java.util.concurrent.TimeoutException. 2023-08-07 13:15:27:834 +0300 [ERROR][%mapdt_tlp_1234%JRaft-StepDownTimer-5][ReplicatorGroupImpl] Fail to check replicator connection to peer=mapdt_tlp_1237, replicatorType=Learner. 2023-08-07 13:15:28:857 +0300 [ERROR][%mapdt_tlp_1234%JRaft-StepDownTimer-1][AbstractClientService] Fail to connect mapdt_tlp_1238, exception: java.util.concurrent.TimeoutException. 2023-08-07 13:15:28:857 +0300 [ERROR][%mapdt_tlp_1234%JRaft-StepDownTimer-1][ReplicatorGroupImpl] Fail to check replicator connection to peer=mapdt_tlp_1238, replicatorType=Learner. 2023-08-07 13:15:29:268 +0300 [ERROR][%mapdt_tlp_1234%JRaft-StepDownTimer-1][AbstractClientService] Fail to connect mapdt_tlp_1237, exception: java.util.concurrent.TimeoutException. 2023-08-07 13:15:29:268 +0300 [ERROR][%mapdt_tlp_1234%JRaft-StepDownTimer-1][ReplicatorGroupImpl] Fail to check replicator connection to peer=mapdt_tlp_1237, replicatorType=Learner. 2023-08-07 13:15:29:338 +0300 [INFO][main][Loza] Stop raft group=metastorage_group 2023-08-07 13:15:29:338 +0300 [INFO][main][NodeImpl] Node shutdown, currTerm=1 state=STATE_LEADER. 2023-08-07 13:15:29:338 +0300 [INFO][main][Replicator] Replicator Replicator [state=Replicate, statInfo=, peerId=mapdt_tlp_1235, type=Follower] is going to quit 2023-08-07 13:15:29:338 +0300 [INFO][%mapdt_tlp_1234%JRaft-FSMCaller-Disruptor-metastorage-_stripe_0-0][StateMachineAdapter] onLeaderStop: status=Status[ESHUTDOWN<1007>: Raft node is going to quit.]. 2023-08-07 13:15:29:339 +0300 [WARNING][%mapdt_tlp_1234%JRaft-Common-Executor-5][Replicator] Replicator Replicator [state=Replicate, statInfo=, peerId=mapdt_tlp_1236, type=Follower] stops sending entries. 2023-08-07 13:15:29:339 +0300 [INFO][%mapdt_tlp_1236%JRaft-Request-Processor-7][NodeImpl] Node received TimeoutNowRequest from mapdt_tlp_1234, term=1 and starts voting. 2023-08-07 13:15:29:339 +0300 [INFO][%mapdt_tlp_1236%JRaft-Request-Processor-7][NodeImpl] Node start vote and grant vote self, term=1. 2023-08-07 13:15:29:339 +0300 [INFO][%mapdt_tlp_1236%JRaft-FSMCaller-Disruptor-metastorage-_stripe_0-0][StateMachineAdapter] onStopFollowing: LeaderChangeContext [leaderId=mapdt_tlp_1234, term=1, status=Status[ERAFTTIMEDOUT<10001>: A follower's leader_id is reset to NULL as it begins to request_vote.]]. 2023-08-07 13:15:29:339 +0300 [INFO][%mapdt_tlp_1234%JRaft-Response-Processor-1][Replicator] Replicator Replicator [state=Replicate, statInfo=, peerId=mapdt_tlp_1236, type=Follower] is going to quit 2023-08-07 13:15:29:340 +0300 [INFO][%mapdt_tlp_1235%JRaft-Request-Processor-14][NodeImpl] Node received RequestVoteRequest from mapdt_tlp_1236, term=2, currTerm=1. 2023-08-07 13:15:29:340 +0300 [INFO][%mapdt_tlp_1235%JRaft-FSMCaller-Disruptor-metastorage-_stripe_0-0][StateMachineAdapter] onStopFollowing: LeaderChangeContext [leaderId=mapdt_tlp_1234, term=1, status=Status[EHIGHERTERMRESPONSE<10008>: Raft node receives higher term RequestVoteRequest.]]. 2023-08-07 13:15:29:342 +0300 [INFO][main][LocalRaftMetaStorage] Save raft meta, path=build\work\MultiActorPlacementDriverTest\tlp_29285\mapdt_tlp_1234_loza\metastorage_group_mapdt_tlp_1234_0\meta, term=1, votedFor=mapdt_tlp_1234, cost time=4 ms 2023-08-07 13:15:29:342 +0300 [INFO][main][FSMCallerImpl] Shutting down FSMCaller... 2023-08-07 13:15:29:342 +0300 [WARNING][%mapdt_tlp_1234%JRaft-Request-Processor-13][NodeImpl] Node is not in active state, currTerm=1. 2023-08-07 13:15:29:342 +0300 [INFO][main][RaftGroupService] Stop the RaftGroupService successfully. 2023-08-07 13:15:29:342 +0300 [WARNING][%mapdt_tlp_1236%JRaft-Response-Processor-2][NodeImpl] Node RequestVote to mapdt_tlp_1234 error: Status[EINVAL<1015>: Node is not in active state, state STATE_SHUTTING.]. 2023-08-07 13:15:29:343 +0300 [INFO][%mapdt_tlp_1234%JRaft-Common-Executor-7][LogThreadPoolExecutor] ThreadPool is terminated: %mapdt_tlp_1234%JRaft-Common-Executor-, org.apache.ignite.raft.jraft.util.MetricThreadPoolExecutor@7ab99a7[Shutting down, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 274]. 2023-08-07 13:15:29:344 +0300 [INFO][%mapdt_tlp_1236%JRaft-Request-Processor-7][LocalRaftMetaStorage] Save raft meta, path=build\work\MultiActorPlacementDriverTest\tlp_29285\mapdt_tlp_1236_loza\metastorage_group_mapdt_tlp_1236_0\meta, term=2, votedFor=mapdt_tlp_1236, cost time=4 ms 2023-08-07 13:15:29:344 +0300 [INFO][%mapdt_tlp_1235%JRaft-Request-Processor-14][LocalRaftMetaStorage] Save raft meta, path=build\work\MultiActorPlacementDriverTest\tlp_29285\mapdt_tlp_1235_loza\metastorage_group_mapdt_tlp_1235_0\meta, term=2, votedFor=, cost time=4 ms 2023-08-07 13:15:29:346 +0300 [INFO][%mapdt_tlp_1234%JRaft-ElectionTimer-3][LogScheduledThreadPoolExecutor] ThreadPool is terminated: %mapdt_tlp_1234%JRaft-ElectionTimer-, org.apache.ignite.raft.jraft.util.MetricScheduledThreadPoolExecutor@422c731f[Shutting down, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 3]. 2023-08-07 13:15:29:347 +0300 [INFO][main][LogScheduledThreadPoolExecutor] ThreadPool is terminated: %mapdt_tlp_1234%JRaft-VoteTimer-, org.apache.ignite.raft.jraft.util.MetricScheduledThreadPoolExecutor@19611337[Shutting down, pool size = 1, active threads = 0, queued tasks = 0, completed tasks = 1]. 2023-08-07 13:15:29:347 +0300 [INFO][%mapdt_tlp_1234%JRaft-Node-Scheduler-19][LogScheduledThreadPoolExecutor] ThreadPool is terminated: %mapdt_tlp_1234%JRaft-Node-Scheduler-, org.apache.ignite.raft.jraft.util.MetricScheduledThreadPoolExecutor@5a9b3eb1[Shutting down, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 241]. 2023-08-07 13:15:29:347 +0300 [INFO][%mapdt_tlp_1235%JRaft-Request-Processor-14][LocalRaftMetaStorage] Save raft meta, path=build\work\MultiActorPlacementDriverTest\tlp_29285\mapdt_tlp_1235_loza\metastorage_group_mapdt_tlp_1235_0\meta, term=2, votedFor=mapdt_tlp_1236, cost time=2 ms 2023-08-07 13:15:29:347 +0300 [INFO][%mapdt_tlp_1234%JRaft-StepDownTimer-7][LogScheduledThreadPoolExecutor] ThreadPool is terminated: %mapdt_tlp_1234%JRaft-StepDownTimer-, org.apache.ignite.raft.jraft.util.MetricScheduledThreadPoolExecutor@1d3a3478[Shutting down, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 12]. 2023-08-07 13:15:29:347 +0300 [INFO][%mapdt_tlp_1234%JRaft-SnapshotTimer-0][LogScheduledThreadPoolExecutor] ThreadPool is terminated: %mapdt_tlp_1234%JRaft-SnapshotTimer-, org.apache.ignite.raft.jraft.util.MetricScheduledThreadPoolExecutor@4ee22cb7[Shutting down, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0]. 2023-08-07 13:15:29:347 +0300 [INFO][%mapdt_tlp_1236%JRaft-Response-Processor-3][NodeImpl] Node become leader of group, term=2, conf=mapdt_tlp_1236,mapdt_tlp_1235,mapdt_tlp_1234,mapdt_tlp_1238/learner,mapdt_tlp_1237/learner, oldConf=. 2023-08-07 13:15:29:347 +0300 [INFO][%mapdt_tlp_1236%JRaft-Response-Processor-3][Replicator] Replicator=Replicator [state=Created, statInfo=, peerId=mapdt_tlp_1235, type=Follower]@mapdt_tlp_1235 is started 2023-08-07 13:15:29:348 +0300 [INFO][%mapdt_tlp_1236%JRaft-Response-Processor-3][Replicator] Replicator=Replicator [state=Created, statInfo=, peerId=mapdt_tlp_1234, type=Follower]@mapdt_tlp_1234 is started 2023-08-07 13:15:29:348 +0300 [INFO][%mapdt_tlp_1234%JRaft-Response-Processor-5][LogThreadPoolExecutor] ThreadPool is terminated: %mapdt_tlp_1234%JRaft-Response-Processor-, org.apache.ignite.raft.jraft.util.MetricThreadPoolExecutor@3f0fcaa8[Shutting down, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 28]. 2023-08-07 13:15:29:348 +0300 [INFO][%mapdt_tlp_1235%JRaft-FSMCaller-Disruptor-metastorage-_stripe_0-0][StateMachineAdapter] onStartFollowing: LeaderChangeContext [leaderId=mapdt_tlp_1236, term=2, status=Status[ENEWLEADER<10011>: Follower receives message from new leader with the same term.]]. 2023-08-07 13:15:29:349 +0300 [INFO][%mapdt_tlp_1234%JRaft-Request-Processor-41][LogThreadPoolExecutor] ThreadPool is terminated: %mapdt_tlp_1234%JRaft-Request-Processor-, org.apache.ignite.raft.jraft.util.MetricThreadPoolExecutor@72752362[Shutting down, pool size = 7, active threads = 0, queued tasks = 0, completed tasks = 206]. 2023-08-07 13:15:29:379 +0300 [INFO][sc-cluster-1236-13323][ScaleCubeTopologyService] Node left [member=ClusterNode [id=24f00a11-c233-431e-bcc5-bde71d7270c8, name=mapdt_tlp_1234, address=192.168.2.14:1234, nodeMetadata=null], eventType=LEAVING] 2023-08-07 13:15:29:379 +0300 [INFO][sc-cluster-1235-13322][ScaleCubeTopologyService] Node left [member=ClusterNode [id=24f00a11-c233-431e-bcc5-bde71d7270c8, name=mapdt_tlp_1234, address=192.168.2.14:1234, nodeMetadata=null], eventType=LEAVING] 2023-08-07 13:15:29:379 +0300 [INFO][sc-cluster-1236-13323][ScaleCubeTopologyService] Topology snapshot [nodes=[mapdt_tlp_1238, mapdt_tlp_1236, mapdt_tlp_1237, mapdt_tlp_1235]] 2023-08-07 13:15:29:379 +0300 [INFO][sc-cluster-1237-13324][ScaleCubeTopologyService] Node left [member=ClusterNode [id=24f00a11-c233-431e-bcc5-bde71d7270c8, name=mapdt_tlp_1234, address=192.168.2.14:1234, nodeMetadata=null], eventType=LEAVING] 2023-08-07 13:15:29:379 +0300 [INFO][sc-cluster-1237-13324][ScaleCubeTopologyService] Topology snapshot [nodes=[mapdt_tlp_1238, mapdt_tlp_1236, mapdt_tlp_1237, mapdt_tlp_1235]] 2023-08-07 13:15:29:379 +0300 [INFO][sc-cluster-1235-13322][ScaleCubeTopologyService] Topology snapshot [nodes=[mapdt_tlp_1238, mapdt_tlp_1236, mapdt_tlp_1237, mapdt_tlp_1235]] 2023-08-07 13:15:29:394 +0300 [INFO][sc-cluster-1238-13325][ScaleCubeTopologyService] Node left [member=ClusterNode [id=24f00a11-c233-431e-bcc5-bde71d7270c8, name=mapdt_tlp_1234, address=192.168.2.14:1234, nodeMetadata=null], eventType=LEAVING] 2023-08-07 13:15:29:394 +0300 [INFO][sc-cluster-1238-13325][ScaleCubeTopologyService] Topology snapshot [nodes=[mapdt_tlp_1238, mapdt_tlp_1236, mapdt_tlp_1237, mapdt_tlp_1235]] 2023-08-07 13:15:29:441 +0300 [INFO][sc-cluster-1234-13321][Transport] Stopping [address=192.168.2.14:1234] 2023-08-07 13:15:29:441 +0300 [INFO][sc-cluster-1234-13321][Transport] Stopped [address=192.168.2.14:1234] 2023-08-07 13:15:29:449 +0300 [INFO][main][Loza] Stop raft group=metastorage_group 2023-08-07 13:15:29:449 +0300 [INFO][main][NodeImpl] Node shutdown, currTerm=2 state=STATE_FOLLOWER. 2023-08-07 13:15:29:449 +0300 [INFO][%mapdt_tlp_1235%JRaft-FSMCaller-Disruptor-metastorage-_stripe_0-0][StateMachineAdapter] onStopFollowing: LeaderChangeContext [leaderId=mapdt_tlp_1236, term=2, status=Status[ESHUTDOWN<1007>: Raft node is going to quit.]]. 2023-08-07 13:15:29:452 +0300 [INFO][main][LocalRaftMetaStorage] Save raft meta, path=build\work\MultiActorPlacementDriverTest\tlp_29285\mapdt_tlp_1235_loza\metastorage_group_mapdt_tlp_1235_0\meta, term=2, votedFor=mapdt_tlp_1236, cost time=2 ms 2023-08-07 13:15:29:452 +0300 [INFO][main][FSMCallerImpl] Shutting down FSMCaller... 2023-08-07 13:15:29:452 +0300 [INFO][main][RaftGroupService] Stop the RaftGroupService successfully. 2023-08-07 13:15:29:453 +0300 [INFO][%mapdt_tlp_1235%JRaft-Common-Executor-3][LogThreadPoolExecutor] ThreadPool is terminated: %mapdt_tlp_1235%JRaft-Common-Executor-, org.apache.ignite.raft.jraft.util.MetricThreadPoolExecutor@7dde0ea4[Shutting down, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 4]. 2023-08-07 13:15:29:456 +0300 [INFO][%mapdt_tlp_1235%JRaft-Node-Scheduler-12][LogScheduledThreadPoolExecutor] ThreadPool is terminated: %mapdt_tlp_1235%JRaft-Node-Scheduler-, org.apache.ignite.raft.jraft.util.MetricScheduledThreadPoolExecutor@5b832fde[Shutting down, pool size = 8, active threads = 0, queued tasks = 0, completed tasks = 15]. 2023-08-07 13:15:29:457 +0300 [INFO][%mapdt_tlp_1235%JRaft-ElectionTimer-3][LogScheduledThreadPoolExecutor] ThreadPool is terminated: %mapdt_tlp_1235%JRaft-ElectionTimer-, org.apache.ignite.raft.jraft.util.MetricScheduledThreadPoolExecutor@b428d58[Shutting down, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 11]. 2023-08-07 13:15:29:457 +0300 [INFO][main][LogScheduledThreadPoolExecutor] ThreadPool is terminated: %mapdt_tlp_1235%JRaft-VoteTimer-, org.apache.ignite.raft.jraft.util.MetricScheduledThreadPoolExecutor@7f4db19[Shutting down, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0]. 2023-08-07 13:15:29:457 +0300 [INFO][main][LogScheduledThreadPoolExecutor] ThreadPool is terminated: %mapdt_tlp_1235%JRaft-StepDownTimer-, org.apache.ignite.raft.jraft.util.MetricScheduledThreadPoolExecutor@6f28a521[Shutting down, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0]. 2023-08-07 13:15:29:457 +0300 [INFO][%mapdt_tlp_1235%JRaft-SnapshotTimer-0][LogScheduledThreadPoolExecutor] ThreadPool is terminated: %mapdt_tlp_1235%JRaft-SnapshotTimer-, org.apache.ignite.raft.jraft.util.MetricScheduledThreadPoolExecutor@4c9dd563[Shutting down, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0]. 2023-08-07 13:15:29:457 +0300 [INFO][main][LogThreadPoolExecutor] ThreadPool is terminated: %mapdt_tlp_1235%JRaft-Response-Processor-, org.apache.ignite.raft.jraft.util.MetricThreadPoolExecutor@3b5f7a13[Shutting down, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0]. 2023-08-07 13:15:29:458 +0300 [INFO][%mapdt_tlp_1235%JRaft-Request-Processor-9][LogThreadPoolExecutor] ThreadPool is terminated: %mapdt_tlp_1235%JRaft-Request-Processor-, org.apache.ignite.raft.jraft.util.MetricThreadPoolExecutor@407afaee[Shutting down, pool size = 4, active threads = 0, queued tasks = 0, completed tasks = 16]. 2023-08-07 13:15:29:476 +0300 [ERROR][%mapdt_tlp_1236%JRaft-Common-Executor-0][AbstractClientService] Fail to connect mapdt_tlp_1234, exception: java.net.ConnectException. 2023-08-07 13:15:29:476 +0300 [WARNING][%mapdt_tlp_1235%MessagingService-inbound--0][IgniteRpcServer] A request execution was rejected [sender=ClusterNode [id=e3819f58-dc89-4385-8fe4-9e2244a17dfe, name=mapdt_tlp_1236, address=192.168.2.14:1236, nodeMetadata=null] req=AppendEntriesRequestImpl [committedIndex=28, data=null, entriesList=null, groupId=metastorage_group, peerId=mapdt_tlp_1235, prevLogIndex=30, prevLogTerm=1, serverId=mapdt_tlp_1236, term=2, timestampLong=110847808600473601] reason=Task org.apache.ignite.raft.jraft.rpc.impl.IgniteRpcServer$RpcMessageHandler$$Lambda$866/0x0000000801032840@7ae80bd3 rejected from org.apache.ignite.raft.jraft.util.MetricThreadPoolExecutor@407afaee[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 16]] 2023-08-07 13:15:29:476 +0300 [ERROR][%mapdt_tlp_1236%JRaft-Common-Executor-1][AbstractClientService] Fail to connect mapdt_tlp_1234, exception: java.net.ConnectException. 2023-08-07 13:15:29:490 +0300 [INFO][sc-cluster-1238-13325][ScaleCubeTopologyService] Node left [member=ClusterNode [id=3cee3aa0-35dc-4595-a69d-fc799255627a, name=mapdt_tlp_1235, address=192.168.2.14:1235, nodeMetadata=null], eventType=LEAVING] 2023-08-07 13:15:29:490 +0300 [INFO][sc-cluster-1237-13324][ScaleCubeTopologyService] Node left [member=ClusterNode [id=3cee3aa0-35dc-4595-a69d-fc799255627a, name=mapdt_tlp_1235, address=192.168.2.14:1235, nodeMetadata=null], eventType=LEAVING] 2023-08-07 13:15:29:490 +0300 [INFO][sc-cluster-1238-13325][ScaleCubeTopologyService] Topology snapshot [nodes=[mapdt_tlp_1238, mapdt_tlp_1236, mapdt_tlp_1237]] 2023-08-07 13:15:29:490 +0300 [INFO][sc-cluster-1236-13323][ScaleCubeTopologyService] Node left [member=ClusterNode [id=3cee3aa0-35dc-4595-a69d-fc799255627a, name=mapdt_tlp_1235, address=192.168.2.14:1235, nodeMetadata=null], eventType=LEAVING] 2023-08-07 13:15:29:490 +0300 [INFO][sc-cluster-1237-13324][ScaleCubeTopologyService] Topology snapshot [nodes=[mapdt_tlp_1238, mapdt_tlp_1236, mapdt_tlp_1237]] 2023-08-07 13:15:29:491 +0300 [INFO][sc-cluster-1236-13323][ScaleCubeTopologyService] Topology snapshot [nodes=[mapdt_tlp_1238, mapdt_tlp_1236, mapdt_tlp_1237]] 2023-08-07 13:15:29:552 +0300 [INFO][sc-cluster-1235-13322][Transport] Stopping [address=192.168.2.14:1235] 2023-08-07 13:15:29:552 +0300 [INFO][sc-cluster-1235-13322][Transport] Stopped [address=192.168.2.14:1235] 2023-08-07 13:15:29:563 +0300 [INFO][main][Loza] Stop raft group=metastorage_group 2023-08-07 13:15:29:600 +0300 [ERROR][%mapdt_tlp_1236%JRaft-Common-Executor-3][AbstractClientService] Fail to connect mapdt_tlp_1234, exception: java.net.ConnectException. 2023-08-07 13:15:29:601 +0300 [ERROR][%mapdt_tlp_1236%JRaft-Common-Executor-4][AbstractClientService] Fail to connect mapdt_tlp_1234, exception: java.net.ConnectException. 2023-08-07 13:15:29:726 +0300 [ERROR][%mapdt_tlp_1236%JRaft-Common-Executor-5][AbstractClientService] Fail to connect mapdt_tlp_1234, exception: java.net.ConnectException. 2023-08-07 13:15:29:727 +0300 [ERROR][%mapdt_tlp_1236%JRaft-Common-Executor-6][AbstractClientService] Fail to connect mapdt_tlp_1234, exception: java.net.ConnectException. 2023-08-07 13:15:29:757 +0300 [ERROR][%mapdt_tlp_1236%JRaft-Response-Processor-3][AbstractClientService] Fail to connect mapdt_tlp_1238, exception: java.util.concurrent.TimeoutException. 2023-08-07 13:15:29:757 +0300 [ERROR][%mapdt_tlp_1236%JRaft-Response-Processor-3][ReplicatorGroupImpl] Fail to check replicator connection to peer=mapdt_tlp_1238, replicatorType=Learner. 2023-08-07 13:15:29:757 +0300 [ERROR][%mapdt_tlp_1236%JRaft-Response-Processor-3][NodeImpl] Fail to add a learner replicator, peer=mapdt_tlp_1238. 2023-08-07 13:15:29:851 +0300 [ERROR][%mapdt_tlp_1236%JRaft-Common-Executor-2][AbstractClientService] Fail to connect mapdt_tlp_1234, exception: java.net.ConnectException. 2023-08-07 13:15:29:852 +0300 [ERROR][%mapdt_tlp_1236%JRaft-Common-Executor-7][AbstractClientService] Fail to connect mapdt_tlp_1234, exception: java.net.ConnectException. 2023-08-07 13:15:29:976 +0300 [ERROR][%mapdt_tlp_1236%JRaft-Common-Executor-0][AbstractClientService] Fail to connect mapdt_tlp_1234, exception: java.net.ConnectException. 2023-08-07 13:15:29:977 +0300 [WARNING][%mapdt_tlp_1236%JRaft-AppendEntries-Processor-1][Replicator] Fail to issue RPC to mapdt_tlp_1234, consecutiveErrorTimes=10, error=Status[EINTERNAL<1004>: Check connection[mapdt_tlp_1234] fail and try to create new one] 2023-08-07 13:15:29:977 +0300 [ERROR][%mapdt_tlp_1236%JRaft-Common-Executor-1][AbstractClientService] Fail to connect mapdt_tlp_1234, exception: java.net.ConnectException. 2023-08-07 13:15:30:101 +0300 [ERROR][%mapdt_tlp_1236%JRaft-Common-Executor-3][AbstractClientService] Fail to connect mapdt_tlp_1234, exception: java.net.ConnectException. 2023-08-07 13:15:30:102 +0300 [ERROR][%mapdt_tlp_1236%JRaft-Common-Executor-4][AbstractClientService] Fail to connect mapdt_tlp_1234, exception: java.net.ConnectException. 2023-08-07 13:15:30:163 +0300 [ERROR][%mapdt_tlp_1236%JRaft-Response-Processor-3][AbstractClientService] Fail to connect mapdt_tlp_1237, exception: java.util.concurrent.TimeoutException. 2023-08-07 13:15:30:163 +0300 [ERROR][%mapdt_tlp_1236%JRaft-Response-Processor-3][ReplicatorGroupImpl] Fail to check replicator connection to peer=mapdt_tlp_1237, replicatorType=Learner. 2023-08-07 13:15:30:163 +0300 [ERROR][%mapdt_tlp_1236%JRaft-Response-Processor-3][NodeImpl] Fail to add a learner replicator, peer=mapdt_tlp_1237. 2023-08-07 13:15:30:164 +0300 [INFO][main][NodeImpl] Node shutdown, currTerm=2 state=STATE_LEADER. 2023-08-07 13:15:30:164 +0300 [ERROR][%mapdt_tlp_1236%JRaft-Common-Executor-6][NodeImpl] Fail to run ConfigurationChangeDone, status: Status[EPERM<1008>: Leader stepped down]. 2023-08-07 13:15:30:164 +0300 [ERROR][%mapdt_tlp_1236%JRaft-Common-Executor-5][AbstractClientService] Fail to connect mapdt_tlp_1235, exception: java.net.ConnectException. 2023-08-07 13:15:30:164 +0300 [INFO][%mapdt_tlp_1236%JRaft-FSMCaller-Disruptor-metastorage-_stripe_0-0][StateMachineAdapter] onLeaderStop: status=Status[ESHUTDOWN<1007>: Raft node is going to quit.]. 2023-08-07 13:15:30:164 +0300 [INFO][main][Replicator] Replicator Replicator [state=Probe, statInfo=, peerId=mapdt_tlp_1234, type=Follower] is going to quit 2023-08-07 13:15:30:165 +0300 [WARNING][%mapdt_tlp_1236%JRaft-Common-Executor-2][Replicator] Replicator Replicator [state=Probe, statInfo=, peerId=mapdt_tlp_1235, type=Follower] stops sending entries. 2023-08-07 13:15:30:165 +0300 [INFO][%mapdt_tlp_1236%JRaft-Response-Processor-19][Replicator] Replicator Replicator [state=Probe, statInfo=, peerId=mapdt_tlp_1235, type=Follower] is going to quit 2023-08-07 13:15:30:170 +0300 [INFO][main][LocalRaftMetaStorage] Save raft meta, path=build\work\MultiActorPlacementDriverTest\tlp_29285\mapdt_tlp_1236_loza\metastorage_group_mapdt_tlp_1236_0\meta, term=2, votedFor=mapdt_tlp_1236, cost time=5 ms 2023-08-07 13:15:30:171 +0300 [INFO][main][FSMCallerImpl] Shutting down FSMCaller... 2023-08-07 13:15:30:171 +0300 [INFO][main][RaftGroupService] Stop the RaftGroupService successfully. 2023-08-07 13:15:30:176 +0300 [INFO][%mapdt_tlp_1236%JRaft-Common-Executor-0][LogThreadPoolExecutor] ThreadPool is terminated: %mapdt_tlp_1236%JRaft-Common-Executor-, org.apache.ignite.raft.jraft.util.MetricThreadPoolExecutor@2291e8e1[Shutting down, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 20]. 2023-08-07 13:15:30:191 +0300 [INFO][%mapdt_tlp_1236%JRaft-Node-Scheduler-15][LogScheduledThreadPoolExecutor] ThreadPool is terminated: %mapdt_tlp_1236%JRaft-Node-Scheduler-, org.apache.ignite.raft.jraft.util.MetricScheduledThreadPoolExecutor@7cdfa9b8[Shutting down, pool size = 13, active threads = 0, queued tasks = 0, completed tasks = 27]. 2023-08-07 13:15:30:192 +0300 [INFO][%mapdt_tlp_1236%JRaft-ElectionTimer-5][LogScheduledThreadPoolExecutor] ThreadPool is terminated: %mapdt_tlp_1236%JRaft-ElectionTimer-, org.apache.ignite.raft.jraft.util.MetricScheduledThreadPoolExecutor@146b6c6b[Shutting down, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 8]. 2023-08-07 13:15:30:192 +0300 [INFO][main][LogScheduledThreadPoolExecutor] ThreadPool is terminated: %mapdt_tlp_1236%JRaft-VoteTimer-, org.apache.ignite.raft.jraft.util.MetricScheduledThreadPoolExecutor@546da21b[Shutting down, pool size = 1, active threads = 0, queued tasks = 0, completed tasks = 0]. 2023-08-07 13:15:30:193 +0300 [INFO][%mapdt_tlp_1236%JRaft-StepDownTimer-0][LogScheduledThreadPoolExecutor] ThreadPool is terminated: %mapdt_tlp_1236%JRaft-StepDownTimer-, org.apache.ignite.raft.jraft.util.MetricScheduledThreadPoolExecutor@333b99b8[Shutting down, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0]. 2023-08-07 13:15:30:193 +0300 [INFO][%mapdt_tlp_1236%JRaft-SnapshotTimer-0][LogScheduledThreadPoolExecutor] ThreadPool is terminated: %mapdt_tlp_1236%JRaft-SnapshotTimer-, org.apache.ignite.raft.jraft.util.MetricScheduledThreadPoolExecutor@fc5efde[Shutting down, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0]. 2023-08-07 13:15:30:194 +0300 [INFO][%mapdt_tlp_1236%JRaft-Response-Processor-7][LogThreadPoolExecutor] ThreadPool is terminated: %mapdt_tlp_1236%JRaft-Response-Processor-, org.apache.ignite.raft.jraft.util.MetricThreadPoolExecutor@21d63d12[Shutting down, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 20]. 2023-08-07 13:15:30:195 +0300 [INFO][%mapdt_tlp_1236%JRaft-Request-Processor-3][LogThreadPoolExecutor] ThreadPool is terminated: %mapdt_tlp_1236%JRaft-Request-Processor-, org.apache.ignite.raft.jraft.util.MetricThreadPoolExecutor@697351e9[Shutting down, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 10]. 2023-08-07 13:15:30:227 +0300 [INFO][sc-cluster-1238-13325][ScaleCubeTopologyService] Node left [member=ClusterNode [id=e3819f58-dc89-4385-8fe4-9e2244a17dfe, name=mapdt_tlp_1236, address=192.168.2.14:1236, nodeMetadata=null], eventType=LEAVING] 2023-08-07 13:15:30:227 +0300 [INFO][sc-cluster-1238-13325][ScaleCubeTopologyService] Topology snapshot [nodes=[mapdt_tlp_1238, mapdt_tlp_1237]] 2023-08-07 13:15:30:227 +0300 [INFO][sc-cluster-1237-13324][ScaleCubeTopologyService] Node left [member=ClusterNode [id=e3819f58-dc89-4385-8fe4-9e2244a17dfe, name=mapdt_tlp_1236, address=192.168.2.14:1236, nodeMetadata=null], eventType=LEAVING] 2023-08-07 13:15:30:227 +0300 [INFO][sc-cluster-1237-13324][ScaleCubeTopologyService] Topology snapshot [nodes=[mapdt_tlp_1238, mapdt_tlp_1237]] 2023-08-07 13:15:30:289 +0300 [INFO][sc-cluster-1236-13323][Transport] Stopping [address=192.168.2.14:1236] 2023-08-07 13:15:30:289 +0300 [INFO][sc-cluster-1236-13323][Transport] Stopped [address=192.168.2.14:1236] 2023-08-07 13:15:30:886 +0300 [INFO][sc-cluster-1237-13324][ScaleCubeTopologyService] Node left [member=ClusterNode [id=24f00a11-c233-431e-bcc5-bde71d7270c8, name=mapdt_tlp_1234, address=192.168.2.14:1234, nodeMetadata=null], eventType=REMOVED] 2023-08-07 13:15:30:886 +0300 [INFO][sc-cluster-1237-13324][ScaleCubeTopologyService] Topology snapshot [nodes=[mapdt_tlp_1238, mapdt_tlp_1237]] 2023-08-07 13:15:30:902 +0300 [INFO][sc-cluster-1238-13325][ScaleCubeTopologyService] Node left [member=ClusterNode [id=24f00a11-c233-431e-bcc5-bde71d7270c8, name=mapdt_tlp_1234, address=192.168.2.14:1234, nodeMetadata=null], eventType=REMOVED] 2023-08-07 13:15:30:902 +0300 [INFO][sc-cluster-1238-13325][ScaleCubeTopologyService] Topology snapshot [nodes=[mapdt_tlp_1238, mapdt_tlp_1237]] 2023-08-07 13:15:30:997 +0300 [INFO][sc-cluster-1237-13324][ScaleCubeTopologyService] Node left [member=ClusterNode [id=3cee3aa0-35dc-4595-a69d-fc799255627a, name=mapdt_tlp_1235, address=192.168.2.14:1235, nodeMetadata=null], eventType=REMOVED] 2023-08-07 13:15:30:997 +0300 [INFO][sc-cluster-1238-13325][ScaleCubeTopologyService] Node left [member=ClusterNode [id=3cee3aa0-35dc-4595-a69d-fc799255627a, name=mapdt_tlp_1235, address=192.168.2.14:1235, nodeMetadata=null], eventType=REMOVED] 2023-08-07 13:15:30:997 +0300 [INFO][sc-cluster-1237-13324][ScaleCubeTopologyService] Topology snapshot [nodes=[mapdt_tlp_1238, mapdt_tlp_1237]] 2023-08-07 13:15:30:997 +0300 [INFO][sc-cluster-1238-13325][ScaleCubeTopologyService] Topology snapshot [nodes=[mapdt_tlp_1238, mapdt_tlp_1237]] 2023-08-07 13:15:31:610 +0300 [INFO][sc-cluster-1238-13325][ScaleCubeTopologyService] Node left [member=ClusterNode [id=c016acd7-457a-47d1-bf28-8cbe0e760468, name=mapdt_tlp_1237, address=192.168.2.14:1237, nodeMetadata=null], eventType=LEAVING] 2023-08-07 13:15:31:610 +0300 [INFO][sc-cluster-1238-13325][ScaleCubeTopologyService] Topology snapshot [nodes=[mapdt_tlp_1238]] 2023-08-07 13:15:31:657 +0300 [INFO][sc-cluster-1237-13324][Transport] Stopping [address=192.168.2.14:1237] 2023-08-07 13:15:31:657 +0300 [INFO][sc-cluster-1237-13324][Transport] Stopped [address=192.168.2.14:1237] 2023-08-07 13:15:31:736 +0300 [INFO][sc-cluster-1238-13325][ScaleCubeTopologyService] Node left [member=ClusterNode [id=e3819f58-dc89-4385-8fe4-9e2244a17dfe, name=mapdt_tlp_1236, address=192.168.2.14:1236, nodeMetadata=null], eventType=REMOVED] 2023-08-07 13:15:31:736 +0300 [INFO][sc-cluster-1238-13325][ScaleCubeTopologyService] Topology snapshot [nodes=[mapdt_tlp_1238]] 2023-08-07 13:15:32:621 +0300 [INFO][sc-cluster-1238-13325][ScaleCubeTopologyService] Node left [member=ClusterNode [id=c016acd7-457a-47d1-bf28-8cbe0e760468, name=mapdt_tlp_1237, address=192.168.2.14:1237, nodeMetadata=null], eventType=REMOVED] 2023-08-07 13:15:32:621 +0300 [INFO][sc-cluster-1238-13325][ScaleCubeTopologyService] Topology snapshot [nodes=[mapdt_tlp_1238]] 2023-08-07 13:15:33:153 +0300 [INFO][sc-cluster-1238-13325][Transport] Stopping [address=192.168.2.14:1238] 2023-08-07 13:15:33:153 +0300 [INFO][sc-cluster-1238-13325][Transport] Stopped [address=192.168.2.14:1238] 2023-08-07 13:15:34:662 +0300 [INFO][main][MultiActorPlacementDriverTest] >>> Stopping test: MultiActorPlacementDriverTest#testLeaseProlong, displayName: testLeaseProlong(), cost: 25460ms. org.opentest4j.AssertionFailedError: Expected :true Actual :false at org.junit.jupiter.api.AssertionFailureBuilder.build(AssertionFailureBuilder.java:151) at org.junit.jupiter.api.AssertionFailureBuilder.buildAndThrow(AssertionFailureBuilder.java:132) at org.junit.jupiter.api.AssertTrue.failNotTrue(AssertTrue.java:63) at org.junit.jupiter.api.AssertTrue.assertTrue(AssertTrue.java:36) at org.junit.jupiter.api.AssertTrue.assertTrue(AssertTrue.java:31) at org.junit.jupiter.api.Assertions.assertTrue(Assertions.java:180) at org.apache.ignite.internal.placementdriver.MultiActorPlacementDriverTest.checkLeaseCreated(MultiActorPlacementDriverTest.java:571) at org.apache.ignite.internal.placementdriver.MultiActorPlacementDriverTest.testLeaseProlong(MultiActorPlacementDriverTest.java:368) at jdk.internal.reflect.GeneratedMethodAccessor16.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:567) at org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:727) at org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60) at org.junit.jupiter.engine.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChain.java:131) at org.junit.jupiter.engine.extension.TimeoutExtension.intercept(TimeoutExtension.java:156) at org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestableMethod(TimeoutExtension.java:147) at org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestMethod(TimeoutExtension.java:86) at org.junit.jupiter.engine.execution.InterceptingExecutableInvoker$ReflectiveInterceptorCall.lambda$ofVoidMethod$0(InterceptingExecutableInvoker.java:103) at org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.lambda$invoke$0(InterceptingExecutableInvoker.java:93) at org.junit.jupiter.engine.execution.InvocationInterceptorChain$InterceptedInvocation.proceed(InvocationInterceptorChain.java:106) at org.junit.jupiter.engine.execution.InvocationInterceptorChain.proceed(InvocationInterceptorChain.java:64) at org.junit.jupiter.engine.execution.InvocationInterceptorChain.chainAndInvoke(InvocationInterceptorChain.java:45) at org.junit.jupiter.engine.execution.InvocationInterceptorChain.invoke(InvocationInterceptorChain.java:37) at org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.invoke(InterceptingExecutableInvoker.java:92) at org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.invoke(InterceptingExecutableInvoker.java:86) at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.lambda$invokeTestMethod$7(TestMethodTestDescriptor.java:217) at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73) at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.invokeTestMethod(TestMethodTestDescriptor.java:213) at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:138) at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:68) at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:151) at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73) at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141) at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137) at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139) at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73) at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138) at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95) at java.base/java.util.ArrayList.forEach(ArrayList.java:1507) at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:41) at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:155) at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73) at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141) at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137) at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139) at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73) at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138) at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95) at java.base/java.util.ArrayList.forEach(ArrayList.java:1507) at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:41) at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:155) at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73) at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141) at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137) at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139) at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73) at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138) at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95) at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.submit(SameThreadHierarchicalTestExecutorService.java:35) at org.junit.platform.engine.support.hierarchical.HierarchicalTestExecutor.execute(HierarchicalTestExecutor.java:57) at org.junit.platform.engine.support.hierarchical.HierarchicalTestEngine.execute(HierarchicalTestEngine.java:54) at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:147) at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:127) at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:90) at org.junit.platform.launcher.core.EngineExecutionOrchestrator.lambda$execute$0(EngineExecutionOrchestrator.java:55) at org.junit.platform.launcher.core.EngineExecutionOrchestrator.withInterceptedStreams(EngineExecutionOrchestrator.java:102) at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:54) at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:114) at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:86) at org.junit.platform.launcher.core.DefaultLauncherSession$DelegatingLauncher.execute(DefaultLauncherSession.java:86) at org.junit.platform.launcher.core.SessionPerRequestLauncher.execute(SessionPerRequestLauncher.java:53) at com.intellij.junit5.JUnit5IdeaTestRunner.startRunnerWithArgs(JUnit5IdeaTestRunner.java:57) at com.intellij.rt.junit.IdeaTestRunner$Repeater$1.execute(IdeaTestRunner.java:38) at com.intellij.rt.execution.junit.TestsRepeater.repeat(TestsRepeater.java:30) at com.intellij.rt.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:35) at com.intellij.rt.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:235) at com.intellij.rt.junit.JUnitStarter.main(JUnitStarter.java:54)