[00:06:05]F: [org.apache.ignite.raft.jraft.core.ItNodeTest] org.apache.ignite.raft.jraft.core.ItNodeTest.testAppendEntriesWhenFollowerIsInErrorState() (26s) [00:06:05] : [org.apache.ignite.raft.jraft.core.ItNodeTest.testAppendEntriesWhenFollowerIsInErrorState()] [Test Output] [2024-01-30T00:06:05,102][INFO ][Test worker][ItNodeTest] >>> Starting test: ItNodeTest#testAppendEntriesWhenFollowerIsInErrorState, displayName: testAppendEntriesWhenFollowerIsInErrorState() [2024-01-30T00:06:05,102][INFO ][Test worker][ItNodeTest] >>>>>>>>>>>>>>> Start test method: testAppendEntriesWhenFollowerIsInErrorState() [2024-01-30T00:06:05,140][INFO ][Test worker][ConnectionManager] Server started [address=/0:0:0:0:0:0:0:0:5003] [2024-01-30T00:06:05,141][INFO ][Test worker][Cluster] [default:int_taewfiies_5003:6939bfc9a1b24909@10.233.90.83:5003][doStart] Starting, config: ClusterConfig[metadata=null, metadataTimeout=1000, metadataCodec=io.scalecube.cluster.metadata.JdkMetadataCodec@4787b3aa, memberId='e939bfc9-a1b2-4909-b37c-148ed686c3ea', memberAlias='int_taewfiies_5003', externalHost='null', externalPort=null, transportConfig=TransportConfig[port=0, clientSecured=false, connectTimeout=1000, messageCodec=io.scalecube.cluster.transport.api.JdkMessageCodec@60eb78bd, maxFrameLength=2097152, transportFactory=org.apache.ignite.network.scalecube.ScaleCubeClusterServiceFactory$2$$Lambda$858/0x0000000800647040@40427cc3, addressMapper=java.util.function.Function$$Lambda$380/0x000000080024f040@2372da56], failureDetectorConfig=FailureDetectorConfig[pingInterval=500, pingTimeout=200, pingReqMembers=1], gossipConfig=GossipConfig[gossipFanout=3, gossipInterval=10, gossipRepeatMult=2, gossipSegmentationThreshold=1000], membershipConfig=MembershipConfig[seedMembers=[10.233.90.83:5003, 10.233.90.83:5004, 10.233.90.83:5005, 10.233.90.83:5006, 10.233.90.83:5007], syncInterval=1000, syncTimeout=3000, suspicionMult=1, namespace='default', removedMembersHistorySize=42]] [2024-01-30T00:06:05,141][INFO ][Test worker][MembershipProtocol] [default:int_taewfiies_5003:6939bfc9a1b24909@10.233.90.83:5003] Making initial Sync to all seed members: [10.233.90.83:5004, 10.233.90.83:5005, 10.233.90.83:5006, 10.233.90.83:5007] [2024-01-30T00:06:05,145][INFO ][sc-cluster-5003-268][Cluster] [default:int_taewfiies_5003:6939bfc9a1b24909@10.233.90.83:5003][doStart] Started [2024-01-30T00:06:05,145][INFO ][Test worker][ScaleCubeTopologyService] Node joined [node=ClusterNodeImpl [id=e939bfc9-a1b2-4909-b37c-148ed686c3ea, name=int_taewfiies_5003, address=10.233.90.83:5003, nodeMetadata=null]] [2024-01-30T00:06:05,145][INFO ][Test worker][ScaleCubeTopologyService] Topology snapshot [nodes=[int_taewfiies_5003]] [2024-01-30T00:06:05,201][INFO ][Test worker][FSMCallerImpl] Starts FSMCaller successfully. [2024-01-30T00:06:05,201][INFO ][Test worker][NodeImpl] Node init, term=0, lastLogId=LogId [index=0, term=0], conf=int_taewfiies_5003,int_taewfiies_5004,int_taewfiies_5005,int_taewfiies_5006,int_taewfiies_5007, oldConf=. [2024-01-30T00:06:05,202][INFO ][Test worker][RaftGroupService] Start the RaftGroupService successfully [2024-01-30T00:06:05,228][INFO ][Test worker][ConnectionManager] Server started [address=/0:0:0:0:0:0:0:0:5004] [2024-01-30T00:06:05,229][INFO ][Test worker][Cluster] [default:int_taewfiies_5004:2b53456118f24b6f@10.233.90.83:5004][doStart] Starting, config: ClusterConfig[metadata=null, metadataTimeout=1000, metadataCodec=io.scalecube.cluster.metadata.JdkMetadataCodec@4787b3aa, memberId='2b534561-18f2-4b6f-98da-a0d11a18c0bb', memberAlias='int_taewfiies_5004', externalHost='null', externalPort=null, transportConfig=TransportConfig[port=0, clientSecured=false, connectTimeout=1000, messageCodec=io.scalecube.cluster.transport.api.JdkMessageCodec@60eb78bd, maxFrameLength=2097152, transportFactory=org.apache.ignite.network.scalecube.ScaleCubeClusterServiceFactory$2$$Lambda$858/0x0000000800647040@5d38b619, addressMapper=java.util.function.Function$$Lambda$380/0x000000080024f040@2372da56], failureDetectorConfig=FailureDetectorConfig[pingInterval=500, pingTimeout=200, pingReqMembers=1], gossipConfig=GossipConfig[gossipFanout=3, gossipInterval=10, gossipRepeatMult=2, gossipSegmentationThreshold=1000], membershipConfig=MembershipConfig[seedMembers=[10.233.90.83:5003, 10.233.90.83:5004, 10.233.90.83:5005, 10.233.90.83:5006, 10.233.90.83:5007], syncInterval=1000, syncTimeout=3000, suspicionMult=1, namespace='default', removedMembersHistorySize=42]] [2024-01-30T00:06:05,229][INFO ][Test worker][MembershipProtocol] [default:int_taewfiies_5004:2b53456118f24b6f@10.233.90.83:5004] Making initial Sync to all seed members: [10.233.90.83:5003, 10.233.90.83:5005, 10.233.90.83:5006, 10.233.90.83:5007] [2024-01-30T00:06:05,237][INFO ][sc-cluster-5003-268][MembershipProtocol] [default:int_taewfiies_5003:6939bfc9a1b24909@10.233.90.83:5003][publishEvent] MembershipEvent[type=ADDED, member=default:int_taewfiies_5004:2b53456118f24b6f@10.233.90.83:5004, oldMetadata=null, newMetadata=7e16449-5, timestamp=2024-01-30T00:06:05.237Z] [2024-01-30T00:06:05,237][INFO ][sc-cluster-5003-268][ScaleCubeTopologyService] Node joined [node=ClusterNodeImpl [id=2b534561-18f2-4b6f-98da-a0d11a18c0bb, name=int_taewfiies_5004, address=10.233.90.83:5004, nodeMetadata=null]] [2024-01-30T00:06:05,237][INFO ][sc-cluster-5003-268][ScaleCubeTopologyService] Topology snapshot [nodes=[int_taewfiies_5003, int_taewfiies_5004]] [2024-01-30T00:06:05,239][INFO ][sc-cluster-5004-269][MembershipProtocol] [default:int_taewfiies_5004:2b53456118f24b6f@10.233.90.83:5004][publishEvent] MembershipEvent[type=ADDED, member=default:int_taewfiies_5003:6939bfc9a1b24909@10.233.90.83:5003, oldMetadata=null, newMetadata=7e16449-5, timestamp=2024-01-30T00:06:05.238Z] [2024-01-30T00:06:05,239][INFO ][sc-cluster-5004-269][ScaleCubeTopologyService] Node joined [node=ClusterNodeImpl [id=e939bfc9-a1b2-4909-b37c-148ed686c3ea, name=int_taewfiies_5003, address=10.233.90.83:5003, nodeMetadata=null]] [2024-01-30T00:06:05,239][INFO ][sc-cluster-5004-269][ScaleCubeTopologyService] Topology snapshot [nodes=[int_taewfiies_5003]] [2024-01-30T00:06:05,239][INFO ][sc-cluster-5004-269][Cluster] [default:int_taewfiies_5004:2b53456118f24b6f@10.233.90.83:5004][doStart] Started [2024-01-30T00:06:05,239][INFO ][Test worker][ScaleCubeTopologyService] Node joined [node=ClusterNodeImpl [id=2b534561-18f2-4b6f-98da-a0d11a18c0bb, name=int_taewfiies_5004, address=10.233.90.83:5004, nodeMetadata=null]] [2024-01-30T00:06:05,239][INFO ][Test worker][ScaleCubeTopologyService] Topology snapshot [nodes=[int_taewfiies_5003, int_taewfiies_5004]] [2024-01-30T00:06:05,254][INFO ][Test worker][FSMCallerImpl] Starts FSMCaller successfully. [2024-01-30T00:06:05,254][INFO ][Test worker][NodeImpl] Node init, term=0, lastLogId=LogId [index=0, term=0], conf=int_taewfiies_5003,int_taewfiies_5004,int_taewfiies_5005,int_taewfiies_5006,int_taewfiies_5007, oldConf=. [2024-01-30T00:06:05,257][INFO ][Test worker][RaftGroupService] Start the RaftGroupService successfully [2024-01-30T00:06:05,279][INFO ][Test worker][ConnectionManager] Server started [address=/0:0:0:0:0:0:0:0:5005] [2024-01-30T00:06:05,280][INFO ][Test worker][Cluster] [default:int_taewfiies_5005:3a0862258ff5467a@10.233.90.83:5005][doStart] Starting, config: ClusterConfig[metadata=null, metadataTimeout=1000, metadataCodec=io.scalecube.cluster.metadata.JdkMetadataCodec@4787b3aa, memberId='ba086225-8ff5-467a-9919-67782d28b656', memberAlias='int_taewfiies_5005', externalHost='null', externalPort=null, transportConfig=TransportConfig[port=0, clientSecured=false, connectTimeout=1000, messageCodec=io.scalecube.cluster.transport.api.JdkMessageCodec@60eb78bd, maxFrameLength=2097152, transportFactory=org.apache.ignite.network.scalecube.ScaleCubeClusterServiceFactory$2$$Lambda$858/0x0000000800647040@7029c9f6, addressMapper=java.util.function.Function$$Lambda$380/0x000000080024f040@2372da56], failureDetectorConfig=FailureDetectorConfig[pingInterval=500, pingTimeout=200, pingReqMembers=1], gossipConfig=GossipConfig[gossipFanout=3, gossipInterval=10, gossipRepeatMult=2, gossipSegmentationThreshold=1000], membershipConfig=MembershipConfig[seedMembers=[10.233.90.83:5003, 10.233.90.83:5004, 10.233.90.83:5005, 10.233.90.83:5006, 10.233.90.83:5007], syncInterval=1000, syncTimeout=3000, suspicionMult=1, namespace='default', removedMembersHistorySize=42]] [2024-01-30T00:06:05,280][INFO ][Test worker][MembershipProtocol] [default:int_taewfiies_5005:3a0862258ff5467a@10.233.90.83:5005] Making initial Sync to all seed members: [10.233.90.83:5003, 10.233.90.83:5004, 10.233.90.83:5006, 10.233.90.83:5007] [2024-01-30T00:06:05,286][INFO ][sc-cluster-5003-268][MembershipProtocol] [default:int_taewfiies_5003:6939bfc9a1b24909@10.233.90.83:5003][publishEvent] MembershipEvent[type=ADDED, member=default:int_taewfiies_5005:3a0862258ff5467a@10.233.90.83:5005, oldMetadata=null, newMetadata=7e16449-5, timestamp=2024-01-30T00:06:05.286Z] [2024-01-30T00:06:05,286][INFO ][sc-cluster-5003-268][ScaleCubeTopologyService] Node joined [node=ClusterNodeImpl [id=ba086225-8ff5-467a-9919-67782d28b656, name=int_taewfiies_5005, address=10.233.90.83:5005, nodeMetadata=null]] [2024-01-30T00:06:05,286][INFO ][sc-cluster-5003-268][ScaleCubeTopologyService] Topology snapshot [nodes=[int_taewfiies_5005, int_taewfiies_5003, int_taewfiies_5004]] [2024-01-30T00:06:05,286][INFO ][sc-cluster-5004-269][MembershipProtocol] [default:int_taewfiies_5004:2b53456118f24b6f@10.233.90.83:5004][publishEvent] MembershipEvent[type=ADDED, member=default:int_taewfiies_5005:3a0862258ff5467a@10.233.90.83:5005, oldMetadata=null, newMetadata=7e16449-5, timestamp=2024-01-30T00:06:05.286Z] [2024-01-30T00:06:05,286][INFO ][sc-cluster-5004-269][ScaleCubeTopologyService] Node joined [node=ClusterNodeImpl [id=ba086225-8ff5-467a-9919-67782d28b656, name=int_taewfiies_5005, address=10.233.90.83:5005, nodeMetadata=null]] [2024-01-30T00:06:05,286][INFO ][sc-cluster-5004-269][ScaleCubeTopologyService] Topology snapshot [nodes=[int_taewfiies_5005, int_taewfiies_5003, int_taewfiies_5004]] [2024-01-30T00:06:05,287][INFO ][sc-cluster-5005-270][MembershipProtocol] [default:int_taewfiies_5005:3a0862258ff5467a@10.233.90.83:5005][publishEvent] MembershipEvent[type=ADDED, member=default:int_taewfiies_5003:6939bfc9a1b24909@10.233.90.83:5003, oldMetadata=null, newMetadata=7e16449-5, timestamp=2024-01-30T00:06:05.287Z] [2024-01-30T00:06:05,288][INFO ][sc-cluster-5005-270][ScaleCubeTopologyService] Node joined [node=ClusterNodeImpl [id=e939bfc9-a1b2-4909-b37c-148ed686c3ea, name=int_taewfiies_5003, address=10.233.90.83:5003, nodeMetadata=null]] [2024-01-30T00:06:05,288][INFO ][sc-cluster-5005-270][ScaleCubeTopologyService] Topology snapshot [nodes=[int_taewfiies_5003]] [2024-01-30T00:06:05,288][INFO ][sc-cluster-5005-270][MembershipProtocol] [default:int_taewfiies_5005:3a0862258ff5467a@10.233.90.83:5005][publishEvent] MembershipEvent[type=ADDED, member=default:int_taewfiies_5004:2b53456118f24b6f@10.233.90.83:5004, oldMetadata=null, newMetadata=7e16449-5, timestamp=2024-01-30T00:06:05.288Z] [2024-01-30T00:06:05,288][INFO ][sc-cluster-5005-270][ScaleCubeTopologyService] Node joined [node=ClusterNodeImpl [id=2b534561-18f2-4b6f-98da-a0d11a18c0bb, name=int_taewfiies_5004, address=10.233.90.83:5004, nodeMetadata=null]] [2024-01-30T00:06:05,288][INFO ][sc-cluster-5005-270][ScaleCubeTopologyService] Topology snapshot [nodes=[int_taewfiies_5003, int_taewfiies_5004]] [2024-01-30T00:06:05,288][INFO ][sc-cluster-5005-270][Cluster] [default:int_taewfiies_5005:3a0862258ff5467a@10.233.90.83:5005][doStart] Started [2024-01-30T00:06:05,288][INFO ][Test worker][ScaleCubeTopologyService] Node joined [node=ClusterNodeImpl [id=ba086225-8ff5-467a-9919-67782d28b656, name=int_taewfiies_5005, address=10.233.90.83:5005, nodeMetadata=null]] [2024-01-30T00:06:05,288][INFO ][Test worker][ScaleCubeTopologyService] Topology snapshot [nodes=[int_taewfiies_5005, int_taewfiies_5003, int_taewfiies_5004]] [2024-01-30T00:06:05,304][INFO ][Test worker][FSMCallerImpl] Starts FSMCaller successfully. [2024-01-30T00:06:05,305][INFO ][Test worker][NodeImpl] Node init, term=0, lastLogId=LogId [index=0, term=0], conf=int_taewfiies_5003,int_taewfiies_5004,int_taewfiies_5005,int_taewfiies_5006,int_taewfiies_5007, oldConf=. [2024-01-30T00:06:05,305][INFO ][Test worker][RaftGroupService] Start the RaftGroupService successfully [2024-01-30T00:06:05,327][INFO ][Test worker][ConnectionManager] Server started [address=/0:0:0:0:0:0:0:0:5006] [2024-01-30T00:06:05,328][INFO ][Test worker][Cluster] [default:int_taewfiies_5006:59b4088774334760@10.233.90.83:5006][doStart] Starting, config: ClusterConfig[metadata=null, metadataTimeout=1000, metadataCodec=io.scalecube.cluster.metadata.JdkMetadataCodec@4787b3aa, memberId='59b40887-7433-4760-9420-837a02c05cb4', memberAlias='int_taewfiies_5006', externalHost='null', externalPort=null, transportConfig=TransportConfig[port=0, clientSecured=false, connectTimeout=1000, messageCodec=io.scalecube.cluster.transport.api.JdkMessageCodec@60eb78bd, maxFrameLength=2097152, transportFactory=org.apache.ignite.network.scalecube.ScaleCubeClusterServiceFactory$2$$Lambda$858/0x0000000800647040@5b748012, addressMapper=java.util.function.Function$$Lambda$380/0x000000080024f040@2372da56], failureDetectorConfig=FailureDetectorConfig[pingInterval=500, pingTimeout=200, pingReqMembers=1], gossipConfig=GossipConfig[gossipFanout=3, gossipInterval=10, gossipRepeatMult=2, gossipSegmentationThreshold=1000], membershipConfig=MembershipConfig[seedMembers=[10.233.90.83:5003, 10.233.90.83:5004, 10.233.90.83:5005, 10.233.90.83:5006, 10.233.90.83:5007], syncInterval=1000, syncTimeout=3000, suspicionMult=1, namespace='default', removedMembersHistorySize=42]] [2024-01-30T00:06:05,328][INFO ][Test worker][MembershipProtocol] [default:int_taewfiies_5006:59b4088774334760@10.233.90.83:5006] Making initial Sync to all seed members: [10.233.90.83:5003, 10.233.90.83:5004, 10.233.90.83:5005, 10.233.90.83:5007] [2024-01-30T00:06:05,334][INFO ][sc-cluster-5003-268][MembershipProtocol] [default:int_taewfiies_5003:6939bfc9a1b24909@10.233.90.83:5003][publishEvent] MembershipEvent[type=ADDED, member=default:int_taewfiies_5006:59b4088774334760@10.233.90.83:5006, oldMetadata=null, newMetadata=7e16449-5, timestamp=2024-01-30T00:06:05.334Z] [2024-01-30T00:06:05,334][INFO ][sc-cluster-5005-270][MembershipProtocol] [default:int_taewfiies_5005:3a0862258ff5467a@10.233.90.83:5005][publishEvent] MembershipEvent[type=ADDED, member=default:int_taewfiies_5006:59b4088774334760@10.233.90.83:5006, oldMetadata=null, newMetadata=7e16449-5, timestamp=2024-01-30T00:06:05.334Z] [2024-01-30T00:06:05,334][INFO ][sc-cluster-5003-268][ScaleCubeTopologyService] Node joined [node=ClusterNodeImpl [id=59b40887-7433-4760-9420-837a02c05cb4, name=int_taewfiies_5006, address=10.233.90.83:5006, nodeMetadata=null]] [2024-01-30T00:06:05,334][INFO ][sc-cluster-5003-268][ScaleCubeTopologyService] Topology snapshot [nodes=[int_taewfiies_5005, int_taewfiies_5006, int_taewfiies_5003, int_taewfiies_5004]] [2024-01-30T00:06:05,334][INFO ][sc-cluster-5005-270][ScaleCubeTopologyService] Node joined [node=ClusterNodeImpl [id=59b40887-7433-4760-9420-837a02c05cb4, name=int_taewfiies_5006, address=10.233.90.83:5006, nodeMetadata=null]] [2024-01-30T00:06:05,335][INFO ][sc-cluster-5005-270][ScaleCubeTopologyService] Topology snapshot [nodes=[int_taewfiies_5005, int_taewfiies_5006, int_taewfiies_5003, int_taewfiies_5004]] [2024-01-30T00:06:05,334][INFO ][sc-cluster-5004-269][MembershipProtocol] [default:int_taewfiies_5004:2b53456118f24b6f@10.233.90.83:5004][publishEvent] MembershipEvent[type=ADDED, member=default:int_taewfiies_5006:59b4088774334760@10.233.90.83:5006, oldMetadata=null, newMetadata=7e16449-5, timestamp=2024-01-30T00:06:05.334Z] [2024-01-30T00:06:05,335][INFO ][sc-cluster-5004-269][ScaleCubeTopologyService] Node joined [node=ClusterNodeImpl [id=59b40887-7433-4760-9420-837a02c05cb4, name=int_taewfiies_5006, address=10.233.90.83:5006, nodeMetadata=null]] [2024-01-30T00:06:05,335][INFO ][sc-cluster-5004-269][ScaleCubeTopologyService] Topology snapshot [nodes=[int_taewfiies_5005, int_taewfiies_5006, int_taewfiies_5003, int_taewfiies_5004]] [2024-01-30T00:06:05,336][INFO ][sc-cluster-5006-271][MembershipProtocol] [default:int_taewfiies_5006:59b4088774334760@10.233.90.83:5006][publishEvent] MembershipEvent[type=ADDED, member=default:int_taewfiies_5004:2b53456118f24b6f@10.233.90.83:5004, oldMetadata=null, newMetadata=7e16449-5, timestamp=2024-01-30T00:06:05.336Z] [2024-01-30T00:06:05,336][INFO ][sc-cluster-5006-271][ScaleCubeTopologyService] Node joined [node=ClusterNodeImpl [id=2b534561-18f2-4b6f-98da-a0d11a18c0bb, name=int_taewfiies_5004, address=10.233.90.83:5004, nodeMetadata=null]] [2024-01-30T00:06:05,336][INFO ][sc-cluster-5006-271][ScaleCubeTopologyService] Topology snapshot [nodes=[int_taewfiies_5004]] [2024-01-30T00:06:05,336][INFO ][sc-cluster-5006-271][MembershipProtocol] [default:int_taewfiies_5006:59b4088774334760@10.233.90.83:5006][publishEvent] MembershipEvent[type=ADDED, member=default:int_taewfiies_5003:6939bfc9a1b24909@10.233.90.83:5003, oldMetadata=null, newMetadata=7e16449-5, timestamp=2024-01-30T00:06:05.336Z] [2024-01-30T00:06:05,336][INFO ][sc-cluster-5006-271][ScaleCubeTopologyService] Node joined [node=ClusterNodeImpl [id=e939bfc9-a1b2-4909-b37c-148ed686c3ea, name=int_taewfiies_5003, address=10.233.90.83:5003, nodeMetadata=null]] [2024-01-30T00:06:05,336][INFO ][sc-cluster-5006-271][ScaleCubeTopologyService] Topology snapshot [nodes=[int_taewfiies_5003, int_taewfiies_5004]] [2024-01-30T00:06:05,337][INFO ][sc-cluster-5006-271][MembershipProtocol] [default:int_taewfiies_5006:59b4088774334760@10.233.90.83:5006][publishEvent] MembershipEvent[type=ADDED, member=default:int_taewfiies_5005:3a0862258ff5467a@10.233.90.83:5005, oldMetadata=null, newMetadata=7e16449-5, timestamp=2024-01-30T00:06:05.337Z] [2024-01-30T00:06:05,337][INFO ][sc-cluster-5006-271][ScaleCubeTopologyService] Node joined [node=ClusterNodeImpl [id=ba086225-8ff5-467a-9919-67782d28b656, name=int_taewfiies_5005, address=10.233.90.83:5005, nodeMetadata=null]] [2024-01-30T00:06:05,337][INFO ][sc-cluster-5006-271][ScaleCubeTopologyService] Topology snapshot [nodes=[int_taewfiies_5005, int_taewfiies_5003, int_taewfiies_5004]] [2024-01-30T00:06:05,337][INFO ][sc-cluster-5006-271][Cluster] [default:int_taewfiies_5006:59b4088774334760@10.233.90.83:5006][doStart] Started [2024-01-30T00:06:05,337][INFO ][Test worker][ScaleCubeTopologyService] Node joined [node=ClusterNodeImpl [id=59b40887-7433-4760-9420-837a02c05cb4, name=int_taewfiies_5006, address=10.233.90.83:5006, nodeMetadata=null]] [2024-01-30T00:06:05,337][INFO ][Test worker][ScaleCubeTopologyService] Topology snapshot [nodes=[int_taewfiies_5005, int_taewfiies_5006, int_taewfiies_5003, int_taewfiies_5004]] [2024-01-30T00:06:05,353][INFO ][Test worker][FSMCallerImpl] Starts FSMCaller successfully. [2024-01-30T00:06:05,354][INFO ][Test worker][NodeImpl] Node init, term=0, lastLogId=LogId [index=0, term=0], conf=int_taewfiies_5003,int_taewfiies_5004,int_taewfiies_5005,int_taewfiies_5006,int_taewfiies_5007, oldConf=. [2024-01-30T00:06:05,354][INFO ][Test worker][RaftGroupService] Start the RaftGroupService successfully [2024-01-30T00:06:05,376][INFO ][Test worker][ConnectionManager] Server started [address=/0:0:0:0:0:0:0:0:5007] [2024-01-30T00:06:05,377][INFO ][Test worker][Cluster] [default:int_taewfiies_5007:1a48dc796fed49a5@10.233.90.83:5007][doStart] Starting, config: ClusterConfig[metadata=null, metadataTimeout=1000, metadataCodec=io.scalecube.cluster.metadata.JdkMetadataCodec@4787b3aa, memberId='9a48dc79-6fed-49a5-99da-fc4b1ec1f94a', memberAlias='int_taewfiies_5007', externalHost='null', externalPort=null, transportConfig=TransportConfig[port=0, clientSecured=false, connectTimeout=1000, messageCodec=io.scalecube.cluster.transport.api.JdkMessageCodec@60eb78bd, maxFrameLength=2097152, transportFactory=org.apache.ignite.network.scalecube.ScaleCubeClusterServiceFactory$2$$Lambda$858/0x0000000800647040@7c1030b3, addressMapper=java.util.function.Function$$Lambda$380/0x000000080024f040@2372da56], failureDetectorConfig=FailureDetectorConfig[pingInterval=500, pingTimeout=200, pingReqMembers=1], gossipConfig=GossipConfig[gossipFanout=3, gossipInterval=10, gossipRepeatMult=2, gossipSegmentationThreshold=1000], membershipConfig=MembershipConfig[seedMembers=[10.233.90.83:5003, 10.233.90.83:5004, 10.233.90.83:5005, 10.233.90.83:5006, 10.233.90.83:5007], syncInterval=1000, syncTimeout=3000, suspicionMult=1, namespace='default', removedMembersHistorySize=42]] [2024-01-30T00:06:05,377][INFO ][Test worker][MembershipProtocol] [default:int_taewfiies_5007:1a48dc796fed49a5@10.233.90.83:5007] Making initial Sync to all seed members: [10.233.90.83:5003, 10.233.90.83:5004, 10.233.90.83:5005, 10.233.90.83:5006] [2024-01-30T00:06:05,387][INFO ][sc-cluster-5004-269][MembershipProtocol] [default:int_taewfiies_5004:2b53456118f24b6f@10.233.90.83:5004][publishEvent] MembershipEvent[type=ADDED, member=default:int_taewfiies_5007:1a48dc796fed49a5@10.233.90.83:5007, oldMetadata=null, newMetadata=7e16449-5, timestamp=2024-01-30T00:06:05.386Z] [2024-01-30T00:06:05,387][INFO ][sc-cluster-5003-268][MembershipProtocol] [default:int_taewfiies_5003:6939bfc9a1b24909@10.233.90.83:5003][publishEvent] MembershipEvent[type=ADDED, member=default:int_taewfiies_5007:1a48dc796fed49a5@10.233.90.83:5007, oldMetadata=null, newMetadata=7e16449-5, timestamp=2024-01-30T00:06:05.387Z] [2024-01-30T00:06:05,387][INFO ][sc-cluster-5004-269][ScaleCubeTopologyService] Node joined [node=ClusterNodeImpl [id=9a48dc79-6fed-49a5-99da-fc4b1ec1f94a, name=int_taewfiies_5007, address=10.233.90.83:5007, nodeMetadata=null]] [2024-01-30T00:06:05,387][INFO ][sc-cluster-5003-268][ScaleCubeTopologyService] Node joined [node=ClusterNodeImpl [id=9a48dc79-6fed-49a5-99da-fc4b1ec1f94a, name=int_taewfiies_5007, address=10.233.90.83:5007, nodeMetadata=null]] [2024-01-30T00:06:05,387][INFO ][sc-cluster-5004-269][ScaleCubeTopologyService] Topology snapshot [nodes=[int_taewfiies_5007, int_taewfiies_5005, int_taewfiies_5006, int_taewfiies_5003, int_taewfiies_5004]] [00:06:05] : [org.apache.ignite.raft.jraft.core.ItNodeTest.testAppendEntriesWhenFollowerIsInErrorState()] [Test Error Output] [2024-01-30T00:06:05,141][WARN ][Test worker][MembershipProtocol] [default:int_taewfiies_5003:6939bfc9a1b24909@10.233.90.83:5003] Filtering out seed address: 10.233.90.83:5003 [2024-01-30T00:06:05,143][WARN ][int_taewfiies_5003-client-1][MembershipProtocol] [default:int_taewfiies_5003:6939bfc9a1b24909@10.233.90.83:5003] Exception on initial Sync, cause: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: /10.233.90.83:5004 [2024-01-30T00:06:05,144][WARN ][int_taewfiies_5003-client-2][MembershipProtocol] [default:int_taewfiies_5003:6939bfc9a1b24909@10.233.90.83:5003] Exception on initial Sync, cause: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: /10.233.90.83:5005 [2024-01-30T00:06:05,144][WARN ][int_taewfiies_5003-client-3][MembershipProtocol] [default:int_taewfiies_5003:6939bfc9a1b24909@10.233.90.83:5003] Exception on initial Sync, cause: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: /10.233.90.83:5006 [2024-01-30T00:06:05,144][WARN ][int_taewfiies_5003-client-4][MembershipProtocol] [default:int_taewfiies_5003:6939bfc9a1b24909@10.233.90.83:5003] Exception on initial Sync, cause: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: /10.233.90.83:5007 [2024-01-30T00:06:05,201][WARN ][Test worker][LocalSnapshotStorage] No data for snapshot reader build/work/ItNodeTest/testAppendEntriesWhenFollowerIsInErrorState_30809148389477213/int_taewfiies_5003/snapshot. [2024-01-30T00:06:05,229][WARN ][Test worker][MembershipProtocol] [default:int_taewfiies_5004:2b53456118f24b6f@10.233.90.83:5004] Filtering out seed address: 10.233.90.83:5004 [2024-01-30T00:06:05,231][WARN ][int_taewfiies_5004-client-2][MembershipProtocol] [default:int_taewfiies_5004:2b53456118f24b6f@10.233.90.83:5004] Exception on initial Sync, cause: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: /10.233.90.83:5005 [2024-01-30T00:06:05,231][WARN ][int_taewfiies_5004-client-3][MembershipProtocol] [default:int_taewfiies_5004:2b53456118f24b6f@10.233.90.83:5004] Exception on initial Sync, cause: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: /10.233.90.83:5006 [2024-01-30T00:06:05,231][WARN ][int_taewfiies_5004-client-4][MembershipProtocol] [default:int_taewfiies_5004:2b53456118f24b6f@10.233.90.83:5004] Exception on initial Sync, cause: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: /10.233.90.83:5007 [2024-01-30T00:06:05,254][WARN ][Test worker][LocalSnapshotStorage] No data for snapshot reader build/work/ItNodeTest/testAppendEntriesWhenFollowerIsInErrorState_30809148389477213/int_taewfiies_5004/snapshot. [2024-01-30T00:06:05,279][WARN ][Test worker][MembershipProtocol] [default:int_taewfiies_5005:3a0862258ff5467a@10.233.90.83:5005] Filtering out seed address: 10.233.90.83:5005 [2024-01-30T00:06:05,282][WARN ][int_taewfiies_5005-client-3][MembershipProtocol] [default:int_taewfiies_5005:3a0862258ff5467a@10.233.90.83:5005] Exception on initial Sync, cause: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: /10.233.90.83:5006 [2024-01-30T00:06:05,282][WARN ][int_taewfiies_5005-client-4][MembershipProtocol] [default:int_taewfiies_5005:3a0862258ff5467a@10.233.90.83:5005] Exception on initial Sync, cause: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: /10.233.90.83:5007 [2024-01-30T00:06:05,304][WARN ][Test worker][LocalSnapshotStorage] No data for snapshot reader build/work/ItNodeTest/testAppendEntriesWhenFollowerIsInErrorState_30809148389477213/int_taewfiies_5005/snapshot. [2024-01-30T00:06:05,328][WARN ][Test worker][MembershipProtocol] [default:int_taewfiies_5006:59b4088774334760@10.233.90.83:5006] Filtering out seed address: 10.233.90.83:5006 [2024-01-30T00:06:05,330][WARN ][int_taewfiies_5006-client-4][MembershipProtocol] [default:int_taewfiies_5006:59b4088774334760@10.233.90.83:5006] Exception on initial Sync, cause: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: /10.233.90.83:5007 [2024-01-30T00:06:05,354][WARN ][Test worker][LocalSnapshotStorage] No data for snapshot reader build/work/ItNodeTest/testAppendEntriesWhenFollowerIsInErrorState_30809148389477213/int_taewfiies_5006/snapshot. [2024-01-30T00:06:05,377][WARN ][Test worker][MembershipProtocol] [default:int_taewfiies_5007:1a48dc796fed49a5@10.233.90.83:5007] Filtering out seed address: 10.233.90.83:5007 [00:06:06] : [org.apache.ignite.raft.jraft.core.ItNodeTest.testAppendEntriesWhenFollowerIsInErrorState()] [Test Output] [2024-01-30T00:06:05,387][INFO ][sc-cluster-5003-268][ScaleCubeTopologyService] Topology snapshot [nodes=[int_taewfiies_5007, int_taewfiies_5005, int_taewfiies_5006, int_taewfiies_5003, int_taewfiies_5004]] [2024-01-30T00:06:05,387][INFO ][sc-cluster-5005-270][MembershipProtocol] [default:int_taewfiies_5005:3a0862258ff5467a@10.233.90.83:5005][publishEvent] MembershipEvent[type=ADDED, member=default:int_taewfiies_5007:1a48dc796fed49a5@10.233.90.83:5007, oldMetadata=null, newMetadata=7e16449-5, timestamp=2024-01-30T00:06:05.387Z] [2024-01-30T00:06:05,387][INFO ][sc-cluster-5005-270][ScaleCubeTopologyService] Node joined [node=ClusterNodeImpl [id=9a48dc79-6fed-49a5-99da-fc4b1ec1f94a, name=int_taewfiies_5007, address=10.233.90.83:5007, nodeMetadata=null]] [2024-01-30T00:06:05,387][INFO ][sc-cluster-5005-270][ScaleCubeTopologyService] Topology snapshot [nodes=[int_taewfiies_5007, int_taewfiies_5005, int_taewfiies_5006, int_taewfiies_5003, int_taewfiies_5004]] [2024-01-30T00:06:05,389][INFO ][sc-cluster-5007-272][MembershipProtocol] [default:int_taewfiies_5007:1a48dc796fed49a5@10.233.90.83:5007][publishEvent] MembershipEvent[type=ADDED, member=default:int_taewfiies_5003:6939bfc9a1b24909@10.233.90.83:5003, oldMetadata=null, newMetadata=7e16449-5, timestamp=2024-01-30T00:06:05.389Z] [2024-01-30T00:06:05,389][INFO ][sc-cluster-5007-272][ScaleCubeTopologyService] Node joined [node=ClusterNodeImpl [id=e939bfc9-a1b2-4909-b37c-148ed686c3ea, name=int_taewfiies_5003, address=10.233.90.83:5003, nodeMetadata=null]] [2024-01-30T00:06:05,389][INFO ][sc-cluster-5007-272][ScaleCubeTopologyService] Topology snapshot [nodes=[int_taewfiies_5003]] [2024-01-30T00:06:05,389][INFO ][sc-cluster-5007-272][MembershipProtocol] [default:int_taewfiies_5007:1a48dc796fed49a5@10.233.90.83:5007][publishEvent] MembershipEvent[type=ADDED, member=default:int_taewfiies_5004:2b53456118f24b6f@10.233.90.83:5004, oldMetadata=null, newMetadata=7e16449-5, timestamp=2024-01-30T00:06:05.389Z] [2024-01-30T00:06:05,389][INFO ][sc-cluster-5007-272][ScaleCubeTopologyService] Node joined [node=ClusterNodeImpl [id=2b534561-18f2-4b6f-98da-a0d11a18c0bb, name=int_taewfiies_5004, address=10.233.90.83:5004, nodeMetadata=null]] [2024-01-30T00:06:05,389][INFO ][sc-cluster-5007-272][ScaleCubeTopologyService] Topology snapshot [nodes=[int_taewfiies_5003, int_taewfiies_5004]] [2024-01-30T00:06:05,389][INFO ][sc-cluster-5007-272][MembershipProtocol] [default:int_taewfiies_5007:1a48dc796fed49a5@10.233.90.83:5007][publishEvent] MembershipEvent[type=ADDED, member=default:int_taewfiies_5005:3a0862258ff5467a@10.233.90.83:5005, oldMetadata=null, newMetadata=7e16449-5, timestamp=2024-01-30T00:06:05.389Z] [2024-01-30T00:06:05,389][INFO ][sc-cluster-5006-271][MembershipProtocol] [default:int_taewfiies_5006:59b4088774334760@10.233.90.83:5006][publishEvent] MembershipEvent[type=ADDED, member=default:int_taewfiies_5007:1a48dc796fed49a5@10.233.90.83:5007, oldMetadata=null, newMetadata=7e16449-5, timestamp=2024-01-30T00:06:05.389Z] [2024-01-30T00:06:05,389][INFO ][sc-cluster-5007-272][ScaleCubeTopologyService] Node joined [node=ClusterNodeImpl [id=ba086225-8ff5-467a-9919-67782d28b656, name=int_taewfiies_5005, address=10.233.90.83:5005, nodeMetadata=null]] [2024-01-30T00:06:05,389][INFO ][sc-cluster-5007-272][ScaleCubeTopologyService] Topology snapshot [nodes=[int_taewfiies_5005, int_taewfiies_5003, int_taewfiies_5004]] [2024-01-30T00:06:05,389][INFO ][sc-cluster-5006-271][ScaleCubeTopologyService] Node joined [node=ClusterNodeImpl [id=9a48dc79-6fed-49a5-99da-fc4b1ec1f94a, name=int_taewfiies_5007, address=10.233.90.83:5007, nodeMetadata=null]] [2024-01-30T00:06:05,389][INFO ][sc-cluster-5006-271][ScaleCubeTopologyService] Topology snapshot [nodes=[int_taewfiies_5007, int_taewfiies_5005, int_taewfiies_5006, int_taewfiies_5003, int_taewfiies_5004]] [2024-01-30T00:06:05,391][INFO ][sc-cluster-5007-272][MembershipProtocol] [default:int_taewfiies_5007:1a48dc796fed49a5@10.233.90.83:5007][publishEvent] MembershipEvent[type=ADDED, member=default:int_taewfiies_5006:59b4088774334760@10.233.90.83:5006, oldMetadata=null, newMetadata=7e16449-5, timestamp=2024-01-30T00:06:05.391Z] [2024-01-30T00:06:05,391][INFO ][sc-cluster-5007-272][ScaleCubeTopologyService] Node joined [node=ClusterNodeImpl [id=59b40887-7433-4760-9420-837a02c05cb4, name=int_taewfiies_5006, address=10.233.90.83:5006, nodeMetadata=null]] [2024-01-30T00:06:05,391][INFO ][sc-cluster-5007-272][ScaleCubeTopologyService] Topology snapshot [nodes=[int_taewfiies_5005, int_taewfiies_5006, int_taewfiies_5003, int_taewfiies_5004]] [2024-01-30T00:06:05,391][INFO ][sc-cluster-5007-272][Cluster] [default:int_taewfiies_5007:1a48dc796fed49a5@10.233.90.83:5007][doStart] Started [2024-01-30T00:06:05,391][INFO ][Test worker][ScaleCubeTopologyService] Node joined [node=ClusterNodeImpl [id=9a48dc79-6fed-49a5-99da-fc4b1ec1f94a, name=int_taewfiies_5007, address=10.233.90.83:5007, nodeMetadata=null]] [2024-01-30T00:06:05,392][INFO ][Test worker][ScaleCubeTopologyService] Topology snapshot [nodes=[int_taewfiies_5007, int_taewfiies_5005, int_taewfiies_5006, int_taewfiies_5003, int_taewfiies_5004]] [2024-01-30T00:06:05,433][INFO ][Test worker][FSMCallerImpl] Starts FSMCaller successfully. [2024-01-30T00:06:05,433][INFO ][Test worker][NodeImpl] Node init, term=0, lastLogId=LogId [index=0, term=0], conf=int_taewfiies_5003,int_taewfiies_5004,int_taewfiies_5005,int_taewfiies_5006,int_taewfiies_5007, oldConf=. [2024-01-30T00:06:05,465][INFO ][Test worker][RaftGroupService] Start the RaftGroupService successfully [2024-01-30T00:06:06,476][INFO ][%int_taewfiies_5004%JRaft-ElectionTimer-0][NodeImpl] Node term 0 start preVote. [2024-01-30T00:06:06,483][INFO ][%int_taewfiies_5005%JRaft-Request-Processor-1][NodeImpl] Node received PreVoteRequest from int_taewfiies_5004, term=1, currTerm=0, granted=true, requestLastLogId=LogId [index=0, term=0], lastLogId=LogId [index=0, term=0]. [2024-01-30T00:06:06,483][INFO ][%int_taewfiies_5007%JRaft-Request-Processor-1][NodeImpl] Node received PreVoteRequest from int_taewfiies_5004, term=1, currTerm=0, granted=true, requestLastLogId=LogId [index=0, term=0], lastLogId=LogId [index=0, term=0]. [2024-01-30T00:06:06,484][INFO ][%int_taewfiies_5003%JRaft-Request-Processor-1][NodeImpl] Node received PreVoteRequest from int_taewfiies_5004, term=1, currTerm=0, granted=true, requestLastLogId=LogId [index=0, term=0], lastLogId=LogId [index=0, term=0]. [2024-01-30T00:06:06,484][INFO ][%int_taewfiies_5004%JRaft-Response-Processor-0][NodeImpl] Node received PreVoteResponse from int_taewfiies_5005, term=0, granted=true. [2024-01-30T00:06:06,484][INFO ][%int_taewfiies_5006%JRaft-Request-Processor-1][NodeImpl] Node received PreVoteRequest from int_taewfiies_5004, term=1, currTerm=0, granted=true, requestLastLogId=LogId [index=0, term=0], lastLogId=LogId [index=0, term=0]. [2024-01-30T00:06:06,484][INFO ][%int_taewfiies_5004%JRaft-Response-Processor-1][NodeImpl] Node received PreVoteResponse from int_taewfiies_5007, term=0, granted=true. [2024-01-30T00:06:06,484][INFO ][%int_taewfiies_5004%JRaft-Response-Processor-1][NodeImpl] Node start vote and grant vote self, term=0. [2024-01-30T00:06:06,486][INFO ][%int_taewfiies_5006%JRaft-Request-Processor-2][NodeImpl] Node received RequestVoteRequest from int_taewfiies_5004, term=1, currTerm=0. [2024-01-30T00:06:06,486][INFO ][%int_taewfiies_5005%JRaft-Request-Processor-2][NodeImpl] Node received RequestVoteRequest from int_taewfiies_5004, term=1, currTerm=0. [2024-01-30T00:06:06,486][INFO ][%int_taewfiies_5003%JRaft-Request-Processor-2][NodeImpl] Node received RequestVoteRequest from int_taewfiies_5004, term=1, currTerm=0. [2024-01-30T00:06:06,486][INFO ][%int_taewfiies_5007%JRaft-Request-Processor-2][NodeImpl] Node received RequestVoteRequest from int_taewfiies_5004, term=1, currTerm=0. [2024-01-30T00:06:06,487][INFO ][%int_taewfiies_5004%JRaft-Response-Processor-1][LocalRaftMetaStorage] Save raft meta, path=build/work/ItNodeTest/testAppendEntriesWhenFollowerIsInErrorState_30809148389477213/int_taewfiies_5004/meta, term=1, votedFor=int_taewfiies_5004, cost time=2 ms [2024-01-30T00:06:06,487][INFO ][%int_taewfiies_5006%JRaft-Request-Processor-2][LocalRaftMetaStorage] Save raft meta, path=build/work/ItNodeTest/testAppendEntriesWhenFollowerIsInErrorState_30809148389477213/int_taewfiies_5006/meta, term=1, votedFor=, cost time=1 ms [2024-01-30T00:06:06,487][INFO ][%int_taewfiies_5005%JRaft-Request-Processor-2][LocalRaftMetaStorage] Save raft meta, path=build/work/ItNodeTest/testAppendEntriesWhenFollowerIsInErrorState_30809148389477213/int_taewfiies_5005/meta, term=1, votedFor=, cost time=1 ms [2024-01-30T00:06:06,487][INFO ][%int_taewfiies_5007%JRaft-Request-Processor-2][LocalRaftMetaStorage] Save raft meta, path=build/work/ItNodeTest/testAppendEntriesWhenFollowerIsInErrorState_30809148389477213/int_taewfiies_5007/meta, term=1, votedFor=, cost time=1 ms [2024-01-30T00:06:06,487][INFO ][%int_taewfiies_5003%JRaft-Request-Processor-2][LocalRaftMetaStorage] Save raft meta, path=build/work/ItNodeTest/testAppendEntriesWhenFollowerIsInErrorState_30809148389477213/int_taewfiies_5003/meta, term=1, votedFor=, cost time=1 ms [2024-01-30T00:06:06,489][INFO ][%int_taewfiies_5006%JRaft-Request-Processor-2][LocalRaftMetaStorage] Save raft meta, path=build/work/ItNodeTest/testAppendEntriesWhenFollowerIsInErrorState_30809148389477213/int_taewfiies_5006/meta, term=1, votedFor=int_taewfiies_5004, cost time=1 ms [2024-01-30T00:06:06,489][INFO ][%int_taewfiies_5007%JRaft-Request-Processor-2][LocalRaftMetaStorage] Save raft meta, path=build/work/ItNodeTest/testAppendEntriesWhenFollowerIsInErrorState_30809148389477213/int_taewfiies_5007/meta, term=1, votedFor=int_taewfiies_5004, cost time=1 ms [2024-01-30T00:06:06,490][INFO ][%int_taewfiies_5005%JRaft-Request-Processor-2][LocalRaftMetaStorage] Save raft meta, path=build/work/ItNodeTest/testAppendEntriesWhenFollowerIsInErrorState_30809148389477213/int_taewfiies_5005/meta, term=1, votedFor=int_taewfiies_5004, cost time=0 ms [2024-01-30T00:06:06,490][INFO ][%int_taewfiies_5003%JRaft-Request-Processor-2][LocalRaftMetaStorage] Save raft meta, path=build/work/ItNodeTest/testAppendEntriesWhenFollowerIsInErrorState_30809148389477213/int_taewfiies_5003/meta, term=1, votedFor=int_taewfiies_5004, cost time=1 ms [2024-01-30T00:06:06,490][INFO ][%int_taewfiies_5004%JRaft-Response-Processor-1][NodeImpl] Node become leader of group, term=1, conf=int_taewfiies_5003,int_taewfiies_5004,int_taewfiies_5005,int_taewfiies_5006,int_taewfiies_5007, oldConf=. [2024-01-30T00:06:06,490][INFO ][%int_taewfiies_5004%JRaft-Response-Processor-1][Replicator] Replicator=Replicator [state=Created, statInfo=, peerId=int_taewfiies_5005, type=Follower]@int_taewfiies_5005 is started [2024-01-30T00:06:06,491][INFO ][%int_taewfiies_5004%JRaft-Response-Processor-1][Replicator] Replicator=Replicator [state=Created, statInfo=, peerId=int_taewfiies_5006, type=Follower]@int_taewfiies_5006 is started [2024-01-30T00:06:06,491][INFO ][%int_taewfiies_5004%JRaft-Response-Processor-1][Replicator] Replicator=Replicator [state=Created, statInfo=, peerId=int_taewfiies_5007, type=Follower]@int_taewfiies_5007 is started [2024-01-30T00:06:06,491][INFO ][%int_taewfiies_5004%JRaft-Response-Processor-1][Replicator] Replicator=Replicator [state=Created, statInfo=, peerId=int_taewfiies_5003, type=Follower]@int_taewfiies_5003 is started [2024-01-30T00:06:06,493][INFO ][%int_taewfiies_5005%JRaft-FSMCaller-Disruptor-_stripe_2-0][StateMachineAdapter] onStartFollowing: LeaderChangeContext [leaderId=int_taewfiies_5004, term=1, status=Status[ENEWLEADER<10011>: Follower receives message from new leader with the same term.]]. [2024-01-30T00:06:06,493][INFO ][%int_taewfiies_5006%JRaft-FSMCaller-Disruptor-_stripe_3-0][StateMachineAdapter] onStartFollowing: LeaderChangeContext [leaderId=int_taewfiies_5004, term=1, status=Status[ENEWLEADER<10011>: Follower receives message from new leader with the same term.]]. [2024-01-30T00:06:06,493][INFO ][%int_taewfiies_5007%JRaft-FSMCaller-Disruptor-_stripe_4-0][StateMachineAdapter] onStartFollowing: LeaderChangeContext [leaderId=int_taewfiies_5004, term=1, status=Status[ENEWLEADER<10011>: Follower receives message from new leader with the same term.]]. [2024-01-30T00:06:06,493][INFO ][%int_taewfiies_5003%JRaft-FSMCaller-Disruptor-_stripe_0-0][StateMachineAdapter] onStartFollowing: LeaderChangeContext [leaderId=int_taewfiies_5004, term=1, status=Status[ENEWLEADER<10011>: Follower receives message from new leader with the same term.]]. [2024-01-30T00:06:06,495][INFO ][%int_taewfiies_5004%JRaft-FSMCaller-Disruptor-_stripe_1-0][StateMachineAdapter] onLeaderStart: term=1. [2024-01-30T00:06:06,501][INFO ][Test worker][ItNodeTest] Set follower JRaftNode [nodeId=] into error state [2024-01-30T00:06:06,504][INFO ][Test worker][LogThreadPoolExecutor] ThreadPool is terminated: %int_taewfiies_5004%JRaft-Request-Processor-, org.apache.ignite.raft.jraft.util.MetricThreadPoolExecutor@69e57355[Shutting down, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0]. [2024-01-30T00:06:06,505][INFO ][Test worker][NodeImpl] Node shutdown, currTerm=1 state=STATE_LEADER. [2024-01-30T00:06:06,505][INFO ][%int_taewfiies_5004%JRaft-FSMCaller-Disruptor-_stripe_1-0][StateMachineAdapter] onLeaderStop: status=Status[ESHUTDOWN<1007>: Raft node is going to quit.]. [2024-01-30T00:06:06,505][INFO ][Test worker][Replicator] Replicator Replicator [state=Replicate, statInfo=, peerId=int_taewfiies_5006, type=Follower] is going to quit [2024-01-30T00:06:06,505][INFO ][Test worker][Replicator] Replicator Replicator [state=Replicate, statInfo=, peerId=int_taewfiies_5007, type=Follower] is going to quit [2024-01-30T00:06:06,505][INFO ][Test worker][Replicator] Replicator Replicator [state=Replicate, statInfo=, peerId=int_taewfiies_5003, type=Follower] is going to quit [2024-01-30T00:06:06,506][INFO ][%int_taewfiies_5005%JRaft-Request-Processor-3][NodeImpl] Node received TimeoutNowRequest from int_taewfiies_5004, term=1 and starts voting. [2024-01-30T00:06:06,506][INFO ][%int_taewfiies_5005%JRaft-Request-Processor-3][NodeImpl] Node start vote and grant vote self, term=1. [2024-01-30T00:06:06,506][INFO ][%int_taewfiies_5005%JRaft-FSMCaller-Disruptor-_stripe_2-0][StateMachineAdapter] onStopFollowing: LeaderChangeContext [leaderId=int_taewfiies_5004, term=1, status=Status[ERAFTTIMEDOUT<10001>: A follower's leader_id is reset to NULL as it begins to request_vote.]]. [2024-01-30T00:06:06,506][INFO ][%int_taewfiies_5004%JRaft-Response-Processor-0][Replicator] Replicator Replicator [state=Replicate, statInfo=, peerId=int_taewfiies_5005, type=Follower] is going to quit [2024-01-30T00:06:06,506][INFO ][Test worker][LocalRaftMetaStorage] Save raft meta, path=build/work/ItNodeTest/testAppendEntriesWhenFollowerIsInErrorState_30809148389477213/int_taewfiies_5004/meta, term=1, votedFor=int_taewfiies_5004, cost time=1 ms [2024-01-30T00:06:06,506][INFO ][Test worker][FSMCallerImpl] Shutting down FSMCaller... [2024-01-30T00:06:06,507][INFO ][%int_taewfiies_5004%JRaft-FSMCaller-Disruptor-_stripe_1-0][StateMachineAdapter] onShutdown. [2024-01-30T00:06:06,507][INFO ][%int_taewfiies_5004%JRaft-ElectionTimer-1][LogScheduledThreadPoolExecutor] ThreadPool is terminated: %int_taewfiies_5004%JRaft-ElectionTimer-, org.apache.ignite.raft.jraft.util.MetricScheduledThreadPoolExecutor@12e04770[Shutting down, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 1]. [2024-01-30T00:06:06,507][INFO ][%int_taewfiies_5004%JRaft-Node-Scheduler-5][LogScheduledThreadPoolExecutor] ThreadPool is terminated: %int_taewfiies_5004%JRaft-Node-Scheduler-, org.apache.ignite.raft.jraft.util.MetricScheduledThreadPoolExecutor@303eb613[Shutting down, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 1]. [2024-01-30T00:06:06,508][INFO ][%int_taewfiies_5004%JRaft-VoteTimer-0][LogScheduledThreadPoolExecutor] ThreadPool is terminated: %int_taewfiies_5004%JRaft-VoteTimer-, org.apache.ignite.raft.jraft.util.MetricScheduledThreadPoolExecutor@661bc49d[Shutting down, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0]. [2024-01-30T00:06:06,508][INFO ][%int_taewfiies_5004%JRaft-StepDownTimer-0][LogScheduledThreadPoolExecutor] ThreadPool is terminated: %int_taewfiies_5004%JRaft-StepDownTimer-, org.apache.ignite.raft.jraft.util.MetricScheduledThreadPoolExecutor@4909c8ee[Shutting down, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0]. [2024-01-30T00:06:06,508][INFO ][%int_taewfiies_5004%JRaft-SnapshotTimer-0][LogScheduledThreadPoolExecutor] ThreadPool is terminated: %int_taewfiies_5004%JRaft-SnapshotTimer-, org.apache.ignite.raft.jraft.util.MetricScheduledThreadPoolExecutor@5b4654fc[Shutting down, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0]. [2024-01-30T00:06:06,508][INFO ][%int_taewfiies_5004%JRaft-Common-Executor-1][LogThreadPoolExecutor] ThreadPool is terminated: %int_taewfiies_5004%JRaft-Common-Executor-, org.apache.ignite.raft.jraft.util.MetricThreadPoolExecutor@5869e9f[Shutting down, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 11]. [2024-01-30T00:06:06,509][INFO ][%int_taewfiies_5005%JRaft-Request-Processor-3][LocalRaftMetaStorage] Save raft meta, path=build/work/ItNodeTest/testAppendEntriesWhenFollowerIsInErrorState_30809148389477213/int_taewfiies_5005/meta, term=2, votedFor=int_taewfiies_5005, cost time=1 ms [2024-01-30T00:06:06,511][INFO ][%int_taewfiies_5006%JRaft-Request-Processor-0][NodeImpl] Node received RequestVoteRequest from int_taewfiies_5005, term=2, currTerm=1. [2024-01-30T00:06:06,511][INFO ][%int_taewfiies_5006%JRaft-FSMCaller-Disruptor-_stripe_3-0][StateMachineAdapter] onStopFollowing: LeaderChangeContext [leaderId=int_taewfiies_5004, term=1, status=Status[EHIGHERTERMRESPONSE<10008>: Raft node receives higher term RequestVoteRequest.]]. [2024-01-30T00:06:06,511][INFO ][%int_taewfiies_5007%JRaft-Request-Processor-0][NodeImpl] Node received RequestVoteRequest from int_taewfiies_5005, term=2, currTerm=1. [2024-01-30T00:06:06,511][INFO ][%int_taewfiies_5007%JRaft-FSMCaller-Disruptor-_stripe_4-0][StateMachineAdapter] onStopFollowing: LeaderChangeContext [leaderId=int_taewfiies_5004, term=1, status=Status[EHIGHERTERMRESPONSE<10008>: Raft node receives higher term RequestVoteRequest.]]. [2024-01-30T00:06:06,512][INFO ][%int_taewfiies_5006%JRaft-Request-Processor-0][LocalRaftMetaStorage] Save raft meta, path=build/work/ItNodeTest/testAppendEntriesWhenFollowerIsInErrorState_30809148389477213/int_taewfiies_5006/meta, term=2, votedFor=, cost time=1 ms [2024-01-30T00:06:06,512][INFO ][%int_taewfiies_5007%JRaft-Request-Processor-0][LocalRaftMetaStorage] Save raft meta, path=build/work/ItNodeTest/testAppendEntriesWhenFollowerIsInErrorState_30809148389477213/int_taewfiies_5007/meta, term=2, votedFor=, cost time=1 ms [2024-01-30T00:06:06,514][INFO ][%int_taewfiies_5004%JRaft-Response-Processor-0][LogThreadPoolExecutor] ThreadPool is terminated: %int_taewfiies_5004%JRaft-Response-Processor-, org.apache.ignite.raft.jraft.util.MetricThreadPoolExecutor@1dc83922[Shutting down, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 13]. [2024-01-30T00:06:06,514][INFO ][%int_taewfiies_5006%JRaft-Request-Processor-0][LocalRaftMetaStorage] Save raft meta, path=build/work/ItNodeTest/testAppendEntriesWhenFollowerIsInErrorState_30809148389477213/int_taewfiies_5006/meta, term=2, votedFor=int_taewfiies_5005, cost time=0 ms [2024-01-30T00:06:06,514][INFO ][%int_taewfiies_5007%JRaft-Request-Processor-0][LocalRaftMetaStorage] Save raft meta, path=build/work/ItNodeTest/testAppendEntriesWhenFollowerIsInErrorState_30809148389477213/int_taewfiies_5007/meta, term=2, votedFor=int_taewfiies_5005, cost time=1 ms [2024-01-30T00:06:06,515][INFO ][Test worker][RaftGroupService] Stop the RaftGroupService successfully. [2024-01-30T00:06:06,515][INFO ][%int_taewfiies_5005%JRaft-Response-Processor-1][NodeImpl] Node become leader of group, term=2, conf=int_taewfiies_5003,int_taewfiies_5004,int_taewfiies_5005,int_taewfiies_5006,int_taewfiies_5007, oldConf=. [2024-01-30T00:06:06,515][INFO ][Test worker][Cluster] [default:int_taewfiies_5004:2b53456118f24b6f@10.233.90.83:5004][doShutdown] Shutting down [2024-01-30T00:06:06,515][INFO ][Test worker][Cluster] [default:int_taewfiies_5004:2b53456118f24b6f@10.233.90.83:5004][leaveCluster] Leaving cluster [2024-01-30T00:06:06,515][INFO ][%int_taewfiies_5005%JRaft-Response-Processor-1][Replicator] Replicator=Replicator [state=Created, statInfo=, peerId=int_taewfiies_5006, type=Follower]@int_taewfiies_5006 is started [00:06:06] : [org.apache.ignite.raft.jraft.core.ItNodeTest.testAppendEntriesWhenFollowerIsInErrorState()] [Test Error Output] [2024-01-30T00:06:05,433][WARN ][Test worker][LocalSnapshotStorage] No data for snapshot reader build/work/ItNodeTest/testAppendEntriesWhenFollowerIsInErrorState_30809148389477213/int_taewfiies_5007/snapshot. [2024-01-30T00:06:06,487][WARN ][%int_taewfiies_5004%JRaft-Response-Processor-2][NodeImpl] Node received invalid PreVoteResponse from int_taewfiies_5003, state not in STATE_FOLLOWER but STATE_CANDIDATE. [2024-01-30T00:06:06,487][WARN ][%int_taewfiies_5004%JRaft-Response-Processor-3][NodeImpl] Node received invalid PreVoteResponse from int_taewfiies_5006, state not in STATE_FOLLOWER but STATE_CANDIDATE. [2024-01-30T00:06:06,492][WARN ][%int_taewfiies_5004%JRaft-Response-Processor-2][NodeImpl] Node received invalid RequestVoteResponse from int_taewfiies_5007, state not in STATE_CANDIDATE but STATE_LEADER. [2024-01-30T00:06:06,493][WARN ][%int_taewfiies_5004%JRaft-Response-Processor-3][NodeImpl] Node received invalid RequestVoteResponse from int_taewfiies_5003, state not in STATE_CANDIDATE but STATE_LEADER. [2024-01-30T00:06:06,502][WARN ][Test worker][NodeImpl] Node got error: Error [type=ERROR_TYPE_STATE_MACHINE, status=Status[UNKNOWN<-1>: Follower has something wrong.]]. [2024-01-30T00:06:06,505][WARN ][%int_taewfiies_5004%JRaft-Common-Executor-2][Replicator] Replicator Replicator [state=Replicate, statInfo=, peerId=int_taewfiies_5005, type=Follower] stops sending entries. [2024-01-30T00:06:06,508][WARN ][%int_taewfiies_5004%MessagingService-inbound-0-0][IgniteRpcServer] A request execution was rejected [sender=ClusterNodeImpl [id=ba086225-8ff5-467a-9919-67782d28b656, name=int_taewfiies_5005, address=10.233.90.83:5005, nodeMetadata=null] req=PingRequestImpl [sendTimestamp=1706573166508] reason=Task org.apache.ignite.raft.jraft.rpc.impl.IgniteRpcServer$RpcMessageHandler$$Lambda$1384/0x0000000800775c40@72031fee rejected from org.apache.ignite.raft.jraft.util.MetricThreadPoolExecutor@69e57355[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0]] [2024-01-30T00:06:06,512][WARN ][%int_taewfiies_5003%JRaft-Request-Processor-0][NodeImpl] Node is not in active state, currTerm=1. [2024-01-30T00:06:06,513][WARN ][%int_taewfiies_5005%JRaft-Response-Processor-3][NodeImpl] Node RequestVote to int_taewfiies_5003 error: Status[EINVAL<1015>: Node is not in active state, state STATE_ERROR.]. [00:06:06] : [org.apache.ignite.raft.jraft.core.ItNodeTest.testAppendEntriesWhenFollowerIsInErrorState()] [Test Output] [2024-01-30T00:06:06,516][INFO ][%int_taewfiies_5005%JRaft-Response-Processor-1][Replicator] Replicator=Replicator [state=Created, statInfo=, peerId=int_taewfiies_5007, type=Follower]@int_taewfiies_5007 is started [2024-01-30T00:06:06,516][INFO ][%int_taewfiies_5005%JRaft-Response-Processor-1][Replicator] Replicator=Replicator [state=Created, statInfo=, peerId=int_taewfiies_5003, type=Follower]@int_taewfiies_5003 is started [2024-01-30T00:06:06,517][INFO ][%int_taewfiies_5006%JRaft-FSMCaller-Disruptor-_stripe_3-0][StateMachineAdapter] onStartFollowing: LeaderChangeContext [leaderId=int_taewfiies_5005, term=2, status=Status[ENEWLEADER<10011>: Follower receives message from new leader with the same term.]]. [2024-01-30T00:06:06,517][INFO ][%int_taewfiies_5007%JRaft-FSMCaller-Disruptor-_stripe_4-0][StateMachineAdapter] onStartFollowing: LeaderChangeContext [leaderId=int_taewfiies_5005, term=2, status=Status[ENEWLEADER<10011>: Follower receives message from new leader with the same term.]]. [2024-01-30T00:06:06,520][INFO ][sc-cluster-5007-272][MembershipProtocol] [default:int_taewfiies_5007:1a48dc796fed49a5@10.233.90.83:5007][publishEvent] MembershipEvent[type=LEAVING, member=default:int_taewfiies_5004:2b53456118f24b6f@10.233.90.83:5004, oldMetadata=null, newMetadata=7e16449-5, timestamp=2024-01-30T00:06:06.520Z] [2024-01-30T00:06:06,520][INFO ][sc-cluster-5003-268][MembershipProtocol] [default:int_taewfiies_5003:6939bfc9a1b24909@10.233.90.83:5003][publishEvent] MembershipEvent[type=LEAVING, member=default:int_taewfiies_5004:2b53456118f24b6f@10.233.90.83:5004, oldMetadata=null, newMetadata=7e16449-5, timestamp=2024-01-30T00:06:06.520Z] [2024-01-30T00:06:06,520][INFO ][sc-cluster-5006-271][MembershipProtocol] [default:int_taewfiies_5006:59b4088774334760@10.233.90.83:5006][publishEvent] MembershipEvent[type=LEAVING, member=default:int_taewfiies_5004:2b53456118f24b6f@10.233.90.83:5004, oldMetadata=null, newMetadata=7e16449-5, timestamp=2024-01-30T00:06:06.520Z] [2024-01-30T00:06:06,520][INFO ][sc-cluster-5007-272][ScaleCubeTopologyService] Node left [member=ClusterNodeImpl [id=2b534561-18f2-4b6f-98da-a0d11a18c0bb, name=int_taewfiies_5004, address=10.233.90.83:5004, nodeMetadata=null], eventType=LEAVING] [00:06:06] : [org.apache.ignite.raft.jraft.core.ItNodeTest.testAppendEntriesWhenFollowerIsInErrorState()] [Test Error Output] [2024-01-30T00:06:06,504][ERROR][%int_taewfiies_5003%JRaft-FSMCaller-Disruptor-_stripe_0-0][StateMachineAdapter] Encountered an error=Status[UNKNOWN<-1>: Follower has something wrong.] on StateMachine org.apache.ignite.raft.jraft.core.MockStateMachine, it's highly recommended to implement this method as raft stops working since some error occurs, you should figure out the cause and repair or remove this node. org.apache.ignite.raft.jraft.error.RaftException: Follower has something wrong. at org.apache.ignite.raft.jraft.core.ItNodeTest.testAppendEntriesWhenFollowerIsInErrorState(ItNodeTest.java:2547) ~[integrationTest/:?] at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?] at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?] at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?] at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?] at org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:727) ~[junit-platform-commons-1.9.3.jar:1.9.3] at org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60) ~[junit-jupiter-engine-5.9.3.jar:5.9.3] at org.junit.jupiter.engine.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChain.java:131) ~[junit-jupiter-engine-5.9.3.jar:5.9.3] at org.junit.jupiter.engine.extension.SameThreadTimeoutInvocation.proceed(SameThreadTimeoutInvocation.java:45) ~[junit-jupiter-engine-5.9.3.jar:5.9.3] at org.junit.jupiter.engine.extension.TimeoutExtension.intercept(TimeoutExtension.java:156) ~[junit-jupiter-engine-5.9.3.jar:5.9.3] at org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestableMethod(TimeoutExtension.java:147) ~[junit-jupiter-engine-5.9.3.jar:5.9.3] at org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestMethod(TimeoutExtension.java:86) ~[junit-jupiter-engine-5.9.3.jar:5.9.3] at org.junit.jupiter.engine.execution.InterceptingExecutableInvoker$ReflectiveInterceptorCall.lambda$ofVoidMethod$0(InterceptingExecutableInvoker.java:103) ~[junit-jupiter-engine-5.9.3.jar:5.9.3] at org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.lambda$invoke$0(InterceptingExecutableInvoker.java:93) ~[junit-jupiter-engine-5.9.3.jar:5.9.3] at org.junit.jupiter.engine.execution.InvocationInterceptorChain$InterceptedInvocation.proceed(InvocationInterceptorChain.java:106) ~[junit-jupiter-engine-5.9.3.jar:5.9.3] at org.junit.jupiter.engine.execution.InvocationInterceptorChain.proceed(InvocationInterceptorChain.java:64) ~[junit-jupiter-engine-5.9.3.jar:5.9.3] at org.junit.jupiter.engine.execution.InvocationInterceptorChain.chainAndInvoke(InvocationInterceptorChain.java:45) ~[junit-jupiter-engine-5.9.3.jar:5.9.3] at org.junit.jupiter.engine.execution.InvocationInterceptorChain.invoke(InvocationInterceptorChain.java:37) ~[junit-jupiter-engine-5.9.3.jar:5.9.3] at org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.invoke(InterceptingExecutableInvoker.java:92) ~[junit-jupiter-engine-5.9.3.jar:5.9.3] at org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.invoke(InterceptingExecutableInvoker.java:86) ~[junit-jupiter-engine-5.9.3.jar:5.9.3] at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.lambda$invokeTestMethod$7(TestMethodTestDescriptor.java:217) ~[junit-jupiter-engine-5.9.3.jar:5.9.3] at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73) ~[junit-platform-engine-1.9.3.jar:1.9.3] at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.invokeTestMethod(TestMethodTestDescriptor.java:213) ~[junit-jupiter-engine-5.9.3.jar:5.9.3] at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:138) ~[junit-jupiter-engine-5.9.3.jar:5.9.3] at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:68) ~[junit-jupiter-engine-5.9.3.jar:5.9.3] at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:151) ~[junit-platform-engine-1.9.3.jar:1.9.3] at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73) ~[junit-platform-engine-1.9.3.jar:1.9.3] at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141) ~[junit-platform-engine-1.9.3.jar:1.9.3] at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137) ~[junit-platform-engine-1.9.3.jar:1.9.3] at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139) ~[junit-platform-engine-1.9.3.jar:1.9.3] at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73) ~[junit-platform-engine-1.9.3.jar:1.9.3] at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138) ~[junit-platform-engine-1.9.3.jar:1.9.3] at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95) ~[junit-platform-engine-1.9.3.jar:1.9.3] at java.util.ArrayList.forEach(ArrayList.java:1541) ~[?:?] at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:41) ~[junit-platform-engine-1.9.3.jar:1.9.3] at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:155) ~[junit-platform-engine-1.9.3.jar:1.9.3] at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73) ~[junit-platform-engine-1.9.3.jar:1.9.3] at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141) ~[junit-platform-engine-1.9.3.jar:1.9.3] at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137) ~[junit-platform-engine-1.9.3.jar:1.9.3] at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139) ~[junit-platform-engine-1.9.3.jar:1.9.3] at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73) ~[junit-platform-engine-1.9.3.jar:1.9.3] at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138) ~[junit-platform-engine-1.9.3.jar:1.9.3] at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95) ~[junit-platform-engine-1.9.3.jar:1.9.3] at java.util.ArrayList.forEach(ArrayList.java:1541) ~[?:?] at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:41) ~[junit-platform-engine-1.9.3.jar:1.9.3] at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:155) ~[junit-platform-engine-1.9.3.jar:1.9.3] at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73) ~[junit-platform-engine-1.9.3.jar:1.9.3] at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141) ~[junit-platform-engine-1.9.3.jar:1.9.3] at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137) ~[junit-platform-engine-1.9.3.jar:1.9.3] at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139) ~[junit-platform-engine-1.9.3.jar:1.9.3] at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73) ~[junit-platform-engine-1.9.3.jar:1.9.3] at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138) ~[junit-platform-engine-1.9.3.jar:1.9.3] at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95) ~[junit-platform-engine-1.9.3.jar:1.9.3] at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.submit(SameThreadHierarchicalTestExecutorService.java:35) ~[junit-platform-engine-1.9.3.jar:1.9.3] at org.junit.platform.engine.support.hierarchical.HierarchicalTestExecutor.execute(HierarchicalTestExecutor.java:57) ~[junit-platform-engine-1.9.3.jar:1.9.3] at org.junit.platform.engine.support.hierarchical.HierarchicalTestEngine.execute(HierarchicalTestEngine.java:54) ~[junit-platform-engine-1.9.3.jar:1.9.3] at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:147) ~[junit-platform-launcher-1.9.3.jar:1.9.3] at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:127) ~[junit-platform-launcher-1.9.3.jar:1.9.3] at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:90) ~[junit-platform-launcher-1.9.3.jar:1.9.3] at org.junit.platform.launcher.core.EngineExecutionOrchestrator.lambda$execute$0(EngineExecutionOrchestrator.java:55) ~[junit-platform-launcher-1.9.3.jar:1.9.3] at org.junit.platform.launcher.core.EngineExecutionOrchestrator.withInterceptedStreams(EngineExecutionOrchestrator.java:102) ~[junit-platform-launcher-1.9.3.jar:1.9.3] at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:54) ~[junit-platform-launcher-1.9.3.jar:1.9.3] at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:114) ~[junit-platform-launcher-1.9.3.jar:1.9.3] at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:86) ~[junit-platform-launcher-1.9.3.jar:1.9.3] at org.junit.platform.launcher.core.DefaultLauncherSession$DelegatingLauncher.execute(DefaultLauncherSession.java:86) ~[junit-platform-launcher-1.9.3.jar:1.9.3] at org.junit.platform.launcher.core.SessionPerRequestLauncher.execute(SessionPerRequestLauncher.java:53) ~[junit-platform-launcher-1.9.3.jar:1.9.3] at org.gradle.api.internal.tasks.testing.junitplatform.JUnitPlatformTestClassProcessor$CollectAllTestClassesExecutor.processAllTestClasses(JUnitPlatformTestClassProcessor.java:99) ~[?:?] at org.gradle.api.internal.tasks.testing.junitplatform.JUnitPlatformTestClassProcessor$CollectAllTestClassesExecutor.access$000(JUnitPlatformTestClassProcessor.java:79) ~[?:?] at org.gradle.api.internal.tasks.testing.junitplatform.JUnitPlatformTestClassProcessor.stop(JUnitPlatformTestClassProcessor.java:75) ~[?:?] at org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.stop(SuiteTestClassProcessor.java:62) ~[?:?] at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?] at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?] at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?] at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?] at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:36) ~[?:?] at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24) ~[?:?] at org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:33) ~[?:?] at org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:94) ~[?:?] at com.sun.proxy.$Proxy5.stop(Unknown Source) ~[?:?] at org.gradle.api.internal.tasks.testing.worker.TestWorker$3.run(TestWorker.java:193) ~[?:?] at org.gradle.api.internal.tasks.testing.worker.TestWorker.executeAndMaintainThreadName(TestWorker.java:129) ~[?:?] at org.gradle.api.internal.tasks.testing.worker.TestWorker.execute(TestWorker.java:100) ~[?:?] at org.gradle.api.internal.tasks.testing.worker.TestWorker.execute(TestWorker.java:60) ~[?:?] at org.gradle.process.internal.worker.child.ActionExecutionWorker.execute(ActionExecutionWorker.java:56) ~[?:?] at org.gradle.process.internal.worker.child.SystemApplicationClassLoaderWorker.call(SystemApplicationClassLoaderWorker.java:113) ~[?:?] at org.gradle.process.internal.worker.child.SystemApplicationClassLoaderWorker.call(SystemApplicationClassLoaderWorker.java:65) ~[?:?] at worker.org.gradle.process.internal.worker.GradleWorkerMain.run(GradleWorkerMain.java:69) ~[gradle-worker.jar:?] at worker.org.gradle.process.internal.worker.GradleWorkerMain.main(GradleWorkerMain.java:74) ~[gradle-worker.jar:?] [2024-01-30T00:06:06,517][WARN ][%int_taewfiies_5003%JRaft-AppendEntries-Processor-1][NodeImpl] Node is not in active state, currTerm=1. [2024-01-30T00:06:06,517][WARN ][%int_taewfiies_5004%MessagingService-inbound-0-0][IgniteRpcServer] A request execution was rejected [sender=ClusterNodeImpl [id=ba086225-8ff5-467a-9919-67782d28b656, name=int_taewfiies_5005, address=10.233.90.83:5005, nodeMetadata=null] req=PingRequestImpl [sendTimestamp=1706573166517] reason=Task org.apache.ignite.raft.jraft.rpc.impl.IgniteRpcServer$RpcMessageHandler$$Lambda$1384/0x0000000800775c40@a7ddd34 rejected from org.apache.ignite.raft.jraft.util.MetricThreadPoolExecutor@69e57355[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0]] [2024-01-30T00:06:06,518][WARN ][%int_taewfiies_5003%JRaft-FSMCaller-Disruptor-_stripe_0-0][NodeImpl] Node got error: Error [type=ERROR_TYPE_STATE_MACHINE, status=Status[UNKNOWN<-1>: Follower has something wrong.]]. [00:06:06] : [org.apache.ignite.raft.jraft.core.ItNodeTest.testAppendEntriesWhenFollowerIsInErrorState()] [Test Output] [2024-01-30T00:06:06,520][INFO ][sc-cluster-5003-268][ScaleCubeTopologyService] Node left [member=ClusterNodeImpl [id=2b534561-18f2-4b6f-98da-a0d11a18c0bb, name=int_taewfiies_5004, address=10.233.90.83:5004, nodeMetadata=null], eventType=LEAVING] [2024-01-30T00:06:06,520][INFO ][sc-cluster-5006-271][ScaleCubeTopologyService] Node left [member=ClusterNodeImpl [id=2b534561-18f2-4b6f-98da-a0d11a18c0bb, name=int_taewfiies_5004, address=10.233.90.83:5004, nodeMetadata=null], eventType=LEAVING] [2024-01-30T00:06:06,520][INFO ][sc-cluster-5007-272][ScaleCubeTopologyService] Topology snapshot [nodes=[int_taewfiies_5007, int_taewfiies_5005, int_taewfiies_5006, int_taewfiies_5003]] [2024-01-30T00:06:06,521][INFO ][sc-cluster-5006-271][ScaleCubeTopologyService] Topology snapshot [nodes=[int_taewfiies_5007, int_taewfiies_5005, int_taewfiies_5006, int_taewfiies_5003]] [2024-01-30T00:06:06,521][INFO ][sc-cluster-5003-268][ScaleCubeTopologyService] Topology snapshot [nodes=[int_taewfiies_5007, int_taewfiies_5005, int_taewfiies_5006, int_taewfiies_5003]] [2024-01-30T00:06:06,522][INFO ][sc-cluster-5005-270][MembershipProtocol] [default:int_taewfiies_5005:3a0862258ff5467a@10.233.90.83:5005][publishEvent] MembershipEvent[type=LEAVING, member=default:int_taewfiies_5004:2b53456118f24b6f@10.233.90.83:5004, oldMetadata=null, newMetadata=7e16449-5, timestamp=2024-01-30T00:06:06.522Z] [2024-01-30T00:06:06,522][INFO ][sc-cluster-5005-270][ScaleCubeTopologyService] Node left [member=ClusterNodeImpl [id=2b534561-18f2-4b6f-98da-a0d11a18c0bb, name=int_taewfiies_5004, address=10.233.90.83:5004, nodeMetadata=null], eventType=LEAVING] [2024-01-30T00:06:06,522][INFO ][sc-cluster-5005-270][ScaleCubeTopologyService] Topology snapshot [nodes=[int_taewfiies_5007, int_taewfiies_5005, int_taewfiies_5006, int_taewfiies_5003]] [2024-01-30T00:06:06,527][INFO ][%int_taewfiies_5003%JRaft-FSMCaller-Disruptor-_stripe_0-0][StateMachineAdapter] onStopFollowing: LeaderChangeContext [leaderId=int_taewfiies_5004, term=1, status=Status[EBADNODE<10009>: Raft node(leader or candidate) is in error.]]. [2024-01-30T00:06:06,589][INFO ][sc-cluster-5004-269][Cluster] [default:int_taewfiies_5004:2b53456118f24b6f@10.233.90.83:5004][leaveCluster] Left cluster [2024-01-30T00:06:06,590][INFO ][sc-cluster-5004-269][Transport] Stopping [address=10.233.90.83:5004] [00:06:06] : [org.apache.ignite.raft.jraft.core.ItNodeTest.testAppendEntriesWhenFollowerIsInErrorState()] [Test Error Output] [2024-01-30T00:06:06,518][WARN ][%int_taewfiies_5003%JRaft-FSMCaller-Disruptor-_stripe_0-0][FSMCallerImpl] FSMCaller already in error status, ignore new error org.apache.ignite.raft.jraft.error.RaftException: Follower has something wrong. at org.apache.ignite.raft.jraft.core.ItNodeTest.testAppendEntriesWhenFollowerIsInErrorState(ItNodeTest.java:2547) ~[integrationTest/:?] at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?] at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?] at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?] at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?] at org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:727) ~[junit-platform-commons-1.9.3.jar:1.9.3] at org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60) ~[junit-jupiter-engine-5.9.3.jar:5.9.3] at org.junit.jupiter.engine.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChain.java:131) ~[junit-jupiter-engine-5.9.3.jar:5.9.3] at org.junit.jupiter.engine.extension.SameThreadTimeoutInvocation.proceed(SameThreadTimeoutInvocation.java:45) ~[junit-jupiter-engine-5.9.3.jar:5.9.3] at org.junit.jupiter.engine.extension.TimeoutExtension.intercept(TimeoutExtension.java:156) ~[junit-jupiter-engine-5.9.3.jar:5.9.3] at org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestableMethod(TimeoutExtension.java:147) ~[junit-jupiter-engine-5.9.3.jar:5.9.3] at org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestMethod(TimeoutExtension.java:86) ~[junit-jupiter-engine-5.9.3.jar:5.9.3] at org.junit.jupiter.engine.execution.InterceptingExecutableInvoker$ReflectiveInterceptorCall.lambda$ofVoidMethod$0(InterceptingExecutableInvoker.java:103) ~[junit-jupiter-engine-5.9.3.jar:5.9.3] at org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.lambda$invoke$0(InterceptingExecutableInvoker.java:93) ~[junit-jupiter-engine-5.9.3.jar:5.9.3] at org.junit.jupiter.engine.execution.InvocationInterceptorChain$InterceptedInvocation.proceed(InvocationInterceptorChain.java:106) ~[junit-jupiter-engine-5.9.3.jar:5.9.3] at org.junit.jupiter.engine.execution.InvocationInterceptorChain.proceed(InvocationInterceptorChain.java:64) ~[junit-jupiter-engine-5.9.3.jar:5.9.3] at org.junit.jupiter.engine.execution.InvocationInterceptorChain.chainAndInvoke(InvocationInterceptorChain.java:45) ~[junit-jupiter-engine-5.9.3.jar:5.9.3] at org.junit.jupiter.engine.execution.InvocationInterceptorChain.invoke(InvocationInterceptorChain.java:37) ~[junit-jupiter-engine-5.9.3.jar:5.9.3] at org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.invoke(InterceptingExecutableInvoker.java:92) ~[junit-jupiter-engine-5.9.3.jar:5.9.3] at org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.invoke(InterceptingExecutableInvoker.java:86) ~[junit-jupiter-engine-5.9.3.jar:5.9.3] at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.lambda$invokeTestMethod$7(TestMethodTestDescriptor.java:217) ~[junit-jupiter-engine-5.9.3.jar:5.9.3] at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73) ~[junit-platform-engine-1.9.3.jar:1.9.3] at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.invokeTestMethod(TestMethodTestDescriptor.java:213) ~[junit-jupiter-engine-5.9.3.jar:5.9.3] at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:138) ~[junit-jupiter-engine-5.9.3.jar:5.9.3] at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:68) ~[junit-jupiter-engine-5.9.3.jar:5.9.3] at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:151) ~[junit-platform-engine-1.9.3.jar:1.9.3] at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73) ~[junit-platform-engine-1.9.3.jar:1.9.3] at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141) ~[junit-platform-engine-1.9.3.jar:1.9.3] at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137) ~[junit-platform-engine-1.9.3.jar:1.9.3] at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139) ~[junit-platform-engine-1.9.3.jar:1.9.3] at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73) ~[junit-platform-engine-1.9.3.jar:1.9.3] at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138) ~[junit-platform-engine-1.9.3.jar:1.9.3] at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95) ~[junit-platform-engine-1.9.3.jar:1.9.3] at java.util.ArrayList.forEach(ArrayList.java:1541) ~[?:?] at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:41) ~[junit-platform-engine-1.9.3.jar:1.9.3] at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:155) ~[junit-platform-engine-1.9.3.jar:1.9.3] at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73) ~[junit-platform-engine-1.9.3.jar:1.9.3] at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141) ~[junit-platform-engine-1.9.3.jar:1.9.3] at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137) ~[junit-platform-engine-1.9.3.jar:1.9.3] at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139) ~[junit-platform-engine-1.9.3.jar:1.9.3] at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73) ~[junit-platform-engine-1.9.3.jar:1.9.3] at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138) ~[junit-platform-engine-1.9.3.jar:1.9.3] at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95) ~[junit-platform-engine-1.9.3.jar:1.9.3] at java.util.ArrayList.forEach(ArrayList.java:1541) ~[?:?] at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:41) ~[junit-platform-engine-1.9.3.jar:1.9.3] at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:155) ~[junit-platform-engine-1.9.3.jar:1.9.3] at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73) ~[junit-platform-engine-1.9.3.jar:1.9.3] at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141) ~[junit-platform-engine-1.9.3.jar:1.9.3] at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137) ~[junit-platform-engine-1.9.3.jar:1.9.3] at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139) ~[junit-platform-engine-1.9.3.jar:1.9.3] at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73) ~[junit-platform-engine-1.9.3.jar:1.9.3] at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138) ~[junit-platform-engine-1.9.3.jar:1.9.3] at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95) ~[junit-platform-engine-1.9.3.jar:1.9.3] at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.submit(SameThreadHierarchicalTestExecutorService.java:35) ~[junit-platform-engine-1.9.3.jar:1.9.3] at org.junit.platform.engine.support.hierarchical.HierarchicalTestExecutor.execute(HierarchicalTestExecutor.java:57) ~[junit-platform-engine-1.9.3.jar:1.9.3] at org.junit.platform.engine.support.hierarchical.HierarchicalTestEngine.execute(HierarchicalTestEngine.java:54) ~[junit-platform-engine-1.9.3.jar:1.9.3] at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:147) ~[junit-platform-launcher-1.9.3.jar:1.9.3] at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:127) ~[junit-platform-launcher-1.9.3.jar:1.9.3] at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:90) ~[junit-platform-launcher-1.9.3.jar:1.9.3] at org.junit.platform.launcher.core.EngineExecutionOrchestrator.lambda$execute$0(EngineExecutionOrchestrator.java:55) ~[junit-platform-launcher-1.9.3.jar:1.9.3] at org.junit.platform.launcher.core.EngineExecutionOrchestrator.withInterceptedStreams(EngineExecutionOrchestrator.java:102) ~[junit-platform-launcher-1.9.3.jar:1.9.3] at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:54) ~[junit-platform-launcher-1.9.3.jar:1.9.3] at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:114) ~[junit-platform-launcher-1.9.3.jar:1.9.3] at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:86) ~[junit-platform-launcher-1.9.3.jar:1.9.3] at org.junit.platform.launcher.core.DefaultLauncherSession$DelegatingLauncher.execute(DefaultLauncherSession.java:86) ~[junit-platform-launcher-1.9.3.jar:1.9.3] at org.junit.platform.launcher.core.SessionPerRequestLauncher.execute(SessionPerRequestLauncher.java:53) ~[junit-platform-launcher-1.9.3.jar:1.9.3] at org.gradle.api.internal.tasks.testing.junitplatform.JUnitPlatformTestClassProcessor$CollectAllTestClassesExecutor.processAllTestClasses(JUnitPlatformTestClassProcessor.java:99) ~[?:?] at org.gradle.api.internal.tasks.testing.junitplatform.JUnitPlatformTestClassProcessor$CollectAllTestClassesExecutor.access$000(JUnitPlatformTestClassProcessor.java:79) ~[?:?] at org.gradle.api.internal.tasks.testing.junitplatform.JUnitPlatformTestClassProcessor.stop(JUnitPlatformTestClassProcessor.java:75) ~[?:?] at org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.stop(SuiteTestClassProcessor.java:62) ~[?:?] at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?] at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?] at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?] at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?] at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:36) ~[?:?] at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24) ~[?:?] at org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:33) ~[?:?] at org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:94) ~[?:?] at com.sun.proxy.$Proxy5.stop(Unknown Source) ~[?:?] at org.gradle.api.internal.tasks.testing.worker.TestWorker$3.run(TestWorker.java:193) ~[?:?] at org.gradle.api.internal.tasks.testing.worker.TestWorker.executeAndMaintainThreadName(TestWorker.java:129) ~[?:?] at org.gradle.api.internal.tasks.testing.worker.TestWorker.execute(TestWorker.java:100) ~[?:?] at org.gradle.api.internal.tasks.testing.worker.TestWorker.execute(TestWorker.java:60) ~[?:?] at org.gradle.process.internal.worker.child.ActionExecutionWorker.execute(ActionExecutionWorker.java:56) ~[?:?] at org.gradle.process.internal.worker.child.SystemApplicationClassLoaderWorker.call(SystemApplicationClassLoaderWorker.java:113) ~[?:?] at org.gradle.process.internal.worker.child.SystemApplicationClassLoaderWorker.call(SystemApplicationClassLoaderWorker.java:65) ~[?:?] at worker.org.gradle.process.internal.worker.GradleWorkerMain.run(GradleWorkerMain.java:69) ~[gradle-worker.jar:?] at worker.org.gradle.process.internal.worker.GradleWorkerMain.main(GradleWorkerMain.java:74) ~[gradle-worker.jar:?] [00:06:07] : [org.apache.ignite.raft.jraft.core.ItNodeTest.testAppendEntriesWhenFollowerIsInErrorState()] [Test Output] [2024-01-30T00:06:06,590][INFO ][sc-cluster-5004-269][Transport] Stopped [address=10.233.90.83:5004] [2024-01-30T00:06:06,590][INFO ][sc-cluster-5004-269][Cluster] [default:int_taewfiies_5004:2b53456118f24b6f@10.233.90.83:5004][doShutdown] Shutdown [2024-01-30T00:06:06,852][INFO ][%int_taewfiies_5005%JRaft-FSMCaller-Disruptor-_stripe_2-0][StateMachineAdapter] onLeaderStart: term=2. [2024-01-30T00:06:06,861][INFO ][Test worker][ItNodeTest] Elect a new leader JRaftNode [nodeId=] [2024-01-30T00:06:06,883][INFO ][Test worker][ItNodeTest] Stop error follower JRaftNode [nodeId=] [2024-01-30T00:06:06,883][INFO ][%int_taewfiies_5003%JRaft-Request-Processor-1][LogThreadPoolExecutor] ThreadPool is terminated: %int_taewfiies_5003%JRaft-Request-Processor-, org.apache.ignite.raft.jraft.util.MetricThreadPoolExecutor@1852941e[Shutting down, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 5]. [2024-01-30T00:06:06,883][INFO ][Test worker][NodeImpl] Node shutdown, currTerm=1 state=STATE_ERROR. [2024-01-30T00:06:06,885][INFO ][Test worker][LocalRaftMetaStorage] Save raft meta, path=build/work/ItNodeTest/testAppendEntriesWhenFollowerIsInErrorState_30809148389477213/int_taewfiies_5003/meta, term=1, votedFor=int_taewfiies_5004, cost time=1 ms [2024-01-30T00:06:06,885][INFO ][Test worker][FSMCallerImpl] Shutting down FSMCaller... [2024-01-30T00:06:06,885][INFO ][%int_taewfiies_5003%JRaft-FSMCaller-Disruptor-_stripe_0-0][StateMachineAdapter] onShutdown. [2024-01-30T00:06:06,886][INFO ][%int_taewfiies_5003%JRaft-Node-Scheduler-0][LogScheduledThreadPoolExecutor] ThreadPool is terminated: %int_taewfiies_5003%JRaft-Node-Scheduler-, org.apache.ignite.raft.jraft.util.MetricScheduledThreadPoolExecutor@65bf0f84[Shutting down, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 1]. [2024-01-30T00:06:06,886][INFO ][%int_taewfiies_5003%JRaft-ElectionTimer-4][LogScheduledThreadPoolExecutor] ThreadPool is terminated: %int_taewfiies_5003%JRaft-ElectionTimer-, org.apache.ignite.raft.jraft.util.MetricScheduledThreadPoolExecutor@1f8dde5b[Shutting down, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0]. [2024-01-30T00:06:06,886][INFO ][Test worker][LogScheduledThreadPoolExecutor] ThreadPool is terminated: %int_taewfiies_5003%JRaft-VoteTimer-, org.apache.ignite.raft.jraft.util.MetricScheduledThreadPoolExecutor@3964b02[Shutting down, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0]. [2024-01-30T00:06:06,887][INFO ][Test worker][LogScheduledThreadPoolExecutor] ThreadPool is terminated: %int_taewfiies_5003%JRaft-StepDownTimer-, org.apache.ignite.raft.jraft.util.MetricScheduledThreadPoolExecutor@7fdafbb3[Shutting down, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0]. [2024-01-30T00:06:06,887][INFO ][%int_taewfiies_5003%JRaft-SnapshotTimer-0][LogScheduledThreadPoolExecutor] ThreadPool is terminated: %int_taewfiies_5003%JRaft-SnapshotTimer-, org.apache.ignite.raft.jraft.util.MetricScheduledThreadPoolExecutor@2a230036[Shutting down, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0]. [2024-01-30T00:06:06,887][INFO ][%int_taewfiies_5003%JRaft-Common-Executor-2][LogThreadPoolExecutor] ThreadPool is terminated: %int_taewfiies_5003%JRaft-Common-Executor-, org.apache.ignite.raft.jraft.util.MetricThreadPoolExecutor@1a4e73e[Shutting down, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 4]. [2024-01-30T00:06:06,892][INFO ][Test worker][LogThreadPoolExecutor] ThreadPool is terminated: %int_taewfiies_5003%JRaft-Response-Processor-, org.apache.ignite.raft.jraft.util.MetricThreadPoolExecutor@2ddedfed[Shutting down, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0]. [2024-01-30T00:06:06,892][INFO ][Test worker][RaftGroupService] Stop the RaftGroupService successfully. [2024-01-30T00:06:06,893][INFO ][Test worker][Cluster] [default:int_taewfiies_5003:6939bfc9a1b24909@10.233.90.83:5003][doShutdown] Shutting down [2024-01-30T00:06:06,893][INFO ][Test worker][Cluster] [default:int_taewfiies_5003:6939bfc9a1b24909@10.233.90.83:5003][leaveCluster] Leaving cluster [2024-01-30T00:06:06,902][INFO ][sc-cluster-5006-271][MembershipProtocol] [default:int_taewfiies_5006:59b4088774334760@10.233.90.83:5006][publishEvent] MembershipEvent[type=LEAVING, member=default:int_taewfiies_5003:6939bfc9a1b24909@10.233.90.83:5003, oldMetadata=null, newMetadata=7e16449-5, timestamp=2024-01-30T00:06:06.902Z] [2024-01-30T00:06:06,902][INFO ][sc-cluster-5006-271][ScaleCubeTopologyService] Node left [member=ClusterNodeImpl [id=e939bfc9-a1b2-4909-b37c-148ed686c3ea, name=int_taewfiies_5003, address=10.233.90.83:5003, nodeMetadata=null], eventType=LEAVING] [2024-01-30T00:06:06,902][INFO ][sc-cluster-5005-270][MembershipProtocol] [default:int_taewfiies_5005:3a0862258ff5467a@10.233.90.83:5005][publishEvent] MembershipEvent[type=LEAVING, member=default:int_taewfiies_5003:6939bfc9a1b24909@10.233.90.83:5003, oldMetadata=null, newMetadata=7e16449-5, timestamp=2024-01-30T00:06:06.902Z] [2024-01-30T00:06:06,902][INFO ][sc-cluster-5006-271][ScaleCubeTopologyService] Topology snapshot [nodes=[int_taewfiies_5007, int_taewfiies_5005, int_taewfiies_5006]] [2024-01-30T00:06:06,902][INFO ][sc-cluster-5005-270][ScaleCubeTopologyService] Node left [member=ClusterNodeImpl [id=e939bfc9-a1b2-4909-b37c-148ed686c3ea, name=int_taewfiies_5003, address=10.233.90.83:5003, nodeMetadata=null], eventType=LEAVING] [2024-01-30T00:06:06,902][INFO ][sc-cluster-5005-270][ScaleCubeTopologyService] Topology snapshot [nodes=[int_taewfiies_5007, int_taewfiies_5005, int_taewfiies_5006]] [2024-01-30T00:06:06,908][INFO ][sc-cluster-5007-272][MembershipProtocol] [default:int_taewfiies_5007:1a48dc796fed49a5@10.233.90.83:5007][publishEvent] MembershipEvent[type=LEAVING, member=default:int_taewfiies_5003:6939bfc9a1b24909@10.233.90.83:5003, oldMetadata=null, newMetadata=7e16449-5, timestamp=2024-01-30T00:06:06.908Z] [2024-01-30T00:06:06,909][INFO ][sc-cluster-5007-272][ScaleCubeTopologyService] Node left [member=ClusterNodeImpl [id=e939bfc9-a1b2-4909-b37c-148ed686c3ea, name=int_taewfiies_5003, address=10.233.90.83:5003, nodeMetadata=null], eventType=LEAVING] [2024-01-30T00:06:06,909][INFO ][sc-cluster-5007-272][ScaleCubeTopologyService] Topology snapshot [nodes=[int_taewfiies_5007, int_taewfiies_5005, int_taewfiies_5006]] [2024-01-30T00:06:06,972][INFO ][sc-cluster-5003-268][Cluster] [default:int_taewfiies_5003:6939bfc9a1b24909@10.233.90.83:5003][leaveCluster] Left cluster [2024-01-30T00:06:06,972][INFO ][sc-cluster-5003-268][Transport] Stopping [address=10.233.90.83:5003] [2024-01-30T00:06:06,972][INFO ][sc-cluster-5003-268][Transport] Stopped [address=10.233.90.83:5003] [2024-01-30T00:06:06,972][INFO ][sc-cluster-5003-268][Cluster] [default:int_taewfiies_5003:6939bfc9a1b24909@10.233.90.83:5003][doShutdown] Shutdown [2024-01-30T00:06:06,978][INFO ][Test worker][ItNodeTest] Restart error follower int_taewfiies_5003 and old leader int_taewfiies_5004 [2024-01-30T00:06:07,015][INFO ][Test worker][ConnectionManager] Server started [address=/0:0:0:0:0:0:0:0:5003] [2024-01-30T00:06:07,016][INFO ][Test worker][Cluster] [default:int_taewfiies_5003:2683329c8fe407e@10.233.90.83:5003][doStart] Starting, config: ClusterConfig[metadata=null, metadataTimeout=1000, metadataCodec=io.scalecube.cluster.metadata.JdkMetadataCodec@4787b3aa, memberId='02683329-c8fe-407e-afee-1c81eceaa9a0', memberAlias='int_taewfiies_5003', externalHost='null', externalPort=null, transportConfig=TransportConfig[port=0, clientSecured=false, connectTimeout=1000, messageCodec=io.scalecube.cluster.transport.api.JdkMessageCodec@60eb78bd, maxFrameLength=2097152, transportFactory=org.apache.ignite.network.scalecube.ScaleCubeClusterServiceFactory$2$$Lambda$858/0x0000000800647040@6252a8f5, addressMapper=java.util.function.Function$$Lambda$380/0x000000080024f040@2372da56], failureDetectorConfig=FailureDetectorConfig[pingInterval=500, pingTimeout=200, pingReqMembers=1], gossipConfig=GossipConfig[gossipFanout=3, gossipInterval=10, gossipRepeatMult=2, gossipSegmentationThreshold=1000], membershipConfig=MembershipConfig[seedMembers=[10.233.90.83:5003, 10.233.90.83:5004, 10.233.90.83:5005, 10.233.90.83:5006, 10.233.90.83:5007], syncInterval=1000, syncTimeout=3000, suspicionMult=1, namespace='default', removedMembersHistorySize=42]] [2024-01-30T00:06:07,016][INFO ][Test worker][MembershipProtocol] [default:int_taewfiies_5003:2683329c8fe407e@10.233.90.83:5003] Making initial Sync to all seed members: [10.233.90.83:5004, 10.233.90.83:5005, 10.233.90.83:5006, 10.233.90.83:5007] [2024-01-30T00:06:07,023][INFO ][sc-cluster-5006-271][MembershipProtocol] [default:int_taewfiies_5006:59b4088774334760@10.233.90.83:5006][publishEvent] MembershipEvent[type=ADDED, member=default:int_taewfiies_5003:2683329c8fe407e@10.233.90.83:5003, oldMetadata=null, newMetadata=7e16449-5, timestamp=2024-01-30T00:06:07.023Z] [2024-01-30T00:06:07,023][INFO ][sc-cluster-5005-270][MembershipProtocol] [default:int_taewfiies_5005:3a0862258ff5467a@10.233.90.83:5005][publishEvent] MembershipEvent[type=ADDED, member=default:int_taewfiies_5003:2683329c8fe407e@10.233.90.83:5003, oldMetadata=null, newMetadata=7e16449-5, timestamp=2024-01-30T00:06:07.023Z] [2024-01-30T00:06:07,023][INFO ][sc-cluster-5007-272][MembershipProtocol] [default:int_taewfiies_5007:1a48dc796fed49a5@10.233.90.83:5007][publishEvent] MembershipEvent[type=ADDED, member=default:int_taewfiies_5003:2683329c8fe407e@10.233.90.83:5003, oldMetadata=null, newMetadata=7e16449-5, timestamp=2024-01-30T00:06:07.023Z] [2024-01-30T00:06:07,023][INFO ][sc-cluster-5006-271][ScaleCubeTopologyService] Node joined [node=ClusterNodeImpl [id=02683329-c8fe-407e-afee-1c81eceaa9a0, name=int_taewfiies_5003, address=10.233.90.83:5003, nodeMetadata=null]] [2024-01-30T00:06:07,023][INFO ][sc-cluster-5006-271][ScaleCubeTopologyService] Topology snapshot [nodes=[int_taewfiies_5007, int_taewfiies_5005, int_taewfiies_5006, int_taewfiies_5003]] [2024-01-30T00:06:07,023][INFO ][sc-cluster-5005-270][ScaleCubeTopologyService] Node joined [node=ClusterNodeImpl [id=02683329-c8fe-407e-afee-1c81eceaa9a0, name=int_taewfiies_5003, address=10.233.90.83:5003, nodeMetadata=null]] [2024-01-30T00:06:07,023][INFO ][sc-cluster-5005-270][ScaleCubeTopologyService] Topology snapshot [nodes=[int_taewfiies_5007, int_taewfiies_5005, int_taewfiies_5006, int_taewfiies_5003]] [2024-01-30T00:06:07,023][INFO ][sc-cluster-5007-272][ScaleCubeTopologyService] Node joined [node=ClusterNodeImpl [id=02683329-c8fe-407e-afee-1c81eceaa9a0, name=int_taewfiies_5003, address=10.233.90.83:5003, nodeMetadata=null]] [2024-01-30T00:06:07,023][INFO ][sc-cluster-5007-272][ScaleCubeTopologyService] Topology snapshot [nodes=[int_taewfiies_5007, int_taewfiies_5005, int_taewfiies_5006, int_taewfiies_5003]] [2024-01-30T00:06:07,025][INFO ][sc-cluster-5003-273][MembershipProtocol] [default:int_taewfiies_5003:2683329c8fe407e@10.233.90.83:5003][publishEvent] MembershipEvent[type=ADDED, member=default:int_taewfiies_5006:59b4088774334760@10.233.90.83:5006, oldMetadata=null, newMetadata=7e16449-5, timestamp=2024-01-30T00:06:07.025Z] [2024-01-30T00:06:07,025][INFO ][sc-cluster-5003-273][ScaleCubeTopologyService] Node joined [node=ClusterNodeImpl [id=59b40887-7433-4760-9420-837a02c05cb4, name=int_taewfiies_5006, address=10.233.90.83:5006, nodeMetadata=null]] [2024-01-30T00:06:07,025][INFO ][sc-cluster-5003-273][ScaleCubeTopologyService] Topology snapshot [nodes=[int_taewfiies_5006]] [2024-01-30T00:06:07,025][INFO ][sc-cluster-5003-273][MembershipProtocol] [default:int_taewfiies_5003:2683329c8fe407e@10.233.90.83:5003][publishEvent] MembershipEvent[type=ADDED, member=default:int_taewfiies_5007:1a48dc796fed49a5@10.233.90.83:5007, oldMetadata=null, newMetadata=7e16449-5, timestamp=2024-01-30T00:06:07.025Z] [2024-01-30T00:06:07,025][INFO ][sc-cluster-5003-273][ScaleCubeTopologyService] Node joined [node=ClusterNodeImpl [id=9a48dc79-6fed-49a5-99da-fc4b1ec1f94a, name=int_taewfiies_5007, address=10.233.90.83:5007, nodeMetadata=null]] [2024-01-30T00:06:07,025][INFO ][sc-cluster-5003-273][ScaleCubeTopologyService] Topology snapshot [nodes=[int_taewfiies_5007, int_taewfiies_5006]] [2024-01-30T00:06:07,025][INFO ][sc-cluster-5003-273][MembershipProtocol] [default:int_taewfiies_5003:2683329c8fe407e@10.233.90.83:5003][publishEvent] MembershipEvent[type=ADDED, member=default:int_taewfiies_5005:3a0862258ff5467a@10.233.90.83:5005, oldMetadata=null, newMetadata=7e16449-5, timestamp=2024-01-30T00:06:07.025Z] [2024-01-30T00:06:07,025][INFO ][sc-cluster-5003-273][ScaleCubeTopologyService] Node joined [node=ClusterNodeImpl [id=ba086225-8ff5-467a-9919-67782d28b656, name=int_taewfiies_5005, address=10.233.90.83:5005, nodeMetadata=null]] [2024-01-30T00:06:07,025][INFO ][sc-cluster-5003-273][ScaleCubeTopologyService] Topology snapshot [nodes=[int_taewfiies_5007, int_taewfiies_5005, int_taewfiies_5006]] [2024-01-30T00:06:07,096][INFO ][sc-cluster-5003-273][Cluster] [default:int_taewfiies_5003:2683329c8fe407e@10.233.90.83:5003][doStart] Started [2024-01-30T00:06:07,097][INFO ][Test worker][ScaleCubeTopologyService] Node joined [node=ClusterNodeImpl [id=02683329-c8fe-407e-afee-1c81eceaa9a0, name=int_taewfiies_5003, address=10.233.90.83:5003, nodeMetadata=null]] [2024-01-30T00:06:07,097][INFO ][Test worker][ScaleCubeTopologyService] Topology snapshot [nodes=[int_taewfiies_5007, int_taewfiies_5005, int_taewfiies_5006, int_taewfiies_5003]] [2024-01-30T00:06:07,120][INFO ][Test worker][FSMCallerImpl] Starts FSMCaller successfully. [2024-01-30T00:06:07,120][INFO ][Test worker][NodeImpl] Node init, term=1, lastLogId=LogId [index=0, term=0], conf=int_taewfiies_5003,int_taewfiies_5004,int_taewfiies_5005,int_taewfiies_5006,int_taewfiies_5007, oldConf=. [2024-01-30T00:06:07,121][INFO ][Test worker][RaftGroupService] Start the RaftGroupService successfully [2024-01-30T00:06:07,125][INFO ][%int_taewfiies_5003%JRaft-AppendEntries-Processor-0][LocalRaftMetaStorage] Save raft meta, path=build/work/ItNodeTest/testAppendEntriesWhenFollowerIsInErrorState_30809148389477213/int_taewfiies_5003/meta, term=2, votedFor=, cost time=1 ms [2024-01-30T00:06:07,126][INFO ][%int_taewfiies_5003%JRaft-FSMCaller-Disruptor-_stripe_0-0][StateMachineAdapter] onStartFollowing: LeaderChangeContext [leaderId=int_taewfiies_5005, term=2, status=Status[ENEWLEADER<10011>: Raft node receives message from new leader with higher term.]]. [2024-01-30T00:06:07,156][INFO ][Test worker][ConnectionManager] Server started [address=/0:0:0:0:0:0:0:0:5004] [2024-01-30T00:06:07,157][INFO ][Test worker][Cluster] [default:int_taewfiies_5004:7d20e3de9a4145a6@10.233.90.83:5004][doStart] Starting, config: ClusterConfig[metadata=null, metadataTimeout=1000, metadataCodec=io.scalecube.cluster.metadata.JdkMetadataCodec@4787b3aa, memberId='7d20e3de-9a41-45a6-818e-ac67c33fbb55', memberAlias='int_taewfiies_5004', externalHost='null', externalPort=null, transportConfig=TransportConfig[port=0, clientSecured=false, connectTimeout=1000, messageCodec=io.scalecube.cluster.transport.api.JdkMessageCodec@60eb78bd, maxFrameLength=2097152, transportFactory=org.apache.ignite.network.scalecube.ScaleCubeClusterServiceFactory$2$$Lambda$858/0x0000000800647040@63640bf6, addressMapper=java.util.function.Function$$Lambda$380/0x000000080024f040@2372da56], failureDetectorConfig=FailureDetectorConfig[pingInterval=500, pingTimeout=200, pingReqMembers=1], gossipConfig=GossipConfig[gossipFanout=3, gossipInterval=10, gossipRepeatMult=2, gossipSegmentationThreshold=1000], membershipConfig=MembershipConfig[seedMembers=[10.233.90.83:5003, 10.233.90.83:5004, 10.233.90.83:5005, 10.233.90.83:5006, 10.233.90.83:5007], syncInterval=1000, syncTimeout=3000, suspicionMult=1, namespace='default', removedMembersHistorySize=42]] [2024-01-30T00:06:07,158][INFO ][Test worker][MembershipProtocol] [default:int_taewfiies_5004:7d20e3de9a4145a6@10.233.90.83:5004] Making initial Sync to all seed members: [10.233.90.83:5003, 10.233.90.83:5005, 10.233.90.83:5006, 10.233.90.83:5007] [2024-01-30T00:06:07,164][INFO ][sc-cluster-5003-273][MembershipProtocol] [default:int_taewfiies_5003:2683329c8fe407e@10.233.90.83:5003][publishEvent] MembershipEvent[type=ADDED, member=default:int_taewfiies_5004:7d20e3de9a4145a6@10.233.90.83:5004, oldMetadata=null, newMetadata=7e16449-5, timestamp=2024-01-30T00:06:07.164Z] [2024-01-30T00:06:07,164][INFO ][sc-cluster-5007-272][MembershipProtocol] [default:int_taewfiies_5007:1a48dc796fed49a5@10.233.90.83:5007][publishEvent] MembershipEvent[type=ADDED, member=default:int_taewfiies_5004:7d20e3de9a4145a6@10.233.90.83:5004, oldMetadata=null, newMetadata=7e16449-5, timestamp=2024-01-30T00:06:07.164Z] [2024-01-30T00:06:07,164][INFO ][sc-cluster-5006-271][MembershipProtocol] [default:int_taewfiies_5006:59b4088774334760@10.233.90.83:5006][publishEvent] MembershipEvent[type=ADDED, member=default:int_taewfiies_5004:7d20e3de9a4145a6@10.233.90.83:5004, oldMetadata=null, newMetadata=7e16449-5, timestamp=2024-01-30T00:06:07.164Z] [2024-01-30T00:06:07,164][INFO ][sc-cluster-5003-273][ScaleCubeTopologyService] Node joined [node=ClusterNodeImpl [id=7d20e3de-9a41-45a6-818e-ac67c33fbb55, name=int_taewfiies_5004, address=10.233.90.83:5004, nodeMetadata=null]] [2024-01-30T00:06:07,164][INFO ][sc-cluster-5007-272][ScaleCubeTopologyService] Node joined [node=ClusterNodeImpl [id=7d20e3de-9a41-45a6-818e-ac67c33fbb55, name=int_taewfiies_5004, address=10.233.90.83:5004, nodeMetadata=null]] [2024-01-30T00:06:07,164][INFO ][sc-cluster-5006-271][ScaleCubeTopologyService] Node joined [node=ClusterNodeImpl [id=7d20e3de-9a41-45a6-818e-ac67c33fbb55, name=int_taewfiies_5004, address=10.233.90.83:5004, nodeMetadata=null]] [2024-01-30T00:06:07,164][INFO ][sc-cluster-5003-273][ScaleCubeTopologyService] Topology snapshot [nodes=[int_taewfiies_5007, int_taewfiies_5005, int_taewfiies_5006, int_taewfiies_5003, int_taewfiies_5004]] [2024-01-30T00:06:07,164][INFO ][sc-cluster-5006-271][ScaleCubeTopologyService] Topology snapshot [nodes=[int_taewfiies_5007, int_taewfiies_5005, int_taewfiies_5006, int_taewfiies_5003, int_taewfiies_5004]] [2024-01-30T00:06:07,164][INFO ][sc-cluster-5007-272][ScaleCubeTopologyService] Topology snapshot [nodes=[int_taewfiies_5007, int_taewfiies_5005, int_taewfiies_5006, int_taewfiies_5003, int_taewfiies_5004]] [2024-01-30T00:06:07,166][INFO ][sc-cluster-5004-274][MembershipProtocol] [default:int_taewfiies_5004:7d20e3de9a4145a6@10.233.90.83:5004][publishEvent] MembershipEvent[type=ADDED, member=default:int_taewfiies_5006:59b4088774334760@10.233.90.83:5006, oldMetadata=null, newMetadata=7e16449-5, timestamp=2024-01-30T00:06:07.166Z] [2024-01-30T00:06:07,167][INFO ][sc-cluster-5004-274][ScaleCubeTopologyService] Node joined [node=ClusterNodeImpl [id=59b40887-7433-4760-9420-837a02c05cb4, name=int_taewfiies_5006, address=10.233.90.83:5006, nodeMetadata=null]] [2024-01-30T00:06:07,167][INFO ][sc-cluster-5004-274][ScaleCubeTopologyService] Topology snapshot [nodes=[int_taewfiies_5006]] [2024-01-30T00:06:07,167][INFO ][sc-cluster-5004-274][MembershipProtocol] [default:int_taewfiies_5004:7d20e3de9a4145a6@10.233.90.83:5004][publishEvent] MembershipEvent[type=ADDED, member=default:int_taewfiies_5003:2683329c8fe407e@10.233.90.83:5003, oldMetadata=null, newMetadata=7e16449-5, timestamp=2024-01-30T00:06:07.167Z] [2024-01-30T00:06:07,167][INFO ][sc-cluster-5004-274][ScaleCubeTopologyService] Node joined [node=ClusterNodeImpl [id=02683329-c8fe-407e-afee-1c81eceaa9a0, name=int_taewfiies_5003, address=10.233.90.83:5003, nodeMetadata=null]] [2024-01-30T00:06:07,167][INFO ][sc-cluster-5004-274][ScaleCubeTopologyService] Topology snapshot [nodes=[int_taewfiies_5006, int_taewfiies_5003]] [00:06:07] : [org.apache.ignite.raft.jraft.core.ItNodeTest.testAppendEntriesWhenFollowerIsInErrorState()] [Test Error Output] [2024-01-30T00:06:06,617][WARN ][%int_taewfiies_5003%JRaft-AppendEntries-Processor-1][NodeImpl] Node is not in active state, currTerm=1. [2024-01-30T00:06:06,618][WARN ][%int_taewfiies_5003%JRaft-AppendEntries-Processor-1][NodeImpl] Node is not in active state, currTerm=1. [2024-01-30T00:06:06,718][WARN ][%int_taewfiies_5003%JRaft-AppendEntries-Processor-1][NodeImpl] Node is not in active state, currTerm=1. [2024-01-30T00:06:06,719][WARN ][%int_taewfiies_5003%JRaft-AppendEntries-Processor-1][NodeImpl] Node is not in active state, currTerm=1. [2024-01-30T00:06:06,819][WARN ][%int_taewfiies_5003%JRaft-AppendEntries-Processor-1][NodeImpl] Node is not in active state, currTerm=1. [2024-01-30T00:06:06,820][WARN ][%int_taewfiies_5003%JRaft-AppendEntries-Processor-1][NodeImpl] Node is not in active state, currTerm=1. [2024-01-30T00:06:06,851][ERROR][%int_taewfiies_5005%JRaft-Response-Processor-1][AbstractClientService] Fail to connect int_taewfiies_5004, exception: java.util.concurrent.TimeoutException. [2024-01-30T00:06:06,851][ERROR][%int_taewfiies_5005%JRaft-Response-Processor-1][ReplicatorGroupImpl] Fail to check replicator connection to peer=int_taewfiies_5004, replicatorType=Follower. [2024-01-30T00:06:06,851][ERROR][%int_taewfiies_5005%JRaft-Response-Processor-1][NodeImpl] Fail to add a replicator, peer=int_taewfiies_5004. [2024-01-30T00:06:06,920][ERROR][%int_taewfiies_5005%JRaft-Common-Executor-2][AbstractClientService] Fail to connect int_taewfiies_5003, exception: java.net.ConnectException. [2024-01-30T00:06:06,920][ERROR][%int_taewfiies_5005%JRaft-Common-Executor-3][AbstractClientService] Fail to connect int_taewfiies_5003, exception: java.net.ConnectException. [2024-01-30T00:06:07,016][WARN ][Test worker][MembershipProtocol] [default:int_taewfiies_5003:2683329c8fe407e@10.233.90.83:5003] Filtering out seed address: 10.233.90.83:5003 [2024-01-30T00:06:07,018][WARN ][int_taewfiies_5003-client-1][MembershipProtocol] [default:int_taewfiies_5003:2683329c8fe407e@10.233.90.83:5003] Exception on initial Sync, cause: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: /10.233.90.83:5004 [2024-01-30T00:06:07,020][ERROR][%int_taewfiies_5005%JRaft-Common-Executor-1][AbstractClientService] Fail to connect int_taewfiies_5003, exception: java.net.ConnectException. [2024-01-30T00:06:07,020][WARN ][%int_taewfiies_5005%JRaft-AppendEntries-Processor-3][Replicator] Fail to issue RPC to int_taewfiies_5003, consecutiveErrorTimes=10, error=Status[EINTERNAL<1004>: Check connection[int_taewfiies_5003] fail and try to create new one] [2024-01-30T00:06:07,021][ERROR][%int_taewfiies_5005%JRaft-Common-Executor-3][AbstractClientService] Fail to connect int_taewfiies_5003, exception: java.net.ConnectException. [2024-01-30T00:06:07,120][WARN ][Test worker][LocalSnapshotStorage] No data for snapshot reader build/work/ItNodeTest/testAppendEntriesWhenFollowerIsInErrorState_30809148389477213/int_taewfiies_5003/snapshot. [2024-01-30T00:06:07,126][WARN ][%int_taewfiies_5003%JRaft-AppendEntries-Processor-0][NodeImpl] Node reject term_unmatched AppendEntriesRequest from int_taewfiies_5005, term=2, prevLogIndex=11, prevLogTerm=1, localPrevLogTerm=0, lastLogIndex=0, entriesSize=0. [2024-01-30T00:06:07,126][WARN ][%int_taewfiies_5003%JRaft-AppendEntries-Processor-0][NodeImpl] Node reject term_unmatched AppendEntriesRequest from int_taewfiies_5005, term=2, prevLogIndex=11, prevLogTerm=1, localPrevLogTerm=0, lastLogIndex=0, entriesSize=0. [2024-01-30T00:06:07,157][WARN ][Test worker][MembershipProtocol] [default:int_taewfiies_5004:7d20e3de9a4145a6@10.233.90.83:5004] Filtering out seed address: 10.233.90.83:5004 [00:06:12] : [org.apache.ignite.raft.jraft.core.ItNodeTest.testAppendEntriesWhenFollowerIsInErrorState()] [Test Output] [2024-01-30T00:06:07,167][INFO ][sc-cluster-5004-274][MembershipProtocol] [default:int_taewfiies_5004:7d20e3de9a4145a6@10.233.90.83:5004][publishEvent] MembershipEvent[type=ADDED, member=default:int_taewfiies_5007:1a48dc796fed49a5@10.233.90.83:5007, oldMetadata=null, newMetadata=7e16449-5, timestamp=2024-01-30T00:06:07.167Z] [2024-01-30T00:06:07,167][INFO ][sc-cluster-5004-274][ScaleCubeTopologyService] Node joined [node=ClusterNodeImpl [id=9a48dc79-6fed-49a5-99da-fc4b1ec1f94a, name=int_taewfiies_5007, address=10.233.90.83:5007, nodeMetadata=null]] [2024-01-30T00:06:07,167][INFO ][sc-cluster-5004-274][ScaleCubeTopologyService] Topology snapshot [nodes=[int_taewfiies_5007, int_taewfiies_5006, int_taewfiies_5003]] [2024-01-30T00:06:07,379][INFO ][sc-cluster-5007-272][MembershipProtocol] [default:int_taewfiies_5007:1a48dc796fed49a5@10.233.90.83:5007] Member left gracefully: default:int_taewfiies_5004:2b53456118f24b6f@10.233.90.83:5004 [2024-01-30T00:06:07,379][INFO ][sc-cluster-5007-272][MembershipProtocol] [default:int_taewfiies_5007:1a48dc796fed49a5@10.233.90.83:5007][publishEvent] MembershipEvent[type=REMOVED, member=default:int_taewfiies_5004:2b53456118f24b6f@10.233.90.83:5004, oldMetadata=7e16449-5, newMetadata=null, timestamp=2024-01-30T00:06:07.379Z] [2024-01-30T00:06:07,379][INFO ][sc-cluster-5007-272][ScaleCubeTopologyService] Node left (noop as it has already reappeared) [member=ClusterNodeImpl [id=2b534561-18f2-4b6f-98da-a0d11a18c0bb, name=int_taewfiies_5004, address=10.233.90.83:5004, nodeMetadata=null], eventType=REMOVED] [2024-01-30T00:06:07,379][INFO ][sc-cluster-5007-272][ScaleCubeTopologyService] Topology snapshot [nodes=[int_taewfiies_5007, int_taewfiies_5005, int_taewfiies_5006, int_taewfiies_5003, int_taewfiies_5004]] [2024-01-30T00:06:07,781][INFO ][sc-cluster-5005-270][MembershipProtocol] [default:int_taewfiies_5005:3a0862258ff5467a@10.233.90.83:5005] Member left gracefully: default:int_taewfiies_5003:6939bfc9a1b24909@10.233.90.83:5003 [2024-01-30T00:06:07,782][INFO ][sc-cluster-5005-270][MembershipProtocol] [default:int_taewfiies_5005:3a0862258ff5467a@10.233.90.83:5005][publishEvent] MembershipEvent[type=REMOVED, member=default:int_taewfiies_5003:6939bfc9a1b24909@10.233.90.83:5003, oldMetadata=7e16449-5, newMetadata=null, timestamp=2024-01-30T00:06:07.781Z] [2024-01-30T00:06:07,782][INFO ][sc-cluster-5005-270][ScaleCubeTopologyService] Node left (noop as it has already reappeared) [member=ClusterNodeImpl [id=e939bfc9-a1b2-4909-b37c-148ed686c3ea, name=int_taewfiies_5003, address=10.233.90.83:5003, nodeMetadata=null], eventType=REMOVED] [2024-01-30T00:06:07,782][INFO ][sc-cluster-5005-270][ScaleCubeTopologyService] Topology snapshot [nodes=[int_taewfiies_5007, int_taewfiies_5005, int_taewfiies_5006, int_taewfiies_5003]] [2024-01-30T00:06:07,907][INFO ][sc-cluster-5007-272][MembershipProtocol] [default:int_taewfiies_5007:1a48dc796fed49a5@10.233.90.83:5007] Member left gracefully: default:int_taewfiies_5003:6939bfc9a1b24909@10.233.90.83:5003 [2024-01-30T00:06:07,907][INFO ][sc-cluster-5007-272][MembershipProtocol] [default:int_taewfiies_5007:1a48dc796fed49a5@10.233.90.83:5007][publishEvent] MembershipEvent[type=REMOVED, member=default:int_taewfiies_5003:6939bfc9a1b24909@10.233.90.83:5003, oldMetadata=7e16449-5, newMetadata=null, timestamp=2024-01-30T00:06:07.907Z] [2024-01-30T00:06:07,907][INFO ][sc-cluster-5007-272][ScaleCubeTopologyService] Node left (noop as it has already reappeared) [member=ClusterNodeImpl [id=e939bfc9-a1b2-4909-b37c-148ed686c3ea, name=int_taewfiies_5003, address=10.233.90.83:5003, nodeMetadata=null], eventType=REMOVED] [2024-01-30T00:06:07,907][INFO ][sc-cluster-5007-272][ScaleCubeTopologyService] Topology snapshot [nodes=[int_taewfiies_5007, int_taewfiies_5005, int_taewfiies_5006, int_taewfiies_5003, int_taewfiies_5004]] [2024-01-30T00:06:08,021][INFO ][sc-cluster-5006-271][MembershipProtocol] [default:int_taewfiies_5006:59b4088774334760@10.233.90.83:5006] Member left gracefully: default:int_taewfiies_5004:2b53456118f24b6f@10.233.90.83:5004 [2024-01-30T00:06:08,021][INFO ][sc-cluster-5006-271][MembershipProtocol] [default:int_taewfiies_5006:59b4088774334760@10.233.90.83:5006][publishEvent] MembershipEvent[type=REMOVED, member=default:int_taewfiies_5004:2b53456118f24b6f@10.233.90.83:5004, oldMetadata=7e16449-5, newMetadata=null, timestamp=2024-01-30T00:06:08.021Z] [2024-01-30T00:06:08,021][INFO ][sc-cluster-5006-271][ScaleCubeTopologyService] Node left (noop as it has already reappeared) [member=ClusterNodeImpl [id=2b534561-18f2-4b6f-98da-a0d11a18c0bb, name=int_taewfiies_5004, address=10.233.90.83:5004, nodeMetadata=null], eventType=REMOVED] [2024-01-30T00:06:08,022][INFO ][sc-cluster-5006-271][ScaleCubeTopologyService] Topology snapshot [nodes=[int_taewfiies_5007, int_taewfiies_5005, int_taewfiies_5006, int_taewfiies_5003, int_taewfiies_5004]] [2024-01-30T00:06:08,022][INFO ][sc-cluster-5005-270][MembershipProtocol] [default:int_taewfiies_5005:3a0862258ff5467a@10.233.90.83:5005] Member left gracefully: default:int_taewfiies_5004:2b53456118f24b6f@10.233.90.83:5004 [2024-01-30T00:06:08,023][INFO ][sc-cluster-5005-270][MembershipProtocol] [default:int_taewfiies_5005:3a0862258ff5467a@10.233.90.83:5005][publishEvent] MembershipEvent[type=REMOVED, member=default:int_taewfiies_5004:2b53456118f24b6f@10.233.90.83:5004, oldMetadata=7e16449-5, newMetadata=null, timestamp=2024-01-30T00:06:08.023Z] [2024-01-30T00:06:08,023][INFO ][sc-cluster-5005-270][ScaleCubeTopologyService] Node left [member=ClusterNodeImpl [id=2b534561-18f2-4b6f-98da-a0d11a18c0bb, name=int_taewfiies_5004, address=10.233.90.83:5004, nodeMetadata=null], eventType=REMOVED] [2024-01-30T00:06:08,023][INFO ][sc-cluster-5005-270][ScaleCubeTopologyService] Topology snapshot [nodes=[int_taewfiies_5007, int_taewfiies_5005, int_taewfiies_5006, int_taewfiies_5003]] [2024-01-30T00:06:08,402][INFO ][sc-cluster-5006-271][MembershipProtocol] [default:int_taewfiies_5006:59b4088774334760@10.233.90.83:5006] Member left gracefully: default:int_taewfiies_5003:6939bfc9a1b24909@10.233.90.83:5003 [2024-01-30T00:06:08,403][INFO ][sc-cluster-5006-271][MembershipProtocol] [default:int_taewfiies_5006:59b4088774334760@10.233.90.83:5006][publishEvent] MembershipEvent[type=REMOVED, member=default:int_taewfiies_5003:6939bfc9a1b24909@10.233.90.83:5003, oldMetadata=7e16449-5, newMetadata=null, timestamp=2024-01-30T00:06:08.403Z] [2024-01-30T00:06:08,403][INFO ][sc-cluster-5006-271][ScaleCubeTopologyService] Node left (noop as it has already reappeared) [member=ClusterNodeImpl [id=e939bfc9-a1b2-4909-b37c-148ed686c3ea, name=int_taewfiies_5003, address=10.233.90.83:5003, nodeMetadata=null], eventType=REMOVED] [2024-01-30T00:06:08,403][INFO ][sc-cluster-5006-271][ScaleCubeTopologyService] Topology snapshot [nodes=[int_taewfiies_5007, int_taewfiies_5005, int_taewfiies_5006, int_taewfiies_5003, int_taewfiies_5004]] [2024-01-30T00:06:10,167][INFO ][sc-cluster-5004-274][Cluster] [default:int_taewfiies_5004:7d20e3de9a4145a6@10.233.90.83:5004][doStart] Started [2024-01-30T00:06:10,167][INFO ][Test worker][ScaleCubeTopologyService] Node joined [node=ClusterNodeImpl [id=7d20e3de-9a41-45a6-818e-ac67c33fbb55, name=int_taewfiies_5004, address=10.233.90.83:5004, nodeMetadata=null]] [2024-01-30T00:06:10,167][INFO ][Test worker][ScaleCubeTopologyService] Topology snapshot [nodes=[int_taewfiies_5007, int_taewfiies_5006, int_taewfiies_5003, int_taewfiies_5004]] [2024-01-30T00:06:10,199][INFO ][Test worker][FSMCallerImpl] Starts FSMCaller successfully. [2024-01-30T00:06:10,200][INFO ][Test worker][NodeImpl] Node init, term=1, lastLogId=LogId [index=0, term=0], conf=int_taewfiies_5003,int_taewfiies_5004,int_taewfiies_5005,int_taewfiies_5006,int_taewfiies_5007, oldConf=. [2024-01-30T00:06:10,200][INFO ][Test worker][RaftGroupService] Start the RaftGroupService successfully [2024-01-30T00:06:10,200][INFO ][Test worker][TestCluster] Start ensureSame, leader=JRaftNode [nodeId=] [2024-01-30T00:06:11,966][INFO ][%int_taewfiies_5004%JRaft-ElectionTimer-0][NodeImpl] Node term 1 start preVote. [2024-01-30T00:06:11,972][INFO ][%int_taewfiies_5006%JRaft-Request-Processor-2][NodeImpl] Node ignore PreVoteRequest from int_taewfiies_5004, term=2, currTerm=2, because the leader int_taewfiies_5005's lease is still valid. [2024-01-30T00:06:11,973][INFO ][%int_taewfiies_5007%JRaft-Request-Processor-2][NodeImpl] Node ignore PreVoteRequest from int_taewfiies_5004, term=2, currTerm=2, because the leader int_taewfiies_5005's lease is still valid. [2024-01-30T00:06:11,973][INFO ][%int_taewfiies_5003%JRaft-Request-Processor-3][NodeImpl] Node ignore PreVoteRequest from int_taewfiies_5004, term=2, currTerm=2, because the leader int_taewfiies_5005's lease is still valid. [2024-01-30T00:06:11,974][INFO ][%int_taewfiies_5004%JRaft-Response-Processor-0][LocalRaftMetaStorage] Save raft meta, path=build/work/ItNodeTest/testAppendEntriesWhenFollowerIsInErrorState_30809148389477213/int_taewfiies_5004/meta, term=2, votedFor=, cost time=1 ms [00:06:12] : [org.apache.ignite.raft.jraft.core.ItNodeTest.testAppendEntriesWhenFollowerIsInErrorState()] [Test Error Output] [2024-01-30T00:06:07,351][ERROR][%int_taewfiies_5005%JRaft-StepDownTimer-0][AbstractClientService] Fail to connect int_taewfiies_5004, exception: java.net.ConnectException. [2024-01-30T00:06:07,351][ERROR][%int_taewfiies_5005%JRaft-StepDownTimer-0][ReplicatorGroupImpl] Fail to check replicator connection to peer=int_taewfiies_5004, replicatorType=Follower. [2024-01-30T00:06:07,852][ERROR][%int_taewfiies_5005%JRaft-StepDownTimer-0][AbstractClientService] Fail to connect int_taewfiies_5004, exception: java.net.ConnectException. [2024-01-30T00:06:07,852][ERROR][%int_taewfiies_5005%JRaft-StepDownTimer-0][ReplicatorGroupImpl] Fail to check replicator connection to peer=int_taewfiies_5004, replicatorType=Follower. [2024-01-30T00:06:08,163][WARN ][sc-cluster-5005-270][MetadataStore] [default:int_taewfiies_5005:3a0862258ff5467a@10.233.90.83:5005][9c7a1b89-e3b3-4a51-8072-992a94870b91] Timeout getting GetMetadataResp from 10.233.90.83:5004 within 1000 ms, cause: java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 1000ms in 'source(MonoDefer)' (and no fallback has been configured) [2024-01-30T00:06:08,164][WARN ][sc-cluster-5005-270][MembershipProtocol] [default:int_taewfiies_5005:3a0862258ff5467a@10.233.90.83:5005][updateMembership][SYNC] Skipping to add/update member: {m: default:int_taewfiies_5004:7d20e3de9a4145a6@10.233.90.83:5004, s: ALIVE, inc: 0}, due to failed fetchMetadata call (cause: java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 1000ms in 'source(MonoDefer)' (and no fallback has been configured)) [2024-01-30T00:06:08,166][WARN ][sc-cluster-5004-274][MetadataStore] [default:int_taewfiies_5004:7d20e3de9a4145a6@10.233.90.83:5004][63da35c5-bf0e-4e6c-ae20-df88569d43e6] Timeout getting GetMetadataResp from 10.233.90.83:5005 within 1000 ms, cause: java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 1000ms in 'source(MonoDefer)' (and no fallback has been configured) [2024-01-30T00:06:08,166][WARN ][sc-cluster-5004-274][MembershipProtocol] [default:int_taewfiies_5004:7d20e3de9a4145a6@10.233.90.83:5004][updateMembership][INITIAL_SYNC] Skipping to add/update member: {m: default:int_taewfiies_5005:3a0862258ff5467a@10.233.90.83:5005, s: ALIVE, inc: 0}, due to failed fetchMetadata call (cause: java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 1000ms in 'source(MonoDefer)' (and no fallback has been configured)) [2024-01-30T00:06:08,166][WARN ][sc-cluster-5004-274][MetadataStore] [default:int_taewfiies_5004:7d20e3de9a4145a6@10.233.90.83:5004][327b37d9-92ca-48c6-9d74-a8b3afa1fcb9] Timeout getting GetMetadataResp from 10.233.90.83:5005 within 1000 ms, cause: java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 1000ms in 'source(MonoDefer)' (and no fallback has been configured) [2024-01-30T00:06:08,166][WARN ][sc-cluster-5004-274][MembershipProtocol] [default:int_taewfiies_5004:7d20e3de9a4145a6@10.233.90.83:5004][updateMembership][INITIAL_SYNC] Skipping to add/update member: {m: default:int_taewfiies_5005:3a0862258ff5467a@10.233.90.83:5005, s: ALIVE, inc: 0}, due to failed fetchMetadata call (cause: java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 1000ms in 'source(MonoDefer)' (and no fallback has been configured)) [2024-01-30T00:06:08,166][WARN ][sc-cluster-5004-274][MetadataStore] [default:int_taewfiies_5004:7d20e3de9a4145a6@10.233.90.83:5004][6d748476-a58c-4cc8-a34d-729680ce6905] Timeout getting GetMetadataResp from 10.233.90.83:5005 within 1000 ms, cause: java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 1000ms in 'source(MonoDefer)' (and no fallback has been configured) [2024-01-30T00:06:08,166][WARN ][sc-cluster-5004-274][MembershipProtocol] [default:int_taewfiies_5004:7d20e3de9a4145a6@10.233.90.83:5004][updateMembership][INITIAL_SYNC] Skipping to add/update member: {m: default:int_taewfiies_5005:3a0862258ff5467a@10.233.90.83:5005, s: ALIVE, inc: 0}, due to failed fetchMetadata call (cause: java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 1000ms in 'source(MonoDefer)' (and no fallback has been configured)) [2024-01-30T00:06:08,167][WARN ][sc-cluster-5005-270][MetadataStore] [default:int_taewfiies_5005:3a0862258ff5467a@10.233.90.83:5005][2c0fc96a-8029-4947-9861-7788e08b8db8] Timeout getting GetMetadataResp from 10.233.90.83:5004 within 1000 ms, cause: java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 1000ms in 'source(MonoDefer)' (and no fallback has been configured) [2024-01-30T00:06:08,167][WARN ][sc-cluster-5005-270][MembershipProtocol] [default:int_taewfiies_5005:3a0862258ff5467a@10.233.90.83:5005][updateMembership][MEMBERSHIP_GOSSIP] Skipping to add/update member: {m: default:int_taewfiies_5004:7d20e3de9a4145a6@10.233.90.83:5004, s: ALIVE, inc: 0}, due to failed fetchMetadata call (cause: java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 1000ms in 'source(MonoDefer)' (and no fallback has been configured)) [2024-01-30T00:06:08,168][WARN ][sc-cluster-5005-270][MetadataStore] [default:int_taewfiies_5005:3a0862258ff5467a@10.233.90.83:5005][20d5397f-b8a9-424b-a3fb-6ebfaa866087] Timeout getting GetMetadataResp from 10.233.90.83:5004 within 1000 ms, cause: java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 1000ms in 'source(MonoDefer)' (and no fallback has been configured) [2024-01-30T00:06:08,168][WARN ][sc-cluster-5005-270][MembershipProtocol] [default:int_taewfiies_5005:3a0862258ff5467a@10.233.90.83:5005][updateMembership][MEMBERSHIP_GOSSIP] Skipping to add/update member: {m: default:int_taewfiies_5004:7d20e3de9a4145a6@10.233.90.83:5004, s: ALIVE, inc: 0}, due to failed fetchMetadata call (cause: java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 1000ms in 'source(MonoDefer)' (and no fallback has been configured)) [2024-01-30T00:06:08,177][WARN ][sc-cluster-5005-270][MetadataStore] [default:int_taewfiies_5005:3a0862258ff5467a@10.233.90.83:5005][3bb000d9-24ab-48ba-8f67-a15dc758e423] Timeout getting GetMetadataResp from 10.233.90.83:5004 within 1000 ms, cause: java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 1000ms in 'source(MonoDefer)' (and no fallback has been configured) [2024-01-30T00:06:08,177][WARN ][sc-cluster-5005-270][MembershipProtocol] [default:int_taewfiies_5005:3a0862258ff5467a@10.233.90.83:5005][updateMembership][MEMBERSHIP_GOSSIP] Skipping to add/update member: {m: default:int_taewfiies_5004:7d20e3de9a4145a6@10.233.90.83:5004, s: ALIVE, inc: 0}, due to failed fetchMetadata call (cause: java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 1000ms in 'source(MonoDefer)' (and no fallback has been configured)) [2024-01-30T00:06:08,290][WARN ][sc-cluster-5005-270][MetadataStore] [default:int_taewfiies_5005:3a0862258ff5467a@10.233.90.83:5005][98730935-5d64-45ad-82aa-1ce59fbbfe6a] Timeout getting GetMetadataResp from 10.233.90.83:5004 within 1000 ms, cause: java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 1000ms in 'source(MonoDefer)' (and no fallback has been configured) [2024-01-30T00:06:08,290][WARN ][sc-cluster-5005-270][MembershipProtocol] [default:int_taewfiies_5005:3a0862258ff5467a@10.233.90.83:5005][updateMembership][SYNC] Skipping to add/update member: {m: default:int_taewfiies_5004:7d20e3de9a4145a6@10.233.90.83:5004, s: ALIVE, inc: 0}, due to failed fetchMetadata call (cause: java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 1000ms in 'source(MonoDefer)' (and no fallback has been configured)) [2024-01-30T00:06:08,352][ERROR][%int_taewfiies_5005%JRaft-StepDownTimer-1][AbstractClientService] Fail to connect int_taewfiies_5004, exception: java.net.ConnectException. [2024-01-30T00:06:08,352][ERROR][%int_taewfiies_5005%JRaft-StepDownTimer-1][ReplicatorGroupImpl] Fail to check replicator connection to peer=int_taewfiies_5004, replicatorType=Follower. [2024-01-30T00:06:08,853][ERROR][%int_taewfiies_5005%JRaft-StepDownTimer-0][AbstractClientService] Fail to connect int_taewfiies_5004, exception: java.net.ConnectException. [2024-01-30T00:06:08,853][ERROR][%int_taewfiies_5005%JRaft-StepDownTimer-0][ReplicatorGroupImpl] Fail to check replicator connection to peer=int_taewfiies_5004, replicatorType=Follower. [2024-01-30T00:06:09,098][WARN ][sc-cluster-5004-274][MetadataStore] [default:int_taewfiies_5004:7d20e3de9a4145a6@10.233.90.83:5004][fb3538dd-6ba9-4a37-b1c0-3bf4ea38e847] Timeout getting GetMetadataResp from 10.233.90.83:5005 within 1000 ms, cause: java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 1000ms in 'source(MonoDefer)' (and no fallback has been configured) [2024-01-30T00:06:09,098][WARN ][sc-cluster-5004-274][MembershipProtocol] [default:int_taewfiies_5004:7d20e3de9a4145a6@10.233.90.83:5004][updateMembership][SYNC] Skipping to add/update member: {m: default:int_taewfiies_5005:3a0862258ff5467a@10.233.90.83:5005, s: ALIVE, inc: 0}, due to failed fetchMetadata call (cause: java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 1000ms in 'source(MonoDefer)' (and no fallback has been configured)) [2024-01-30T00:06:09,290][WARN ][sc-cluster-5005-270][MetadataStore] [default:int_taewfiies_5005:3a0862258ff5467a@10.233.90.83:5005][c506aa26-095d-4aa4-9c8d-430a4c1b75d2] Timeout getting GetMetadataResp from 10.233.90.83:5004 within 1000 ms, cause: java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 1000ms in 'source(MonoDefer)' (and no fallback has been configured) [2024-01-30T00:06:09,290][WARN ][sc-cluster-5005-270][MembershipProtocol] [default:int_taewfiies_5005:3a0862258ff5467a@10.233.90.83:5005][updateMembership][SYNC] Skipping to add/update member: {m: default:int_taewfiies_5004:7d20e3de9a4145a6@10.233.90.83:5004, s: ALIVE, inc: 0}, due to failed fetchMetadata call (cause: java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 1000ms in 'source(MonoDefer)' (and no fallback has been configured)) [2024-01-30T00:06:09,353][ERROR][%int_taewfiies_5005%JRaft-StepDownTimer-2][AbstractClientService] Fail to connect int_taewfiies_5004, exception: java.net.ConnectException. [2024-01-30T00:06:09,354][ERROR][%int_taewfiies_5005%JRaft-StepDownTimer-2][ReplicatorGroupImpl] Fail to check replicator connection to peer=int_taewfiies_5004, replicatorType=Follower. [2024-01-30T00:06:09,854][ERROR][%int_taewfiies_5005%JRaft-StepDownTimer-1][AbstractClientService] Fail to connect int_taewfiies_5004, exception: java.net.ConnectException. [2024-01-30T00:06:09,854][ERROR][%int_taewfiies_5005%JRaft-StepDownTimer-1][ReplicatorGroupImpl] Fail to check replicator connection to peer=int_taewfiies_5004, replicatorType=Follower. [2024-01-30T00:06:10,098][WARN ][sc-cluster-5005-270][MetadataStore] [default:int_taewfiies_5005:3a0862258ff5467a@10.233.90.83:5005][621d90df-59f5-4aac-a902-9045f1ad0ab0] Timeout getting GetMetadataResp from 10.233.90.83:5004 within 1000 ms, cause: java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 1000ms in 'source(MonoDefer)' (and no fallback has been configured) [2024-01-30T00:06:10,098][WARN ][sc-cluster-5005-270][MembershipProtocol] [default:int_taewfiies_5005:3a0862258ff5467a@10.233.90.83:5005][updateMembership][SYNC] Skipping to add/update member: {m: default:int_taewfiies_5004:7d20e3de9a4145a6@10.233.90.83:5004, s: ALIVE, inc: 0}, due to failed fetchMetadata call (cause: java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 1000ms in 'source(MonoDefer)' (and no fallback has been configured)) [2024-01-30T00:06:10,166][WARN ][sc-cluster-5004-274][MembershipProtocol] [default:int_taewfiies_5004:7d20e3de9a4145a6@10.233.90.83:5004] Exception on initial SyncAck, cause: java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 3000ms in 'limitRequest' (and no fallback has been configured) [2024-01-30T00:06:10,200][WARN ][Test worker][LocalSnapshotStorage] No data for snapshot reader build/work/ItNodeTest/testAppendEntriesWhenFollowerIsInErrorState_30809148389477213/int_taewfiies_5004/snapshot. [2024-01-30T00:06:10,290][WARN ][sc-cluster-5005-270][MetadataStore] [default:int_taewfiies_5005:3a0862258ff5467a@10.233.90.83:5005][eccc13ac-18c1-4601-8d17-9c169d807eee] Timeout getting GetMetadataResp from 10.233.90.83:5004 within 1000 ms, cause: java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 1000ms in 'source(MonoDefer)' (and no fallback has been configured) [2024-01-30T00:06:10,290][WARN ][sc-cluster-5005-270][MembershipProtocol] [default:int_taewfiies_5005:3a0862258ff5467a@10.233.90.83:5005][updateMembership][SYNC] Skipping to add/update member: {m: default:int_taewfiies_5004:7d20e3de9a4145a6@10.233.90.83:5004, s: ALIVE, inc: 0}, due to failed fetchMetadata call (cause: java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 1000ms in 'source(MonoDefer)' (and no fallback has been configured)) [2024-01-30T00:06:10,338][WARN ][sc-cluster-5005-270][MetadataStore] [default:int_taewfiies_5005:3a0862258ff5467a@10.233.90.83:5005][b2c67ff6-26d6-4e20-b3e6-5a136ad40986] Timeout getting GetMetadataResp from 10.233.90.83:5004 within 1000 ms, cause: java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 1000ms in 'source(MonoDefer)' (and no fallback has been configured) [2024-01-30T00:06:10,338][WARN ][sc-cluster-5005-270][MembershipProtocol] [default:int_taewfiies_5005:3a0862258ff5467a@10.233.90.83:5005][updateMembership][SYNC] Skipping to add/update member: {m: default:int_taewfiies_5004:7d20e3de9a4145a6@10.233.90.83:5004, s: ALIVE, inc: 0}, due to failed fetchMetadata call (cause: java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 1000ms in 'source(MonoDefer)' (and no fallback has been configured)) [2024-01-30T00:06:10,354][ERROR][%int_taewfiies_5005%JRaft-StepDownTimer-3][AbstractClientService] Fail to connect int_taewfiies_5004, exception: java.net.ConnectException. [2024-01-30T00:06:10,355][ERROR][%int_taewfiies_5005%JRaft-StepDownTimer-3][ReplicatorGroupImpl] Fail to check replicator connection to peer=int_taewfiies_5004, replicatorType=Follower. [2024-01-30T00:06:10,855][ERROR][%int_taewfiies_5005%JRaft-StepDownTimer-0][AbstractClientService] Fail to connect int_taewfiies_5004, exception: java.net.ConnectException. [2024-01-30T00:06:10,855][ERROR][%int_taewfiies_5005%JRaft-StepDownTimer-0][ReplicatorGroupImpl] Fail to check replicator connection to peer=int_taewfiies_5004, replicatorType=Follower. [2024-01-30T00:06:11,338][WARN ][sc-cluster-5005-270][MetadataStore] [default:int_taewfiies_5005:3a0862258ff5467a@10.233.90.83:5005][a1795178-60d3-45e8-952b-db264647504d] Timeout getting GetMetadataResp from 10.233.90.83:5004 within 1000 ms, cause: java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 1000ms in 'source(MonoDefer)' (and no fallback has been configured) [2024-01-30T00:06:11,338][WARN ][sc-cluster-5005-270][MembershipProtocol] [default:int_taewfiies_5005:3a0862258ff5467a@10.233.90.83:5005][updateMembership][SYNC] Skipping to add/update member: {m: default:int_taewfiies_5004:7d20e3de9a4145a6@10.233.90.83:5004, s: ALIVE, inc: 0}, due to failed fetchMetadata call (cause: java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 1000ms in 'source(MonoDefer)' (and no fallback has been configured)) [2024-01-30T00:06:11,356][ERROR][%int_taewfiies_5005%JRaft-StepDownTimer-0][AbstractClientService] Fail to connect int_taewfiies_5004, exception: java.net.ConnectException. [2024-01-30T00:06:11,356][ERROR][%int_taewfiies_5005%JRaft-StepDownTimer-0][ReplicatorGroupImpl] Fail to check replicator connection to peer=int_taewfiies_5004, replicatorType=Follower. [2024-01-30T00:06:11,856][ERROR][%int_taewfiies_5005%JRaft-StepDownTimer-0][AbstractClientService] Fail to connect int_taewfiies_5004, exception: java.net.ConnectException. [2024-01-30T00:06:11,856][ERROR][%int_taewfiies_5005%JRaft-StepDownTimer-0][ReplicatorGroupImpl] Fail to check replicator connection to peer=int_taewfiies_5004, replicatorType=Follower. [2024-01-30T00:06:11,973][WARN ][%int_taewfiies_5004%JRaft-Response-Processor-0][NodeImpl] Node received invalid PreVoteResponse from int_taewfiies_5006, term 2, expect=1. [2024-01-30T00:06:11,975][WARN ][%int_taewfiies_5004%JRaft-Response-Processor-1][NodeImpl] Node received invalid PreVoteResponse from int_taewfiies_5007, term=1, currTerm=2. [2024-01-30T00:06:11,975][WARN ][%int_taewfiies_5004%JRaft-Response-Processor-2][NodeImpl] Node received invalid PreVoteResponse from int_taewfiies_5003, term=1, currTerm=2. [2024-01-30T00:06:12,098][WARN ][sc-cluster-5005-270][MetadataStore] [default:int_taewfiies_5005:3a0862258ff5467a@10.233.90.83:5005][b097de88-f716-4d5c-9033-9779254320e7] Timeout getting GetMetadataResp from 10.233.90.83:5004 within 1000 ms, cause: java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 1000ms in 'source(MonoDefer)' (and no fallback has been configured) [2024-01-30T00:06:12,098][WARN ][sc-cluster-5005-270][MembershipProtocol] [default:int_taewfiies_5005:3a0862258ff5467a@10.233.90.83:5005][updateMembership][SYNC] Skipping to add/update member: {m: default:int_taewfiies_5004:7d20e3de9a4145a6@10.233.90.83:5004, s: ALIVE, inc: 0}, due to failed fetchMetadata call (cause: java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 1000ms in 'source(MonoDefer)' (and no fallback has been configured)) [2024-01-30T00:06:12,168][WARN ][sc-cluster-5004-274][MetadataStore] [default:int_taewfiies_5004:7d20e3de9a4145a6@10.233.90.83:5004][f4aa1e8c-0a9a-41bf-a322-c12e585e7d0c] Timeout getting GetMetadataResp from 10.233.90.83:5005 within 1000 ms, cause: java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 1000ms in 'source(MonoDefer)' (and no fallback has been configured) [2024-01-30T00:06:12,168][WARN ][sc-cluster-5004-274][MembershipProtocol] [default:int_taewfiies_5004:7d20e3de9a4145a6@10.233.90.83:5004][updateMembership][SYNC] Skipping to add/update member: {m: default:int_taewfiies_5005:3a0862258ff5467a@10.233.90.83:5005, s: ALIVE, inc: 0}, due to failed fetchMetadata call (cause: java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 1000ms in 'source(MonoDefer)' (and no fallback has been configured)) [2024-01-30T00:06:12,290][WARN ][sc-cluster-5005-270][MetadataStore] [default:int_taewfiies_5005:3a0862258ff5467a@10.233.90.83:5005][5897f467-bb29-4998-ad91-aa8cdb484c8b] Timeout getting GetMetadataResp from 10.233.90.83:5004 within 1000 ms, cause: java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 1000ms in 'source(MonoDefer)' (and no fallback has been configured) [2024-01-30T00:06:12,290][WARN ][sc-cluster-5005-270][MembershipProtocol] [default:int_taewfiies_5005:3a0862258ff5467a@10.233.90.83:5005][updateMembership][SYNC] Skipping to add/update member: {m: default:int_taewfiies_5004:7d20e3de9a4145a6@10.233.90.83:5004, s: ALIVE, inc: 0}, due to failed fetchMetadata call (cause: java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 1000ms in 'source(MonoDefer)' (and no fallback has been configured)) [2024-01-30T00:06:12,338][WARN ][sc-cluster-5004-274][MetadataStore] [default:int_taewfiies_5004:7d20e3de9a4145a6@10.233.90.83:5004][6a7b277c-9ecb-4f08-abe0-93cbb0cb1fa5] Timeout getting GetMetadataResp from 10.233.90.83:5005 within 1000 ms, cause: java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 1000ms in 'source(MonoDefer)' (and no fallback has been configured) [2024-01-30T00:06:12,338][WARN ][sc-cluster-5004-274][MembershipProtocol] [default:int_taewfiies_5004:7d20e3de9a4145a6@10.233.90.83:5004][updateMembership][SYNC] Skipping to add/update member: {m: default:int_taewfiies_5005:3a0862258ff5467a@10.233.90.83:5005, s: ALIVE, inc: 0}, due to failed fetchMetadata call (cause: java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 1000ms in 'source(MonoDefer)' (and no fallback has been configured)) [2024-01-30T00:06:12,357][ERROR][%int_taewfiies_5005%JRaft-StepDownTimer-0][AbstractClientService] Fail to connect int_taewfiies_5004, exception: java.net.ConnectException. [00:06:18] : [org.apache.ignite.raft.jraft.core.ItNodeTest.testAppendEntriesWhenFollowerIsInErrorState()] [Test Output] [2024-01-30T00:06:13,438][INFO ][%int_taewfiies_5004%JRaft-ElectionTimer-0][NodeImpl] Unsuccessful election round number 2 [2024-01-30T00:06:13,438][INFO ][%int_taewfiies_5004%JRaft-ElectionTimer-0][NodeImpl] Node term 2 start preVote. [2024-01-30T00:06:13,439][INFO ][%int_taewfiies_5006%JRaft-Request-Processor-3][NodeImpl] Node ignore PreVoteRequest from int_taewfiies_5004, term=3, currTerm=2, because the leader int_taewfiies_5005's lease is still valid. [2024-01-30T00:06:13,439][INFO ][%int_taewfiies_5007%JRaft-Request-Processor-3][NodeImpl] Node ignore PreVoteRequest from int_taewfiies_5004, term=3, currTerm=2, because the leader int_taewfiies_5005's lease is still valid. [2024-01-30T00:06:13,439][INFO ][%int_taewfiies_5003%JRaft-Request-Processor-0][NodeImpl] Node ignore PreVoteRequest from int_taewfiies_5004, term=3, currTerm=2, because the leader int_taewfiies_5005's lease is still valid. [2024-01-30T00:06:13,439][INFO ][%int_taewfiies_5004%JRaft-Response-Processor-0][NodeImpl] Node received PreVoteResponse from int_taewfiies_5006, term=2, granted=false. [2024-01-30T00:06:13,439][INFO ][%int_taewfiies_5004%JRaft-Response-Processor-0][NodeImpl] Node received PreVoteResponse from int_taewfiies_5003, term=2, granted=false. [2024-01-30T00:06:13,439][INFO ][%int_taewfiies_5004%JRaft-Response-Processor-1][NodeImpl] Node received PreVoteResponse from int_taewfiies_5007, term=2, granted=false. [2024-01-30T00:06:14,832][INFO ][%int_taewfiies_5004%JRaft-ElectionTimer-2][NodeImpl] Unsuccessful election round number 3 [2024-01-30T00:06:14,832][INFO ][%int_taewfiies_5004%JRaft-ElectionTimer-2][NodeImpl] Node term 2 start preVote. [2024-01-30T00:06:14,833][INFO ][%int_taewfiies_5007%JRaft-Request-Processor-0][NodeImpl] Node ignore PreVoteRequest from int_taewfiies_5004, term=3, currTerm=2, because the leader int_taewfiies_5005's lease is still valid. [2024-01-30T00:06:14,833][INFO ][%int_taewfiies_5006%JRaft-Request-Processor-0][NodeImpl] Node ignore PreVoteRequest from int_taewfiies_5004, term=3, currTerm=2, because the leader int_taewfiies_5005's lease is still valid. [2024-01-30T00:06:14,833][INFO ][%int_taewfiies_5003%JRaft-Request-Processor-1][NodeImpl] Node ignore PreVoteRequest from int_taewfiies_5004, term=3, currTerm=2, because the leader int_taewfiies_5005's lease is still valid. [2024-01-30T00:06:14,834][INFO ][%int_taewfiies_5004%JRaft-Response-Processor-2][NodeImpl] Node received PreVoteResponse from int_taewfiies_5007, term=2, granted=false. [2024-01-30T00:06:14,834][INFO ][%int_taewfiies_5004%JRaft-Response-Processor-0][NodeImpl] Node received PreVoteResponse from int_taewfiies_5006, term=2, granted=false. [2024-01-30T00:06:14,834][INFO ][%int_taewfiies_5004%JRaft-Response-Processor-1][NodeImpl] Node received PreVoteResponse from int_taewfiies_5003, term=2, granted=false. [2024-01-30T00:06:15,833][INFO ][%int_taewfiies_5004%JRaft-ElectionTimer-0][NodeImpl] Unsuccessful election round number 4 [2024-01-30T00:06:15,833][INFO ][%int_taewfiies_5004%JRaft-ElectionTimer-0][NodeImpl] Node reset electionTimeout, currTimer 2 state STATE_FOLLOWER new electionTimeout 2000. [2024-01-30T00:06:15,834][INFO ][%int_taewfiies_5004%JRaft-ElectionTimer-0][NodeImpl] Election timeout was adjusted according to ExponentialBackoffTimeoutStrategy [maxTimeout=11000, roundsWithoutAdjusting=3] [2024-01-30T00:06:15,834][INFO ][%int_taewfiies_5004%JRaft-ElectionTimer-0][NodeImpl] Node term 2 start preVote. [2024-01-30T00:06:15,834][INFO ][%int_taewfiies_5006%JRaft-Request-Processor-1][NodeImpl] Node ignore PreVoteRequest from int_taewfiies_5004, term=3, currTerm=2, because the leader int_taewfiies_5005's lease is still valid. [2024-01-30T00:06:15,835][INFO ][%int_taewfiies_5007%JRaft-Request-Processor-1][NodeImpl] Node ignore PreVoteRequest from int_taewfiies_5004, term=3, currTerm=2, because the leader int_taewfiies_5005's lease is still valid. [2024-01-30T00:06:15,835][INFO ][%int_taewfiies_5003%JRaft-Request-Processor-2][NodeImpl] Node ignore PreVoteRequest from int_taewfiies_5004, term=3, currTerm=2, because the leader int_taewfiies_5005's lease is still valid. [2024-01-30T00:06:15,835][INFO ][%int_taewfiies_5004%JRaft-Response-Processor-2][NodeImpl] Node received PreVoteResponse from int_taewfiies_5006, term=2, granted=false. [2024-01-30T00:06:15,835][INFO ][%int_taewfiies_5004%JRaft-Response-Processor-0][NodeImpl] Node received PreVoteResponse from int_taewfiies_5007, term=2, granted=false. [2024-01-30T00:06:15,835][INFO ][%int_taewfiies_5004%JRaft-Response-Processor-1][NodeImpl] Node received PreVoteResponse from int_taewfiies_5003, term=2, granted=false. [2024-01-30T00:06:17,858][INFO ][%int_taewfiies_5004%JRaft-ElectionTimer-1][NodeImpl] Unsuccessful election round number 5 [2024-01-30T00:06:17,859][INFO ][%int_taewfiies_5004%JRaft-ElectionTimer-1][NodeImpl] Node reset electionTimeout, currTimer 2 state STATE_FOLLOWER new electionTimeout 4000. [2024-01-30T00:06:17,859][INFO ][%int_taewfiies_5004%JRaft-ElectionTimer-1][NodeImpl] Election timeout was adjusted according to ExponentialBackoffTimeoutStrategy [maxTimeout=11000, roundsWithoutAdjusting=3] [2024-01-30T00:06:17,859][INFO ][%int_taewfiies_5004%JRaft-ElectionTimer-1][NodeImpl] Node term 2 start preVote. [2024-01-30T00:06:17,860][INFO ][%int_taewfiies_5006%JRaft-Request-Processor-2][NodeImpl] Node ignore PreVoteRequest from int_taewfiies_5004, term=3, currTerm=2, because the leader int_taewfiies_5005's lease is still valid. [2024-01-30T00:06:17,860][INFO ][%int_taewfiies_5007%JRaft-Request-Processor-2][NodeImpl] Node ignore PreVoteRequest from int_taewfiies_5004, term=3, currTerm=2, because the leader int_taewfiies_5005's lease is still valid. [2024-01-30T00:06:17,860][INFO ][%int_taewfiies_5003%JRaft-Request-Processor-3][NodeImpl] Node ignore PreVoteRequest from int_taewfiies_5004, term=3, currTerm=2, because the leader int_taewfiies_5005's lease is still valid. [2024-01-30T00:06:17,860][INFO ][%int_taewfiies_5004%JRaft-Response-Processor-2][NodeImpl] Node received PreVoteResponse from int_taewfiies_5006, term=2, granted=false. [2024-01-30T00:06:17,860][INFO ][%int_taewfiies_5004%JRaft-Response-Processor-0][NodeImpl] Node received PreVoteResponse from int_taewfiies_5007, term=2, granted=false. [2024-01-30T00:06:17,861][INFO ][%int_taewfiies_5004%JRaft-Response-Processor-1][NodeImpl] Node received PreVoteResponse from int_taewfiies_5003, term=2, granted=false. [00:06:18] : [org.apache.ignite.raft.jraft.core.ItNodeTest.testAppendEntriesWhenFollowerIsInErrorState()] [Test Error Output] [2024-01-30T00:06:12,357][ERROR][%int_taewfiies_5005%JRaft-StepDownTimer-0][ReplicatorGroupImpl] Fail to check replicator connection to peer=int_taewfiies_5004, replicatorType=Follower. [2024-01-30T00:06:12,857][ERROR][%int_taewfiies_5005%JRaft-StepDownTimer-0][AbstractClientService] Fail to connect int_taewfiies_5004, exception: java.net.ConnectException. [2024-01-30T00:06:12,857][ERROR][%int_taewfiies_5005%JRaft-StepDownTimer-0][ReplicatorGroupImpl] Fail to check replicator connection to peer=int_taewfiies_5004, replicatorType=Follower. [2024-01-30T00:06:13,098][WARN ][sc-cluster-5004-274][MetadataStore] [default:int_taewfiies_5004:7d20e3de9a4145a6@10.233.90.83:5004][4dde6222-ab6d-42d4-8fe7-c2a810a68245] Timeout getting GetMetadataResp from 10.233.90.83:5005 within 1000 ms, cause: java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 1000ms in 'source(MonoDefer)' (and no fallback has been configured) [2024-01-30T00:06:13,098][WARN ][sc-cluster-5004-274][MembershipProtocol] [default:int_taewfiies_5004:7d20e3de9a4145a6@10.233.90.83:5004][updateMembership][SYNC] Skipping to add/update member: {m: default:int_taewfiies_5005:3a0862258ff5467a@10.233.90.83:5005, s: ALIVE, inc: 0}, due to failed fetchMetadata call (cause: java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 1000ms in 'source(MonoDefer)' (and no fallback has been configured)) [2024-01-30T00:06:13,168][WARN ][sc-cluster-5004-274][MetadataStore] [default:int_taewfiies_5004:7d20e3de9a4145a6@10.233.90.83:5004][75d3ca9e-a946-4b35-9dff-46ee2fa52495] Timeout getting GetMetadataResp from 10.233.90.83:5005 within 1000 ms, cause: java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 1000ms in 'source(MonoDefer)' (and no fallback has been configured) [2024-01-30T00:06:13,168][WARN ][sc-cluster-5004-274][MembershipProtocol] [default:int_taewfiies_5004:7d20e3de9a4145a6@10.233.90.83:5004][updateMembership][SYNC] Skipping to add/update member: {m: default:int_taewfiies_5005:3a0862258ff5467a@10.233.90.83:5005, s: ALIVE, inc: 0}, due to failed fetchMetadata call (cause: java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 1000ms in 'source(MonoDefer)' (and no fallback has been configured)) [2024-01-30T00:06:13,290][WARN ][sc-cluster-5005-270][MetadataStore] [default:int_taewfiies_5005:3a0862258ff5467a@10.233.90.83:5005][78520d34-5ec9-4845-846f-81461ae103b7] Timeout getting GetMetadataResp from 10.233.90.83:5004 within 1000 ms, cause: java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 1000ms in 'source(MonoDefer)' (and no fallback has been configured) [2024-01-30T00:06:13,290][WARN ][sc-cluster-5005-270][MembershipProtocol] [default:int_taewfiies_5005:3a0862258ff5467a@10.233.90.83:5005][updateMembership][SYNC] Skipping to add/update member: {m: default:int_taewfiies_5004:7d20e3de9a4145a6@10.233.90.83:5004, s: ALIVE, inc: 0}, due to failed fetchMetadata call (cause: java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 1000ms in 'source(MonoDefer)' (and no fallback has been configured)) [2024-01-30T00:06:13,338][WARN ][sc-cluster-5004-274][MetadataStore] [default:int_taewfiies_5004:7d20e3de9a4145a6@10.233.90.83:5004][b3cfa5fb-650d-4ba8-a546-91c0b98ffe72] Timeout getting GetMetadataResp from 10.233.90.83:5005 within 1000 ms, cause: java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 1000ms in 'source(MonoDefer)' (and no fallback has been configured) [2024-01-30T00:06:13,338][WARN ][sc-cluster-5004-274][MembershipProtocol] [default:int_taewfiies_5004:7d20e3de9a4145a6@10.233.90.83:5004][updateMembership][SYNC] Skipping to add/update member: {m: default:int_taewfiies_5005:3a0862258ff5467a@10.233.90.83:5005, s: ALIVE, inc: 0}, due to failed fetchMetadata call (cause: java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 1000ms in 'source(MonoDefer)' (and no fallback has been configured)) [2024-01-30T00:06:13,358][ERROR][%int_taewfiies_5005%JRaft-StepDownTimer-0][AbstractClientService] Fail to connect int_taewfiies_5004, exception: java.net.ConnectException. [2024-01-30T00:06:13,358][ERROR][%int_taewfiies_5005%JRaft-StepDownTimer-0][ReplicatorGroupImpl] Fail to check replicator connection to peer=int_taewfiies_5004, replicatorType=Follower. [2024-01-30T00:06:13,392][WARN ][sc-cluster-5004-274][MetadataStore] [default:int_taewfiies_5004:7d20e3de9a4145a6@10.233.90.83:5004][7005db1d-f912-4c15-8c3f-6aae470e1da9] Timeout getting GetMetadataResp from 10.233.90.83:5005 within 1000 ms, cause: java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 1000ms in 'source(MonoDefer)' (and no fallback has been configured) [2024-01-30T00:06:13,393][WARN ][sc-cluster-5004-274][MembershipProtocol] [default:int_taewfiies_5004:7d20e3de9a4145a6@10.233.90.83:5004][updateMembership][SYNC] Skipping to add/update member: {m: default:int_taewfiies_5005:3a0862258ff5467a@10.233.90.83:5005, s: ALIVE, inc: 0}, due to failed fetchMetadata call (cause: java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 1000ms in 'source(MonoDefer)' (and no fallback has been configured)) [2024-01-30T00:06:13,858][ERROR][%int_taewfiies_5005%JRaft-StepDownTimer-3][AbstractClientService] Fail to connect int_taewfiies_5004, exception: java.net.ConnectException. [2024-01-30T00:06:13,858][ERROR][%int_taewfiies_5005%JRaft-StepDownTimer-3][ReplicatorGroupImpl] Fail to check replicator connection to peer=int_taewfiies_5004, replicatorType=Follower. [2024-01-30T00:06:14,168][WARN ][sc-cluster-5004-274][MetadataStore] [default:int_taewfiies_5004:7d20e3de9a4145a6@10.233.90.83:5004][14c8c2b0-bce1-4063-899b-153b0620529c] Timeout getting GetMetadataResp from 10.233.90.83:5005 within 1000 ms, cause: java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 1000ms in 'source(MonoDefer)' (and no fallback has been configured) [2024-01-30T00:06:14,169][WARN ][sc-cluster-5004-274][MembershipProtocol] [default:int_taewfiies_5004:7d20e3de9a4145a6@10.233.90.83:5004][updateMembership][SYNC] Skipping to add/update member: {m: default:int_taewfiies_5005:3a0862258ff5467a@10.233.90.83:5005, s: ALIVE, inc: 0}, due to failed fetchMetadata call (cause: java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 1000ms in 'source(MonoDefer)' (and no fallback has been configured)) [2024-01-30T00:06:14,290][WARN ][sc-cluster-5005-270][MetadataStore] [default:int_taewfiies_5005:3a0862258ff5467a@10.233.90.83:5005][683f0b99-d8b6-4271-9eb8-e7503aa4f0a1] Timeout getting GetMetadataResp from 10.233.90.83:5004 within 1000 ms, cause: java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 1000ms in 'source(MonoDefer)' (and no fallback has been configured) [2024-01-30T00:06:14,290][WARN ][sc-cluster-5005-270][MembershipProtocol] [default:int_taewfiies_5005:3a0862258ff5467a@10.233.90.83:5005][updateMembership][SYNC] Skipping to add/update member: {m: default:int_taewfiies_5004:7d20e3de9a4145a6@10.233.90.83:5004, s: ALIVE, inc: 0}, due to failed fetchMetadata call (cause: java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 1000ms in 'source(MonoDefer)' (and no fallback has been configured)) [2024-01-30T00:06:14,338][WARN ][sc-cluster-5005-270][MetadataStore] [default:int_taewfiies_5005:3a0862258ff5467a@10.233.90.83:5005][cf0ce37d-c1e4-48a1-9cc8-1c7e64446933] Timeout getting GetMetadataResp from 10.233.90.83:5004 within 1000 ms, cause: java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 1000ms in 'source(MonoDefer)' (and no fallback has been configured) [2024-01-30T00:06:14,338][WARN ][sc-cluster-5005-270][MembershipProtocol] [default:int_taewfiies_5005:3a0862258ff5467a@10.233.90.83:5005][updateMembership][SYNC] Skipping to add/update member: {m: default:int_taewfiies_5004:7d20e3de9a4145a6@10.233.90.83:5004, s: ALIVE, inc: 0}, due to failed fetchMetadata call (cause: java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 1000ms in 'source(MonoDefer)' (and no fallback has been configured)) [2024-01-30T00:06:14,359][ERROR][%int_taewfiies_5005%JRaft-StepDownTimer-3][AbstractClientService] Fail to connect int_taewfiies_5004, exception: java.net.ConnectException. [2024-01-30T00:06:14,359][ERROR][%int_taewfiies_5005%JRaft-StepDownTimer-3][ReplicatorGroupImpl] Fail to check replicator connection to peer=int_taewfiies_5004, replicatorType=Follower. [2024-01-30T00:06:14,393][WARN ][sc-cluster-5005-270][MetadataStore] [default:int_taewfiies_5005:3a0862258ff5467a@10.233.90.83:5005][991733a3-008e-4fae-b5e8-919e017ed0b5] Timeout getting GetMetadataResp from 10.233.90.83:5004 within 1000 ms, cause: java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 1000ms in 'source(MonoDefer)' (and no fallback has been configured) [2024-01-30T00:06:14,393][WARN ][sc-cluster-5005-270][MembershipProtocol] [default:int_taewfiies_5005:3a0862258ff5467a@10.233.90.83:5005][updateMembership][SYNC] Skipping to add/update member: {m: default:int_taewfiies_5004:7d20e3de9a4145a6@10.233.90.83:5004, s: ALIVE, inc: 0}, due to failed fetchMetadata call (cause: java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 1000ms in 'source(MonoDefer)' (and no fallback has been configured)) [2024-01-30T00:06:14,859][ERROR][%int_taewfiies_5005%JRaft-StepDownTimer-3][AbstractClientService] Fail to connect int_taewfiies_5004, exception: java.net.ConnectException. [2024-01-30T00:06:14,859][ERROR][%int_taewfiies_5005%JRaft-StepDownTimer-3][ReplicatorGroupImpl] Fail to check replicator connection to peer=int_taewfiies_5004, replicatorType=Follower. [2024-01-30T00:06:15,098][WARN ][sc-cluster-5005-270][MetadataStore] [default:int_taewfiies_5005:3a0862258ff5467a@10.233.90.83:5005][7b7b0122-f6ff-4eca-b89e-8dfaf1c0c11a] Timeout getting GetMetadataResp from 10.233.90.83:5004 within 1000 ms, cause: java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 1000ms in 'source(MonoDefer)' (and no fallback has been configured) [2024-01-30T00:06:15,098][WARN ][sc-cluster-5005-270][MembershipProtocol] [default:int_taewfiies_5005:3a0862258ff5467a@10.233.90.83:5005][updateMembership][SYNC] Skipping to add/update member: {m: default:int_taewfiies_5004:7d20e3de9a4145a6@10.233.90.83:5004, s: ALIVE, inc: 0}, due to failed fetchMetadata call (cause: java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 1000ms in 'source(MonoDefer)' (and no fallback has been configured)) [2024-01-30T00:06:15,168][WARN ][sc-cluster-5004-274][MetadataStore] [default:int_taewfiies_5004:7d20e3de9a4145a6@10.233.90.83:5004][747ec5ef-1cc0-45df-93a7-32ab373616e9] Timeout getting GetMetadataResp from 10.233.90.83:5005 within 1000 ms, cause: java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 1000ms in 'source(MonoDefer)' (and no fallback has been configured) [2024-01-30T00:06:15,169][WARN ][sc-cluster-5004-274][MembershipProtocol] [default:int_taewfiies_5004:7d20e3de9a4145a6@10.233.90.83:5004][updateMembership][SYNC] Skipping to add/update member: {m: default:int_taewfiies_5005:3a0862258ff5467a@10.233.90.83:5005, s: ALIVE, inc: 0}, due to failed fetchMetadata call (cause: java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 1000ms in 'source(MonoDefer)' (and no fallback has been configured)) [2024-01-30T00:06:15,290][WARN ][sc-cluster-5005-270][MetadataStore] [default:int_taewfiies_5005:3a0862258ff5467a@10.233.90.83:5005][bb2545a1-ade4-4a33-935f-65d19c7544a9] Timeout getting GetMetadataResp from 10.233.90.83:5004 within 1000 ms, cause: java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 1000ms in 'source(MonoDefer)' (and no fallback has been configured) [2024-01-30T00:06:15,290][WARN ][sc-cluster-5005-270][MembershipProtocol] [default:int_taewfiies_5005:3a0862258ff5467a@10.233.90.83:5005][updateMembership][SYNC] Skipping to add/update member: {m: default:int_taewfiies_5004:7d20e3de9a4145a6@10.233.90.83:5004, s: ALIVE, inc: 0}, due to failed fetchMetadata call (cause: java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 1000ms in 'source(MonoDefer)' (and no fallback has been configured)) [2024-01-30T00:06:15,359][ERROR][%int_taewfiies_5005%JRaft-StepDownTimer-3][AbstractClientService] Fail to connect int_taewfiies_5004, exception: java.net.ConnectException. [2024-01-30T00:06:15,359][ERROR][%int_taewfiies_5005%JRaft-StepDownTimer-3][ReplicatorGroupImpl] Fail to check replicator connection to peer=int_taewfiies_5004, replicatorType=Follower. [2024-01-30T00:06:15,860][ERROR][%int_taewfiies_5005%JRaft-StepDownTimer-3][AbstractClientService] Fail to connect int_taewfiies_5004, exception: java.net.ConnectException. [2024-01-30T00:06:15,860][ERROR][%int_taewfiies_5005%JRaft-StepDownTimer-3][ReplicatorGroupImpl] Fail to check replicator connection to peer=int_taewfiies_5004, replicatorType=Follower. [2024-01-30T00:06:16,168][WARN ][sc-cluster-5004-274][MetadataStore] [default:int_taewfiies_5004:7d20e3de9a4145a6@10.233.90.83:5004][ad031371-1e29-4abf-a2bf-38a609bab771] Timeout getting GetMetadataResp from 10.233.90.83:5005 within 1000 ms, cause: java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 1000ms in 'source(MonoDefer)' (and no fallback has been configured) [2024-01-30T00:06:16,168][WARN ][sc-cluster-5004-274][MembershipProtocol] [default:int_taewfiies_5004:7d20e3de9a4145a6@10.233.90.83:5004][updateMembership][SYNC] Skipping to add/update member: {m: default:int_taewfiies_5005:3a0862258ff5467a@10.233.90.83:5005, s: ALIVE, inc: 0}, due to failed fetchMetadata call (cause: java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 1000ms in 'source(MonoDefer)' (and no fallback has been configured)) [2024-01-30T00:06:16,290][WARN ][sc-cluster-5005-270][MetadataStore] [default:int_taewfiies_5005:3a0862258ff5467a@10.233.90.83:5005][43af788d-92db-491f-aa76-7d5afde32b17] Timeout getting GetMetadataResp from 10.233.90.83:5004 within 1000 ms, cause: java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 1000ms in 'source(MonoDefer)' (and no fallback has been configured) [2024-01-30T00:06:16,290][WARN ][sc-cluster-5005-270][MembershipProtocol] [default:int_taewfiies_5005:3a0862258ff5467a@10.233.90.83:5005][updateMembership][SYNC] Skipping to add/update member: {m: default:int_taewfiies_5004:7d20e3de9a4145a6@10.233.90.83:5004, s: ALIVE, inc: 0}, due to failed fetchMetadata call (cause: java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 1000ms in 'source(MonoDefer)' (and no fallback has been configured)) [2024-01-30T00:06:16,338][WARN ][sc-cluster-5004-274][MetadataStore] [default:int_taewfiies_5004:7d20e3de9a4145a6@10.233.90.83:5004][94658300-56df-4ab3-88ba-c1157528b073] Timeout getting GetMetadataResp from 10.233.90.83:5005 within 1000 ms, cause: java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 1000ms in 'source(MonoDefer)' (and no fallback has been configured) [2024-01-30T00:06:16,338][WARN ][sc-cluster-5004-274][MembershipProtocol] [default:int_taewfiies_5004:7d20e3de9a4145a6@10.233.90.83:5004][updateMembership][SYNC] Skipping to add/update member: {m: default:int_taewfiies_5005:3a0862258ff5467a@10.233.90.83:5005, s: ALIVE, inc: 0}, due to failed fetchMetadata call (cause: java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 1000ms in 'source(MonoDefer)' (and no fallback has been configured)) [2024-01-30T00:06:16,360][ERROR][%int_taewfiies_5005%JRaft-StepDownTimer-3][AbstractClientService] Fail to connect int_taewfiies_5004, exception: java.net.ConnectException. [2024-01-30T00:06:16,360][ERROR][%int_taewfiies_5005%JRaft-StepDownTimer-3][ReplicatorGroupImpl] Fail to check replicator connection to peer=int_taewfiies_5004, replicatorType=Follower. [2024-01-30T00:06:16,861][ERROR][%int_taewfiies_5005%JRaft-StepDownTimer-6][AbstractClientService] Fail to connect int_taewfiies_5004, exception: java.net.ConnectException. [2024-01-30T00:06:16,861][ERROR][%int_taewfiies_5005%JRaft-StepDownTimer-6][ReplicatorGroupImpl] Fail to check replicator connection to peer=int_taewfiies_5004, replicatorType=Follower. [2024-01-30T00:06:17,098][WARN ][sc-cluster-5005-270][MetadataStore] [default:int_taewfiies_5005:3a0862258ff5467a@10.233.90.83:5005][5a0bd7a8-ca53-4c21-a4b8-2bbccb0db3f9] Timeout getting GetMetadataResp from 10.233.90.83:5004 within 1000 ms, cause: java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 1000ms in 'source(MonoDefer)' (and no fallback has been configured) [2024-01-30T00:06:17,098][WARN ][sc-cluster-5005-270][MembershipProtocol] [default:int_taewfiies_5005:3a0862258ff5467a@10.233.90.83:5005][updateMembership][SYNC] Skipping to add/update member: {m: default:int_taewfiies_5004:7d20e3de9a4145a6@10.233.90.83:5004, s: ALIVE, inc: 0}, due to failed fetchMetadata call (cause: java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 1000ms in 'source(MonoDefer)' (and no fallback has been configured)) [2024-01-30T00:06:17,168][WARN ][sc-cluster-5004-274][MetadataStore] [default:int_taewfiies_5004:7d20e3de9a4145a6@10.233.90.83:5004][19ff6a35-1fe4-4bf4-96ae-b67c5ebeb362] Timeout getting GetMetadataResp from 10.233.90.83:5005 within 1000 ms, cause: java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 1000ms in 'source(MonoDefer)' (and no fallback has been configured) [2024-01-30T00:06:17,168][WARN ][sc-cluster-5004-274][MembershipProtocol] [default:int_taewfiies_5004:7d20e3de9a4145a6@10.233.90.83:5004][updateMembership][SYNC] Skipping to add/update member: {m: default:int_taewfiies_5005:3a0862258ff5467a@10.233.90.83:5005, s: ALIVE, inc: 0}, due to failed fetchMetadata call (cause: java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 1000ms in 'source(MonoDefer)' (and no fallback has been configured)) [2024-01-30T00:06:17,290][WARN ][sc-cluster-5005-270][MetadataStore] [default:int_taewfiies_5005:3a0862258ff5467a@10.233.90.83:5005][a16b54ae-4b5f-4fb3-932b-7085f3cf9ebd] Timeout getting GetMetadataResp from 10.233.90.83:5004 within 1000 ms, cause: java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 1000ms in 'source(MonoDefer)' (and no fallback has been configured) [2024-01-30T00:06:17,290][WARN ][sc-cluster-5005-270][MembershipProtocol] [default:int_taewfiies_5005:3a0862258ff5467a@10.233.90.83:5005][updateMembership][SYNC] Skipping to add/update member: {m: default:int_taewfiies_5004:7d20e3de9a4145a6@10.233.90.83:5004, s: ALIVE, inc: 0}, due to failed fetchMetadata call (cause: java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 1000ms in 'source(MonoDefer)' (and no fallback has been configured)) [2024-01-30T00:06:17,338][WARN ][sc-cluster-5004-274][MetadataStore] [default:int_taewfiies_5004:7d20e3de9a4145a6@10.233.90.83:5004][123cef8b-b7f6-49b4-ab7a-af38e8b591a1] Timeout getting GetMetadataResp from 10.233.90.83:5005 within 1000 ms, cause: java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 1000ms in 'source(MonoDefer)' (and no fallback has been configured) [2024-01-30T00:06:17,338][WARN ][sc-cluster-5004-274][MembershipProtocol] [default:int_taewfiies_5004:7d20e3de9a4145a6@10.233.90.83:5004][updateMembership][SYNC] Skipping to add/update member: {m: default:int_taewfiies_5005:3a0862258ff5467a@10.233.90.83:5005, s: ALIVE, inc: 0}, due to failed fetchMetadata call (cause: java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 1000ms in 'source(MonoDefer)' (and no fallback has been configured)) [2024-01-30T00:06:17,361][ERROR][%int_taewfiies_5005%JRaft-StepDownTimer-6][AbstractClientService] Fail to connect int_taewfiies_5004, exception: java.net.ConnectException. [2024-01-30T00:06:17,362][ERROR][%int_taewfiies_5005%JRaft-StepDownTimer-6][ReplicatorGroupImpl] Fail to check replicator connection to peer=int_taewfiies_5004, replicatorType=Follower. [2024-01-30T00:06:17,862][ERROR][%int_taewfiies_5005%JRaft-StepDownTimer-6][AbstractClientService] Fail to connect int_taewfiies_5004, exception: java.net.ConnectException. [2024-01-30T00:06:17,862][ERROR][%int_taewfiies_5005%JRaft-StepDownTimer-6][ReplicatorGroupImpl] Fail to check replicator connection to peer=int_taewfiies_5004, replicatorType=Follower. [2024-01-30T00:06:18,168][WARN ][sc-cluster-5004-274][MetadataStore] [default:int_taewfiies_5004:7d20e3de9a4145a6@10.233.90.83:5004][38b5ef45-923f-4a2d-be4e-730074d2a3d0] Timeout getting GetMetadataResp from 10.233.90.83:5005 within 1000 ms, cause: java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 1000ms in 'source(MonoDefer)' (and no fallback has been configured) [2024-01-30T00:06:18,168][WARN ][sc-cluster-5004-274][MembershipProtocol] [default:int_taewfiies_5004:7d20e3de9a4145a6@10.233.90.83:5004][updateMembership][SYNC] Skipping to add/update member: {m: default:int_taewfiies_5005:3a0862258ff5467a@10.233.90.83:5005, s: ALIVE, inc: 0}, due to failed fetchMetadata call (cause: java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 1000ms in 'source(MonoDefer)' (and no fallback has been configured)) [2024-01-30T00:06:18,338][WARN ][sc-cluster-5004-274][MetadataStore] [default:int_taewfiies_5004:7d20e3de9a4145a6@10.233.90.83:5004][ab2dcb11-c7dc-4445-9f72-8dbc4929cf03] Timeout getting GetMetadataResp from 10.233.90.83:5005 within 1000 ms, cause: java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 1000ms in 'source(MonoDefer)' (and no fallback has been configured) [2024-01-30T00:06:18,338][WARN ][sc-cluster-5004-274][MembershipProtocol] [default:int_taewfiies_5004:7d20e3de9a4145a6@10.233.90.83:5004][updateMembership][SYNC] Skipping to add/update member: {m: default:int_taewfiies_5005:3a0862258ff5467a@10.233.90.83:5005, s: ALIVE, inc: 0}, due to failed fetchMetadata call (cause: java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 1000ms in 'source(MonoDefer)' (and no fallback has been configured)) [00:06:26] : [org.apache.ignite.raft.jraft.core.ItNodeTest.testAppendEntriesWhenFollowerIsInErrorState()] [Test Output] [2024-01-30T00:06:18,687][INFO ][%int_taewfiies_5004%JRaft-ElectionTimer-3][NodeImpl] Unsuccessful election round number 6 [2024-01-30T00:06:18,687][INFO ][%int_taewfiies_5004%JRaft-ElectionTimer-3][NodeImpl] Node reset electionTimeout, currTimer 2 state STATE_FOLLOWER new electionTimeout 8000. [2024-01-30T00:06:18,687][INFO ][%int_taewfiies_5004%JRaft-ElectionTimer-3][NodeImpl] Election timeout was adjusted according to ExponentialBackoffTimeoutStrategy [maxTimeout=11000, roundsWithoutAdjusting=3] [2024-01-30T00:06:18,687][INFO ][%int_taewfiies_5004%JRaft-ElectionTimer-3][NodeImpl] Node term 2 start preVote. [2024-01-30T00:06:18,688][INFO ][%int_taewfiies_5003%JRaft-Request-Processor-0][NodeImpl] Node ignore PreVoteRequest from int_taewfiies_5004, term=3, currTerm=2, because the leader int_taewfiies_5005's lease is still valid. [2024-01-30T00:06:18,688][INFO ][%int_taewfiies_5006%JRaft-Request-Processor-3][NodeImpl] Node ignore PreVoteRequest from int_taewfiies_5004, term=3, currTerm=2, because the leader int_taewfiies_5005's lease is still valid. [2024-01-30T00:06:18,688][INFO ][%int_taewfiies_5007%JRaft-Request-Processor-3][NodeImpl] Node ignore PreVoteRequest from int_taewfiies_5004, term=3, currTerm=2, because the leader int_taewfiies_5005's lease is still valid. [2024-01-30T00:06:18,688][INFO ][%int_taewfiies_5004%JRaft-Response-Processor-2][NodeImpl] Node received PreVoteResponse from int_taewfiies_5003, term=2, granted=false. [2024-01-30T00:06:18,688][INFO ][%int_taewfiies_5004%JRaft-Response-Processor-0][NodeImpl] Node received PreVoteResponse from int_taewfiies_5006, term=2, granted=false. [2024-01-30T00:06:18,688][INFO ][%int_taewfiies_5004%JRaft-Response-Processor-1][NodeImpl] Node received PreVoteResponse from int_taewfiies_5007, term=2, granted=false. [2024-01-30T00:06:22,623][INFO ][%int_taewfiies_5004%JRaft-ElectionTimer-4][NodeImpl] Unsuccessful election round number 7 [2024-01-30T00:06:22,623][INFO ][%int_taewfiies_5004%JRaft-ElectionTimer-4][NodeImpl] Node reset electionTimeout, currTimer 2 state STATE_FOLLOWER new electionTimeout 11000. [2024-01-30T00:06:22,623][INFO ][%int_taewfiies_5004%JRaft-ElectionTimer-4][NodeImpl] Election timeout was adjusted according to ExponentialBackoffTimeoutStrategy [maxTimeout=11000, roundsWithoutAdjusting=3] [2024-01-30T00:06:22,624][INFO ][%int_taewfiies_5004%JRaft-ElectionTimer-4][NodeImpl] Node term 2 start preVote. [2024-01-30T00:06:22,624][INFO ][%int_taewfiies_5003%JRaft-Request-Processor-1][NodeImpl] Node ignore PreVoteRequest from int_taewfiies_5004, term=3, currTerm=2, because the leader int_taewfiies_5005's lease is still valid. [2024-01-30T00:06:22,624][INFO ][%int_taewfiies_5006%JRaft-Request-Processor-0][NodeImpl] Node ignore PreVoteRequest from int_taewfiies_5004, term=3, currTerm=2, because the leader int_taewfiies_5005's lease is still valid. [2024-01-30T00:06:22,624][INFO ][%int_taewfiies_5007%JRaft-Request-Processor-0][NodeImpl] Node ignore PreVoteRequest from int_taewfiies_5004, term=3, currTerm=2, because the leader int_taewfiies_5005's lease is still valid. [2024-01-30T00:06:22,625][INFO ][%int_taewfiies_5004%JRaft-Response-Processor-2][NodeImpl] Node received PreVoteResponse from int_taewfiies_5003, term=2, granted=false. [2024-01-30T00:06:22,625][INFO ][%int_taewfiies_5004%JRaft-Response-Processor-2][NodeImpl] Node received PreVoteResponse from int_taewfiies_5006, term=2, granted=false. [2024-01-30T00:06:22,625][INFO ][%int_taewfiies_5004%JRaft-Response-Processor-0][NodeImpl] Node received PreVoteResponse from int_taewfiies_5007, term=2, granted=false. [00:06:26] : [org.apache.ignite.raft.jraft.core.ItNodeTest.testAppendEntriesWhenFollowerIsInErrorState()] [Test Error Output] [2024-01-30T00:06:18,362][ERROR][%int_taewfiies_5005%JRaft-StepDownTimer-6][AbstractClientService] Fail to connect int_taewfiies_5004, exception: java.net.ConnectException. [2024-01-30T00:06:18,362][ERROR][%int_taewfiies_5005%JRaft-StepDownTimer-6][ReplicatorGroupImpl] Fail to check replicator connection to peer=int_taewfiies_5004, replicatorType=Follower. [2024-01-30T00:06:18,393][WARN ][sc-cluster-5005-270][MetadataStore] [default:int_taewfiies_5005:3a0862258ff5467a@10.233.90.83:5005][04bc50ae-11d7-4aa5-bb79-e4e975bd0a6e] Timeout getting GetMetadataResp from 10.233.90.83:5004 within 1000 ms, cause: java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 1000ms in 'source(MonoDefer)' (and no fallback has been configured) [2024-01-30T00:06:18,393][WARN ][sc-cluster-5005-270][MembershipProtocol] [default:int_taewfiies_5005:3a0862258ff5467a@10.233.90.83:5005][updateMembership][SYNC] Skipping to add/update member: {m: default:int_taewfiies_5004:7d20e3de9a4145a6@10.233.90.83:5004, s: ALIVE, inc: 0}, due to failed fetchMetadata call (cause: java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 1000ms in 'source(MonoDefer)' (and no fallback has been configured)) [2024-01-30T00:06:18,863][ERROR][%int_taewfiies_5005%JRaft-StepDownTimer-6][AbstractClientService] Fail to connect int_taewfiies_5004, exception: java.net.ConnectException. [2024-01-30T00:06:18,863][ERROR][%int_taewfiies_5005%JRaft-StepDownTimer-6][ReplicatorGroupImpl] Fail to check replicator connection to peer=int_taewfiies_5004, replicatorType=Follower. [2024-01-30T00:06:19,168][WARN ][sc-cluster-5004-274][MetadataStore] [default:int_taewfiies_5004:7d20e3de9a4145a6@10.233.90.83:5004][6f8e5fb5-2b1b-4027-92c3-f35412ede358] Timeout getting GetMetadataResp from 10.233.90.83:5005 within 1000 ms, cause: java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 1000ms in 'source(MonoDefer)' (and no fallback has been configured) [2024-01-30T00:06:19,168][WARN ][sc-cluster-5004-274][MembershipProtocol] [default:int_taewfiies_5004:7d20e3de9a4145a6@10.233.90.83:5004][updateMembership][SYNC] Skipping to add/update member: {m: default:int_taewfiies_5005:3a0862258ff5467a@10.233.90.83:5005, s: ALIVE, inc: 0}, due to failed fetchMetadata call (cause: java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 1000ms in 'source(MonoDefer)' (and no fallback has been configured)) [2024-01-30T00:06:19,290][WARN ][sc-cluster-5005-270][MetadataStore] [default:int_taewfiies_5005:3a0862258ff5467a@10.233.90.83:5005][40c57af9-56e1-4368-b747-650e296f60b4] Timeout getting GetMetadataResp from 10.233.90.83:5004 within 1000 ms, cause: java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 1000ms in 'source(MonoDefer)' (and no fallback has been configured) [2024-01-30T00:06:19,290][WARN ][sc-cluster-5005-270][MembershipProtocol] [default:int_taewfiies_5005:3a0862258ff5467a@10.233.90.83:5005][updateMembership][SYNC] Skipping to add/update member: {m: default:int_taewfiies_5004:7d20e3de9a4145a6@10.233.90.83:5004, s: ALIVE, inc: 0}, due to failed fetchMetadata call (cause: java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 1000ms in 'source(MonoDefer)' (and no fallback has been configured)) [2024-01-30T00:06:19,338][WARN ][sc-cluster-5005-270][MetadataStore] [default:int_taewfiies_5005:3a0862258ff5467a@10.233.90.83:5005][1ef4abf7-21fd-44d6-b56e-756a2821f1b2] Timeout getting GetMetadataResp from 10.233.90.83:5004 within 1000 ms, cause: java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 1000ms in 'source(MonoDefer)' (and no fallback has been configured) [2024-01-30T00:06:19,338][WARN ][sc-cluster-5005-270][MembershipProtocol] [default:int_taewfiies_5005:3a0862258ff5467a@10.233.90.83:5005][updateMembership][SYNC] Skipping to add/update member: {m: default:int_taewfiies_5004:7d20e3de9a4145a6@10.233.90.83:5004, s: ALIVE, inc: 0}, due to failed fetchMetadata call (cause: java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 1000ms in 'source(MonoDefer)' (and no fallback has been configured)) [2024-01-30T00:06:19,363][ERROR][%int_taewfiies_5005%JRaft-StepDownTimer-6][AbstractClientService] Fail to connect int_taewfiies_5004, exception: java.net.ConnectException. [2024-01-30T00:06:19,363][ERROR][%int_taewfiies_5005%JRaft-StepDownTimer-6][ReplicatorGroupImpl] Fail to check replicator connection to peer=int_taewfiies_5004, replicatorType=Follower. [2024-01-30T00:06:19,864][ERROR][%int_taewfiies_5005%JRaft-StepDownTimer-6][AbstractClientService] Fail to connect int_taewfiies_5004, exception: java.net.ConnectException. [2024-01-30T00:06:19,864][ERROR][%int_taewfiies_5005%JRaft-StepDownTimer-6][ReplicatorGroupImpl] Fail to check replicator connection to peer=int_taewfiies_5004, replicatorType=Follower. [2024-01-30T00:06:20,168][WARN ][sc-cluster-5004-274][MetadataStore] [default:int_taewfiies_5004:7d20e3de9a4145a6@10.233.90.83:5004][f376f088-392e-4168-a53e-e67521ca165c] Timeout getting GetMetadataResp from 10.233.90.83:5005 within 1000 ms, cause: java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 1000ms in 'source(MonoDefer)' (and no fallback has been configured) [2024-01-30T00:06:20,168][WARN ][sc-cluster-5004-274][MembershipProtocol] [default:int_taewfiies_5004:7d20e3de9a4145a6@10.233.90.83:5004][updateMembership][SYNC] Skipping to add/update member: {m: default:int_taewfiies_5005:3a0862258ff5467a@10.233.90.83:5005, s: ALIVE, inc: 0}, due to failed fetchMetadata call (cause: java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 1000ms in 'source(MonoDefer)' (and no fallback has been configured)) [2024-01-30T00:06:20,290][WARN ][sc-cluster-5005-270][MetadataStore] [default:int_taewfiies_5005:3a0862258ff5467a@10.233.90.83:5005][7b3e5c5d-4f5a-409b-81b8-2aa955daa302] Timeout getting GetMetadataResp from 10.233.90.83:5004 within 1000 ms, cause: java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 1000ms in 'source(MonoDefer)' (and no fallback has been configured) [2024-01-30T00:06:20,290][WARN ][sc-cluster-5005-270][MembershipProtocol] [default:int_taewfiies_5005:3a0862258ff5467a@10.233.90.83:5005][updateMembership][SYNC] Skipping to add/update member: {m: default:int_taewfiies_5004:7d20e3de9a4145a6@10.233.90.83:5004, s: ALIVE, inc: 0}, due to failed fetchMetadata call (cause: java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 1000ms in 'source(MonoDefer)' (and no fallback has been configured)) [2024-01-30T00:06:20,364][ERROR][%int_taewfiies_5005%JRaft-StepDownTimer-6][AbstractClientService] Fail to connect int_taewfiies_5004, exception: java.net.ConnectException. [2024-01-30T00:06:20,364][ERROR][%int_taewfiies_5005%JRaft-StepDownTimer-6][ReplicatorGroupImpl] Fail to check replicator connection to peer=int_taewfiies_5004, replicatorType=Follower. [2024-01-30T00:06:20,865][ERROR][%int_taewfiies_5005%JRaft-StepDownTimer-6][AbstractClientService] Fail to connect int_taewfiies_5004, exception: java.net.ConnectException. [2024-01-30T00:06:20,865][ERROR][%int_taewfiies_5005%JRaft-StepDownTimer-6][ReplicatorGroupImpl] Fail to check replicator connection to peer=int_taewfiies_5004, replicatorType=Follower. [2024-01-30T00:06:21,098][WARN ][sc-cluster-5005-270][MetadataStore] [default:int_taewfiies_5005:3a0862258ff5467a@10.233.90.83:5005][bd426d03-7f9b-44b6-a285-1086f78e0e2c] Timeout getting GetMetadataResp from 10.233.90.83:5004 within 1000 ms, cause: java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 1000ms in 'source(MonoDefer)' (and no fallback has been configured) [2024-01-30T00:06:21,098][WARN ][sc-cluster-5005-270][MembershipProtocol] [default:int_taewfiies_5005:3a0862258ff5467a@10.233.90.83:5005][updateMembership][SYNC] Skipping to add/update member: {m: default:int_taewfiies_5004:7d20e3de9a4145a6@10.233.90.83:5004, s: ALIVE, inc: 0}, due to failed fetchMetadata call (cause: java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 1000ms in 'source(MonoDefer)' (and no fallback has been configured)) [2024-01-30T00:06:21,168][WARN ][sc-cluster-5004-274][MetadataStore] [default:int_taewfiies_5004:7d20e3de9a4145a6@10.233.90.83:5004][8e216d43-4ed5-41ff-a285-faad87b2cb5d] Timeout getting GetMetadataResp from 10.233.90.83:5005 within 1000 ms, cause: java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 1000ms in 'source(MonoDefer)' (and no fallback has been configured) [2024-01-30T00:06:21,168][WARN ][sc-cluster-5004-274][MembershipProtocol] [default:int_taewfiies_5004:7d20e3de9a4145a6@10.233.90.83:5004][updateMembership][SYNC] Skipping to add/update member: {m: default:int_taewfiies_5005:3a0862258ff5467a@10.233.90.83:5005, s: ALIVE, inc: 0}, due to failed fetchMetadata call (cause: java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 1000ms in 'source(MonoDefer)' (and no fallback has been configured)) [2024-01-30T00:06:21,290][WARN ][sc-cluster-5005-270][MetadataStore] [default:int_taewfiies_5005:3a0862258ff5467a@10.233.90.83:5005][7cdeddcd-fb27-45a0-8260-2631e74edeb4] Timeout getting GetMetadataResp from 10.233.90.83:5004 within 1000 ms, cause: java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 1000ms in 'source(MonoDefer)' (and no fallback has been configured) [2024-01-30T00:06:21,290][WARN ][sc-cluster-5005-270][MembershipProtocol] [default:int_taewfiies_5005:3a0862258ff5467a@10.233.90.83:5005][updateMembership][SYNC] Skipping to add/update member: {m: default:int_taewfiies_5004:7d20e3de9a4145a6@10.233.90.83:5004, s: ALIVE, inc: 0}, due to failed fetchMetadata call (cause: java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 1000ms in 'source(MonoDefer)' (and no fallback has been configured)) [2024-01-30T00:06:21,338][WARN ][sc-cluster-5004-274][MetadataStore] [default:int_taewfiies_5004:7d20e3de9a4145a6@10.233.90.83:5004][426dd11e-b75b-4b34-b347-bc1d49aaf49c] Timeout getting GetMetadataResp from 10.233.90.83:5005 within 1000 ms, cause: java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 1000ms in 'source(MonoDefer)' (and no fallback has been configured) [2024-01-30T00:06:21,338][WARN ][sc-cluster-5004-274][MembershipProtocol] [default:int_taewfiies_5004:7d20e3de9a4145a6@10.233.90.83:5004][updateMembership][SYNC] Skipping to add/update member: {m: default:int_taewfiies_5005:3a0862258ff5467a@10.233.90.83:5005, s: ALIVE, inc: 0}, due to failed fetchMetadata call (cause: java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 1000ms in 'source(MonoDefer)' (and no fallback has been configured)) [2024-01-30T00:06:21,365][ERROR][%int_taewfiies_5005%JRaft-StepDownTimer-6][AbstractClientService] Fail to connect int_taewfiies_5004, exception: java.net.ConnectException. [2024-01-30T00:06:21,365][ERROR][%int_taewfiies_5005%JRaft-StepDownTimer-6][ReplicatorGroupImpl] Fail to check replicator connection to peer=int_taewfiies_5004, replicatorType=Follower. [2024-01-30T00:06:21,393][WARN ][sc-cluster-5004-274][MetadataStore] [default:int_taewfiies_5004:7d20e3de9a4145a6@10.233.90.83:5004][3749ccaf-27eb-412c-8966-ab2261c9d26b] Timeout getting GetMetadataResp from 10.233.90.83:5005 within 1000 ms, cause: java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 1000ms in 'source(MonoDefer)' (and no fallback has been configured) [2024-01-30T00:06:21,393][WARN ][sc-cluster-5004-274][MembershipProtocol] [default:int_taewfiies_5004:7d20e3de9a4145a6@10.233.90.83:5004][updateMembership][SYNC] Skipping to add/update member: {m: default:int_taewfiies_5005:3a0862258ff5467a@10.233.90.83:5005, s: ALIVE, inc: 0}, due to failed fetchMetadata call (cause: java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 1000ms in 'source(MonoDefer)' (and no fallback has been configured)) [2024-01-30T00:06:21,865][ERROR][%int_taewfiies_5005%JRaft-StepDownTimer-6][AbstractClientService] Fail to connect int_taewfiies_5004, exception: java.net.ConnectException. [2024-01-30T00:06:21,866][ERROR][%int_taewfiies_5005%JRaft-StepDownTimer-6][ReplicatorGroupImpl] Fail to check replicator connection to peer=int_taewfiies_5004, replicatorType=Follower. [2024-01-30T00:06:22,168][WARN ][sc-cluster-5004-274][MetadataStore] [default:int_taewfiies_5004:7d20e3de9a4145a6@10.233.90.83:5004][fa1a01fb-a829-44be-a74a-48f2a5fac899] Timeout getting GetMetadataResp from 10.233.90.83:5005 within 1000 ms, cause: java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 1000ms in 'source(MonoDefer)' (and no fallback has been configured) [2024-01-30T00:06:22,168][WARN ][sc-cluster-5004-274][MembershipProtocol] [default:int_taewfiies_5004:7d20e3de9a4145a6@10.233.90.83:5004][updateMembership][SYNC] Skipping to add/update member: {m: default:int_taewfiies_5005:3a0862258ff5467a@10.233.90.83:5005, s: ALIVE, inc: 0}, due to failed fetchMetadata call (cause: java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 1000ms in 'source(MonoDefer)' (and no fallback has been configured)) [2024-01-30T00:06:22,290][WARN ][sc-cluster-5005-270][MetadataStore] [default:int_taewfiies_5005:3a0862258ff5467a@10.233.90.83:5005][6c5282e9-2163-4646-b424-df2ca9c194e6] Timeout getting GetMetadataResp from 10.233.90.83:5004 within 1000 ms, cause: java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 1000ms in 'source(MonoDefer)' (and no fallback has been configured) [2024-01-30T00:06:22,290][WARN ][sc-cluster-5005-270][MembershipProtocol] [default:int_taewfiies_5005:3a0862258ff5467a@10.233.90.83:5005][updateMembership][SYNC] Skipping to add/update member: {m: default:int_taewfiies_5004:7d20e3de9a4145a6@10.233.90.83:5004, s: ALIVE, inc: 0}, due to failed fetchMetadata call (cause: java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 1000ms in 'source(MonoDefer)' (and no fallback has been configured)) [2024-01-30T00:06:22,366][ERROR][%int_taewfiies_5005%JRaft-StepDownTimer-6][AbstractClientService] Fail to connect int_taewfiies_5004, exception: java.net.ConnectException. [2024-01-30T00:06:22,366][ERROR][%int_taewfiies_5005%JRaft-StepDownTimer-6][ReplicatorGroupImpl] Fail to check replicator connection to peer=int_taewfiies_5004, replicatorType=Follower. [2024-01-30T00:06:22,866][ERROR][%int_taewfiies_5005%JRaft-StepDownTimer-6][AbstractClientService] Fail to connect int_taewfiies_5004, exception: java.net.ConnectException. [2024-01-30T00:06:22,866][ERROR][%int_taewfiies_5005%JRaft-StepDownTimer-6][ReplicatorGroupImpl] Fail to check replicator connection to peer=int_taewfiies_5004, replicatorType=Follower. [2024-01-30T00:06:23,168][WARN ][sc-cluster-5004-274][MetadataStore] [default:int_taewfiies_5004:7d20e3de9a4145a6@10.233.90.83:5004][fd9f2dec-0bd6-44a0-aa72-d0f08f2d8b4e] Timeout getting GetMetadataResp from 10.233.90.83:5005 within 1000 ms, cause: java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 1000ms in 'source(MonoDefer)' (and no fallback has been configured) [2024-01-30T00:06:23,168][WARN ][sc-cluster-5004-274][MembershipProtocol] [default:int_taewfiies_5004:7d20e3de9a4145a6@10.233.90.83:5004][updateMembership][SYNC] Skipping to add/update member: {m: default:int_taewfiies_5005:3a0862258ff5467a@10.233.90.83:5005, s: ALIVE, inc: 0}, due to failed fetchMetadata call (cause: java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 1000ms in 'source(MonoDefer)' (and no fallback has been configured)) [2024-01-30T00:06:23,338][WARN ][sc-cluster-5005-270][MetadataStore] [default:int_taewfiies_5005:3a0862258ff5467a@10.233.90.83:5005][2aeebd1d-58c3-408a-a2b5-1d46b4044d2c] Timeout getting GetMetadataResp from 10.233.90.83:5004 within 1000 ms, cause: java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 1000ms in 'source(MonoDefer)' (and no fallback has been configured) [2024-01-30T00:06:23,338][WARN ][sc-cluster-5005-270][MembershipProtocol] [default:int_taewfiies_5005:3a0862258ff5467a@10.233.90.83:5005][updateMembership][SYNC] Skipping to add/update member: {m: default:int_taewfiies_5004:7d20e3de9a4145a6@10.233.90.83:5004, s: ALIVE, inc: 0}, due to failed fetchMetadata call (cause: java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 1000ms in 'source(MonoDefer)' (and no fallback has been configured)) [2024-01-30T00:06:23,367][ERROR][%int_taewfiies_5005%JRaft-StepDownTimer-1][AbstractClientService] Fail to connect int_taewfiies_5004, exception: java.net.ConnectException. [2024-01-30T00:06:23,367][ERROR][%int_taewfiies_5005%JRaft-StepDownTimer-1][ReplicatorGroupImpl] Fail to check replicator connection to peer=int_taewfiies_5004, replicatorType=Follower. [2024-01-30T00:06:23,393][WARN ][sc-cluster-5004-274][MetadataStore] [default:int_taewfiies_5004:7d20e3de9a4145a6@10.233.90.83:5004][1be06047-bfb3-420c-9cb2-bf6b14d51721] Timeout getting GetMetadataResp from 10.233.90.83:5005 within 1000 ms, cause: java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 1000ms in 'source(MonoDefer)' (and no fallback has been configured) [2024-01-30T00:06:23,393][WARN ][sc-cluster-5004-274][MembershipProtocol] [default:int_taewfiies_5004:7d20e3de9a4145a6@10.233.90.83:5004][updateMembership][SYNC] Skipping to add/update member: {m: default:int_taewfiies_5005:3a0862258ff5467a@10.233.90.83:5005, s: ALIVE, inc: 0}, due to failed fetchMetadata call (cause: java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 1000ms in 'source(MonoDefer)' (and no fallback has been configured)) [2024-01-30T00:06:23,867][ERROR][%int_taewfiies_5005%JRaft-StepDownTimer-1][AbstractClientService] Fail to connect int_taewfiies_5004, exception: java.net.ConnectException. [2024-01-30T00:06:23,867][ERROR][%int_taewfiies_5005%JRaft-StepDownTimer-1][ReplicatorGroupImpl] Fail to check replicator connection to peer=int_taewfiies_5004, replicatorType=Follower. [2024-01-30T00:06:24,168][WARN ][sc-cluster-5004-274][MetadataStore] [default:int_taewfiies_5004:7d20e3de9a4145a6@10.233.90.83:5004][a8890809-1d3b-42c0-8218-ca288b987080] Timeout getting GetMetadataResp from 10.233.90.83:5005 within 1000 ms, cause: java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 1000ms in 'source(MonoDefer)' (and no fallback has been configured) [2024-01-30T00:06:24,169][WARN ][sc-cluster-5004-274][MembershipProtocol] [default:int_taewfiies_5004:7d20e3de9a4145a6@10.233.90.83:5004][updateMembership][SYNC] Skipping to add/update member: {m: default:int_taewfiies_5005:3a0862258ff5467a@10.233.90.83:5005, s: ALIVE, inc: 0}, due to failed fetchMetadata call (cause: java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 1000ms in 'source(MonoDefer)' (and no fallback has been configured)) [2024-01-30T00:06:24,290][WARN ][sc-cluster-5005-270][MetadataStore] [default:int_taewfiies_5005:3a0862258ff5467a@10.233.90.83:5005][1d243490-2b8b-49f6-852e-0a733b99f534] Timeout getting GetMetadataResp from 10.233.90.83:5004 within 1000 ms, cause: java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 1000ms in 'source(MonoDefer)' (and no fallback has been configured) [2024-01-30T00:06:24,290][WARN ][sc-cluster-5005-270][MembershipProtocol] [default:int_taewfiies_5005:3a0862258ff5467a@10.233.90.83:5005][updateMembership][SYNC] Skipping to add/update member: {m: default:int_taewfiies_5004:7d20e3de9a4145a6@10.233.90.83:5004, s: ALIVE, inc: 0}, due to failed fetchMetadata call (cause: java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 1000ms in 'source(MonoDefer)' (and no fallback has been configured)) [2024-01-30T00:06:24,338][WARN ][sc-cluster-5005-270][MetadataStore] [default:int_taewfiies_5005:3a0862258ff5467a@10.233.90.83:5005][479ab1ff-70b5-414d-9269-6892d83c66b7] Timeout getting GetMetadataResp from 10.233.90.83:5004 within 1000 ms, cause: java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 1000ms in 'source(MonoDefer)' (and no fallback has been configured) [2024-01-30T00:06:24,338][WARN ][sc-cluster-5005-270][MembershipProtocol] [default:int_taewfiies_5005:3a0862258ff5467a@10.233.90.83:5005][updateMembership][SYNC] Skipping to add/update member: {m: default:int_taewfiies_5004:7d20e3de9a4145a6@10.233.90.83:5004, s: ALIVE, inc: 0}, due to failed fetchMetadata call (cause: java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 1000ms in 'source(MonoDefer)' (and no fallback has been configured)) [2024-01-30T00:06:24,367][ERROR][%int_taewfiies_5005%JRaft-StepDownTimer-1][AbstractClientService] Fail to connect int_taewfiies_5004, exception: java.net.ConnectException. [2024-01-30T00:06:24,367][ERROR][%int_taewfiies_5005%JRaft-StepDownTimer-1][ReplicatorGroupImpl] Fail to check replicator connection to peer=int_taewfiies_5004, replicatorType=Follower. [2024-01-30T00:06:24,393][WARN ][sc-cluster-5004-274][MetadataStore] [default:int_taewfiies_5004:7d20e3de9a4145a6@10.233.90.83:5004][d547b051-8ec0-4d3f-a820-bb926156002f] Timeout getting GetMetadataResp from 10.233.90.83:5005 within 1000 ms, cause: java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 1000ms in 'source(MonoDefer)' (and no fallback has been configured) [2024-01-30T00:06:24,393][WARN ][sc-cluster-5004-274][MembershipProtocol] [default:int_taewfiies_5004:7d20e3de9a4145a6@10.233.90.83:5004][updateMembership][SYNC] Skipping to add/update member: {m: default:int_taewfiies_5005:3a0862258ff5467a@10.233.90.83:5005, s: ALIVE, inc: 0}, due to failed fetchMetadata call (cause: java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 1000ms in 'source(MonoDefer)' (and no fallback has been configured)) [2024-01-30T00:06:24,868][ERROR][%int_taewfiies_5005%JRaft-StepDownTimer-1][AbstractClientService] Fail to connect int_taewfiies_5004, exception: java.net.ConnectException. [2024-01-30T00:06:24,868][ERROR][%int_taewfiies_5005%JRaft-StepDownTimer-1][ReplicatorGroupImpl] Fail to check replicator connection to peer=int_taewfiies_5004, replicatorType=Follower. [2024-01-30T00:06:25,168][WARN ][sc-cluster-5004-274][MetadataStore] [default:int_taewfiies_5004:7d20e3de9a4145a6@10.233.90.83:5004][7733b294-7c56-4e15-a74c-104c09df09de] Timeout getting GetMetadataResp from 10.233.90.83:5005 within 1000 ms, cause: java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 1000ms in 'source(MonoDefer)' (and no fallback has been configured) [2024-01-30T00:06:25,169][WARN ][sc-cluster-5004-274][MembershipProtocol] [default:int_taewfiies_5004:7d20e3de9a4145a6@10.233.90.83:5004][updateMembership][SYNC] Skipping to add/update member: {m: default:int_taewfiies_5005:3a0862258ff5467a@10.233.90.83:5005, s: ALIVE, inc: 0}, due to failed fetchMetadata call (cause: java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 1000ms in 'source(MonoDefer)' (and no fallback has been configured)) [2024-01-30T00:06:25,290][WARN ][sc-cluster-5005-270][MetadataStore] [default:int_taewfiies_5005:3a0862258ff5467a@10.233.90.83:5005][17826785-5a2a-4f87-b965-83939bb3b2ba] Timeout getting GetMetadataResp from 10.233.90.83:5004 within 1000 ms, cause: java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 1000ms in 'source(MonoDefer)' (and no fallback has been configured) [2024-01-30T00:06:25,290][WARN ][sc-cluster-5005-270][MembershipProtocol] [default:int_taewfiies_5005:3a0862258ff5467a@10.233.90.83:5005][updateMembership][SYNC] Skipping to add/update member: {m: default:int_taewfiies_5004:7d20e3de9a4145a6@10.233.90.83:5004, s: ALIVE, inc: 0}, due to failed fetchMetadata call (cause: java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 1000ms in 'source(MonoDefer)' (and no fallback has been configured)) [2024-01-30T00:06:25,368][ERROR][%int_taewfiies_5005%JRaft-StepDownTimer-1][AbstractClientService] Fail to connect int_taewfiies_5004, exception: java.net.ConnectException. [2024-01-30T00:06:25,368][ERROR][%int_taewfiies_5005%JRaft-StepDownTimer-1][ReplicatorGroupImpl] Fail to check replicator connection to peer=int_taewfiies_5004, replicatorType=Follower. [2024-01-30T00:06:25,393][WARN ][sc-cluster-5004-274][MetadataStore] [default:int_taewfiies_5004:7d20e3de9a4145a6@10.233.90.83:5004][6b79757d-c042-4442-86d0-e30ae3d54a89] Timeout getting GetMetadataResp from 10.233.90.83:5005 within 1000 ms, cause: java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 1000ms in 'source(MonoDefer)' (and no fallback has been configured) [2024-01-30T00:06:25,393][WARN ][sc-cluster-5004-274][MembershipProtocol] [default:int_taewfiies_5004:7d20e3de9a4145a6@10.233.90.83:5004][updateMembership][SYNC] Skipping to add/update member: {m: default:int_taewfiies_5005:3a0862258ff5467a@10.233.90.83:5005, s: ALIVE, inc: 0}, due to failed fetchMetadata call (cause: java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 1000ms in 'source(MonoDefer)' (and no fallback has been configured)) [2024-01-30T00:06:25,868][ERROR][%int_taewfiies_5005%JRaft-StepDownTimer-1][AbstractClientService] Fail to connect int_taewfiies_5004, exception: java.net.ConnectException. [2024-01-30T00:06:25,869][ERROR][%int_taewfiies_5005%JRaft-StepDownTimer-1][ReplicatorGroupImpl] Fail to check replicator connection to peer=int_taewfiies_5004, replicatorType=Follower. [2024-01-30T00:06:26,168][WARN ][sc-cluster-5004-274][MetadataStore] [default:int_taewfiies_5004:7d20e3de9a4145a6@10.233.90.83:5004][d04c1a78-5f76-4057-9930-e4e547ae8f22] Timeout getting GetMetadataResp from 10.233.90.83:5005 within 1000 ms, cause: java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 1000ms in 'source(MonoDefer)' (and no fallback has been configured) [2024-01-30T00:06:26,169][WARN ][sc-cluster-5004-274][MembershipProtocol] [default:int_taewfiies_5004:7d20e3de9a4145a6@10.233.90.83:5004][updateMembership][SYNC] Skipping to add/update member: {m: default:int_taewfiies_5005:3a0862258ff5467a@10.233.90.83:5005, s: ALIVE, inc: 0}, due to failed fetchMetadata call (cause: java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 1000ms in 'source(MonoDefer)' (and no fallback has been configured)) [2024-01-30T00:06:26,290][WARN ][sc-cluster-5005-270][MetadataStore] [default:int_taewfiies_5005:3a0862258ff5467a@10.233.90.83:5005][f1e581d6-dff0-4191-9c04-c53beb276954] Timeout getting GetMetadataResp from 10.233.90.83:5004 within 1000 ms, cause: java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 1000ms in 'source(MonoDefer)' (and no fallback has been configured) [00:06:30] : [org.apache.ignite.raft.jraft.core.ItNodeTest.testAppendEntriesWhenFollowerIsInErrorState()] [Test Output] [2024-01-30T00:06:27,598][INFO ][%int_taewfiies_5004%JRaft-ElectionTimer-5][NodeImpl] Unsuccessful election round number 8 [2024-01-30T00:06:27,598][INFO ][%int_taewfiies_5004%JRaft-ElectionTimer-5][NodeImpl] Node term 2 start preVote. [2024-01-30T00:06:27,599][INFO ][%int_taewfiies_5007%JRaft-Request-Processor-1][NodeImpl] Node ignore PreVoteRequest from int_taewfiies_5004, term=3, currTerm=2, because the leader int_taewfiies_5005's lease is still valid. [2024-01-30T00:06:27,599][INFO ][%int_taewfiies_5006%JRaft-Request-Processor-1][NodeImpl] Node ignore PreVoteRequest from int_taewfiies_5004, term=3, currTerm=2, because the leader int_taewfiies_5005's lease is still valid. [2024-01-30T00:06:27,599][INFO ][%int_taewfiies_5003%JRaft-Request-Processor-2][NodeImpl] Node ignore PreVoteRequest from int_taewfiies_5004, term=3, currTerm=2, because the leader int_taewfiies_5005's lease is still valid. [2024-01-30T00:06:27,600][INFO ][%int_taewfiies_5004%JRaft-Response-Processor-1][NodeImpl] Node received PreVoteResponse from int_taewfiies_5007, term=2, granted=false. [2024-01-30T00:06:27,600][INFO ][%int_taewfiies_5004%JRaft-Response-Processor-2][NodeImpl] Node received PreVoteResponse from int_taewfiies_5006, term=2, granted=false. [2024-01-30T00:06:27,600][INFO ][%int_taewfiies_5004%JRaft-Response-Processor-0][NodeImpl] Node received PreVoteResponse from int_taewfiies_5003, term=2, granted=false. [2024-01-30T00:06:30,201][INFO ][Test worker][TestCluster] End ensureSame, leader=JRaftNode [nodeId=] [2024-01-30T00:06:30,204][INFO ][%int_taewfiies_5005%JRaft-Request-Processor-1][LogThreadPoolExecutor] ThreadPool is terminated: %int_taewfiies_5005%JRaft-Request-Processor-, org.apache.ignite.raft.jraft.util.MetricThreadPoolExecutor@5f3c7d81[Shutting down, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 4]. [2024-01-30T00:06:30,204][INFO ][Test worker][NodeImpl] Node shutdown, currTerm=2 state=STATE_LEADER. [2024-01-30T00:06:30,204][INFO ][Test worker][Replicator] Replicator Replicator [state=Replicate, statInfo=, peerId=int_taewfiies_5007, type=Follower] is going to quit [2024-01-30T00:06:30,204][INFO ][%int_taewfiies_5005%JRaft-FSMCaller-Disruptor-_stripe_2-0][StateMachineAdapter] onLeaderStop: status=Status[ESHUTDOWN<1007>: Raft node is going to quit.]. [2024-01-30T00:06:30,204][INFO ][Test worker][Replicator] Replicator Replicator [state=Replicate, statInfo=, peerId=int_taewfiies_5003, type=Follower] is going to quit [2024-01-30T00:06:30,205][INFO ][%int_taewfiies_5006%JRaft-Request-Processor-2][NodeImpl] Node received TimeoutNowRequest from int_taewfiies_5005, term=2 and starts voting. [2024-01-30T00:06:30,205][INFO ][%int_taewfiies_5006%JRaft-Request-Processor-2][NodeImpl] Node start vote and grant vote self, term=2. [2024-01-30T00:06:30,205][INFO ][%int_taewfiies_5005%JRaft-Response-Processor-0][Replicator] Replicator Replicator [state=Replicate, statInfo=, peerId=int_taewfiies_5006, type=Follower] is going to quit [2024-01-30T00:06:30,206][INFO ][%int_taewfiies_5006%JRaft-FSMCaller-Disruptor-_stripe_3-0][StateMachineAdapter] onStopFollowing: LeaderChangeContext [leaderId=int_taewfiies_5005, term=2, status=Status[ERAFTTIMEDOUT<10001>: A follower's leader_id is reset to NULL as it begins to request_vote.]]. [2024-01-30T00:06:30,206][INFO ][Test worker][LocalRaftMetaStorage] Save raft meta, path=build/work/ItNodeTest/testAppendEntriesWhenFollowerIsInErrorState_30809148389477213/int_taewfiies_5005/meta, term=2, votedFor=int_taewfiies_5005, cost time=1 ms [2024-01-30T00:06:30,206][INFO ][Test worker][FSMCallerImpl] Shutting down FSMCaller... [2024-01-30T00:06:30,206][INFO ][%int_taewfiies_5005%JRaft-FSMCaller-Disruptor-_stripe_2-0][StateMachineAdapter] onShutdown. [2024-01-30T00:06:30,207][INFO ][%int_taewfiies_5005%JRaft-Node-Scheduler-3][LogScheduledThreadPoolExecutor] ThreadPool is terminated: %int_taewfiies_5005%JRaft-Node-Scheduler-, org.apache.ignite.raft.jraft.util.MetricScheduledThreadPoolExecutor@5837cf28[Shutting down, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 732]. [2024-01-30T00:06:30,207][INFO ][%int_taewfiies_5005%JRaft-ElectionTimer-4][LogScheduledThreadPoolExecutor] ThreadPool is terminated: %int_taewfiies_5005%JRaft-ElectionTimer-, org.apache.ignite.raft.jraft.util.MetricScheduledThreadPoolExecutor@7a1081fb[Shutting down, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 4]. [2024-01-30T00:06:30,207][INFO ][%int_taewfiies_5006%JRaft-Request-Processor-2][LocalRaftMetaStorage] Save raft meta, path=build/work/ItNodeTest/testAppendEntriesWhenFollowerIsInErrorState_30809148389477213/int_taewfiies_5006/meta, term=3, votedFor=int_taewfiies_5006, cost time=1 ms [2024-01-30T00:06:30,207][INFO ][%int_taewfiies_5005%JRaft-VoteTimer-0][LogScheduledThreadPoolExecutor] ThreadPool is terminated: %int_taewfiies_5005%JRaft-VoteTimer-, org.apache.ignite.raft.jraft.util.MetricScheduledThreadPoolExecutor@198089a1[Shutting down, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 1]. [2024-01-30T00:06:30,209][INFO ][%int_taewfiies_5004%JRaft-Request-Processor-1][NodeImpl] Node received RequestVoteRequest from int_taewfiies_5006, term=3, currTerm=2. [2024-01-30T00:06:30,210][INFO ][%int_taewfiies_5005%JRaft-StepDownTimer-3][LogScheduledThreadPoolExecutor] ThreadPool is terminated: %int_taewfiies_5005%JRaft-StepDownTimer-, org.apache.ignite.raft.jraft.util.MetricScheduledThreadPoolExecutor@4c31517e[Shutting down, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 46]. [2024-01-30T00:06:30,210][INFO ][%int_taewfiies_5007%JRaft-Request-Processor-3][NodeImpl] Node received RequestVoteRequest from int_taewfiies_5006, term=3, currTerm=2. [2024-01-30T00:06:30,210][INFO ][%int_taewfiies_5005%JRaft-SnapshotTimer-0][LogScheduledThreadPoolExecutor] ThreadPool is terminated: %int_taewfiies_5005%JRaft-SnapshotTimer-, org.apache.ignite.raft.jraft.util.MetricScheduledThreadPoolExecutor@4b611649[Shutting down, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0]. [2024-01-30T00:06:30,210][INFO ][%int_taewfiies_5007%JRaft-FSMCaller-Disruptor-_stripe_4-0][StateMachineAdapter] onStopFollowing: LeaderChangeContext [leaderId=int_taewfiies_5005, term=2, status=Status[EHIGHERTERMRESPONSE<10008>: Raft node receives higher term RequestVoteRequest.]]. [2024-01-30T00:06:30,211][INFO ][%int_taewfiies_5004%JRaft-Request-Processor-1][LocalRaftMetaStorage] Save raft meta, path=build/work/ItNodeTest/testAppendEntriesWhenFollowerIsInErrorState_30809148389477213/int_taewfiies_5004/meta, term=3, votedFor=, cost time=1 ms [2024-01-30T00:06:30,211][INFO ][%int_taewfiies_5005%JRaft-Common-Executor-2][LogThreadPoolExecutor] ThreadPool is terminated: %int_taewfiies_5005%JRaft-Common-Executor-, org.apache.ignite.raft.jraft.util.MetricThreadPoolExecutor@dcf6baf[Shutting down, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 718]. [2024-01-30T00:06:30,211][INFO ][%int_taewfiies_5003%JRaft-Request-Processor-0][NodeImpl] Node received RequestVoteRequest from int_taewfiies_5006, term=3, currTerm=2. [2024-01-30T00:06:30,211][INFO ][%int_taewfiies_5007%JRaft-Request-Processor-3][LocalRaftMetaStorage] Save raft meta, path=build/work/ItNodeTest/testAppendEntriesWhenFollowerIsInErrorState_30809148389477213/int_taewfiies_5007/meta, term=3, votedFor=, cost time=1 ms [2024-01-30T00:06:30,211][INFO ][%int_taewfiies_5003%JRaft-FSMCaller-Disruptor-_stripe_0-0][StateMachineAdapter] onStopFollowing: LeaderChangeContext [leaderId=int_taewfiies_5005, term=2, status=Status[EHIGHERTERMRESPONSE<10008>: Raft node receives higher term RequestVoteRequest.]]. [2024-01-30T00:06:30,212][INFO ][%int_taewfiies_5004%JRaft-Request-Processor-1][LocalRaftMetaStorage] Save raft meta, path=build/work/ItNodeTest/testAppendEntriesWhenFollowerIsInErrorState_30809148389477213/int_taewfiies_5004/meta, term=3, votedFor=int_taewfiies_5006, cost time=1 ms [2024-01-30T00:06:30,212][INFO ][%int_taewfiies_5003%JRaft-Request-Processor-0][LocalRaftMetaStorage] Save raft meta, path=build/work/ItNodeTest/testAppendEntriesWhenFollowerIsInErrorState_30809148389477213/int_taewfiies_5003/meta, term=3, votedFor=, cost time=1 ms [2024-01-30T00:06:30,213][INFO ][%int_taewfiies_5007%JRaft-Request-Processor-3][LocalRaftMetaStorage] Save raft meta, path=build/work/ItNodeTest/testAppendEntriesWhenFollowerIsInErrorState_30809148389477213/int_taewfiies_5007/meta, term=3, votedFor=int_taewfiies_5006, cost time=0 ms [2024-01-30T00:06:30,213][INFO ][%int_taewfiies_5006%JRaft-Response-Processor-0][NodeImpl] Node become leader of group, term=3, conf=int_taewfiies_5003,int_taewfiies_5004,int_taewfiies_5005,int_taewfiies_5006,int_taewfiies_5007, oldConf=. [2024-01-30T00:06:30,213][INFO ][%int_taewfiies_5003%JRaft-Request-Processor-0][LocalRaftMetaStorage] Save raft meta, path=build/work/ItNodeTest/testAppendEntriesWhenFollowerIsInErrorState_30809148389477213/int_taewfiies_5003/meta, term=3, votedFor=int_taewfiies_5006, cost time=1 ms [2024-01-30T00:06:30,214][INFO ][%int_taewfiies_5005%JRaft-Response-Processor-1][LogThreadPoolExecutor] ThreadPool is terminated: %int_taewfiies_5005%JRaft-Response-Processor-, org.apache.ignite.raft.jraft.util.MetricThreadPoolExecutor@f71abcb[Shutting down, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 61]. [2024-01-30T00:06:30,215][INFO ][Test worker][RaftGroupService] Stop the RaftGroupService successfully. [2024-01-30T00:06:30,215][INFO ][Test worker][Cluster] [default:int_taewfiies_5005:3a0862258ff5467a@10.233.90.83:5005][doShutdown] Shutting down [2024-01-30T00:06:30,215][INFO ][Test worker][Cluster] [default:int_taewfiies_5005:3a0862258ff5467a@10.233.90.83:5005][leaveCluster] Leaving cluster [2024-01-30T00:06:30,220][INFO ][sc-cluster-5006-271][MembershipProtocol] [default:int_taewfiies_5006:59b4088774334760@10.233.90.83:5006][publishEvent] MembershipEvent[type=LEAVING, member=default:int_taewfiies_5005:3a0862258ff5467a@10.233.90.83:5005, oldMetadata=null, newMetadata=7e16449-5, timestamp=2024-01-30T00:06:30.220Z] [2024-01-30T00:06:30,221][INFO ][sc-cluster-5003-273][MembershipProtocol] [default:int_taewfiies_5003:2683329c8fe407e@10.233.90.83:5003][publishEvent] MembershipEvent[type=LEAVING, member=default:int_taewfiies_5005:3a0862258ff5467a@10.233.90.83:5005, oldMetadata=null, newMetadata=7e16449-5, timestamp=2024-01-30T00:06:30.220Z] [2024-01-30T00:06:30,221][INFO ][sc-cluster-5007-272][MembershipProtocol] [default:int_taewfiies_5007:1a48dc796fed49a5@10.233.90.83:5007][publishEvent] MembershipEvent[type=LEAVING, member=default:int_taewfiies_5005:3a0862258ff5467a@10.233.90.83:5005, oldMetadata=null, newMetadata=7e16449-5, timestamp=2024-01-30T00:06:30.220Z] [2024-01-30T00:06:30,221][INFO ][sc-cluster-5006-271][ScaleCubeTopologyService] Node left [member=ClusterNodeImpl [id=ba086225-8ff5-467a-9919-67782d28b656, name=int_taewfiies_5005, address=10.233.90.83:5005, nodeMetadata=null], eventType=LEAVING] [2024-01-30T00:06:30,221][INFO ][sc-cluster-5003-273][ScaleCubeTopologyService] Node left [member=ClusterNodeImpl [id=ba086225-8ff5-467a-9919-67782d28b656, name=int_taewfiies_5005, address=10.233.90.83:5005, nodeMetadata=null], eventType=LEAVING] [2024-01-30T00:06:30,221][INFO ][sc-cluster-5007-272][ScaleCubeTopologyService] Node left [member=ClusterNodeImpl [id=ba086225-8ff5-467a-9919-67782d28b656, name=int_taewfiies_5005, address=10.233.90.83:5005, nodeMetadata=null], eventType=LEAVING] [2024-01-30T00:06:30,221][INFO ][sc-cluster-5006-271][ScaleCubeTopologyService] Topology snapshot [nodes=[int_taewfiies_5007, int_taewfiies_5006, int_taewfiies_5003, int_taewfiies_5004]] [2024-01-30T00:06:30,221][INFO ][sc-cluster-5007-272][ScaleCubeTopologyService] Topology snapshot [nodes=[int_taewfiies_5007, int_taewfiies_5006, int_taewfiies_5003, int_taewfiies_5004]] [2024-01-30T00:06:30,222][INFO ][sc-cluster-5003-273][ScaleCubeTopologyService] Topology snapshot [nodes=[int_taewfiies_5007, int_taewfiies_5006, int_taewfiies_5003, int_taewfiies_5004]] [2024-01-30T00:06:30,290][INFO ][sc-cluster-5005-270][Cluster] [default:int_taewfiies_5005:3a0862258ff5467a@10.233.90.83:5005][leaveCluster] Left cluster [2024-01-30T00:06:30,290][INFO ][sc-cluster-5005-270][Transport] Stopping [address=10.233.90.83:5005] [2024-01-30T00:06:30,290][INFO ][sc-cluster-5005-270][Transport] Stopped [address=10.233.90.83:5005] [2024-01-30T00:06:30,291][INFO ][sc-cluster-5005-270][Cluster] [default:int_taewfiies_5005:3a0862258ff5467a@10.233.90.83:5005][doShutdown] Shutdown [2024-01-30T00:06:30,298][INFO ][%int_taewfiies_5006%JRaft-Request-Processor-3][LogThreadPoolExecutor] ThreadPool is terminated: %int_taewfiies_5006%JRaft-Request-Processor-, org.apache.ignite.raft.jraft.util.MetricThreadPoolExecutor@21267c89[Shutting down, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 15]. [00:06:30] : [org.apache.ignite.raft.jraft.core.ItNodeTest.testAppendEntriesWhenFollowerIsInErrorState()] [Test Error Output] [2024-01-30T00:06:26,290][WARN ][sc-cluster-5005-270][MembershipProtocol] [default:int_taewfiies_5005:3a0862258ff5467a@10.233.90.83:5005][updateMembership][SYNC] Skipping to add/update member: {m: default:int_taewfiies_5004:7d20e3de9a4145a6@10.233.90.83:5004, s: ALIVE, inc: 0}, due to failed fetchMetadata call (cause: java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 1000ms in 'source(MonoDefer)' (and no fallback has been configured)) [2024-01-30T00:06:26,369][ERROR][%int_taewfiies_5005%JRaft-StepDownTimer-1][AbstractClientService] Fail to connect int_taewfiies_5004, exception: java.net.ConnectException. [2024-01-30T00:06:26,369][ERROR][%int_taewfiies_5005%JRaft-StepDownTimer-1][ReplicatorGroupImpl] Fail to check replicator connection to peer=int_taewfiies_5004, replicatorType=Follower. [2024-01-30T00:06:26,393][WARN ][sc-cluster-5004-274][MetadataStore] [default:int_taewfiies_5004:7d20e3de9a4145a6@10.233.90.83:5004][aa5ebddd-63d3-4f09-8687-5bbea18592a8] Timeout getting GetMetadataResp from 10.233.90.83:5005 within 1000 ms, cause: java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 1000ms in 'source(MonoDefer)' (and no fallback has been configured) [2024-01-30T00:06:26,393][WARN ][sc-cluster-5004-274][MembershipProtocol] [default:int_taewfiies_5004:7d20e3de9a4145a6@10.233.90.83:5004][updateMembership][SYNC] Skipping to add/update member: {m: default:int_taewfiies_5005:3a0862258ff5467a@10.233.90.83:5005, s: ALIVE, inc: 0}, due to failed fetchMetadata call (cause: java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 1000ms in 'source(MonoDefer)' (and no fallback has been configured)) [2024-01-30T00:06:26,869][ERROR][%int_taewfiies_5005%JRaft-StepDownTimer-1][AbstractClientService] Fail to connect int_taewfiies_5004, exception: java.net.ConnectException. [2024-01-30T00:06:26,869][ERROR][%int_taewfiies_5005%JRaft-StepDownTimer-1][ReplicatorGroupImpl] Fail to check replicator connection to peer=int_taewfiies_5004, replicatorType=Follower. [2024-01-30T00:06:27,098][WARN ][sc-cluster-5004-274][MetadataStore] [default:int_taewfiies_5004:7d20e3de9a4145a6@10.233.90.83:5004][651bd07a-b2ba-4f83-9878-6038f9a99400] Timeout getting GetMetadataResp from 10.233.90.83:5005 within 1000 ms, cause: java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 1000ms in 'source(MonoDefer)' (and no fallback has been configured) [2024-01-30T00:06:27,098][WARN ][sc-cluster-5004-274][MembershipProtocol] [default:int_taewfiies_5004:7d20e3de9a4145a6@10.233.90.83:5004][updateMembership][SYNC] Skipping to add/update member: {m: default:int_taewfiies_5005:3a0862258ff5467a@10.233.90.83:5005, s: ALIVE, inc: 0}, due to failed fetchMetadata call (cause: java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 1000ms in 'source(MonoDefer)' (and no fallback has been configured)) [2024-01-30T00:06:27,168][WARN ][sc-cluster-5004-274][MetadataStore] [default:int_taewfiies_5004:7d20e3de9a4145a6@10.233.90.83:5004][91ed72c4-24da-41dd-9b14-ab09d7044a3b] Timeout getting GetMetadataResp from 10.233.90.83:5005 within 1000 ms, cause: java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 1000ms in 'source(MonoDefer)' (and no fallback has been configured) [2024-01-30T00:06:27,169][WARN ][sc-cluster-5004-274][MembershipProtocol] [default:int_taewfiies_5004:7d20e3de9a4145a6@10.233.90.83:5004][updateMembership][SYNC] Skipping to add/update member: {m: default:int_taewfiies_5005:3a0862258ff5467a@10.233.90.83:5005, s: ALIVE, inc: 0}, due to failed fetchMetadata call (cause: java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 1000ms in 'source(MonoDefer)' (and no fallback has been configured)) [2024-01-30T00:06:27,290][WARN ][sc-cluster-5005-270][MetadataStore] [default:int_taewfiies_5005:3a0862258ff5467a@10.233.90.83:5005][69e1b385-6e70-4c96-af89-2abab0f4032a] Timeout getting GetMetadataResp from 10.233.90.83:5004 within 1000 ms, cause: java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 1000ms in 'source(MonoDefer)' (and no fallback has been configured) [2024-01-30T00:06:27,290][WARN ][sc-cluster-5005-270][MembershipProtocol] [default:int_taewfiies_5005:3a0862258ff5467a@10.233.90.83:5005][updateMembership][SYNC] Skipping to add/update member: {m: default:int_taewfiies_5004:7d20e3de9a4145a6@10.233.90.83:5004, s: ALIVE, inc: 0}, due to failed fetchMetadata call (cause: java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 1000ms in 'source(MonoDefer)' (and no fallback has been configured)) [2024-01-30T00:06:27,370][ERROR][%int_taewfiies_5005%JRaft-StepDownTimer-1][AbstractClientService] Fail to connect int_taewfiies_5004, exception: java.net.ConnectException. [2024-01-30T00:06:27,370][ERROR][%int_taewfiies_5005%JRaft-StepDownTimer-1][ReplicatorGroupImpl] Fail to check replicator connection to peer=int_taewfiies_5004, replicatorType=Follower. [2024-01-30T00:06:27,393][WARN ][sc-cluster-5004-274][MetadataStore] [default:int_taewfiies_5004:7d20e3de9a4145a6@10.233.90.83:5004][7722d930-40be-4450-a9d7-7c8acc9939c9] Timeout getting GetMetadataResp from 10.233.90.83:5005 within 1000 ms, cause: java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 1000ms in 'source(MonoDefer)' (and no fallback has been configured) [2024-01-30T00:06:27,393][WARN ][sc-cluster-5004-274][MembershipProtocol] [default:int_taewfiies_5004:7d20e3de9a4145a6@10.233.90.83:5004][updateMembership][SYNC] Skipping to add/update member: {m: default:int_taewfiies_5005:3a0862258ff5467a@10.233.90.83:5005, s: ALIVE, inc: 0}, due to failed fetchMetadata call (cause: java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 1000ms in 'source(MonoDefer)' (and no fallback has been configured)) [2024-01-30T00:06:27,870][ERROR][%int_taewfiies_5005%JRaft-StepDownTimer-1][AbstractClientService] Fail to connect int_taewfiies_5004, exception: java.net.ConnectException. [2024-01-30T00:06:27,870][ERROR][%int_taewfiies_5005%JRaft-StepDownTimer-1][ReplicatorGroupImpl] Fail to check replicator connection to peer=int_taewfiies_5004, replicatorType=Follower. [2024-01-30T00:06:28,168][WARN ][sc-cluster-5004-274][MetadataStore] [default:int_taewfiies_5004:7d20e3de9a4145a6@10.233.90.83:5004][d573fae1-bee6-4ebf-8d45-c368c3d23e2b] Timeout getting GetMetadataResp from 10.233.90.83:5005 within 1000 ms, cause: java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 1000ms in 'source(MonoDefer)' (and no fallback has been configured) [2024-01-30T00:06:28,168][WARN ][sc-cluster-5004-274][MembershipProtocol] [default:int_taewfiies_5004:7d20e3de9a4145a6@10.233.90.83:5004][updateMembership][SYNC] Skipping to add/update member: {m: default:int_taewfiies_5005:3a0862258ff5467a@10.233.90.83:5005, s: ALIVE, inc: 0}, due to failed fetchMetadata call (cause: java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 1000ms in 'source(MonoDefer)' (and no fallback has been configured)) [2024-01-30T00:06:28,291][WARN ][sc-cluster-5005-270][MetadataStore] [default:int_taewfiies_5005:3a0862258ff5467a@10.233.90.83:5005][313e97f1-02f2-4bb2-9ab5-cfebc952f279] Timeout getting GetMetadataResp from 10.233.90.83:5004 within 1000 ms, cause: java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 1000ms in 'source(MonoDefer)' (and no fallback has been configured) [2024-01-30T00:06:28,291][WARN ][sc-cluster-5005-270][MembershipProtocol] [default:int_taewfiies_5005:3a0862258ff5467a@10.233.90.83:5005][updateMembership][SYNC] Skipping to add/update member: {m: default:int_taewfiies_5004:7d20e3de9a4145a6@10.233.90.83:5004, s: ALIVE, inc: 0}, due to failed fetchMetadata call (cause: java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 1000ms in 'source(MonoDefer)' (and no fallback has been configured)) [2024-01-30T00:06:28,370][ERROR][%int_taewfiies_5005%JRaft-StepDownTimer-1][AbstractClientService] Fail to connect int_taewfiies_5004, exception: java.net.ConnectException. [2024-01-30T00:06:28,371][ERROR][%int_taewfiies_5005%JRaft-StepDownTimer-1][ReplicatorGroupImpl] Fail to check replicator connection to peer=int_taewfiies_5004, replicatorType=Follower. [2024-01-30T00:06:28,871][ERROR][%int_taewfiies_5005%JRaft-StepDownTimer-1][AbstractClientService] Fail to connect int_taewfiies_5004, exception: java.net.ConnectException. [2024-01-30T00:06:28,871][ERROR][%int_taewfiies_5005%JRaft-StepDownTimer-1][ReplicatorGroupImpl] Fail to check replicator connection to peer=int_taewfiies_5004, replicatorType=Follower. [2024-01-30T00:06:29,168][WARN ][sc-cluster-5004-274][MetadataStore] [default:int_taewfiies_5004:7d20e3de9a4145a6@10.233.90.83:5004][d1b689a3-cc07-4577-b598-1f0f52edb042] Timeout getting GetMetadataResp from 10.233.90.83:5005 within 1000 ms, cause: java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 1000ms in 'source(MonoDefer)' (and no fallback has been configured) [2024-01-30T00:06:29,169][WARN ][sc-cluster-5004-274][MembershipProtocol] [default:int_taewfiies_5004:7d20e3de9a4145a6@10.233.90.83:5004][updateMembership][SYNC] Skipping to add/update member: {m: default:int_taewfiies_5005:3a0862258ff5467a@10.233.90.83:5005, s: ALIVE, inc: 0}, due to failed fetchMetadata call (cause: java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 1000ms in 'source(MonoDefer)' (and no fallback has been configured)) [2024-01-30T00:06:29,371][ERROR][%int_taewfiies_5005%JRaft-StepDownTimer-1][AbstractClientService] Fail to connect int_taewfiies_5004, exception: java.net.ConnectException. [2024-01-30T00:06:29,371][ERROR][%int_taewfiies_5005%JRaft-StepDownTimer-1][ReplicatorGroupImpl] Fail to check replicator connection to peer=int_taewfiies_5004, replicatorType=Follower. [2024-01-30T00:06:29,393][WARN ][sc-cluster-5005-270][MetadataStore] [default:int_taewfiies_5005:3a0862258ff5467a@10.233.90.83:5005][54b722f7-3cff-469c-8558-fd70a98604c8] Timeout getting GetMetadataResp from 10.233.90.83:5004 within 1000 ms, cause: java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 1000ms in 'source(MonoDefer)' (and no fallback has been configured) [2024-01-30T00:06:29,393][WARN ][sc-cluster-5005-270][MembershipProtocol] [default:int_taewfiies_5005:3a0862258ff5467a@10.233.90.83:5005][updateMembership][SYNC] Skipping to add/update member: {m: default:int_taewfiies_5004:7d20e3de9a4145a6@10.233.90.83:5004, s: ALIVE, inc: 0}, due to failed fetchMetadata call (cause: java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 1000ms in 'source(MonoDefer)' (and no fallback has been configured)) [2024-01-30T00:06:29,872][ERROR][%int_taewfiies_5005%JRaft-StepDownTimer-1][AbstractClientService] Fail to connect int_taewfiies_5004, exception: java.net.ConnectException. [2024-01-30T00:06:29,872][ERROR][%int_taewfiies_5005%JRaft-StepDownTimer-1][ReplicatorGroupImpl] Fail to check replicator connection to peer=int_taewfiies_5004, replicatorType=Follower. [2024-01-30T00:06:30,098][WARN ][sc-cluster-5005-270][MetadataStore] [default:int_taewfiies_5005:3a0862258ff5467a@10.233.90.83:5005][eaaa957c-89c4-4462-8f5e-e875bcf24274] Timeout getting GetMetadataResp from 10.233.90.83:5004 within 1000 ms, cause: java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 1000ms in 'source(MonoDefer)' (and no fallback has been configured) [2024-01-30T00:06:30,098][WARN ][sc-cluster-5005-270][MembershipProtocol] [default:int_taewfiies_5005:3a0862258ff5467a@10.233.90.83:5005][updateMembership][SYNC] Skipping to add/update member: {m: default:int_taewfiies_5004:7d20e3de9a4145a6@10.233.90.83:5004, s: ALIVE, inc: 0}, due to failed fetchMetadata call (cause: java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 1000ms in 'source(MonoDefer)' (and no fallback has been configured)) [2024-01-30T00:06:30,168][WARN ][sc-cluster-5004-274][MetadataStore] [default:int_taewfiies_5004:7d20e3de9a4145a6@10.233.90.83:5004][151e9043-eb21-4ac3-9e1e-fdcb29554b50] Timeout getting GetMetadataResp from 10.233.90.83:5005 within 1000 ms, cause: java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 1000ms in 'source(MonoDefer)' (and no fallback has been configured) [2024-01-30T00:06:30,168][WARN ][sc-cluster-5004-274][MembershipProtocol] [default:int_taewfiies_5004:7d20e3de9a4145a6@10.233.90.83:5004][updateMembership][SYNC] Skipping to add/update member: {m: default:int_taewfiies_5005:3a0862258ff5467a@10.233.90.83:5005, s: ALIVE, inc: 0}, due to failed fetchMetadata call (cause: java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 1000ms in 'source(MonoDefer)' (and no fallback has been configured)) [2024-01-30T00:06:30,205][WARN ][%int_taewfiies_5005%JRaft-Common-Executor-0][Replicator] Replicator Replicator [state=Replicate, statInfo=, peerId=int_taewfiies_5006, type=Follower] stops sending entries. [2024-01-30T00:06:30,206][WARN ][%int_taewfiies_5005%MessagingService-inbound-0-0][IgniteRpcServer] A request execution was rejected [sender=ClusterNodeImpl [id=59b40887-7433-4760-9420-837a02c05cb4, name=int_taewfiies_5006, address=10.233.90.83:5006, nodeMetadata=null] req=PingRequestImpl [sendTimestamp=1706573190206] reason=Task org.apache.ignite.raft.jraft.rpc.impl.IgniteRpcServer$RpcMessageHandler$$Lambda$1384/0x0000000800775c40@10d270c0 rejected from org.apache.ignite.raft.jraft.util.MetricThreadPoolExecutor@5f3c7d81[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 4]] [2024-01-30T00:06:30,214][WARN ][%int_taewfiies_5005%MessagingService-inbound-0-0][IgniteRpcServer] A request execution was rejected [sender=ClusterNodeImpl [id=59b40887-7433-4760-9420-837a02c05cb4, name=int_taewfiies_5006, address=10.233.90.83:5006, nodeMetadata=null] req=PingRequestImpl [sendTimestamp=1706573190213] reason=Task org.apache.ignite.raft.jraft.rpc.impl.IgniteRpcServer$RpcMessageHandler$$Lambda$1384/0x0000000800775c40@7144559 rejected from org.apache.ignite.raft.jraft.util.MetricThreadPoolExecutor@5f3c7d81[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 4]] [2024-01-30T00:06:30,547][ERROR][%int_taewfiies_5006%JRaft-Response-Processor-0][AbstractClientService] Fail to connect int_taewfiies_5005, exception: java.util.concurrent.TimeoutException. [00:06:31] : [org.apache.ignite.raft.jraft.core.ItNodeTest.testAppendEntriesWhenFollowerIsInErrorState()] [Test Output] [2024-01-30T00:06:30,547][INFO ][%int_taewfiies_5006%JRaft-Response-Processor-0][Replicator] Replicator=Replicator [state=Created, statInfo=, peerId=int_taewfiies_5007, type=Follower]@int_taewfiies_5007 is started [2024-01-30T00:06:30,547][INFO ][%int_taewfiies_5006%JRaft-Response-Processor-0][Replicator] Replicator=Replicator [state=Created, statInfo=, peerId=int_taewfiies_5003, type=Follower]@int_taewfiies_5003 is started [2024-01-30T00:06:30,547][INFO ][%int_taewfiies_5006%JRaft-Response-Processor-0][Replicator] Replicator=Replicator [state=Created, statInfo=, peerId=int_taewfiies_5004, type=Follower]@int_taewfiies_5004 is started [2024-01-30T00:06:30,548][INFO ][Test worker][NodeImpl] Node shutdown, currTerm=3 state=STATE_LEADER. [2024-01-30T00:06:30,548][INFO ][%int_taewfiies_5003%JRaft-FSMCaller-Disruptor-_stripe_0-0][StateMachineAdapter] onStartFollowing: LeaderChangeContext [leaderId=int_taewfiies_5006, term=3, status=Status[ENEWLEADER<10011>: Follower receives message from new leader with the same term.]]. [2024-01-30T00:06:30,549][INFO ][%int_taewfiies_5007%JRaft-FSMCaller-Disruptor-_stripe_4-0][StateMachineAdapter] onStartFollowing: LeaderChangeContext [leaderId=int_taewfiies_5006, term=3, status=Status[ENEWLEADER<10011>: Follower receives message from new leader with the same term.]]. [2024-01-30T00:06:30,549][INFO ][%int_taewfiies_5004%JRaft-AppendEntries-Processor-0][NodeImpl] Election timeout was reset to initial value due to successful leader election. [2024-01-30T00:06:30,549][INFO ][%int_taewfiies_5006%JRaft-FSMCaller-Disruptor-_stripe_3-0][StateMachineAdapter] onLeaderStop: status=Status[ESHUTDOWN<1007>: Raft node is going to quit.]. [2024-01-30T00:06:30,549][INFO ][%int_taewfiies_5004%JRaft-AppendEntries-Processor-0][NodeImpl] Node reset electionTimeout, currTimer 3 state STATE_FOLLOWER new electionTimeout 1000. [2024-01-30T00:06:30,549][INFO ][%int_taewfiies_5004%JRaft-FSMCaller-Disruptor-_stripe_1-0][StateMachineAdapter] onStartFollowing: LeaderChangeContext [leaderId=int_taewfiies_5006, term=3, status=Status[ENEWLEADER<10011>: Follower receives message from new leader with the same term.]]. [2024-01-30T00:06:30,549][INFO ][Test worker][Replicator] Replicator Replicator [state=Probe, statInfo=, peerId=int_taewfiies_5003, type=Follower] is going to quit [2024-01-30T00:06:30,549][INFO ][Test worker][Replicator] Replicator Replicator [state=Probe, statInfo=, peerId=int_taewfiies_5004, type=Follower] is going to quit [2024-01-30T00:06:30,550][INFO ][%int_taewfiies_5007%JRaft-Request-Processor-0][NodeImpl] Node received TimeoutNowRequest from int_taewfiies_5006, term=3 and starts voting. [2024-01-30T00:06:30,550][INFO ][%int_taewfiies_5007%JRaft-Request-Processor-0][NodeImpl] Node start vote and grant vote self, term=3. [2024-01-30T00:06:30,551][INFO ][%int_taewfiies_5007%JRaft-FSMCaller-Disruptor-_stripe_4-0][StateMachineAdapter] onStopFollowing: LeaderChangeContext [leaderId=int_taewfiies_5006, term=3, status=Status[ERAFTTIMEDOUT<10001>: A follower's leader_id is reset to NULL as it begins to request_vote.]]. [2024-01-30T00:06:30,551][INFO ][%int_taewfiies_5006%JRaft-Response-Processor-2][Replicator] Replicator Replicator [state=Replicate, statInfo=, peerId=int_taewfiies_5007, type=Follower] is going to quit [2024-01-30T00:06:30,551][INFO ][Test worker][LocalRaftMetaStorage] Save raft meta, path=build/work/ItNodeTest/testAppendEntriesWhenFollowerIsInErrorState_30809148389477213/int_taewfiies_5006/meta, term=3, votedFor=int_taewfiies_5006, cost time=1 ms [2024-01-30T00:06:30,551][INFO ][Test worker][FSMCallerImpl] Shutting down FSMCaller... [2024-01-30T00:06:30,551][INFO ][%int_taewfiies_5006%JRaft-FSMCaller-Disruptor-_stripe_3-0][StateMachineAdapter] onShutdown. [2024-01-30T00:06:30,552][INFO ][%int_taewfiies_5006%JRaft-Node-Scheduler-2][LogScheduledThreadPoolExecutor] ThreadPool is terminated: %int_taewfiies_5006%JRaft-Node-Scheduler-, org.apache.ignite.raft.jraft.util.MetricScheduledThreadPoolExecutor@6f69d59f[Shutting down, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 25]. [2024-01-30T00:06:30,552][INFO ][%int_taewfiies_5007%JRaft-Request-Processor-0][LocalRaftMetaStorage] Save raft meta, path=build/work/ItNodeTest/testAppendEntriesWhenFollowerIsInErrorState_30809148389477213/int_taewfiies_5007/meta, term=4, votedFor=int_taewfiies_5007, cost time=1 ms [2024-01-30T00:06:30,552][INFO ][%int_taewfiies_5006%JRaft-ElectionTimer-4][LogScheduledThreadPoolExecutor] ThreadPool is terminated: %int_taewfiies_5006%JRaft-ElectionTimer-, org.apache.ignite.raft.jraft.util.MetricScheduledThreadPoolExecutor@295cc9b2[Shutting down, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 21]. [2024-01-30T00:06:30,553][INFO ][%int_taewfiies_5006%JRaft-VoteTimer-0][LogScheduledThreadPoolExecutor] ThreadPool is terminated: %int_taewfiies_5006%JRaft-VoteTimer-, org.apache.ignite.raft.jraft.util.MetricScheduledThreadPoolExecutor@4f439e70[Shutting down, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0]. [2024-01-30T00:06:30,554][INFO ][%int_taewfiies_5004%JRaft-Request-Processor-3][NodeImpl] Node received RequestVoteRequest from int_taewfiies_5007, term=4, currTerm=3. [2024-01-30T00:06:30,554][INFO ][%int_taewfiies_5004%JRaft-FSMCaller-Disruptor-_stripe_1-0][StateMachineAdapter] onStopFollowing: LeaderChangeContext [leaderId=int_taewfiies_5006, term=3, status=Status[EHIGHERTERMRESPONSE<10008>: Raft node receives higher term RequestVoteRequest.]]. [2024-01-30T00:06:30,554][INFO ][%int_taewfiies_5006%JRaft-StepDownTimer-0][LogScheduledThreadPoolExecutor] ThreadPool is terminated: %int_taewfiies_5006%JRaft-StepDownTimer-, org.apache.ignite.raft.jraft.util.MetricScheduledThreadPoolExecutor@30d28144[Shutting down, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0]. [2024-01-30T00:06:30,555][INFO ][%int_taewfiies_5006%JRaft-SnapshotTimer-0][LogScheduledThreadPoolExecutor] ThreadPool is terminated: %int_taewfiies_5006%JRaft-SnapshotTimer-, org.apache.ignite.raft.jraft.util.MetricScheduledThreadPoolExecutor@1f4e9246[Shutting down, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0]. [2024-01-30T00:06:30,555][INFO ][%int_taewfiies_5006%JRaft-Common-Executor-1][LogThreadPoolExecutor] ThreadPool is terminated: %int_taewfiies_5006%JRaft-Common-Executor-, org.apache.ignite.raft.jraft.util.MetricThreadPoolExecutor@8499f21[Shutting down, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 6]. [2024-01-30T00:06:30,555][INFO ][%int_taewfiies_5004%JRaft-Request-Processor-3][LocalRaftMetaStorage] Save raft meta, path=build/work/ItNodeTest/testAppendEntriesWhenFollowerIsInErrorState_30809148389477213/int_taewfiies_5004/meta, term=4, votedFor=, cost time=1 ms [2024-01-30T00:06:30,556][INFO ][%int_taewfiies_5003%JRaft-Request-Processor-2][NodeImpl] Node received RequestVoteRequest from int_taewfiies_5007, term=4, currTerm=3. [2024-01-30T00:06:30,556][INFO ][%int_taewfiies_5003%JRaft-FSMCaller-Disruptor-_stripe_0-0][StateMachineAdapter] onStopFollowing: LeaderChangeContext [leaderId=int_taewfiies_5006, term=3, status=Status[EHIGHERTERMRESPONSE<10008>: Raft node receives higher term RequestVoteRequest.]]. [2024-01-30T00:06:30,556][INFO ][%int_taewfiies_5004%JRaft-Request-Processor-3][LocalRaftMetaStorage] Save raft meta, path=build/work/ItNodeTest/testAppendEntriesWhenFollowerIsInErrorState_30809148389477213/int_taewfiies_5004/meta, term=4, votedFor=int_taewfiies_5007, cost time=1 ms [2024-01-30T00:06:30,557][INFO ][%int_taewfiies_5003%JRaft-Request-Processor-2][LocalRaftMetaStorage] Save raft meta, path=build/work/ItNodeTest/testAppendEntriesWhenFollowerIsInErrorState_30809148389477213/int_taewfiies_5003/meta, term=4, votedFor=, cost time=1 ms [2024-01-30T00:06:30,558][INFO ][%int_taewfiies_5003%JRaft-Request-Processor-2][LocalRaftMetaStorage] Save raft meta, path=build/work/ItNodeTest/testAppendEntriesWhenFollowerIsInErrorState_30809148389477213/int_taewfiies_5003/meta, term=4, votedFor=int_taewfiies_5007, cost time=1 ms [2024-01-30T00:06:30,558][INFO ][%int_taewfiies_5007%JRaft-Response-Processor-0][NodeImpl] Node become leader of group, term=4, conf=int_taewfiies_5003,int_taewfiies_5004,int_taewfiies_5005,int_taewfiies_5006,int_taewfiies_5007, oldConf=. [2024-01-30T00:06:30,559][INFO ][%int_taewfiies_5006%JRaft-Response-Processor-2][LogThreadPoolExecutor] ThreadPool is terminated: %int_taewfiies_5006%JRaft-Response-Processor-, org.apache.ignite.raft.jraft.util.MetricThreadPoolExecutor@35719d4c[Shutting down, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 9]. [2024-01-30T00:06:30,560][INFO ][Test worker][RaftGroupService] Stop the RaftGroupService successfully. [2024-01-30T00:06:30,560][INFO ][Test worker][Cluster] [default:int_taewfiies_5006:59b4088774334760@10.233.90.83:5006][doShutdown] Shutting down [2024-01-30T00:06:30,560][INFO ][Test worker][Cluster] [default:int_taewfiies_5006:59b4088774334760@10.233.90.83:5006][leaveCluster] Leaving cluster [2024-01-30T00:06:30,569][INFO ][sc-cluster-5003-273][MembershipProtocol] [default:int_taewfiies_5003:2683329c8fe407e@10.233.90.83:5003][publishEvent] MembershipEvent[type=LEAVING, member=default:int_taewfiies_5006:59b4088774334760@10.233.90.83:5006, oldMetadata=null, newMetadata=7e16449-5, timestamp=2024-01-30T00:06:30.569Z] [2024-01-30T00:06:30,569][INFO ][sc-cluster-5004-274][MembershipProtocol] [default:int_taewfiies_5004:7d20e3de9a4145a6@10.233.90.83:5004][publishEvent] MembershipEvent[type=LEAVING, member=default:int_taewfiies_5006:59b4088774334760@10.233.90.83:5006, oldMetadata=null, newMetadata=7e16449-5, timestamp=2024-01-30T00:06:30.569Z] [2024-01-30T00:06:30,569][INFO ][sc-cluster-5003-273][ScaleCubeTopologyService] Node left [member=ClusterNodeImpl [id=59b40887-7433-4760-9420-837a02c05cb4, name=int_taewfiies_5006, address=10.233.90.83:5006, nodeMetadata=null], eventType=LEAVING] [2024-01-30T00:06:30,569][INFO ][sc-cluster-5004-274][ScaleCubeTopologyService] Node left [member=ClusterNodeImpl [id=59b40887-7433-4760-9420-837a02c05cb4, name=int_taewfiies_5006, address=10.233.90.83:5006, nodeMetadata=null], eventType=LEAVING] [2024-01-30T00:06:30,569][INFO ][sc-cluster-5003-273][ScaleCubeTopologyService] Topology snapshot [nodes=[int_taewfiies_5007, int_taewfiies_5003, int_taewfiies_5004]] [2024-01-30T00:06:30,569][INFO ][sc-cluster-5004-274][ScaleCubeTopologyService] Topology snapshot [nodes=[int_taewfiies_5007, int_taewfiies_5003, int_taewfiies_5004]] [2024-01-30T00:06:30,578][INFO ][sc-cluster-5007-272][MembershipProtocol] [default:int_taewfiies_5007:1a48dc796fed49a5@10.233.90.83:5007][publishEvent] MembershipEvent[type=LEAVING, member=default:int_taewfiies_5006:59b4088774334760@10.233.90.83:5006, oldMetadata=null, newMetadata=7e16449-5, timestamp=2024-01-30T00:06:30.578Z] [2024-01-30T00:06:30,578][INFO ][sc-cluster-5007-272][ScaleCubeTopologyService] Node left [member=ClusterNodeImpl [id=59b40887-7433-4760-9420-837a02c05cb4, name=int_taewfiies_5006, address=10.233.90.83:5006, nodeMetadata=null], eventType=LEAVING] [2024-01-30T00:06:30,578][INFO ][sc-cluster-5007-272][ScaleCubeTopologyService] Topology snapshot [nodes=[int_taewfiies_5007, int_taewfiies_5003, int_taewfiies_5004]] [2024-01-30T00:06:30,638][INFO ][sc-cluster-5006-271][Cluster] [default:int_taewfiies_5006:59b4088774334760@10.233.90.83:5006][leaveCluster] Left cluster [2024-01-30T00:06:30,639][INFO ][sc-cluster-5006-271][Transport] Stopping [address=10.233.90.83:5006] [2024-01-30T00:06:30,639][INFO ][sc-cluster-5006-271][Transport] Stopped [address=10.233.90.83:5006] [2024-01-30T00:06:30,639][INFO ][sc-cluster-5006-271][Cluster] [default:int_taewfiies_5006:59b4088774334760@10.233.90.83:5006][doShutdown] Shutdown [2024-01-30T00:06:30,645][INFO ][%int_taewfiies_5007%JRaft-Request-Processor-3][LogThreadPoolExecutor] ThreadPool is terminated: %int_taewfiies_5007%JRaft-Request-Processor-, org.apache.ignite.raft.jraft.util.MetricThreadPoolExecutor@15203660[Shutting down, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 17]. [2024-01-30T00:06:30,893][INFO ][%int_taewfiies_5007%JRaft-Response-Processor-0][Replicator] Replicator=Replicator [state=Created, statInfo=, peerId=int_taewfiies_5003, type=Follower]@int_taewfiies_5003 is started [2024-01-30T00:06:30,893][INFO ][%int_taewfiies_5007%JRaft-Response-Processor-0][Replicator] Replicator=Replicator [state=Created, statInfo=, peerId=int_taewfiies_5004, type=Follower]@int_taewfiies_5004 is started [2024-01-30T00:06:30,893][INFO ][Test worker][NodeImpl] Node shutdown, currTerm=4 state=STATE_LEADER. [2024-01-30T00:06:30,894][INFO ][%int_taewfiies_5003%JRaft-FSMCaller-Disruptor-_stripe_0-0][StateMachineAdapter] onStartFollowing: LeaderChangeContext [leaderId=int_taewfiies_5007, term=4, status=Status[ENEWLEADER<10011>: Follower receives message from new leader with the same term.]]. [2024-01-30T00:06:30,894][INFO ][%int_taewfiies_5007%JRaft-FSMCaller-Disruptor-_stripe_4-0][StateMachineAdapter] onLeaderStop: status=Status[ESHUTDOWN<1007>: Raft node is going to quit.]. [2024-01-30T00:06:30,894][INFO ][%int_taewfiies_5004%JRaft-FSMCaller-Disruptor-_stripe_1-0][StateMachineAdapter] onStartFollowing: LeaderChangeContext [leaderId=int_taewfiies_5007, term=4, status=Status[ENEWLEADER<10011>: Follower receives message from new leader with the same term.]]. [2024-01-30T00:06:30,894][INFO ][Test worker][Replicator] Replicator Replicator [state=Probe, statInfo=, peerId=int_taewfiies_5004, type=Follower] is going to quit [2024-01-30T00:06:30,895][INFO ][%int_taewfiies_5003%JRaft-Request-Processor-3][NodeImpl] Node received TimeoutNowRequest from int_taewfiies_5007, term=4 and starts voting. [2024-01-30T00:06:30,895][INFO ][%int_taewfiies_5003%JRaft-Request-Processor-3][NodeImpl] Node start vote and grant vote self, term=4. [2024-01-30T00:06:30,895][INFO ][%int_taewfiies_5003%JRaft-FSMCaller-Disruptor-_stripe_0-0][StateMachineAdapter] onStopFollowing: LeaderChangeContext [leaderId=int_taewfiies_5007, term=4, status=Status[ERAFTTIMEDOUT<10001>: A follower's leader_id is reset to NULL as it begins to request_vote.]]. [2024-01-30T00:06:30,895][INFO ][%int_taewfiies_5007%JRaft-Response-Processor-1][Replicator] Replicator Replicator [state=Replicate, statInfo=, peerId=int_taewfiies_5003, type=Follower] is going to quit [2024-01-30T00:06:30,896][INFO ][Test worker][LocalRaftMetaStorage] Save raft meta, path=build/work/ItNodeTest/testAppendEntriesWhenFollowerIsInErrorState_30809148389477213/int_taewfiies_5007/meta, term=4, votedFor=int_taewfiies_5007, cost time=1 ms [2024-01-30T00:06:30,896][INFO ][Test worker][FSMCallerImpl] Shutting down FSMCaller... [2024-01-30T00:06:30,896][INFO ][%int_taewfiies_5007%JRaft-FSMCaller-Disruptor-_stripe_4-0][StateMachineAdapter] onShutdown. [2024-01-30T00:06:30,897][INFO ][%int_taewfiies_5003%JRaft-Request-Processor-3][LocalRaftMetaStorage] Save raft meta, path=build/work/ItNodeTest/testAppendEntriesWhenFollowerIsInErrorState_30809148389477213/int_taewfiies_5003/meta, term=5, votedFor=int_taewfiies_5003, cost time=1 ms [2024-01-30T00:06:30,897][INFO ][Test worker][LogScheduledThreadPoolExecutor] ThreadPool is terminated: %int_taewfiies_5007%JRaft-Node-Scheduler-, org.apache.ignite.raft.jraft.util.MetricScheduledThreadPoolExecutor@6fcf2ddd[Shutting down, pool size = 8, active threads = 0, queued tasks = 0, completed tasks = 25]. [2024-01-30T00:06:30,897][INFO ][%int_taewfiies_5004%JRaft-Request-Processor-1][NodeImpl] Node received RequestVoteRequest from int_taewfiies_5003, term=5, currTerm=4. [2024-01-30T00:06:30,898][INFO ][%int_taewfiies_5004%JRaft-FSMCaller-Disruptor-_stripe_1-0][StateMachineAdapter] onStopFollowing: LeaderChangeContext [leaderId=int_taewfiies_5007, term=4, status=Status[EHIGHERTERMRESPONSE<10008>: Raft node receives higher term RequestVoteRequest.]]. [2024-01-30T00:06:30,898][INFO ][%int_taewfiies_5007%JRaft-ElectionTimer-7][LogScheduledThreadPoolExecutor] ThreadPool is terminated: %int_taewfiies_5007%JRaft-ElectionTimer-, org.apache.ignite.raft.jraft.util.MetricScheduledThreadPoolExecutor@2dfb1480[Shutting down, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 22]. [2024-01-30T00:06:30,898][INFO ][%int_taewfiies_5007%JRaft-VoteTimer-0][LogScheduledThreadPoolExecutor] ThreadPool is terminated: %int_taewfiies_5007%JRaft-VoteTimer-, org.apache.ignite.raft.jraft.util.MetricScheduledThreadPoolExecutor@6cd5454c[Shutting down, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0]. [2024-01-30T00:06:30,898][INFO ][%int_taewfiies_5004%JRaft-Request-Processor-1][LocalRaftMetaStorage] Save raft meta, path=build/work/ItNodeTest/testAppendEntriesWhenFollowerIsInErrorState_30809148389477213/int_taewfiies_5004/meta, term=5, votedFor=, cost time=1 ms [2024-01-30T00:06:30,898][INFO ][%int_taewfiies_5007%JRaft-StepDownTimer-0][LogScheduledThreadPoolExecutor] ThreadPool is terminated: %int_taewfiies_5007%JRaft-StepDownTimer-, org.apache.ignite.raft.jraft.util.MetricScheduledThreadPoolExecutor@34c49a08[Shutting down, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0]. [2024-01-30T00:06:30,899][INFO ][%int_taewfiies_5007%JRaft-SnapshotTimer-0][LogScheduledThreadPoolExecutor] ThreadPool is terminated: %int_taewfiies_5007%JRaft-SnapshotTimer-, org.apache.ignite.raft.jraft.util.MetricScheduledThreadPoolExecutor@babdac[Shutting down, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0]. [2024-01-30T00:06:30,899][INFO ][%int_taewfiies_5007%JRaft-Common-Executor-2][LogThreadPoolExecutor] ThreadPool is terminated: %int_taewfiies_5007%JRaft-Common-Executor-, org.apache.ignite.raft.jraft.util.MetricThreadPoolExecutor@126cf4dc[Shutting down, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 6]. [2024-01-30T00:06:30,899][INFO ][%int_taewfiies_5004%JRaft-Request-Processor-1][LocalRaftMetaStorage] Save raft meta, path=build/work/ItNodeTest/testAppendEntriesWhenFollowerIsInErrorState_30809148389477213/int_taewfiies_5004/meta, term=5, votedFor=int_taewfiies_5003, cost time=1 ms [2024-01-30T00:06:30,903][INFO ][%int_taewfiies_5007%JRaft-Response-Processor-0][LogThreadPoolExecutor] ThreadPool is terminated: %int_taewfiies_5007%JRaft-Response-Processor-, org.apache.ignite.raft.jraft.util.MetricThreadPoolExecutor@584f4cf[Shutting down, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 9]. [2024-01-30T00:06:30,904][INFO ][Test worker][RaftGroupService] Stop the RaftGroupService successfully. [2024-01-30T00:06:30,904][INFO ][Test worker][Cluster] [default:int_taewfiies_5007:1a48dc796fed49a5@10.233.90.83:5007][doShutdown] Shutting down [2024-01-30T00:06:30,904][INFO ][Test worker][Cluster] [default:int_taewfiies_5007:1a48dc796fed49a5@10.233.90.83:5007][leaveCluster] Leaving cluster [00:06:31] : [org.apache.ignite.raft.jraft.core.ItNodeTest.testAppendEntriesWhenFollowerIsInErrorState()] [Test Error Output] [2024-01-30T00:06:30,547][ERROR][%int_taewfiies_5006%JRaft-Response-Processor-0][ReplicatorGroupImpl] Fail to check replicator connection to peer=int_taewfiies_5005, replicatorType=Follower. [2024-01-30T00:06:30,547][ERROR][%int_taewfiies_5006%JRaft-Response-Processor-0][NodeImpl] Fail to add a replicator, peer=int_taewfiies_5005. [2024-01-30T00:06:30,548][WARN ][%int_taewfiies_5006%JRaft-Response-Processor-1][NodeImpl] Node received invalid RequestVoteResponse from int_taewfiies_5003, state not in STATE_CANDIDATE but STATE_LEADER. [2024-01-30T00:06:30,549][ERROR][%int_taewfiies_5006%JRaft-Common-Executor-0][NodeImpl] Fail to run ConfigurationChangeDone, status: Status[EPERM<1008>: Leader stepped down]. [2024-01-30T00:06:30,549][WARN ][%int_taewfiies_5004%JRaft-AppendEntries-Processor-0][NodeImpl] Node reject term_unmatched AppendEntriesRequest from int_taewfiies_5006, term=3, prevLogIndex=22, prevLogTerm=2, localPrevLogTerm=0, lastLogIndex=0, entriesSize=0. [2024-01-30T00:06:30,550][WARN ][%int_taewfiies_5006%JRaft-Common-Executor-2][Replicator] Replicator Replicator [state=Replicate, statInfo=, peerId=int_taewfiies_5007, type=Follower] stops sending entries. [2024-01-30T00:06:30,551][WARN ][%int_taewfiies_5006%MessagingService-inbound-0-0][IgniteRpcServer] A request execution was rejected [sender=ClusterNodeImpl [id=9a48dc79-6fed-49a5-99da-fc4b1ec1f94a, name=int_taewfiies_5007, address=10.233.90.83:5007, nodeMetadata=null] req=PingRequestImpl [sendTimestamp=1706573190551] reason=Task org.apache.ignite.raft.jraft.rpc.impl.IgniteRpcServer$RpcMessageHandler$$Lambda$1384/0x0000000800775c40@7c0df6b4 rejected from org.apache.ignite.raft.jraft.util.MetricThreadPoolExecutor@21267c89[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 15]] [2024-01-30T00:06:30,559][ERROR][%int_taewfiies_5007%JRaft-Response-Processor-0][AbstractClientService] Fail to connect int_taewfiies_5005, exception: java.net.ConnectException. [2024-01-30T00:06:30,559][ERROR][%int_taewfiies_5007%JRaft-Response-Processor-0][ReplicatorGroupImpl] Fail to check replicator connection to peer=int_taewfiies_5005, replicatorType=Follower. [2024-01-30T00:06:30,559][ERROR][%int_taewfiies_5007%JRaft-Response-Processor-0][NodeImpl] Fail to add a replicator, peer=int_taewfiies_5005. [2024-01-30T00:06:30,559][WARN ][%int_taewfiies_5006%MessagingService-inbound-0-0][IgniteRpcServer] A request execution was rejected [sender=ClusterNodeImpl [id=9a48dc79-6fed-49a5-99da-fc4b1ec1f94a, name=int_taewfiies_5007, address=10.233.90.83:5007, nodeMetadata=null] req=PingRequestImpl [sendTimestamp=1706573190559] reason=Task org.apache.ignite.raft.jraft.rpc.impl.IgniteRpcServer$RpcMessageHandler$$Lambda$1384/0x0000000800775c40@58a32d48 rejected from org.apache.ignite.raft.jraft.util.MetricThreadPoolExecutor@21267c89[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 15]] [2024-01-30T00:06:30,892][ERROR][%int_taewfiies_5007%JRaft-Response-Processor-0][AbstractClientService] Fail to connect int_taewfiies_5006, exception: java.util.concurrent.TimeoutException. [2024-01-30T00:06:30,892][ERROR][%int_taewfiies_5007%JRaft-Response-Processor-0][ReplicatorGroupImpl] Fail to check replicator connection to peer=int_taewfiies_5006, replicatorType=Follower. [2024-01-30T00:06:30,893][ERROR][%int_taewfiies_5007%JRaft-Response-Processor-0][NodeImpl] Fail to add a replicator, peer=int_taewfiies_5006. [2024-01-30T00:06:30,894][WARN ][%int_taewfiies_5003%JRaft-AppendEntries-Processor-2][NodeImpl] Node reject term_unmatched AppendEntriesRequest from int_taewfiies_5007, term=4, prevLogIndex=23, prevLogTerm=3, localPrevLogTerm=0, lastLogIndex=22, entriesSize=0. [2024-01-30T00:06:30,894][WARN ][%int_taewfiies_5004%JRaft-AppendEntries-Processor-1][NodeImpl] Node reject term_unmatched AppendEntriesRequest from int_taewfiies_5007, term=4, prevLogIndex=23, prevLogTerm=3, localPrevLogTerm=0, lastLogIndex=0, entriesSize=0. [2024-01-30T00:06:30,894][ERROR][%int_taewfiies_5007%JRaft-Common-Executor-0][NodeImpl] Fail to run ConfigurationChangeDone, status: Status[EPERM<1008>: Leader stepped down]. [2024-01-30T00:06:30,896][WARN ][%int_taewfiies_5007%MessagingService-inbound-0-0][IgniteRpcServer] A request execution was rejected [sender=ClusterNodeImpl [id=02683329-c8fe-407e-afee-1c81eceaa9a0, name=int_taewfiies_5003, address=10.233.90.83:5003, nodeMetadata=null] req=PingRequestImpl [sendTimestamp=1706573190896] reason=Task org.apache.ignite.raft.jraft.rpc.impl.IgniteRpcServer$RpcMessageHandler$$Lambda$1384/0x0000000800775c40@2e6c1342 rejected from org.apache.ignite.raft.jraft.util.MetricThreadPoolExecutor@15203660[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 17]] [2024-01-30T00:06:30,897][WARN ][%int_taewfiies_5003%JRaft-AppendEntries-Processor-2][NodeImpl] Node ignore stale AppendEntriesRequest from int_taewfiies_5007, term=4, currTerm=5. [00:06:31] : [org.apache.ignite.raft.jraft.core.ItNodeTest.testAppendEntriesWhenFollowerIsInErrorState()] [Test Output] [2024-01-30T00:06:30,908][INFO ][sc-cluster-5004-274][MembershipProtocol] [default:int_taewfiies_5004:7d20e3de9a4145a6@10.233.90.83:5004][publishEvent] MembershipEvent[type=LEAVING, member=default:int_taewfiies_5007:1a48dc796fed49a5@10.233.90.83:5007, oldMetadata=null, newMetadata=7e16449-5, timestamp=2024-01-30T00:06:30.908Z] [2024-01-30T00:06:30,908][INFO ][sc-cluster-5003-273][MembershipProtocol] [default:int_taewfiies_5003:2683329c8fe407e@10.233.90.83:5003][publishEvent] MembershipEvent[type=LEAVING, member=default:int_taewfiies_5007:1a48dc796fed49a5@10.233.90.83:5007, oldMetadata=null, newMetadata=7e16449-5, timestamp=2024-01-30T00:06:30.908Z] [2024-01-30T00:06:30,908][INFO ][sc-cluster-5004-274][ScaleCubeTopologyService] Node left [member=ClusterNodeImpl [id=9a48dc79-6fed-49a5-99da-fc4b1ec1f94a, name=int_taewfiies_5007, address=10.233.90.83:5007, nodeMetadata=null], eventType=LEAVING] [2024-01-30T00:06:30,908][INFO ][sc-cluster-5003-273][ScaleCubeTopologyService] Node left [member=ClusterNodeImpl [id=9a48dc79-6fed-49a5-99da-fc4b1ec1f94a, name=int_taewfiies_5007, address=10.233.90.83:5007, nodeMetadata=null], eventType=LEAVING] [2024-01-30T00:06:30,908][INFO ][sc-cluster-5004-274][ScaleCubeTopologyService] Topology snapshot [nodes=[int_taewfiies_5003, int_taewfiies_5004]] [2024-01-30T00:06:30,908][INFO ][sc-cluster-5003-273][ScaleCubeTopologyService] Topology snapshot [nodes=[int_taewfiies_5003, int_taewfiies_5004]] [2024-01-30T00:06:30,977][INFO ][sc-cluster-5007-272][Cluster] [default:int_taewfiies_5007:1a48dc796fed49a5@10.233.90.83:5007][leaveCluster] Left cluster [2024-01-30T00:06:30,977][INFO ][sc-cluster-5007-272][Transport] Stopping [address=10.233.90.83:5007] [2024-01-30T00:06:30,978][INFO ][sc-cluster-5007-272][Transport] Stopped [address=10.233.90.83:5007] [2024-01-30T00:06:30,978][INFO ][sc-cluster-5007-272][Cluster] [default:int_taewfiies_5007:1a48dc796fed49a5@10.233.90.83:5007][doShutdown] Shutdown [2024-01-30T00:06:30,984][INFO ][%int_taewfiies_5003%JRaft-Request-Processor-2][LogThreadPoolExecutor] ThreadPool is terminated: %int_taewfiies_5003%JRaft-Request-Processor-, org.apache.ignite.raft.jraft.util.MetricThreadPoolExecutor@36742447[Shutting down, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 16]. [2024-01-30T00:06:30,984][INFO ][Test worker][NodeImpl] Node shutdown, currTerm=5 state=STATE_CANDIDATE. [2024-01-30T00:06:30,985][INFO ][Test worker][LocalRaftMetaStorage] Save raft meta, path=build/work/ItNodeTest/testAppendEntriesWhenFollowerIsInErrorState_30809148389477213/int_taewfiies_5003/meta, term=5, votedFor=int_taewfiies_5003, cost time=1 ms [2024-01-30T00:06:30,985][INFO ][Test worker][FSMCallerImpl] Shutting down FSMCaller... [2024-01-30T00:06:30,986][INFO ][%int_taewfiies_5003%JRaft-FSMCaller-Disruptor-_stripe_0-0][StateMachineAdapter] onShutdown. [2024-01-30T00:06:30,987][INFO ][%int_taewfiies_5003%JRaft-Node-Scheduler-4][LogScheduledThreadPoolExecutor] ThreadPool is terminated: %int_taewfiies_5003%JRaft-Node-Scheduler-, org.apache.ignite.raft.jraft.util.MetricScheduledThreadPoolExecutor@29cb5821[Shutting down, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 23]. [2024-01-30T00:06:30,987][INFO ][%int_taewfiies_5003%JRaft-ElectionTimer-3][LogScheduledThreadPoolExecutor] ThreadPool is terminated: %int_taewfiies_5003%JRaft-ElectionTimer-, org.apache.ignite.raft.jraft.util.MetricScheduledThreadPoolExecutor@34be6e11[Shutting down, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 15]. [2024-01-30T00:06:30,988][INFO ][%int_taewfiies_5003%JRaft-VoteTimer-0][LogScheduledThreadPoolExecutor] ThreadPool is terminated: %int_taewfiies_5003%JRaft-VoteTimer-, org.apache.ignite.raft.jraft.util.MetricScheduledThreadPoolExecutor@341436dc[Shutting down, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0]. [2024-01-30T00:06:30,988][INFO ][Test worker][LogScheduledThreadPoolExecutor] ThreadPool is terminated: %int_taewfiies_5003%JRaft-StepDownTimer-, org.apache.ignite.raft.jraft.util.MetricScheduledThreadPoolExecutor@77c8538f[Shutting down, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0]. [2024-01-30T00:06:30,988][INFO ][%int_taewfiies_5003%JRaft-SnapshotTimer-0][LogScheduledThreadPoolExecutor] ThreadPool is terminated: %int_taewfiies_5003%JRaft-SnapshotTimer-, org.apache.ignite.raft.jraft.util.MetricScheduledThreadPoolExecutor@35819db0[Shutting down, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0]. [2024-01-30T00:06:30,989][INFO ][%int_taewfiies_5003%JRaft-Common-Executor-2][LogThreadPoolExecutor] ThreadPool is terminated: %int_taewfiies_5003%JRaft-Common-Executor-, org.apache.ignite.raft.jraft.util.MetricThreadPoolExecutor@7bc406c3[Shutting down, pool size = 1, active threads = 0, queued tasks = 0, completed tasks = 4]. [2024-01-30T00:06:30,993][INFO ][%int_taewfiies_5003%JRaft-Response-Processor-3][LogThreadPoolExecutor] ThreadPool is terminated: %int_taewfiies_5003%JRaft-Response-Processor-, org.apache.ignite.raft.jraft.util.MetricThreadPoolExecutor@4bf0509a[Shutting down, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 4]. [2024-01-30T00:06:30,993][INFO ][Test worker][RaftGroupService] Stop the RaftGroupService successfully. [2024-01-30T00:06:30,993][INFO ][Test worker][Cluster] [default:int_taewfiies_5003:2683329c8fe407e@10.233.90.83:5003][doShutdown] Shutting down [2024-01-30T00:06:30,993][INFO ][Test worker][Cluster] [default:int_taewfiies_5003:2683329c8fe407e@10.233.90.83:5003][leaveCluster] Leaving cluster [2024-01-30T00:06:31,007][INFO ][sc-cluster-5004-274][MembershipProtocol] [default:int_taewfiies_5004:7d20e3de9a4145a6@10.233.90.83:5004][publishEvent] MembershipEvent[type=LEAVING, member=default:int_taewfiies_5003:2683329c8fe407e@10.233.90.83:5003, oldMetadata=null, newMetadata=7e16449-5, timestamp=2024-01-30T00:06:31.007Z] [2024-01-30T00:06:31,007][INFO ][sc-cluster-5004-274][ScaleCubeTopologyService] Node left [member=ClusterNodeImpl [id=02683329-c8fe-407e-afee-1c81eceaa9a0, name=int_taewfiies_5003, address=10.233.90.83:5003, nodeMetadata=null], eventType=LEAVING] [2024-01-30T00:06:31,007][INFO ][sc-cluster-5004-274][ScaleCubeTopologyService] Topology snapshot [nodes=[int_taewfiies_5004]] [2024-01-30T00:06:31,067][INFO ][sc-cluster-5003-273][Cluster] [default:int_taewfiies_5003:2683329c8fe407e@10.233.90.83:5003][leaveCluster] Left cluster [2024-01-30T00:06:31,067][INFO ][sc-cluster-5003-273][Transport] Stopping [address=10.233.90.83:5003] [2024-01-30T00:06:31,067][INFO ][sc-cluster-5003-273][Transport] Stopped [address=10.233.90.83:5003] [2024-01-30T00:06:31,067][INFO ][sc-cluster-5003-273][Cluster] [default:int_taewfiies_5003:2683329c8fe407e@10.233.90.83:5003][doShutdown] Shutdown [2024-01-30T00:06:31,073][INFO ][%int_taewfiies_5004%JRaft-Request-Processor-2][LogThreadPoolExecutor] ThreadPool is terminated: %int_taewfiies_5004%JRaft-Request-Processor-, org.apache.ignite.raft.jraft.util.MetricThreadPoolExecutor@509c744c[Shutting down, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 6]. [2024-01-30T00:06:31,073][INFO ][Test worker][NodeImpl] Node shutdown, currTerm=5 state=STATE_FOLLOWER. [2024-01-30T00:06:31,075][INFO ][Test worker][LocalRaftMetaStorage] Save raft meta, path=build/work/ItNodeTest/testAppendEntriesWhenFollowerIsInErrorState_30809148389477213/int_taewfiies_5004/meta, term=5, votedFor=int_taewfiies_5003, cost time=1 ms [2024-01-30T00:06:31,075][INFO ][Test worker][FSMCallerImpl] Shutting down FSMCaller... [2024-01-30T00:06:31,075][INFO ][%int_taewfiies_5004%JRaft-FSMCaller-Disruptor-_stripe_1-0][StateMachineAdapter] onShutdown. [2024-01-30T00:06:31,076][INFO ][%int_taewfiies_5004%JRaft-Node-Scheduler-7][LogScheduledThreadPoolExecutor] ThreadPool is terminated: %int_taewfiies_5004%JRaft-Node-Scheduler-, org.apache.ignite.raft.jraft.util.MetricScheduledThreadPoolExecutor@5e40e863[Shutting down, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 20]. [2024-01-30T00:06:31,277][INFO ][%int_taewfiies_5004%JRaft-ElectionTimer-6][LogScheduledThreadPoolExecutor] ThreadPool is terminated: %int_taewfiies_5004%JRaft-ElectionTimer-, org.apache.ignite.raft.jraft.util.MetricScheduledThreadPoolExecutor@3be50d24[Shutting down, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 11]. [2024-01-30T00:06:31,277][INFO ][Test worker][LogScheduledThreadPoolExecutor] ThreadPool is terminated: %int_taewfiies_5004%JRaft-VoteTimer-, org.apache.ignite.raft.jraft.util.MetricScheduledThreadPoolExecutor@55941b3f[Shutting down, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0]. [2024-01-30T00:06:31,277][INFO ][Test worker][LogScheduledThreadPoolExecutor] ThreadPool is terminated: %int_taewfiies_5004%JRaft-StepDownTimer-, org.apache.ignite.raft.jraft.util.MetricScheduledThreadPoolExecutor@76599cfc[Shutting down, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0]. [2024-01-30T00:06:31,278][INFO ][%int_taewfiies_5004%JRaft-SnapshotTimer-0][LogScheduledThreadPoolExecutor] ThreadPool is terminated: %int_taewfiies_5004%JRaft-SnapshotTimer-, org.apache.ignite.raft.jraft.util.MetricScheduledThreadPoolExecutor@532a0d5f[Shutting down, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0]. [2024-01-30T00:06:31,278][INFO ][%int_taewfiies_5004%JRaft-Common-Executor-0][LogThreadPoolExecutor] ThreadPool is terminated: %int_taewfiies_5004%JRaft-Common-Executor-, org.apache.ignite.raft.jraft.util.MetricThreadPoolExecutor@d2446e3[Shutting down, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 4]. [2024-01-30T00:06:31,282][INFO ][%int_taewfiies_5004%JRaft-Response-Processor-0][LogThreadPoolExecutor] ThreadPool is terminated: %int_taewfiies_5004%JRaft-Response-Processor-, org.apache.ignite.raft.jraft.util.MetricThreadPoolExecutor@86a616c[Shutting down, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 35]. [2024-01-30T00:06:31,283][INFO ][Test worker][RaftGroupService] Stop the RaftGroupService successfully. [2024-01-30T00:06:31,283][INFO ][Test worker][Cluster] [default:int_taewfiies_5004:7d20e3de9a4145a6@10.233.90.83:5004][doShutdown] Shutting down [2024-01-30T00:06:31,283][INFO ][Test worker][Cluster] [default:int_taewfiies_5004:7d20e3de9a4145a6@10.233.90.83:5004][leaveCluster] Leaving cluster [2024-01-30T00:06:31,358][INFO ][sc-cluster-5004-274][Cluster] [default:int_taewfiies_5004:7d20e3de9a4145a6@10.233.90.83:5004][leaveCluster] Left cluster [2024-01-30T00:06:31,358][INFO ][sc-cluster-5004-274][Transport] Stopping [address=10.233.90.83:5004] [2024-01-30T00:06:31,358][INFO ][sc-cluster-5004-274][Transport] Stopped [address=10.233.90.83:5004] [2024-01-30T00:06:31,358][INFO ][sc-cluster-5004-274][Cluster] [default:int_taewfiies_5004:7d20e3de9a4145a6@10.233.90.83:5004][doShutdown] Shutdown [2024-01-30T00:06:31,367][INFO ][Test worker][ItNodeTest] >>>>>>>>>>>>>>> End test method: testAppendEntriesWhenFollowerIsInErrorState(), cost:26265 ms.