java.lang.IllegalStateException java.lang.IllegalStateException: No user log between index:1 and last_applied_index:1 at org.apache.ignite.raft.jraft.core.ITNodeTest.readCommittedUserLog(ITNodeTest.java:2728) ------- Stderr: ------- 2021-07-28 10:11:10:527 +0300 [main] INFO ITNodeTest - >>>>>>>>>>>>>>> Start test method: readCommittedUserLog() Jul 28, 2021 10:11:10 AM org.apache.ignite.lang.IgniteLogger logInternal INFO: Connection created [address=/0.0.0.0:5003] 2021-07-28 10:11:10:540 +0300 [main] INFO Cluster - [null][doStart] Starting, config: ClusterConfig[metadata=null, metadataTimeout=1000, metadataCodec=io.scalecube.cluster.metadata.JdkMetadataCodec@476aac9, memberAlias='172.17.0.3:5003', externalHost='null', externalPort=null, transportConfig=TransportConfig[port=0, isSecured=false, connectTimeout=1000, messageCodec=io.scalecube.cluster.transport.api.JdkMessageCodec@6cea706c, maxFrameLength=2097152, transportFactory=org.apache.ignite.network.scalecube.DelegatingTransportFactory@2045a469], failureDetectorConfig=FailureDetectorConfig[pingInterval=500, pingTimeout=200, pingReqMembers=1], gossipConfig=GossipConfig[gossipFanout=3, gossipInterval=10, gossipRepeatMult=2, gossipSegmentationThreshold=1000], membershipConfig=MembershipConfig[seedMembers=[172.17.0.3:5003, 172.17.0.3:5004, 172.17.0.3:5005], syncInterval=1000, syncTimeout=3000, suspicionMult=1, namespace='default', removedMembersHistorySize=42]] 2021-07-28 10:11:10:540 +0300 [main] WARN MembershipProtocol - [default:172.17.0.3:5003:9ac3bcc0-dffd-4634-9556-370d2d9e7a8c@172.17.0.3:5003] Filtering out seed address: 172.17.0.3:5003 2021-07-28 10:11:10:540 +0300 [main] INFO MembershipProtocol - [default:172.17.0.3:5003:9ac3bcc0-dffd-4634-9556-370d2d9e7a8c@172.17.0.3:5003] Making initial Sync to all seed members: [172.17.0.3:5004, 172.17.0.3:5005] 2021-07-28 10:11:10:542 +0300 [sc-cluster-5003-109] INFO Cluster - [default:172.17.0.3:5003:9ac3bcc0-dffd-4634-9556-370d2d9e7a8c@172.17.0.3:5003][doStart] Started Jul 28, 2021 10:11:10 AM org.apache.ignite.lang.IgniteLogger logInternal INFO: Node joined: ClusterNode [id=9ac3bcc0-dffd-4634-9556-370d2d9e7a8c, name=172.17.0.3:5003, address=172.17.0.3:5003] Jul 28, 2021 10:11:10 AM org.apache.ignite.lang.IgniteLogger logInternal INFO: Topology snapshot [nodes=1] ^-- ClusterNode [id=9ac3bcc0-dffd-4634-9556-370d2d9e7a8c, name=172.17.0.3:5003, address=172.17.0.3:5003] 2021-07-28 10:11:10:546 +0300 [main] INFO FSMCallerImpl - Starts FSMCaller successfully. 2021-07-28 10:11:10:547 +0300 [main] WARN LocalSnapshotStorage - No data for snapshot reader target/work/ITNodeTest/readCommittedUserLog_1627456270527/172.17.0.3_5003/snapshot. 2021-07-28 10:11:10:549 +0300 [main] INFO NodeImpl - Node init, term=0, lastLogId=LogId [index=0, term=0], conf=172.17.0.3:5003,172.17.0.3:5004,172.17.0.3:5005, oldConf=. 2021-07-28 10:11:10:550 +0300 [main] INFO RaftGroupService - Start the RaftGroupService successfully Jul 28, 2021 10:11:10 AM org.apache.ignite.lang.IgniteLogger logInternal INFO: Connection created [address=/0.0.0.0:5004] 2021-07-28 10:11:10:557 +0300 [main] INFO Cluster - [null][doStart] Starting, config: ClusterConfig[metadata=null, metadataTimeout=1000, metadataCodec=io.scalecube.cluster.metadata.JdkMetadataCodec@476aac9, memberAlias='172.17.0.3:5004', externalHost='null', externalPort=null, transportConfig=TransportConfig[port=0, isSecured=false, connectTimeout=1000, messageCodec=io.scalecube.cluster.transport.api.JdkMessageCodec@6cea706c, maxFrameLength=2097152, transportFactory=org.apache.ignite.network.scalecube.DelegatingTransportFactory@542aba71], failureDetectorConfig=FailureDetectorConfig[pingInterval=500, pingTimeout=200, pingReqMembers=1], gossipConfig=GossipConfig[gossipFanout=3, gossipInterval=10, gossipRepeatMult=2, gossipSegmentationThreshold=1000], membershipConfig=MembershipConfig[seedMembers=[172.17.0.3:5003, 172.17.0.3:5004, 172.17.0.3:5005], syncInterval=1000, syncTimeout=3000, suspicionMult=1, namespace='default', removedMembersHistorySize=42]] 2021-07-28 10:11:10:558 +0300 [main] WARN MembershipProtocol - [default:172.17.0.3:5004:194bc875-2be5-4fa4-af7a-ad7fdb6f8195@172.17.0.3:5004] Filtering out seed address: 172.17.0.3:5004 2021-07-28 10:11:10:558 +0300 [main] INFO MembershipProtocol - [default:172.17.0.3:5004:194bc875-2be5-4fa4-af7a-ad7fdb6f8195@172.17.0.3:5004] Making initial Sync to all seed members: [172.17.0.3:5003, 172.17.0.3:5005] 2021-07-28 10:11:10:564 +0300 [sc-cluster-5003-109] INFO MembershipProtocol - [default:172.17.0.3:5003:9ac3bcc0-dffd-4634-9556-370d2d9e7a8c@172.17.0.3:5003][publishEvent] MembershipEvent[type=ADDED, member=default:172.17.0.3:5004:194bc875-2be5-4fa4-af7a-ad7fdb6f8195@172.17.0.3:5004, oldMetadata=null, newMetadata=7e16449-5, timestamp=2021-07-28T07:11:10.564Z] Jul 28, 2021 10:11:10 AM org.apache.ignite.lang.IgniteLogger logInternal INFO: Node joined: ClusterNode [id=194bc875-2be5-4fa4-af7a-ad7fdb6f8195, name=172.17.0.3:5004, address=172.17.0.3:5004] Jul 28, 2021 10:11:10 AM org.apache.ignite.lang.IgniteLogger logInternal INFO: Topology snapshot [nodes=2] ^-- ClusterNode [id=194bc875-2be5-4fa4-af7a-ad7fdb6f8195, name=172.17.0.3:5004, address=172.17.0.3:5004] ^-- ClusterNode [id=9ac3bcc0-dffd-4634-9556-370d2d9e7a8c, name=172.17.0.3:5003, address=172.17.0.3:5003] 2021-07-28 10:11:10:566 +0300 [sc-cluster-5004-110] INFO MembershipProtocol - [default:172.17.0.3:5004:194bc875-2be5-4fa4-af7a-ad7fdb6f8195@172.17.0.3:5004][publishEvent] MembershipEvent[type=ADDED, member=default:172.17.0.3:5003:9ac3bcc0-dffd-4634-9556-370d2d9e7a8c@172.17.0.3:5003, oldMetadata=null, newMetadata=7e16449-5, timestamp=2021-07-28T07:11:10.566Z] Jul 28, 2021 10:11:10 AM org.apache.ignite.lang.IgniteLogger logInternal INFO: Node joined: ClusterNode [id=9ac3bcc0-dffd-4634-9556-370d2d9e7a8c, name=172.17.0.3:5003, address=172.17.0.3:5003] Jul 28, 2021 10:11:10 AM org.apache.ignite.lang.IgniteLogger logInternal INFO: Topology snapshot [nodes=1] ^-- ClusterNode [id=9ac3bcc0-dffd-4634-9556-370d2d9e7a8c, name=172.17.0.3:5003, address=172.17.0.3:5003] 2021-07-28 10:11:10:566 +0300 [sc-cluster-5004-110] INFO Cluster - [default:172.17.0.3:5004:194bc875-2be5-4fa4-af7a-ad7fdb6f8195@172.17.0.3:5004][doStart] Started Jul 28, 2021 10:11:10 AM org.apache.ignite.lang.IgniteLogger logInternal INFO: Node joined: ClusterNode [id=194bc875-2be5-4fa4-af7a-ad7fdb6f8195, name=172.17.0.3:5004, address=172.17.0.3:5004] Jul 28, 2021 10:11:10 AM org.apache.ignite.lang.IgniteLogger logInternal INFO: Topology snapshot [nodes=2] ^-- ClusterNode [id=194bc875-2be5-4fa4-af7a-ad7fdb6f8195, name=172.17.0.3:5004, address=172.17.0.3:5004] ^-- ClusterNode [id=9ac3bcc0-dffd-4634-9556-370d2d9e7a8c, name=172.17.0.3:5003, address=172.17.0.3:5003] 2021-07-28 10:11:10:571 +0300 [main] INFO FSMCallerImpl - Starts FSMCaller successfully. 2021-07-28 10:11:10:571 +0300 [main] WARN LocalSnapshotStorage - No data for snapshot reader target/work/ITNodeTest/readCommittedUserLog_1627456270527/172.17.0.3_5004/snapshot. 2021-07-28 10:11:10:573 +0300 [main] INFO NodeImpl - Node init, term=0, lastLogId=LogId [index=0, term=0], conf=172.17.0.3:5003,172.17.0.3:5004,172.17.0.3:5005, oldConf=. 2021-07-28 10:11:10:573 +0300 [main] INFO RaftGroupService - Start the RaftGroupService successfully Jul 28, 2021 10:11:10 AM org.apache.ignite.lang.IgniteLogger logInternal INFO: Connection created [address=/0.0.0.0:5005] 2021-07-28 10:11:10:581 +0300 [main] INFO Cluster - [null][doStart] Starting, config: ClusterConfig[metadata=null, metadataTimeout=1000, metadataCodec=io.scalecube.cluster.metadata.JdkMetadataCodec@476aac9, memberAlias='172.17.0.3:5005', externalHost='null', externalPort=null, transportConfig=TransportConfig[port=0, isSecured=false, connectTimeout=1000, messageCodec=io.scalecube.cluster.transport.api.JdkMessageCodec@6cea706c, maxFrameLength=2097152, transportFactory=org.apache.ignite.network.scalecube.DelegatingTransportFactory@33c590], failureDetectorConfig=FailureDetectorConfig[pingInterval=500, pingTimeout=200, pingReqMembers=1], gossipConfig=GossipConfig[gossipFanout=3, gossipInterval=10, gossipRepeatMult=2, gossipSegmentationThreshold=1000], membershipConfig=MembershipConfig[seedMembers=[172.17.0.3:5003, 172.17.0.3:5004, 172.17.0.3:5005], syncInterval=1000, syncTimeout=3000, suspicionMult=1, namespace='default', removedMembersHistorySize=42]] 2021-07-28 10:11:10:581 +0300 [main] WARN MembershipProtocol - [default:172.17.0.3:5005:5f2cd128-1aa8-498a-9115-0373e15c97a2@172.17.0.3:5005] Filtering out seed address: 172.17.0.3:5005 2021-07-28 10:11:10:582 +0300 [main] INFO MembershipProtocol - [default:172.17.0.3:5005:5f2cd128-1aa8-498a-9115-0373e15c97a2@172.17.0.3:5005] Making initial Sync to all seed members: [172.17.0.3:5003, 172.17.0.3:5004] 2021-07-28 10:11:10:587 +0300 [sc-cluster-5003-109] INFO MembershipProtocol - [default:172.17.0.3:5003:9ac3bcc0-dffd-4634-9556-370d2d9e7a8c@172.17.0.3:5003][publishEvent] MembershipEvent[type=ADDED, member=default:172.17.0.3:5005:5f2cd128-1aa8-498a-9115-0373e15c97a2@172.17.0.3:5005, oldMetadata=null, newMetadata=7e16449-5, timestamp=2021-07-28T07:11:10.587Z] Jul 28, 2021 10:11:10 AM org.apache.ignite.lang.IgniteLogger logInternal INFO: Node joined: ClusterNode [id=5f2cd128-1aa8-498a-9115-0373e15c97a2, name=172.17.0.3:5005, address=172.17.0.3:5005] Jul 28, 2021 10:11:10 AM org.apache.ignite.lang.IgniteLogger logInternal INFO: Topology snapshot [nodes=3] ^-- ClusterNode [id=5f2cd128-1aa8-498a-9115-0373e15c97a2, name=172.17.0.3:5005, address=172.17.0.3:5005] ^-- ClusterNode [id=194bc875-2be5-4fa4-af7a-ad7fdb6f8195, name=172.17.0.3:5004, address=172.17.0.3:5004] ^-- ClusterNode [id=9ac3bcc0-dffd-4634-9556-370d2d9e7a8c, name=172.17.0.3:5003, address=172.17.0.3:5003] 2021-07-28 10:11:10:588 +0300 [sc-cluster-5005-111] INFO MembershipProtocol - [default:172.17.0.3:5005:5f2cd128-1aa8-498a-9115-0373e15c97a2@172.17.0.3:5005][publishEvent] MembershipEvent[type=ADDED, member=default:172.17.0.3:5003:9ac3bcc0-dffd-4634-9556-370d2d9e7a8c@172.17.0.3:5003, oldMetadata=null, newMetadata=7e16449-5, timestamp=2021-07-28T07:11:10.588Z] Jul 28, 2021 10:11:10 AM org.apache.ignite.lang.IgniteLogger logInternal INFO: Node joined: ClusterNode [id=9ac3bcc0-dffd-4634-9556-370d2d9e7a8c, name=172.17.0.3:5003, address=172.17.0.3:5003] Jul 28, 2021 10:11:10 AM org.apache.ignite.lang.IgniteLogger logInternal INFO: Topology snapshot [nodes=1] ^-- ClusterNode [id=9ac3bcc0-dffd-4634-9556-370d2d9e7a8c, name=172.17.0.3:5003, address=172.17.0.3:5003] 2021-07-28 10:11:10:591 +0300 [sc-cluster-5005-111] INFO MembershipProtocol - [default:172.17.0.3:5005:5f2cd128-1aa8-498a-9115-0373e15c97a2@172.17.0.3:5005][publishEvent] MembershipEvent[type=ADDED, member=default:172.17.0.3:5004:194bc875-2be5-4fa4-af7a-ad7fdb6f8195@172.17.0.3:5004, oldMetadata=null, newMetadata=7e16449-5, timestamp=2021-07-28T07:11:10.591Z] 2021-07-28 10:11:10:591 +0300 [sc-cluster-5004-110] INFO MembershipProtocol - [default:172.17.0.3:5004:194bc875-2be5-4fa4-af7a-ad7fdb6f8195@172.17.0.3:5004][publishEvent] MembershipEvent[type=ADDED, member=default:172.17.0.3:5005:5f2cd128-1aa8-498a-9115-0373e15c97a2@172.17.0.3:5005, oldMetadata=null, newMetadata=7e16449-5, timestamp=2021-07-28T07:11:10.591Z] Jul 28, 2021 10:11:10 AM org.apache.ignite.lang.IgniteLogger logInternal INFO: Node joined: ClusterNode [id=194bc875-2be5-4fa4-af7a-ad7fdb6f8195, name=172.17.0.3:5004, address=172.17.0.3:5004] Jul 28, 2021 10:11:10 AM org.apache.ignite.lang.IgniteLogger logInternal INFO: Node joined: ClusterNode [id=5f2cd128-1aa8-498a-9115-0373e15c97a2, name=172.17.0.3:5005, address=172.17.0.3:5005] Jul 28, 2021 10:11:10 AM org.apache.ignite.lang.IgniteLogger logInternal INFO: Topology snapshot [nodes=2] ^-- ClusterNode [id=194bc875-2be5-4fa4-af7a-ad7fdb6f8195, name=172.17.0.3:5004, address=172.17.0.3:5004] ^-- ClusterNode [id=9ac3bcc0-dffd-4634-9556-370d2d9e7a8c, name=172.17.0.3:5003, address=172.17.0.3:5003] Jul 28, 2021 10:11:10 AM org.apache.ignite.lang.IgniteLogger logInternal INFO: Topology snapshot [nodes=3] ^-- ClusterNode [id=5f2cd128-1aa8-498a-9115-0373e15c97a2, name=172.17.0.3:5005, address=172.17.0.3:5005] ^-- ClusterNode [id=194bc875-2be5-4fa4-af7a-ad7fdb6f8195, name=172.17.0.3:5004, address=172.17.0.3:5004] ^-- ClusterNode [id=9ac3bcc0-dffd-4634-9556-370d2d9e7a8c, name=172.17.0.3:5003, address=172.17.0.3:5003] 2021-07-28 10:11:10:592 +0300 [sc-cluster-5005-111] INFO Cluster - [default:172.17.0.3:5005:5f2cd128-1aa8-498a-9115-0373e15c97a2@172.17.0.3:5005][doStart] Started Jul 28, 2021 10:11:10 AM org.apache.ignite.lang.IgniteLogger logInternal INFO: Node joined: ClusterNode [id=5f2cd128-1aa8-498a-9115-0373e15c97a2, name=172.17.0.3:5005, address=172.17.0.3:5005] Jul 28, 2021 10:11:10 AM org.apache.ignite.lang.IgniteLogger logInternal INFO: Topology snapshot [nodes=3] ^-- ClusterNode [id=5f2cd128-1aa8-498a-9115-0373e15c97a2, name=172.17.0.3:5005, address=172.17.0.3:5005] ^-- ClusterNode [id=194bc875-2be5-4fa4-af7a-ad7fdb6f8195, name=172.17.0.3:5004, address=172.17.0.3:5004] ^-- ClusterNode [id=9ac3bcc0-dffd-4634-9556-370d2d9e7a8c, name=172.17.0.3:5003, address=172.17.0.3:5003] 2021-07-28 10:11:10:596 +0300 [main] INFO FSMCallerImpl - Starts FSMCaller successfully. 2021-07-28 10:11:10:596 +0300 [main] WARN LocalSnapshotStorage - No data for snapshot reader target/work/ITNodeTest/readCommittedUserLog_1627456270527/172.17.0.3_5005/snapshot. 2021-07-28 10:11:10:598 +0300 [main] INFO NodeImpl - Node init, term=0, lastLogId=LogId [index=0, term=0], conf=172.17.0.3:5003,172.17.0.3:5004,172.17.0.3:5005, oldConf=. 2021-07-28 10:11:10:601 +0300 [main] INFO RaftGroupService - Start the RaftGroupService successfully 2021-07-28 10:11:11:210 +0300 [JRaft-ElectionTimer-172.17.0.3:5004-0] INFO NodeImpl - Node term 0 start preVote. 2021-07-28 10:11:11:212 +0300 [JRaft-Request-Processor-172.17.0.3:5005-1] INFO NodeImpl - Node received PreVoteRequest from 172.17.0.3:5004, term=1, currTerm=0, granted=true, requestLastLogId=LogId [index=0, term=0], lastLogId=LogId [index=0, term=0]. 2021-07-28 10:11:11:215 +0300 [JRaft-Response-Processor-172.17.0.3:5004-1] INFO NodeImpl - Node received PreVoteResponse from 172.17.0.3:5005, term=0, granted=true. 2021-07-28 10:11:11:215 +0300 [JRaft-Response-Processor-172.17.0.3:5004-1] INFO NodeImpl - Node start vote and grant vote self, term=0. 2021-07-28 10:11:11:219 +0300 [JRaft-Request-Processor-172.17.0.3:5003-1] INFO NodeImpl - Node received PreVoteRequest from 172.17.0.3:5004, term=1, currTerm=0, granted=true, requestLastLogId=LogId [index=0, term=0], lastLogId=LogId [index=0, term=0]. 2021-07-28 10:11:11:220 +0300 [JRaft-Request-Processor-172.17.0.3:5003-2] INFO NodeImpl - Node received RequestVoteRequest from 172.17.0.3:5004, term=1, currTerm=0. 2021-07-28 10:11:11:220 +0300 [JRaft-Request-Processor-172.17.0.3:5005-2] INFO NodeImpl - Node received RequestVoteRequest from 172.17.0.3:5004, term=1, currTerm=0. 2021-07-28 10:11:11:227 +0300 [JRaft-Response-Processor-172.17.0.3:5004-1] INFO LocalRaftMetaStorage - Save raft meta, path=target/work/ITNodeTest/readCommittedUserLog_1627456270527/172.17.0.3_5004/meta, term=1, votedFor=172.17.0.3:5004, cost time=8 ms 2021-07-28 10:11:11:227 +0300 [JRaft-Request-Processor-172.17.0.3:5005-2] INFO LocalRaftMetaStorage - Save raft meta, path=target/work/ITNodeTest/readCommittedUserLog_1627456270527/172.17.0.3_5005/meta, term=1, votedFor=0.0.0.0:0, cost time=7 ms 2021-07-28 10:11:11:227 +0300 [JRaft-Request-Processor-172.17.0.3:5003-2] INFO LocalRaftMetaStorage - Save raft meta, path=target/work/ITNodeTest/readCommittedUserLog_1627456270527/172.17.0.3_5003/meta, term=1, votedFor=0.0.0.0:0, cost time=8 ms 2021-07-28 10:11:11:228 +0300 [JRaft-Response-Processor-172.17.0.3:5004-3] WARN NodeImpl - Node received invalid PreVoteResponse from 172.17.0.3:5003, state not in STATE_FOLLOWER but STATE_CANDIDATE. 2021-07-28 10:11:11:248 +0300 [JRaft-Request-Processor-172.17.0.3:5005-2] INFO LocalRaftMetaStorage - Save raft meta, path=target/work/ITNodeTest/readCommittedUserLog_1627456270527/172.17.0.3_5005/meta, term=1, votedFor=172.17.0.3:5004, cost time=21 ms 2021-07-28 10:11:11:248 +0300 [JRaft-Request-Processor-172.17.0.3:5003-2] INFO LocalRaftMetaStorage - Save raft meta, path=target/work/ITNodeTest/readCommittedUserLog_1627456270527/172.17.0.3_5003/meta, term=1, votedFor=172.17.0.3:5004, cost time=21 ms 2021-07-28 10:11:11:250 +0300 [JRaft-Response-Processor-172.17.0.3:5004-4] INFO NodeImpl - Node become leader of group, term=1, conf=172.17.0.3:5003,172.17.0.3:5004,172.17.0.3:5005, oldConf=. 2021-07-28 10:11:11:250 +0300 [JRaft-Response-Processor-172.17.0.3:5004-4] INFO Replicator - Replicator=Replicator [state=null, statInfo=, peerId=172.17.0.3:5005, type=Follower]@172.17.0.3:5005 is started 2021-07-28 10:11:11:254 +0300 [JRaft-Response-Processor-172.17.0.3:5004-4] INFO Replicator - Replicator=Replicator [state=null, statInfo=, peerId=172.17.0.3:5003, type=Follower]@172.17.0.3:5003 is started 2021-07-28 10:11:11:255 +0300 [JRaft-FSMCaller-Disruptor-172.17.0.3:5005-0] INFO StateMachineAdapter - onStartFollowing: LeaderChangeContext [leaderId=172.17.0.3:5004, term=1, status=Status[ENEWLEADER<10011>: Follower receives message from new leader with the same term.]]. 2021-07-28 10:11:11:256 +0300 [JRaft-Response-Processor-172.17.0.3:5004-5] WARN NodeImpl - Node received invalid RequestVoteResponse from 172.17.0.3:5003, state not in STATE_CANDIDATE but STATE_LEADER. 2021-07-28 10:11:11:257 +0300 [JRaft-FSMCaller-Disruptor-172.17.0.3:5003-0] INFO StateMachineAdapter - onStartFollowing: LeaderChangeContext [leaderId=172.17.0.3:5004, term=1, status=Status[ENEWLEADER<10011>: Follower receives message from new leader with the same term.]]. 2021-07-28 10:11:11:258 +0300 [JRaft-FSMCaller-Disruptor-172.17.0.3:5004-0] INFO StateMachineAdapter - onLeaderStart: term=1. 2021-07-28 10:11:11:273 +0300 [main] INFO NodeImpl - Node shutdown, currTerm=1 state=STATE_FOLLOWER. 2021-07-28 10:11:11:274 +0300 [JRaft-FSMCaller-Disruptor-172.17.0.3:5003-0] INFO StateMachineAdapter - onStopFollowing: LeaderChangeContext [leaderId=172.17.0.3:5004, term=1, status=Status[ESHUTDOWN<1007>: Raft node is going to quit.]]. 2021-07-28 10:11:11:277 +0300 [main] INFO LocalRaftMetaStorage - Save raft meta, path=target/work/ITNodeTest/readCommittedUserLog_1627456270527/172.17.0.3_5003/meta, term=1, votedFor=172.17.0.3:5004, cost time=3 ms 2021-07-28 10:11:11:277 +0300 [main] INFO FSMCallerImpl - Shutting down FSMCaller... 2021-07-28 10:11:11:277 +0300 [main] INFO StateMachineAdapter - onShutdown. 2021-07-28 10:11:11:278 +0300 [main] INFO LogScheduledThreadPoolExecutor - ThreadPool is terminated: JRaft-Node-Scheduler-172.17.0.3:5003-, org.apache.ignite.raft.jraft.util.MetricScheduledThreadPoolExecutor@3ea4dcc5[Shutting down, pool size = 1, active threads = 0, queued tasks = 0, completed tasks = 0]. 2021-07-28 10:11:11:278 +0300 [main] INFO RepeatedTimer - Destroy timer: RepeatedTimer{timeout=null, stopped=true, running=false, destroyed=true, invoking=false, timeoutMs=600, name='JRaft-ElectionTimer-172.17.0.3:5003-'}. 2021-07-28 10:11:11:278 +0300 [main] INFO RepeatedTimer - Destroy timer: RepeatedTimer{timeout=null, stopped=true, running=false, destroyed=true, invoking=false, timeoutMs=600, name='JRaft-VoteTimer-172.17.0.3:5003-'}. 2021-07-28 10:11:11:278 +0300 [main] INFO RepeatedTimer - Destroy timer: RepeatedTimer{timeout=null, stopped=true, running=false, destroyed=true, invoking=false, timeoutMs=300, name='JRaft-StepDownTimer-172.17.0.3:5003-'}. 2021-07-28 10:11:11:278 +0300 [main] INFO RepeatedTimer - Destroy timer: RepeatedTimer{timeout=null, stopped=true, running=false, destroyed=true, invoking=false, timeoutMs=300000, name='JRaft-SnapshotTimer-172.17.0.3:5003-'}. 2021-07-28 10:11:11:278 +0300 [main] INFO RaftGroupService - Stop the RaftGroupService successfully. 2021-07-28 10:11:11:278 +0300 [main] INFO LogThreadPoolExecutor - ThreadPool is terminated: JRaft-Response-Processor-172.17.0.3:5003-, org.apache.ignite.raft.jraft.util.MetricThreadPoolExecutor@73611fbb[Shutting down, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0]. 2021-07-28 10:11:11:326 +0300 [JRaft-Common-Executor-172.17.0.3:5003-1] INFO LogThreadPoolExecutor - ThreadPool is terminated: JRaft-Common-Executor-172.17.0.3:5003-, org.apache.ignite.raft.jraft.util.MetricThreadPoolExecutor@2a647052[Shutting down, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 4]. 2021-07-28 10:11:11:326 +0300 [main] INFO Cluster - [default:172.17.0.3:5003:9ac3bcc0-dffd-4634-9556-370d2d9e7a8c@172.17.0.3:5003][doShutdown] Shutting down 2021-07-28 10:11:11:327 +0300 [main] INFO Cluster - [default:172.17.0.3:5003:9ac3bcc0-dffd-4634-9556-370d2d9e7a8c@172.17.0.3:5003][leaveCluster] Leaving cluster 2021-07-28 10:11:11:331 +0300 [sc-cluster-5005-111] INFO MembershipProtocol - [default:172.17.0.3:5005:5f2cd128-1aa8-498a-9115-0373e15c97a2@172.17.0.3:5005][publishEvent] MembershipEvent[type=LEAVING, member=default:172.17.0.3:5003:9ac3bcc0-dffd-4634-9556-370d2d9e7a8c@172.17.0.3:5003, oldMetadata=null, newMetadata=7e16449-5, timestamp=2021-07-28T07:11:11.331Z] 2021-07-28 10:11:11:331 +0300 [sc-cluster-5004-110] INFO MembershipProtocol - [default:172.17.0.3:5004:194bc875-2be5-4fa4-af7a-ad7fdb6f8195@172.17.0.3:5004][publishEvent] MembershipEvent[type=LEAVING, member=default:172.17.0.3:5003:9ac3bcc0-dffd-4634-9556-370d2d9e7a8c@172.17.0.3:5003, oldMetadata=null, newMetadata=7e16449-5, timestamp=2021-07-28T07:11:11.331Z] Jul 28, 2021 10:11:11 AM org.apache.ignite.lang.IgniteLogger logInternal INFO: Topology snapshot [nodes=3] ^-- ClusterNode [id=5f2cd128-1aa8-498a-9115-0373e15c97a2, name=172.17.0.3:5005, address=172.17.0.3:5005] ^-- ClusterNode [id=194bc875-2be5-4fa4-af7a-ad7fdb6f8195, name=172.17.0.3:5004, address=172.17.0.3:5004] ^-- ClusterNode [id=9ac3bcc0-dffd-4634-9556-370d2d9e7a8c, name=172.17.0.3:5003, address=172.17.0.3:5003] Jul 28, 2021 10:11:11 AM org.apache.ignite.lang.IgniteLogger logInternal INFO: Topology snapshot [nodes=3] ^-- ClusterNode [id=5f2cd128-1aa8-498a-9115-0373e15c97a2, name=172.17.0.3:5005, address=172.17.0.3:5005] ^-- ClusterNode [id=194bc875-2be5-4fa4-af7a-ad7fdb6f8195, name=172.17.0.3:5004, address=172.17.0.3:5004] ^-- ClusterNode [id=9ac3bcc0-dffd-4634-9556-370d2d9e7a8c, name=172.17.0.3:5003, address=172.17.0.3:5003] 2021-07-28 10:11:11:381 +0300 [sc-cluster-5003-109] INFO Cluster - [default:172.17.0.3:5003:9ac3bcc0-dffd-4634-9556-370d2d9e7a8c@172.17.0.3:5003][leaveCluster] Left cluster Jul 28, 2021 10:11:11 AM org.apache.ignite.lang.IgniteLogger logInternal INFO: Stopping 172.17.0.3:5003 Jul 28, 2021 10:11:11 AM org.apache.ignite.lang.IgniteLogger logInternal INFO: Stopped 172.17.0.3:5003 2021-07-28 10:11:11:381 +0300 [sc-cluster-5003-109] INFO Cluster - [default:172.17.0.3:5003:9ac3bcc0-dffd-4634-9556-370d2d9e7a8c@172.17.0.3:5003][doShutdown] Shutdown 2021-07-28 10:11:11:430 +0300 [main] INFO NodeImpl - Node shutdown, currTerm=1 state=STATE_LEADER. 2021-07-28 10:11:11:430 +0300 [JRaft-FSMCaller-Disruptor-172.17.0.3:5004-0] INFO StateMachineAdapter - onLeaderStop: status=Status[ESHUTDOWN<1007>: Raft node is going to quit.]. 2021-07-28 10:11:11:439 +0300 [main] INFO Replicator - Replicator Replicator [state=Probe, statInfo=, peerId=172.17.0.3:5003, type=Follower] is going to quit 2021-07-28 10:11:11:439 +0300 [JRaft-Common-Executor-172.17.0.3:5004-7] WARN Replicator - Replicator Replicator [state=Replicate, statInfo=, peerId=172.17.0.3:5005, type=Follower] stops sending entries. 2021-07-28 10:11:11:440 +0300 [JRaft-Request-Processor-172.17.0.3:5005-3] INFO NodeImpl - Node start vote and grant vote self, term=1. 2021-07-28 10:11:11:440 +0300 [JRaft-FSMCaller-Disruptor-172.17.0.3:5005-0] INFO StateMachineAdapter - onStopFollowing: LeaderChangeContext [leaderId=172.17.0.3:5004, term=1, status=Status[ERAFTTIMEDOUT<10001>: A follower's leader_id is reset to NULL as it begins to request_vote.]]. 2021-07-28 10:11:11:441 +0300 [JRaft-Response-Processor-172.17.0.3:5004-6] INFO Replicator - Replicator Replicator [state=Replicate, statInfo=, peerId=172.17.0.3:5005, type=Follower] is going to quit 2021-07-28 10:11:11:443 +0300 [JRaft-Request-Processor-172.17.0.3:5005-3] ERROR AbstractClientService - Fail to connect 172.17.0.3:5003, exception: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: /172.17.0.3:5003. 2021-07-28 10:11:11:443 +0300 [JRaft-Request-Processor-172.17.0.3:5005-3] WARN NodeImpl - Node channel init failed, address=172.17.0.3:5003. 2021-07-28 10:11:11:675 +0300 [JRaft-Request-Processor-172.17.0.3:5005-3] INFO LocalRaftMetaStorage - Save raft meta, path=target/work/ITNodeTest/readCommittedUserLog_1627456270527/172.17.0.3_5005/meta, term=2, votedFor=172.17.0.3:5005, cost time=232 ms 2021-07-28 10:11:11:675 +0300 [main] INFO LocalRaftMetaStorage - Save raft meta, path=target/work/ITNodeTest/readCommittedUserLog_1627456270527/172.17.0.3_5004/meta, term=1, votedFor=172.17.0.3:5004, cost time=236 ms 2021-07-28 10:11:11:675 +0300 [JRaft-Request-Processor-172.17.0.3:5005-3] INFO NodeImpl - Node received TimeoutNowRequest from 172.17.0.3:5004, term=1. 2021-07-28 10:11:11:675 +0300 [main] INFO FSMCallerImpl - Shutting down FSMCaller... 2021-07-28 10:11:11:676 +0300 [main] INFO StateMachineAdapter - onShutdown. 2021-07-28 10:11:11:676 +0300 [JRaft-Request-Processor-172.17.0.3:5004-1] WARN NodeImpl - Node is not in active state, currTerm=1. 2021-07-28 10:11:11:676 +0300 [JRaft-Node-Scheduler-172.17.0.3:5004-1] INFO LogScheduledThreadPoolExecutor - ThreadPool is terminated: JRaft-Node-Scheduler-172.17.0.3:5004-, org.apache.ignite.raft.jraft.util.MetricScheduledThreadPoolExecutor@3c881fe3[Shutting down, pool size = 5, active threads = 0, queued tasks = 0, completed tasks = 6]. 2021-07-28 10:11:11:676 +0300 [main] INFO RepeatedTimer - Destroy timer: RepeatedTimer{timeout=null, stopped=true, running=false, destroyed=true, invoking=false, timeoutMs=600, name='JRaft-ElectionTimer-172.17.0.3:5004-'}. 2021-07-28 10:11:11:677 +0300 [main] INFO RepeatedTimer - Destroy timer: RepeatedTimer{timeout=null, stopped=true, running=false, destroyed=true, invoking=false, timeoutMs=600, name='JRaft-VoteTimer-172.17.0.3:5004-'}. 2021-07-28 10:11:11:677 +0300 [main] INFO RepeatedTimer - Destroy timer: RepeatedTimer{timeout=null, stopped=true, running=false, destroyed=true, invoking=false, timeoutMs=300, name='JRaft-StepDownTimer-172.17.0.3:5004-'}. 2021-07-28 10:11:11:677 +0300 [main] INFO RepeatedTimer - Destroy timer: RepeatedTimer{timeout=null, stopped=true, running=false, destroyed=true, invoking=false, timeoutMs=300000, name='JRaft-SnapshotTimer-172.17.0.3:5004-'}. 2021-07-28 10:11:11:677 +0300 [main] INFO RaftGroupService - Stop the RaftGroupService successfully. 2021-07-28 10:11:11:677 +0300 [JRaft-Response-Processor-172.17.0.3:5005-2] WARN NodeImpl - Node RequestVote to 172.17.0.3:5004 error: Status[EINVAL<1015>: Node is not in active state, state STATE_SHUTTING.]. 2021-07-28 10:11:11:678 +0300 [JRaft-Response-Processor-172.17.0.3:5004-6] INFO LogThreadPoolExecutor - ThreadPool is terminated: JRaft-Response-Processor-172.17.0.3:5004-, org.apache.ignite.raft.jraft.util.MetricThreadPoolExecutor@58814de4[Shutting down, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 7]. 2021-07-28 10:11:11:692 +0300 [main] INFO Cluster - [default:172.17.0.3:5004:194bc875-2be5-4fa4-af7a-ad7fdb6f8195@172.17.0.3:5004][doShutdown] Shutting down 2021-07-28 10:11:11:692 +0300 [JRaft-Common-Executor-172.17.0.3:5004-10] INFO LogThreadPoolExecutor - ThreadPool is terminated: JRaft-Common-Executor-172.17.0.3:5004-, org.apache.ignite.raft.jraft.util.MetricThreadPoolExecutor@58cf37b8[Shutting down, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 11]. 2021-07-28 10:11:11:693 +0300 [main] INFO Cluster - [default:172.17.0.3:5004:194bc875-2be5-4fa4-af7a-ad7fdb6f8195@172.17.0.3:5004][leaveCluster] Leaving cluster 2021-07-28 10:11:11:699 +0300 [sc-cluster-5005-111] INFO MembershipProtocol - [default:172.17.0.3:5005:5f2cd128-1aa8-498a-9115-0373e15c97a2@172.17.0.3:5005][publishEvent] MembershipEvent[type=LEAVING, member=default:172.17.0.3:5004:194bc875-2be5-4fa4-af7a-ad7fdb6f8195@172.17.0.3:5004, oldMetadata=null, newMetadata=7e16449-5, timestamp=2021-07-28T07:11:11.699Z] Jul 28, 2021 10:11:11 AM org.apache.ignite.lang.IgniteLogger logInternal INFO: Topology snapshot [nodes=3] ^-- ClusterNode [id=5f2cd128-1aa8-498a-9115-0373e15c97a2, name=172.17.0.3:5005, address=172.17.0.3:5005] ^-- ClusterNode [id=194bc875-2be5-4fa4-af7a-ad7fdb6f8195, name=172.17.0.3:5004, address=172.17.0.3:5004] ^-- ClusterNode [id=9ac3bcc0-dffd-4634-9556-370d2d9e7a8c, name=172.17.0.3:5003, address=172.17.0.3:5003] 2021-07-28 10:11:11:749 +0300 [sc-cluster-5004-110] INFO Cluster - [default:172.17.0.3:5004:194bc875-2be5-4fa4-af7a-ad7fdb6f8195@172.17.0.3:5004][leaveCluster] Left cluster Jul 28, 2021 10:11:11 AM org.apache.ignite.lang.IgniteLogger logInternal INFO: Stopping 172.17.0.3:5004 Jul 28, 2021 10:11:11 AM org.apache.ignite.lang.IgniteLogger logInternal INFO: Stopped 172.17.0.3:5004 2021-07-28 10:11:11:749 +0300 [sc-cluster-5004-110] INFO Cluster - [default:172.17.0.3:5004:194bc875-2be5-4fa4-af7a-ad7fdb6f8195@172.17.0.3:5004][doShutdown] Shutdown 2021-07-28 10:11:11:789 +0300 [main] INFO NodeImpl - Node shutdown, currTerm=2 state=STATE_CANDIDATE. 2021-07-28 10:11:12:175 +0300 [main] INFO LocalRaftMetaStorage - Save raft meta, path=target/work/ITNodeTest/readCommittedUserLog_1627456270527/172.17.0.3_5005/meta, term=2, votedFor=172.17.0.3:5005, cost time=385 ms 2021-07-28 10:11:12:175 +0300 [main] INFO FSMCallerImpl - Shutting down FSMCaller... 2021-07-28 10:11:12:176 +0300 [main] INFO StateMachineAdapter - onShutdown. 2021-07-28 10:11:12:176 +0300 [main] INFO RepeatedTimer - Destroy timer: RepeatedTimer{timeout=null, stopped=true, running=false, destroyed=true, invoking=false, timeoutMs=600, name='JRaft-ElectionTimer-172.17.0.3:5005-'}. 2021-07-28 10:11:12:176 +0300 [JRaft-Node-Scheduler-172.17.0.3:5005-1] INFO LogScheduledThreadPoolExecutor - ThreadPool is terminated: JRaft-Node-Scheduler-172.17.0.3:5005-, org.apache.ignite.raft.jraft.util.MetricScheduledThreadPoolExecutor@64c1e61e[Shutting down, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 1]. 2021-07-28 10:11:12:176 +0300 [main] INFO RepeatedTimer - Destroy timer: RepeatedTimer{timeout=null, stopped=true, running=false, destroyed=true, invoking=false, timeoutMs=600, name='JRaft-VoteTimer-172.17.0.3:5005-'}. 2021-07-28 10:11:12:176 +0300 [main] INFO RepeatedTimer - Destroy timer: RepeatedTimer{timeout=null, stopped=true, running=false, destroyed=true, invoking=false, timeoutMs=300, name='JRaft-StepDownTimer-172.17.0.3:5005-'}. 2021-07-28 10:11:12:177 +0300 [main] INFO RepeatedTimer - Destroy timer: RepeatedTimer{timeout=null, stopped=true, running=false, destroyed=true, invoking=false, timeoutMs=300000, name='JRaft-SnapshotTimer-172.17.0.3:5005-'}. 2021-07-28 10:11:12:177 +0300 [main] INFO RaftGroupService - Stop the RaftGroupService successfully. 2021-07-28 10:11:12:177 +0300 [JRaft-Response-Processor-172.17.0.3:5005-1] INFO LogThreadPoolExecutor - ThreadPool is terminated: JRaft-Response-Processor-172.17.0.3:5005-, org.apache.ignite.raft.jraft.util.MetricThreadPoolExecutor@4b43227b[Shutting down, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 3]. 2021-07-28 10:11:12:228 +0300 [main] INFO Cluster - [default:172.17.0.3:5005:5f2cd128-1aa8-498a-9115-0373e15c97a2@172.17.0.3:5005][doShutdown] Shutting down 2021-07-28 10:11:12:229 +0300 [main] INFO Cluster - [default:172.17.0.3:5005:5f2cd128-1aa8-498a-9115-0373e15c97a2@172.17.0.3:5005][leaveCluster] Leaving cluster 2021-07-28 10:11:12:234 +0300 [JRaft-Common-Executor-172.17.0.3:5005-1] INFO LogThreadPoolExecutor - ThreadPool is terminated: JRaft-Common-Executor-172.17.0.3:5005-, org.apache.ignite.raft.jraft.util.MetricThreadPoolExecutor@53889855[Shutting down, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 4]. 2021-07-28 10:11:12:282 +0300 [sc-cluster-5005-111] INFO Cluster - [default:172.17.0.3:5005:5f2cd128-1aa8-498a-9115-0373e15c97a2@172.17.0.3:5005][leaveCluster] Left cluster Jul 28, 2021 10:11:12 AM org.apache.ignite.lang.IgniteLogger logInternal INFO: Stopping 172.17.0.3:5005 Jul 28, 2021 10:11:12 AM org.apache.ignite.lang.IgniteLogger logInternal INFO: Stopped 172.17.0.3:5005 2021-07-28 10:11:12:283 +0300 [sc-cluster-5005-111] INFO Cluster - [default:172.17.0.3:5005:5f2cd128-1aa8-498a-9115-0373e15c97a2@172.17.0.3:5005][doShutdown] Shutdown 2021-07-28 10:11:12:324 +0300 [main] INFO ITNodeTest - >>>>>>>>>>>>>>> End test method: readCommittedUserLog(), cost:1797 ms.