[==========] Running 9 tests from 1 test case. [----------] Global test environment set-up. [----------] 9 tests from RaftConsensusQuorumTest [ RUN ] RaftConsensusQuorumTest.TestFollowersReplicateAndCommitMessage WARNING: Logging before InitGoogleLogging() is written to STDERR I0920 01:06:07.139546 233 fs_manager.cc:262] Metadata directory not provided I0920 01:06:07.139874 233 fs_manager.cc:268] Using write-ahead log directory (fs_wal_dir) as metadata directory I0920 01:06:07.143172 233 fs_manager.cc:608] Generated new instance metadata in path /tmp/dist-test-taskEr7w4V/test-tmp/raft_consensus_quorum-test.0.RaftConsensusQuorumTest.TestFollowersReplicateAndCommitMessage.1568941567107070-233/peer-0-root/instance: uuid: "10fe51950a6f413d88f91d3bfd891554" format_stamp: "Formatted at 2019-09-20 01:06:07 on dist-test-slave-dist-test-slave-9x72" I0920 01:06:07.158522 233 fs_manager.cc:509] Time spent creating directory manager: real 0.015s user 0.012s sys 0.002s I0920 01:06:07.159734 233 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0920 01:06:07.160296 233 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0920 01:06:07.170004 233 fs_report.cc:352] FS layout report -------------------- wal directory: metadata directory: 1 data directories: /tmp/dist-test-taskEr7w4V/test-tmp/raft_consensus_quorum-test.0.RaftConsensusQuorumTest.TestFollowersReplicateAndCommitMessage.1568941567107070-233/peer-0-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0920 01:06:07.170486 233 fs_manager.cc:425] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s I0920 01:06:07.170650 233 fs_manager.cc:442] Opened local filesystem: /tmp/dist-test-taskEr7w4V/test-tmp/raft_consensus_quorum-test.0.RaftConsensusQuorumTest.TestFollowersReplicateAndCommitMessage.1568941567107070-233/peer-0-root uuid: "10fe51950a6f413d88f91d3bfd891554" format_stamp: "Formatted at 2019-09-20 01:06:07 on dist-test-slave-dist-test-slave-9x72" I0920 01:06:07.171710 233 log.cc:525] T TestTablet P 10fe51950a6f413d88f91d3bfd891554: Log is configured to *not* fsync() on all Append() calls I0920 01:06:07.181075 233 fs_manager.cc:262] Metadata directory not provided I0920 01:06:07.181229 233 fs_manager.cc:268] Using write-ahead log directory (fs_wal_dir) as metadata directory I0920 01:06:07.182009 233 fs_manager.cc:608] Generated new instance metadata in path /tmp/dist-test-taskEr7w4V/test-tmp/raft_consensus_quorum-test.0.RaftConsensusQuorumTest.TestFollowersReplicateAndCommitMessage.1568941567107070-233/peer-1-root/instance: uuid: "d2597a2195994ee59827504b1e0aacd0" format_stamp: "Formatted at 2019-09-20 01:06:07 on dist-test-slave-dist-test-slave-9x72" I0920 01:06:07.191217 233 fs_manager.cc:509] Time spent creating directory manager: real 0.009s user 0.010s sys 0.000s I0920 01:06:07.192461 233 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0920 01:06:07.199542 233 fs_report.cc:352] FS layout report -------------------- wal directory: metadata directory: 1 data directories: /tmp/dist-test-taskEr7w4V/test-tmp/raft_consensus_quorum-test.0.RaftConsensusQuorumTest.TestFollowersReplicateAndCommitMessage.1568941567107070-233/peer-1-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0920 01:06:07.199837 233 fs_manager.cc:425] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s I0920 01:06:07.199968 233 fs_manager.cc:442] Opened local filesystem: /tmp/dist-test-taskEr7w4V/test-tmp/raft_consensus_quorum-test.0.RaftConsensusQuorumTest.TestFollowersReplicateAndCommitMessage.1568941567107070-233/peer-1-root uuid: "d2597a2195994ee59827504b1e0aacd0" format_stamp: "Formatted at 2019-09-20 01:06:07 on dist-test-slave-dist-test-slave-9x72" I0920 01:06:07.208426 233 fs_manager.cc:262] Metadata directory not provided I0920 01:06:07.208583 233 fs_manager.cc:268] Using write-ahead log directory (fs_wal_dir) as metadata directory I0920 01:06:07.209354 233 fs_manager.cc:608] Generated new instance metadata in path /tmp/dist-test-taskEr7w4V/test-tmp/raft_consensus_quorum-test.0.RaftConsensusQuorumTest.TestFollowersReplicateAndCommitMessage.1568941567107070-233/peer-2-root/instance: uuid: "0fb3b9d84842460db0d64f8b78d1dc9a" format_stamp: "Formatted at 2019-09-20 01:06:07 on dist-test-slave-dist-test-slave-9x72" I0920 01:06:07.217293 233 fs_manager.cc:509] Time spent creating directory manager: real 0.008s user 0.008s sys 0.000s I0920 01:06:07.218399 233 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0920 01:06:07.225747 233 fs_report.cc:352] FS layout report -------------------- wal directory: metadata directory: 1 data directories: /tmp/dist-test-taskEr7w4V/test-tmp/raft_consensus_quorum-test.0.RaftConsensusQuorumTest.TestFollowersReplicateAndCommitMessage.1568941567107070-233/peer-2-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0920 01:06:07.226143 233 fs_manager.cc:425] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s I0920 01:06:07.226325 233 fs_manager.cc:442] Opened local filesystem: /tmp/dist-test-taskEr7w4V/test-tmp/raft_consensus_quorum-test.0.RaftConsensusQuorumTest.TestFollowersReplicateAndCommitMessage.1568941567107070-233/peer-2-root uuid: "0fb3b9d84842460db0d64f8b78d1dc9a" format_stamp: "Formatted at 2019-09-20 01:06:07 on dist-test-slave-dist-test-slave-9x72" W0920 01:06:08.134562 233 thread.cc:599] rpc reactor (reactor) Time spent creating pthread: real 0.859s user 0.246s sys 0.585s W0920 01:06:08.134783 233 thread.cc:566] rpc reactor (reactor) Time spent starting thread: real 0.859s user 0.246s sys 0.585s I0920 01:06:08.226173 233 raft_consensus.cc:340] T TestTablet P 10fe51950a6f413d88f91d3bfd891554 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 peers { permanent_uuid: "10fe51950a6f413d88f91d3bfd891554" member_type: VOTER last_known_addr { host: "peer-0.fake-domain-for-tests" port: 0 } } peers { permanent_uuid: "d2597a2195994ee59827504b1e0aacd0" member_type: VOTER last_known_addr { host: "peer-1.fake-domain-for-tests" port: 0 } } peers { permanent_uuid: "0fb3b9d84842460db0d64f8b78d1dc9a" member_type: VOTER last_known_addr { host: "peer-2.fake-domain-for-tests" port: 0 } } I0920 01:06:08.226939 233 raft_consensus.cc:705] T TestTablet P 10fe51950a6f413d88f91d3bfd891554 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10fe51950a6f413d88f91d3bfd891554, State: Initialized, Role: FOLLOWER I0920 01:06:08.227428 233 consensus_queue.cc:229] T TestTablet P 10fe51950a6f413d88f91d3bfd891554 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "10fe51950a6f413d88f91d3bfd891554" member_type: VOTER last_known_addr { host: "peer-0.fake-domain-for-tests" port: 0 } } peers { permanent_uuid: "d2597a2195994ee59827504b1e0aacd0" member_type: VOTER last_known_addr { host: "peer-1.fake-domain-for-tests" port: 0 } } peers { permanent_uuid: "0fb3b9d84842460db0d64f8b78d1dc9a" member_type: VOTER last_known_addr { host: "peer-2.fake-domain-for-tests" port: 0 } } I0920 01:06:08.239451 233 raft_consensus.cc:340] T TestTablet P d2597a2195994ee59827504b1e0aacd0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 peers { permanent_uuid: "10fe51950a6f413d88f91d3bfd891554" member_type: VOTER last_known_addr { host: "peer-0.fake-domain-for-tests" port: 0 } } peers { permanent_uuid: "d2597a2195994ee59827504b1e0aacd0" member_type: VOTER last_known_addr { host: "peer-1.fake-domain-for-tests" port: 0 } } peers { permanent_uuid: "0fb3b9d84842460db0d64f8b78d1dc9a" member_type: VOTER last_known_addr { host: "peer-2.fake-domain-for-tests" port: 0 } } I0920 01:06:08.239938 233 raft_consensus.cc:705] T TestTablet P d2597a2195994ee59827504b1e0aacd0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2597a2195994ee59827504b1e0aacd0, State: Initialized, Role: FOLLOWER I0920 01:06:08.240270 233 consensus_queue.cc:229] T TestTablet P d2597a2195994ee59827504b1e0aacd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "10fe51950a6f413d88f91d3bfd891554" member_type: VOTER last_known_addr { host: "peer-0.fake-domain-for-tests" port: 0 } } peers { permanent_uuid: "d2597a2195994ee59827504b1e0aacd0" member_type: VOTER last_known_addr { host: "peer-1.fake-domain-for-tests" port: 0 } } peers { permanent_uuid: "0fb3b9d84842460db0d64f8b78d1dc9a" member_type: VOTER last_known_addr { host: "peer-2.fake-domain-for-tests" port: 0 } } I0920 01:06:08.250180 233 raft_consensus.cc:340] T TestTablet P 0fb3b9d84842460db0d64f8b78d1dc9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 peers { permanent_uuid: "10fe51950a6f413d88f91d3bfd891554" member_type: VOTER last_known_addr { host: "peer-0.fake-domain-for-tests" port: 0 } } peers { permanent_uuid: "d2597a2195994ee59827504b1e0aacd0" member_type: VOTER last_known_addr { host: "peer-1.fake-domain-for-tests" port: 0 } } peers { permanent_uuid: "0fb3b9d84842460db0d64f8b78d1dc9a" member_type: VOTER last_known_addr { host: "peer-2.fake-domain-for-tests" port: 0 } } I0920 01:06:08.250649 233 raft_consensus.cc:705] T TestTablet P 0fb3b9d84842460db0d64f8b78d1dc9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fb3b9d84842460db0d64f8b78d1dc9a, State: Initialized, Role: FOLLOWER I0920 01:06:08.251008 233 consensus_queue.cc:229] T TestTablet P 0fb3b9d84842460db0d64f8b78d1dc9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "10fe51950a6f413d88f91d3bfd891554" member_type: VOTER last_known_addr { host: "peer-0.fake-domain-for-tests" port: 0 } } peers { permanent_uuid: "d2597a2195994ee59827504b1e0aacd0" member_type: VOTER last_known_addr { host: "peer-1.fake-domain-for-tests" port: 0 } } peers { permanent_uuid: "0fb3b9d84842460db0d64f8b78d1dc9a" member_type: VOTER last_known_addr { host: "peer-2.fake-domain-for-tests" port: 0 } } I0920 01:06:08.251714 233 raft_consensus.cc:405] T TestTablet P 0fb3b9d84842460db0d64f8b78d1dc9a [term 0 FOLLOWER]: Emulating election... I0920 01:06:08.251874 233 raft_consensus.cc:2890] T TestTablet P 0fb3b9d84842460db0d64f8b78d1dc9a [term 0 FOLLOWER]: Advancing to term 1 I0920 01:06:08.281486 233 raft_consensus.cc:667] T TestTablet P 0fb3b9d84842460db0d64f8b78d1dc9a [term 1 LEADER]: Becoming Leader. State: Replica: 0fb3b9d84842460db0d64f8b78d1dc9a, State: Running, Role: LEADER I0920 01:06:08.282117 233 consensus_queue.cc:206] T TestTablet P 0fb3b9d84842460db0d64f8b78d1dc9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "10fe51950a6f413d88f91d3bfd891554" member_type: VOTER last_known_addr { host: "peer-0.fake-domain-for-tests" port: 0 } } peers { permanent_uuid: "d2597a2195994ee59827504b1e0aacd0" member_type: VOTER last_known_addr { host: "peer-1.fake-domain-for-tests" port: 0 } } peers { permanent_uuid: "0fb3b9d84842460db0d64f8b78d1dc9a" member_type: VOTER last_known_addr { host: "peer-2.fake-domain-for-tests" port: 0 } } I0920 01:06:08.291160 264 raft_consensus.cc:2890] T TestTablet P d2597a2195994ee59827504b1e0aacd0 [term 0 FOLLOWER]: Advancing to term 1 I0920 01:06:08.298748 264 raft_consensus.cc:1184] T TestTablet P d2597a2195994ee59827504b1e0aacd0 [term 1 FOLLOWER]: Refusing update from remote peer 0fb3b9d84842460db0d64f8b78d1dc9a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0920 01:06:08.294746 265 raft_consensus.cc:2890] T TestTablet P 10fe51950a6f413d88f91d3bfd891554 [term 0 FOLLOWER]: Advancing to term 1 I0920 01:06:08.328393 253 consensus_queue.cc:984] T TestTablet P 0fb3b9d84842460db0d64f8b78d1dc9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d2597a2195994ee59827504b1e0aacd0" member_type: VOTER last_known_addr { host: "peer-1.fake-domain-for-tests" port: 0 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s I0920 01:06:08.330902 265 raft_consensus.cc:1184] T TestTablet P 10fe51950a6f413d88f91d3bfd891554 [term 1 FOLLOWER]: Refusing update from remote peer 0fb3b9d84842460db0d64f8b78d1dc9a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0920 01:06:08.331522 253 consensus_queue.cc:984] T TestTablet P 0fb3b9d84842460db0d64f8b78d1dc9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "10fe51950a6f413d88f91d3bfd891554" member_type: VOTER last_known_addr { host: "peer-0.fake-domain-for-tests" port: 0 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s I0920 01:06:08.340595 233 raft_consensus.cc:2147] T TestTablet P 10fe51950a6f413d88f91d3bfd891554 [term 1 FOLLOWER]: Raft consensus shutting down. I0920 01:06:08.341015 233 raft_consensus.cc:2174] T TestTablet P 10fe51950a6f413d88f91d3bfd891554 [term 1 FOLLOWER]: Raft consensus is shut down! I0920 01:06:08.341250 233 raft_consensus.cc:2147] T TestTablet P d2597a2195994ee59827504b1e0aacd0 [term 1 FOLLOWER]: Raft consensus shutting down. I0920 01:06:08.341475 233 raft_consensus.cc:2174] T TestTablet P d2597a2195994ee59827504b1e0aacd0 [term 1 FOLLOWER]: Raft consensus is shut down! I0920 01:06:08.341645 233 raft_consensus.cc:2147] T TestTablet P 0fb3b9d84842460db0d64f8b78d1dc9a [term 1 LEADER]: Raft consensus shutting down. I0920 01:06:08.342125 233 raft_consensus.cc:2174] T TestTablet P 0fb3b9d84842460db0d64f8b78d1dc9a [term 1 FOLLOWER]: Raft consensus is shut down! [ OK ] RaftConsensusQuorumTest.TestFollowersReplicateAndCommitMessage (1240 ms) [ RUN ] RaftConsensusQuorumTest.TestFollowersReplicateAndCommitSequence I0920 01:06:08.374454 233 fs_manager.cc:262] Metadata directory not provided I0920 01:06:08.374630 233 fs_manager.cc:268] Using write-ahead log directory (fs_wal_dir) as metadata directory I0920 01:06:08.375551 233 fs_manager.cc:608] Generated new instance metadata in path /tmp/dist-test-taskEr7w4V/test-tmp/raft_consensus_quorum-test.0.RaftConsensusQuorumTest.TestFollowersReplicateAndCommitSequence.1568941567107070-233/peer-0-root/instance: uuid: "efbe8ff62e8c4356bb4efb0d388a13b1" format_stamp: "Formatted at 2019-09-20 01:06:08 on dist-test-slave-dist-test-slave-9x72" I0920 01:06:08.384547 233 fs_manager.cc:509] Time spent creating directory manager: real 0.009s user 0.006s sys 0.003s I0920 01:06:08.385754 233 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0920 01:06:08.392676 233 fs_report.cc:352] FS layout report -------------------- wal directory: metadata directory: 1 data directories: /tmp/dist-test-taskEr7w4V/test-tmp/raft_consensus_quorum-test.0.RaftConsensusQuorumTest.TestFollowersReplicateAndCommitSequence.1568941567107070-233/peer-0-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0920 01:06:08.393031 233 fs_manager.cc:425] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s I0920 01:06:08.393162 233 fs_manager.cc:442] Opened local filesystem: /tmp/dist-test-taskEr7w4V/test-tmp/raft_consensus_quorum-test.0.RaftConsensusQuorumTest.TestFollowersReplicateAndCommitSequence.1568941567107070-233/peer-0-root uuid: "efbe8ff62e8c4356bb4efb0d388a13b1" format_stamp: "Formatted at 2019-09-20 01:06:08 on dist-test-slave-dist-test-slave-9x72" I0920 01:06:08.401214 233 fs_manager.cc:262] Metadata directory not provided I0920 01:06:08.401379 233 fs_manager.cc:268] Using write-ahead log directory (fs_wal_dir) as metadata directory I0920 01:06:08.402163 233 fs_manager.cc:608] Generated new instance metadata in path /tmp/dist-test-taskEr7w4V/test-tmp/raft_consensus_quorum-test.0.RaftConsensusQuorumTest.TestFollowersReplicateAndCommitSequence.1568941567107070-233/peer-1-root/instance: uuid: "d0393a5a815949c0b1cd668f30ed3922" format_stamp: "Formatted at 2019-09-20 01:06:08 on dist-test-slave-dist-test-slave-9x72" I0920 01:06:08.409407 233 fs_manager.cc:509] Time spent creating directory manager: real 0.007s user 0.008s sys 0.000s I0920 01:06:08.410444 233 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0920 01:06:08.416301 233 fs_report.cc:352] FS layout report -------------------- wal directory: metadata directory: 1 data directories: /tmp/dist-test-taskEr7w4V/test-tmp/raft_consensus_quorum-test.0.RaftConsensusQuorumTest.TestFollowersReplicateAndCommitSequence.1568941567107070-233/peer-1-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0920 01:06:08.416647 233 fs_manager.cc:425] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s I0920 01:06:08.416769 233 fs_manager.cc:442] Opened local filesystem: /tmp/dist-test-taskEr7w4V/test-tmp/raft_consensus_quorum-test.0.RaftConsensusQuorumTest.TestFollowersReplicateAndCommitSequence.1568941567107070-233/peer-1-root uuid: "d0393a5a815949c0b1cd668f30ed3922" format_stamp: "Formatted at 2019-09-20 01:06:08 on dist-test-slave-dist-test-slave-9x72" I0920 01:06:08.423832 233 fs_manager.cc:262] Metadata directory not provided I0920 01:06:08.423995 233 fs_manager.cc:268] Using write-ahead log directory (fs_wal_dir) as metadata directory I0920 01:06:08.424779 233 fs_manager.cc:608] Generated new instance metadata in path /tmp/dist-test-taskEr7w4V/test-tmp/raft_consensus_quorum-test.0.RaftConsensusQuorumTest.TestFollowersReplicateAndCommitSequence.1568941567107070-233/peer-2-root/instance: uuid: "8f4d9cd6ac0f4f459a5ced71d21d6a5a" format_stamp: "Formatted at 2019-09-20 01:06:08 on dist-test-slave-dist-test-slave-9x72" I0920 01:06:08.432469 233 fs_manager.cc:509] Time spent creating directory manager: real 0.007s user 0.008s sys 0.000s I0920 01:06:08.433538 233 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0920 01:06:08.439553 233 fs_report.cc:352] FS layout report -------------------- wal directory: metadata directory: 1 data directories: /tmp/dist-test-taskEr7w4V/test-tmp/raft_consensus_quorum-test.0.RaftConsensusQuorumTest.TestFollowersReplicateAndCommitSequence.1568941567107070-233/peer-2-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0920 01:06:08.439896 233 fs_manager.cc:425] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s I0920 01:06:08.440059 233 fs_manager.cc:442] Opened local filesystem: /tmp/dist-test-taskEr7w4V/test-tmp/raft_consensus_quorum-test.0.RaftConsensusQuorumTest.TestFollowersReplicateAndCommitSequence.1568941567107070-233/peer-2-root uuid: "8f4d9cd6ac0f4f459a5ced71d21d6a5a" format_stamp: "Formatted at 2019-09-20 01:06:08 on dist-test-slave-dist-test-slave-9x72" I0920 01:06:08.454686 233 raft_consensus.cc:340] T TestTablet P efbe8ff62e8c4356bb4efb0d388a13b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 peers { permanent_uuid: "efbe8ff62e8c4356bb4efb0d388a13b1" member_type: VOTER last_known_addr { host: "peer-0.fake-domain-for-tests" port: 0 } } peers { permanent_uuid: "d0393a5a815949c0b1cd668f30ed3922" member_type: VOTER last_known_addr { host: "peer-1.fake-domain-for-tests" port: 0 } } peers { permanent_uuid: "8f4d9cd6ac0f4f459a5ced71d21d6a5a" member_type: VOTER last_known_addr { host: "peer-2.fake-domain-for-tests" port: 0 } } I0920 01:06:08.455135 233 raft_consensus.cc:705] T TestTablet P efbe8ff62e8c4356bb4efb0d388a13b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efbe8ff62e8c4356bb4efb0d388a13b1, State: Initialized, Role: FOLLOWER I0920 01:06:08.455479 233 consensus_queue.cc:229] T TestTablet P efbe8ff62e8c4356bb4efb0d388a13b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "efbe8ff62e8c4356bb4efb0d388a13b1" member_type: VOTER last_known_addr { host: "peer-0.fake-domain-for-tests" port: 0 } } peers { permanent_uuid: "d0393a5a815949c0b1cd668f30ed3922" member_type: VOTER last_known_addr { host: "peer-1.fake-domain-for-tests" port: 0 } } peers { permanent_uuid: "8f4d9cd6ac0f4f459a5ced71d21d6a5a" member_type: VOTER last_known_addr { host: "peer-2.fake-domain-for-tests" port: 0 } } I0920 01:06:08.462046 233 raft_consensus.cc:340] T TestTablet P d0393a5a815949c0b1cd668f30ed3922 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 peers { permanent_uuid: "efbe8ff62e8c4356bb4efb0d388a13b1" member_type: VOTER last_known_addr { host: "peer-0.fake-domain-for-tests" port: 0 } } peers { permanent_uuid: "d0393a5a815949c0b1cd668f30ed3922" member_type: VOTER last_known_addr { host: "peer-1.fake-domain-for-tests" port: 0 } } peers { permanent_uuid: "8f4d9cd6ac0f4f459a5ced71d21d6a5a" member_type: VOTER last_known_addr { host: "peer-2.fake-domain-for-tests" port: 0 } } I0920 01:06:08.462498 233 raft_consensus.cc:705] T TestTablet P d0393a5a815949c0b1cd668f30ed3922 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0393a5a815949c0b1cd668f30ed3922, State: Initialized, Role: FOLLOWER I0920 01:06:08.462787 233 consensus_queue.cc:229] T TestTablet P d0393a5a815949c0b1cd668f30ed3922 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "efbe8ff62e8c4356bb4efb0d388a13b1" member_type: VOTER last_known_addr { host: "peer-0.fake-domain-for-tests" port: 0 } } peers { permanent_uuid: "d0393a5a815949c0b1cd668f30ed3922" member_type: VOTER last_known_addr { host: "peer-1.fake-domain-for-tests" port: 0 } } peers { permanent_uuid: "8f4d9cd6ac0f4f459a5ced71d21d6a5a" member_type: VOTER last_known_addr { host: "peer-2.fake-domain-for-tests" port: 0 } } I0920 01:06:08.470396 233 raft_consensus.cc:340] T TestTablet P 8f4d9cd6ac0f4f459a5ced71d21d6a5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 peers { permanent_uuid: "efbe8ff62e8c4356bb4efb0d388a13b1" member_type: VOTER last_known_addr { host: "peer-0.fake-domain-for-tests" port: 0 } } peers { permanent_uuid: "d0393a5a815949c0b1cd668f30ed3922" member_type: VOTER last_known_addr { host: "peer-1.fake-domain-for-tests" port: 0 } } peers { permanent_uuid: "8f4d9cd6ac0f4f459a5ced71d21d6a5a" member_type: VOTER last_known_addr { host: "peer-2.fake-domain-for-tests" port: 0 } } I0920 01:06:08.470850 233 raft_consensus.cc:705] T TestTablet P 8f4d9cd6ac0f4f459a5ced71d21d6a5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f4d9cd6ac0f4f459a5ced71d21d6a5a, State: Initialized, Role: FOLLOWER I0920 01:06:08.471153 233 consensus_queue.cc:229] T TestTablet P 8f4d9cd6ac0f4f459a5ced71d21d6a5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "efbe8ff62e8c4356bb4efb0d388a13b1" member_type: VOTER last_known_addr { host: "peer-0.fake-domain-for-tests" port: 0 } } peers { permanent_uuid: "d0393a5a815949c0b1cd668f30ed3922" member_type: VOTER last_known_addr { host: "peer-1.fake-domain-for-tests" port: 0 } } peers { permanent_uuid: "8f4d9cd6ac0f4f459a5ced71d21d6a5a" member_type: VOTER last_known_addr { host: "peer-2.fake-domain-for-tests" port: 0 } } I0920 01:06:08.471735 233 raft_consensus.cc:405] T TestTablet P 8f4d9cd6ac0f4f459a5ced71d21d6a5a [term 0 FOLLOWER]: Emulating election... I0920 01:06:08.471873 233 raft_consensus.cc:2890] T TestTablet P 8f4d9cd6ac0f4f459a5ced71d21d6a5a [term 0 FOLLOWER]: Advancing to term 1 I0920 01:06:08.474877 233 raft_consensus.cc:667] T TestTablet P 8f4d9cd6ac0f4f459a5ced71d21d6a5a [term 1 LEADER]: Becoming Leader. State: Replica: 8f4d9cd6ac0f4f459a5ced71d21d6a5a, State: Running, Role: LEADER I0920 01:06:08.475220 233 consensus_queue.cc:206] T TestTablet P 8f4d9cd6ac0f4f459a5ced71d21d6a5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "efbe8ff62e8c4356bb4efb0d388a13b1" member_type: VOTER last_known_addr { host: "peer-0.fake-domain-for-tests" port: 0 } } peers { permanent_uuid: "d0393a5a815949c0b1cd668f30ed3922" member_type: VOTER last_known_addr { host: "peer-1.fake-domain-for-tests" port: 0 } } peers { permanent_uuid: "8f4d9cd6ac0f4f459a5ced71d21d6a5a" member_type: VOTER last_known_addr { host: "peer-2.fake-domain-for-tests" port: 0 } } I0920 01:06:08.481475 294 raft_consensus.cc:2890] T TestTablet P d0393a5a815949c0b1cd668f30ed3922 [term 0 FOLLOWER]: Advancing to term 1 I0920 01:06:08.484097 295 raft_consensus.cc:2890] T TestTablet P efbe8ff62e8c4356bb4efb0d388a13b1 [term 0 FOLLOWER]: Advancing to term 1 I0920 01:06:08.488765 294 raft_consensus.cc:1184] T TestTablet P d0393a5a815949c0b1cd668f30ed3922 [term 1 FOLLOWER]: Refusing update from remote peer 8f4d9cd6ac0f4f459a5ced71d21d6a5a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0920 01:06:08.489156 295 raft_consensus.cc:1184] T TestTablet P efbe8ff62e8c4356bb4efb0d388a13b1 [term 1 FOLLOWER]: Refusing update from remote peer 8f4d9cd6ac0f4f459a5ced71d21d6a5a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0920 01:06:08.489791 293 consensus_queue.cc:984] T TestTablet P 8f4d9cd6ac0f4f459a5ced71d21d6a5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0393a5a815949c0b1cd668f30ed3922" member_type: VOTER last_known_addr { host: "peer-1.fake-domain-for-tests" port: 0 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0920 01:06:08.490505 283 consensus_queue.cc:984] T TestTablet P 8f4d9cd6ac0f4f459a5ced71d21d6a5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "efbe8ff62e8c4356bb4efb0d388a13b1" member_type: VOTER last_known_addr { host: "peer-0.fake-domain-for-tests" port: 0 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s I0920 01:06:08.831518 233 raft_consensus.cc:2147] T TestTablet P efbe8ff62e8c4356bb4efb0d388a13b1 [term 1 FOLLOWER]: Raft consensus shutting down. I0920 01:06:08.831782 233 raft_consensus.cc:2174] T TestTablet P efbe8ff62e8c4356bb4efb0d388a13b1 [term 1 FOLLOWER]: Raft consensus is shut down! I0920 01:06:08.831935 233 raft_consensus.cc:2147] T TestTablet P d0393a5a815949c0b1cd668f30ed3922 [term 1 FOLLOWER]: Raft consensus shutting down. I0920 01:06:08.832090 233 raft_consensus.cc:2174] T TestTablet P d0393a5a815949c0b1cd668f30ed3922 [term 1 FOLLOWER]: Raft consensus is shut down! I0920 01:06:08.832264 233 raft_consensus.cc:2147] T TestTablet P 8f4d9cd6ac0f4f459a5ced71d21d6a5a [term 1 LEADER]: Raft consensus shutting down. I0920 01:06:08.832563 233 raft_consensus.cc:2174] T TestTablet P 8f4d9cd6ac0f4f459a5ced71d21d6a5a [term 1 FOLLOWER]: Raft consensus is shut down! [ OK ] RaftConsensusQuorumTest.TestFollowersReplicateAndCommitSequence (537 ms) [ RUN ] RaftConsensusQuorumTest.TestConsensusContinuesIfAMinorityFallsBehind I0920 01:06:08.910951 233 fs_manager.cc:262] Metadata directory not provided I0920 01:06:08.911113 233 fs_manager.cc:268] Using write-ahead log directory (fs_wal_dir) as metadata directory I0920 01:06:08.912000 233 fs_manager.cc:608] Generated new instance metadata in path /tmp/dist-test-taskEr7w4V/test-tmp/raft_consensus_quorum-test.0.RaftConsensusQuorumTest.TestConsensusContinuesIfAMinorityFallsBehind.1568941567107070-233/peer-0-root/instance: uuid: "17fcde7f158c4fe2b9b78aabef4ad20c" format_stamp: "Formatted at 2019-09-20 01:06:08 on dist-test-slave-dist-test-slave-9x72" I0920 01:06:08.919602 233 fs_manager.cc:509] Time spent creating directory manager: real 0.007s user 0.008s sys 0.000s I0920 01:06:08.920749 233 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0920 01:06:08.926497 233 fs_report.cc:352] FS layout report -------------------- wal directory: metadata directory: 1 data directories: /tmp/dist-test-taskEr7w4V/test-tmp/raft_consensus_quorum-test.0.RaftConsensusQuorumTest.TestConsensusContinuesIfAMinorityFallsBehind.1568941567107070-233/peer-0-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0920 01:06:08.926834 233 fs_manager.cc:425] Time spent opening block manager: real 0.002s user 0.000s sys 0.000s I0920 01:06:08.926936 233 fs_manager.cc:442] Opened local filesystem: /tmp/dist-test-taskEr7w4V/test-tmp/raft_consensus_quorum-test.0.RaftConsensusQuorumTest.TestConsensusContinuesIfAMinorityFallsBehind.1568941567107070-233/peer-0-root uuid: "17fcde7f158c4fe2b9b78aabef4ad20c" format_stamp: "Formatted at 2019-09-20 01:06:08 on dist-test-slave-dist-test-slave-9x72" I0920 01:06:08.934445 233 fs_manager.cc:262] Metadata directory not provided I0920 01:06:08.934607 233 fs_manager.cc:268] Using write-ahead log directory (fs_wal_dir) as metadata directory I0920 01:06:08.935451 233 fs_manager.cc:608] Generated new instance metadata in path /tmp/dist-test-taskEr7w4V/test-tmp/raft_consensus_quorum-test.0.RaftConsensusQuorumTest.TestConsensusContinuesIfAMinorityFallsBehind.1568941567107070-233/peer-1-root/instance: uuid: "33a35b5c6cc24f6db9a9541fe06ce281" format_stamp: "Formatted at 2019-09-20 01:06:08 on dist-test-slave-dist-test-slave-9x72" I0920 01:06:08.942757 233 fs_manager.cc:509] Time spent creating directory manager: real 0.007s user 0.005s sys 0.001s I0920 01:06:08.943881 233 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0920 01:06:08.949671 233 fs_report.cc:352] FS layout report -------------------- wal directory: metadata directory: 1 data directories: /tmp/dist-test-taskEr7w4V/test-tmp/raft_consensus_quorum-test.0.RaftConsensusQuorumTest.TestConsensusContinuesIfAMinorityFallsBehind.1568941567107070-233/peer-1-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0920 01:06:08.949965 233 fs_manager.cc:425] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s I0920 01:06:08.950070 233 fs_manager.cc:442] Opened local filesystem: /tmp/dist-test-taskEr7w4V/test-tmp/raft_consensus_quorum-test.0.RaftConsensusQuorumTest.TestConsensusContinuesIfAMinorityFallsBehind.1568941567107070-233/peer-1-root uuid: "33a35b5c6cc24f6db9a9541fe06ce281" format_stamp: "Formatted at 2019-09-20 01:06:08 on dist-test-slave-dist-test-slave-9x72" I0920 01:06:08.957247 233 fs_manager.cc:262] Metadata directory not provided I0920 01:06:08.957414 233 fs_manager.cc:268] Using write-ahead log directory (fs_wal_dir) as metadata directory I0920 01:06:08.958218 233 fs_manager.cc:608] Generated new instance metadata in path /tmp/dist-test-taskEr7w4V/test-tmp/raft_consensus_quorum-test.0.RaftConsensusQuorumTest.TestConsensusContinuesIfAMinorityFallsBehind.1568941567107070-233/peer-2-root/instance: uuid: "51455ae8ca8a478b97f86fb10d50d8f0" format_stamp: "Formatted at 2019-09-20 01:06:08 on dist-test-slave-dist-test-slave-9x72" I0920 01:06:08.965824 233 fs_manager.cc:509] Time spent creating directory manager: real 0.007s user 0.004s sys 0.004s I0920 01:06:08.966892 233 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0920 01:06:08.972852 233 fs_report.cc:352] FS layout report -------------------- wal directory: metadata directory: 1 data directories: /tmp/dist-test-taskEr7w4V/test-tmp/raft_consensus_quorum-test.0.RaftConsensusQuorumTest.TestConsensusContinuesIfAMinorityFallsBehind.1568941567107070-233/peer-2-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0920 01:06:08.973189 233 fs_manager.cc:425] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s I0920 01:06:08.973306 233 fs_manager.cc:442] Opened local filesystem: /tmp/dist-test-taskEr7w4V/test-tmp/raft_consensus_quorum-test.0.RaftConsensusQuorumTest.TestConsensusContinuesIfAMinorityFallsBehind.1568941567107070-233/peer-2-root uuid: "51455ae8ca8a478b97f86fb10d50d8f0" format_stamp: "Formatted at 2019-09-20 01:06:08 on dist-test-slave-dist-test-slave-9x72" I0920 01:06:08.988523 233 raft_consensus.cc:340] T TestTablet P 17fcde7f158c4fe2b9b78aabef4ad20c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 peers { permanent_uuid: "17fcde7f158c4fe2b9b78aabef4ad20c" member_type: VOTER last_known_addr { host: "peer-0.fake-domain-for-tests" port: 0 } } peers { permanent_uuid: "33a35b5c6cc24f6db9a9541fe06ce281" member_type: VOTER last_known_addr { host: "peer-1.fake-domain-for-tests" port: 0 } } peers { permanent_uuid: "51455ae8ca8a478b97f86fb10d50d8f0" member_type: VOTER last_known_addr { host: "peer-2.fake-domain-for-tests" port: 0 } } I0920 01:06:08.988961 233 raft_consensus.cc:705] T TestTablet P 17fcde7f158c4fe2b9b78aabef4ad20c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17fcde7f158c4fe2b9b78aabef4ad20c, State: Initialized, Role: FOLLOWER I0920 01:06:08.989203 233 consensus_queue.cc:229] T TestTablet P 17fcde7f158c4fe2b9b78aabef4ad20c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "17fcde7f158c4fe2b9b78aabef4ad20c" member_type: VOTER last_known_addr { host: "peer-0.fake-domain-for-tests" port: 0 } } peers { permanent_uuid: "33a35b5c6cc24f6db9a9541fe06ce281" member_type: VOTER last_known_addr { host: "peer-1.fake-domain-for-tests" port: 0 } } peers { permanent_uuid: "51455ae8ca8a478b97f86fb10d50d8f0" member_type: VOTER last_known_addr { host: "peer-2.fake-domain-for-tests" port: 0 } } I0920 01:06:08.994544 233 raft_consensus.cc:340] T TestTablet P 33a35b5c6cc24f6db9a9541fe06ce281 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 peers { permanent_uuid: "17fcde7f158c4fe2b9b78aabef4ad20c" member_type: VOTER last_known_addr { host: "peer-0.fake-domain-for-tests" port: 0 } } peers { permanent_uuid: "33a35b5c6cc24f6db9a9541fe06ce281" member_type: VOTER last_known_addr { host: "peer-1.fake-domain-for-tests" port: 0 } } peers { permanent_uuid: "51455ae8ca8a478b97f86fb10d50d8f0" member_type: VOTER last_known_addr { host: "peer-2.fake-domain-for-tests" port: 0 } } I0920 01:06:08.994964 233 raft_consensus.cc:705] T TestTablet P 33a35b5c6cc24f6db9a9541fe06ce281 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33a35b5c6cc24f6db9a9541fe06ce281, State: Initialized, Role: FOLLOWER I0920 01:06:08.995203 233 consensus_queue.cc:229] T TestTablet P 33a35b5c6cc24f6db9a9541fe06ce281 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "17fcde7f158c4fe2b9b78aabef4ad20c" member_type: VOTER last_known_addr { host: "peer-0.fake-domain-for-tests" port: 0 } } peers { permanent_uuid: "33a35b5c6cc24f6db9a9541fe06ce281" member_type: VOTER last_known_addr { host: "peer-1.fake-domain-for-tests" port: 0 } } peers { permanent_uuid: "51455ae8ca8a478b97f86fb10d50d8f0" member_type: VOTER last_known_addr { host: "peer-2.fake-domain-for-tests" port: 0 } } I0920 01:06:08.999691 233 raft_consensus.cc:340] T TestTablet P 51455ae8ca8a478b97f86fb10d50d8f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 peers { permanent_uuid: "17fcde7f158c4fe2b9b78aabef4ad20c" member_type: VOTER last_known_addr { host: "peer-0.fake-domain-for-tests" port: 0 } } peers { permanent_uuid: "33a35b5c6cc24f6db9a9541fe06ce281" member_type: VOTER last_known_addr { host: "peer-1.fake-domain-for-tests" port: 0 } } peers { permanent_uuid: "51455ae8ca8a478b97f86fb10d50d8f0" member_type: VOTER last_known_addr { host: "peer-2.fake-domain-for-tests" port: 0 } } I0920 01:06:09.000110 233 raft_consensus.cc:705] T TestTablet P 51455ae8ca8a478b97f86fb10d50d8f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51455ae8ca8a478b97f86fb10d50d8f0, State: Initialized, Role: FOLLOWER I0920 01:06:09.000360 233 consensus_queue.cc:229] T TestTablet P 51455ae8ca8a478b97f86fb10d50d8f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "17fcde7f158c4fe2b9b78aabef4ad20c" member_type: VOTER last_known_addr { host: "peer-0.fake-domain-for-tests" port: 0 } } peers { permanent_uuid: "33a35b5c6cc24f6db9a9541fe06ce281" member_type: VOTER last_known_addr { host: "peer-1.fake-domain-for-tests" port: 0 } } peers { permanent_uuid: "51455ae8ca8a478b97f86fb10d50d8f0" member_type: VOTER last_known_addr { host: "peer-2.fake-domain-for-tests" port: 0 } } I0920 01:06:09.000880 233 raft_consensus.cc:405] T TestTablet P 51455ae8ca8a478b97f86fb10d50d8f0 [term 0 FOLLOWER]: Emulating election... I0920 01:06:09.001176 233 raft_consensus.cc:2890] T TestTablet P 51455ae8ca8a478b97f86fb10d50d8f0 [term 0 FOLLOWER]: Advancing to term 1 I0920 01:06:09.004567 233 raft_consensus.cc:667] T TestTablet P 51455ae8ca8a478b97f86fb10d50d8f0 [term 1 LEADER]: Becoming Leader. State: Replica: 51455ae8ca8a478b97f86fb10d50d8f0, State: Running, Role: LEADER I0920 01:06:09.004927 233 consensus_queue.cc:206] T TestTablet P 51455ae8ca8a478b97f86fb10d50d8f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "17fcde7f158c4fe2b9b78aabef4ad20c" member_type: VOTER last_known_addr { host: "peer-0.fake-domain-for-tests" port: 0 } } peers { permanent_uuid: "33a35b5c6cc24f6db9a9541fe06ce281" member_type: VOTER last_known_addr { host: "peer-1.fake-domain-for-tests" port: 0 } } peers { permanent_uuid: "51455ae8ca8a478b97f86fb10d50d8f0" member_type: VOTER last_known_addr { host: "peer-2.fake-domain-for-tests" port: 0 } } I0920 01:06:09.009918 325 raft_consensus.cc:2890] T TestTablet P 33a35b5c6cc24f6db9a9541fe06ce281 [term 0 FOLLOWER]: Advancing to term 1 I0920 01:06:09.013990 325 raft_consensus.cc:1184] T TestTablet P 33a35b5c6cc24f6db9a9541fe06ce281 [term 1 FOLLOWER]: Refusing update from remote peer 51455ae8ca8a478b97f86fb10d50d8f0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0920 01:06:09.015244 324 consensus_queue.cc:984] T TestTablet P 51455ae8ca8a478b97f86fb10d50d8f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "33a35b5c6cc24f6db9a9541fe06ce281" member_type: VOTER last_known_addr { host: "peer-1.fake-domain-for-tests" port: 0 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0920 01:06:09.040063 326 raft_consensus.cc:2890] T TestTablet P 17fcde7f158c4fe2b9b78aabef4ad20c [term 0 FOLLOWER]: Advancing to term 1 I0920 01:06:09.044090 326 raft_consensus.cc:1184] T TestTablet P 17fcde7f158c4fe2b9b78aabef4ad20c [term 1 FOLLOWER]: Refusing update from remote peer 51455ae8ca8a478b97f86fb10d50d8f0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0920 01:06:09.044926 324 consensus_queue.cc:984] T TestTablet P 51455ae8ca8a478b97f86fb10d50d8f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "17fcde7f158c4fe2b9b78aabef4ad20c" member_type: VOTER last_known_addr { host: "peer-0.fake-domain-for-tests" port: 0 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0920 01:06:09.049846 233 raft_consensus.cc:2147] T TestTablet P 17fcde7f158c4fe2b9b78aabef4ad20c [term 1 FOLLOWER]: Raft consensus shutting down. I0920 01:06:09.050168 233 raft_consensus.cc:2174] T TestTablet P 17fcde7f158c4fe2b9b78aabef4ad20c [term 1 FOLLOWER]: Raft consensus is shut down! I0920 01:06:09.050364 233 raft_consensus.cc:2147] T TestTablet P 33a35b5c6cc24f6db9a9541fe06ce281 [term 1 FOLLOWER]: Raft consensus shutting down. I0920 01:06:09.050547 233 raft_consensus.cc:2174] T TestTablet P 33a35b5c6cc24f6db9a9541fe06ce281 [term 1 FOLLOWER]: Raft consensus is shut down! I0920 01:06:09.050685 233 raft_consensus.cc:2147] T TestTablet P 51455ae8ca8a478b97f86fb10d50d8f0 [term 1 LEADER]: Raft consensus shutting down. I0920 01:06:09.051017 233 raft_consensus.cc:2174] T TestTablet P 51455ae8ca8a478b97f86fb10d50d8f0 [term 1 FOLLOWER]: Raft consensus is shut down! [ OK ] RaftConsensusQuorumTest.TestConsensusContinuesIfAMinorityFallsBehind (180 ms) [ RUN ] RaftConsensusQuorumTest.TestConsensusStopsIfAMajorityFallsBehind I0920 01:06:09.090653 233 fs_manager.cc:262] Metadata directory not provided I0920 01:06:09.090821 233 fs_manager.cc:268] Using write-ahead log directory (fs_wal_dir) as metadata directory I0920 01:06:09.091753 233 fs_manager.cc:608] Generated new instance metadata in path /tmp/dist-test-taskEr7w4V/test-tmp/raft_consensus_quorum-test.0.RaftConsensusQuorumTest.TestConsensusStopsIfAMajorityFallsBehind.1568941567107070-233/peer-0-root/instance: uuid: "af647ac3afe249df80c6daba15c1c2f2" format_stamp: "Formatted at 2019-09-20 01:06:09 on dist-test-slave-dist-test-slave-9x72" I0920 01:06:09.099840 233 fs_manager.cc:509] Time spent creating directory manager: real 0.008s user 0.008s sys 0.000s I0920 01:06:09.101198 233 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0920 01:06:09.107687 233 fs_report.cc:352] FS layout report -------------------- wal directory: metadata directory: 1 data directories: /tmp/dist-test-taskEr7w4V/test-tmp/raft_consensus_quorum-test.0.RaftConsensusQuorumTest.TestConsensusStopsIfAMajorityFallsBehind.1568941567107070-233/peer-0-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0920 01:06:09.108067 233 fs_manager.cc:425] Time spent opening block manager: real 0.002s user 0.000s sys 0.000s I0920 01:06:09.108204 233 fs_manager.cc:442] Opened local filesystem: /tmp/dist-test-taskEr7w4V/test-tmp/raft_consensus_quorum-test.0.RaftConsensusQuorumTest.TestConsensusStopsIfAMajorityFallsBehind.1568941567107070-233/peer-0-root uuid: "af647ac3afe249df80c6daba15c1c2f2" format_stamp: "Formatted at 2019-09-20 01:06:09 on dist-test-slave-dist-test-slave-9x72" I0920 01:06:09.115264 233 fs_manager.cc:262] Metadata directory not provided I0920 01:06:09.115483 233 fs_manager.cc:268] Using write-ahead log directory (fs_wal_dir) as metadata directory I0920 01:06:09.116333 233 fs_manager.cc:608] Generated new instance metadata in path /tmp/dist-test-taskEr7w4V/test-tmp/raft_consensus_quorum-test.0.RaftConsensusQuorumTest.TestConsensusStopsIfAMajorityFallsBehind.1568941567107070-233/peer-1-root/instance: uuid: "80c942e351ab4627a152d9339b91c49c" format_stamp: "Formatted at 2019-09-20 01:06:09 on dist-test-slave-dist-test-slave-9x72" I0920 01:06:09.123834 233 fs_manager.cc:509] Time spent creating directory manager: real 0.007s user 0.007s sys 0.000s I0920 01:06:09.125092 233 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0920 01:06:09.131630 233 fs_report.cc:352] FS layout report -------------------- wal directory: metadata directory: 1 data directories: /tmp/dist-test-taskEr7w4V/test-tmp/raft_consensus_quorum-test.0.RaftConsensusQuorumTest.TestConsensusStopsIfAMajorityFallsBehind.1568941567107070-233/peer-1-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0920 01:06:09.132030 233 fs_manager.cc:425] Time spent opening block manager: real 0.002s user 0.000s sys 0.000s I0920 01:06:09.132169 233 fs_manager.cc:442] Opened local filesystem: /tmp/dist-test-taskEr7w4V/test-tmp/raft_consensus_quorum-test.0.RaftConsensusQuorumTest.TestConsensusStopsIfAMajorityFallsBehind.1568941567107070-233/peer-1-root uuid: "80c942e351ab4627a152d9339b91c49c" format_stamp: "Formatted at 2019-09-20 01:06:09 on dist-test-slave-dist-test-slave-9x72" I0920 01:06:09.139333 233 fs_manager.cc:262] Metadata directory not provided I0920 01:06:09.139497 233 fs_manager.cc:268] Using write-ahead log directory (fs_wal_dir) as metadata directory I0920 01:06:09.140305 233 fs_manager.cc:608] Generated new instance metadata in path /tmp/dist-test-taskEr7w4V/test-tmp/raft_consensus_quorum-test.0.RaftConsensusQuorumTest.TestConsensusStopsIfAMajorityFallsBehind.1568941567107070-233/peer-2-root/instance: uuid: "3f8dac7b0c654cc5bfca7d9e8c1b7211" format_stamp: "Formatted at 2019-09-20 01:06:09 on dist-test-slave-dist-test-slave-9x72" I0920 01:06:09.147991 233 fs_manager.cc:509] Time spent creating directory manager: real 0.007s user 0.007s sys 0.000s I0920 01:06:09.149211 233 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0920 01:06:09.155530 233 fs_report.cc:352] FS layout report -------------------- wal directory: metadata directory: 1 data directories: /tmp/dist-test-taskEr7w4V/test-tmp/raft_consensus_quorum-test.0.RaftConsensusQuorumTest.TestConsensusStopsIfAMajorityFallsBehind.1568941567107070-233/peer-2-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0920 01:06:09.155838 233 fs_manager.cc:425] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s I0920 01:06:09.156096 233 fs_manager.cc:442] Opened local filesystem: /tmp/dist-test-taskEr7w4V/test-tmp/raft_consensus_quorum-test.0.RaftConsensusQuorumTest.TestConsensusStopsIfAMajorityFallsBehind.1568941567107070-233/peer-2-root uuid: "3f8dac7b0c654cc5bfca7d9e8c1b7211" format_stamp: "Formatted at 2019-09-20 01:06:09 on dist-test-slave-dist-test-slave-9x72" I0920 01:06:09.171237 233 raft_consensus.cc:340] T TestTablet P af647ac3afe249df80c6daba15c1c2f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 peers { permanent_uuid: "af647ac3afe249df80c6daba15c1c2f2" member_type: VOTER last_known_addr { host: "peer-0.fake-domain-for-tests" port: 0 } } peers { permanent_uuid: "80c942e351ab4627a152d9339b91c49c" member_type: VOTER last_known_addr { host: "peer-1.fake-domain-for-tests" port: 0 } } peers { permanent_uuid: "3f8dac7b0c654cc5bfca7d9e8c1b7211" member_type: VOTER last_known_addr { host: "peer-2.fake-domain-for-tests" port: 0 } } I0920 01:06:09.171883 233 raft_consensus.cc:705] T TestTablet P af647ac3afe249df80c6daba15c1c2f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af647ac3afe249df80c6daba15c1c2f2, State: Initialized, Role: FOLLOWER I0920 01:06:09.172258 233 consensus_queue.cc:229] T TestTablet P af647ac3afe249df80c6daba15c1c2f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "af647ac3afe249df80c6daba15c1c2f2" member_type: VOTER last_known_addr { host: "peer-0.fake-domain-for-tests" port: 0 } } peers { permanent_uuid: "80c942e351ab4627a152d9339b91c49c" member_type: VOTER last_known_addr { host: "peer-1.fake-domain-for-tests" port: 0 } } peers { permanent_uuid: "3f8dac7b0c654cc5bfca7d9e8c1b7211" member_type: VOTER last_known_addr { host: "peer-2.fake-domain-for-tests" port: 0 } } I0920 01:06:09.177443 233 raft_consensus.cc:340] T TestTablet P 80c942e351ab4627a152d9339b91c49c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 peers { permanent_uuid: "af647ac3afe249df80c6daba15c1c2f2" member_type: VOTER last_known_addr { host: "peer-0.fake-domain-for-tests" port: 0 } } peers { permanent_uuid: "80c942e351ab4627a152d9339b91c49c" member_type: VOTER last_known_addr { host: "peer-1.fake-domain-for-tests" port: 0 } } peers { permanent_uuid: "3f8dac7b0c654cc5bfca7d9e8c1b7211" member_type: VOTER last_known_addr { host: "peer-2.fake-domain-for-tests" port: 0 } } I0920 01:06:09.177914 233 raft_consensus.cc:705] T TestTablet P 80c942e351ab4627a152d9339b91c49c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80c942e351ab4627a152d9339b91c49c, State: Initialized, Role: FOLLOWER I0920 01:06:09.178208 233 consensus_queue.cc:229] T TestTablet P 80c942e351ab4627a152d9339b91c49c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "af647ac3afe249df80c6daba15c1c2f2" member_type: VOTER last_known_addr { host: "peer-0.fake-domain-for-tests" port: 0 } } peers { permanent_uuid: "80c942e351ab4627a152d9339b91c49c" member_type: VOTER last_known_addr { host: "peer-1.fake-domain-for-tests" port: 0 } } peers { permanent_uuid: "3f8dac7b0c654cc5bfca7d9e8c1b7211" member_type: VOTER last_known_addr { host: "peer-2.fake-domain-for-tests" port: 0 } } I0920 01:06:09.182648 233 raft_consensus.cc:340] T TestTablet P 3f8dac7b0c654cc5bfca7d9e8c1b7211 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 peers { permanent_uuid: "af647ac3afe249df80c6daba15c1c2f2" member_type: VOTER last_known_addr { host: "peer-0.fake-domain-for-tests" port: 0 } } peers { permanent_uuid: "80c942e351ab4627a152d9339b91c49c" member_type: VOTER last_known_addr { host: "peer-1.fake-domain-for-tests" port: 0 } } peers { permanent_uuid: "3f8dac7b0c654cc5bfca7d9e8c1b7211" member_type: VOTER last_known_addr { host: "peer-2.fake-domain-for-tests" port: 0 } } I0920 01:06:09.183095 233 raft_consensus.cc:705] T TestTablet P 3f8dac7b0c654cc5bfca7d9e8c1b7211 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f8dac7b0c654cc5bfca7d9e8c1b7211, State: Initialized, Role: FOLLOWER I0920 01:06:09.183420 233 consensus_queue.cc:229] T TestTablet P 3f8dac7b0c654cc5bfca7d9e8c1b7211 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "af647ac3afe249df80c6daba15c1c2f2" member_type: VOTER last_known_addr { host: "peer-0.fake-domain-for-tests" port: 0 } } peers { permanent_uuid: "80c942e351ab4627a152d9339b91c49c" member_type: VOTER last_known_addr { host: "peer-1.fake-domain-for-tests" port: 0 } } peers { permanent_uuid: "3f8dac7b0c654cc5bfca7d9e8c1b7211" member_type: VOTER last_known_addr { host: "peer-2.fake-domain-for-tests" port: 0 } } I0920 01:06:09.183970 233 raft_consensus.cc:405] T TestTablet P 3f8dac7b0c654cc5bfca7d9e8c1b7211 [term 0 FOLLOWER]: Emulating election... I0920 01:06:09.184110 233 raft_consensus.cc:2890] T TestTablet P 3f8dac7b0c654cc5bfca7d9e8c1b7211 [term 0 FOLLOWER]: Advancing to term 1 I0920 01:06:09.187317 233 raft_consensus.cc:667] T TestTablet P 3f8dac7b0c654cc5bfca7d9e8c1b7211 [term 1 LEADER]: Becoming Leader. State: Replica: 3f8dac7b0c654cc5bfca7d9e8c1b7211, State: Running, Role: LEADER I0920 01:06:09.187676 233 consensus_queue.cc:206] T TestTablet P 3f8dac7b0c654cc5bfca7d9e8c1b7211 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "af647ac3afe249df80c6daba15c1c2f2" member_type: VOTER last_known_addr { host: "peer-0.fake-domain-for-tests" port: 0 } } peers { permanent_uuid: "80c942e351ab4627a152d9339b91c49c" member_type: VOTER last_known_addr { host: "peer-1.fake-domain-for-tests" port: 0 } } peers { permanent_uuid: "3f8dac7b0c654cc5bfca7d9e8c1b7211" member_type: VOTER last_known_addr { host: "peer-2.fake-domain-for-tests" port: 0 } } I0920 01:06:09.692821 354 raft_consensus.cc:2890] T TestTablet P 80c942e351ab4627a152d9339b91c49c [term 0 FOLLOWER]: Advancing to term 1 I0920 01:06:09.692976 355 raft_consensus.cc:2890] T TestTablet P af647ac3afe249df80c6daba15c1c2f2 [term 0 FOLLOWER]: Advancing to term 1 I0920 01:06:09.697341 355 raft_consensus.cc:1184] T TestTablet P af647ac3afe249df80c6daba15c1c2f2 [term 1 FOLLOWER]: Refusing update from remote peer 3f8dac7b0c654cc5bfca7d9e8c1b7211: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0920 01:06:09.697355 354 raft_consensus.cc:1184] T TestTablet P 80c942e351ab4627a152d9339b91c49c [term 1 FOLLOWER]: Refusing update from remote peer 3f8dac7b0c654cc5bfca7d9e8c1b7211: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0920 01:06:09.699200 357 consensus_queue.cc:984] T TestTablet P 3f8dac7b0c654cc5bfca7d9e8c1b7211 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af647ac3afe249df80c6daba15c1c2f2" member_type: VOTER last_known_addr { host: "peer-0.fake-domain-for-tests" port: 0 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0920 01:06:09.699981 358 consensus_queue.cc:984] T TestTablet P 3f8dac7b0c654cc5bfca7d9e8c1b7211 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80c942e351ab4627a152d9339b91c49c" member_type: VOTER last_known_addr { host: "peer-1.fake-domain-for-tests" port: 0 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0920 01:06:09.706418 233 raft_consensus.cc:2147] T TestTablet P af647ac3afe249df80c6daba15c1c2f2 [term 1 FOLLOWER]: Raft consensus shutting down. I0920 01:06:09.706645 233 raft_consensus.cc:2174] T TestTablet P af647ac3afe249df80c6daba15c1c2f2 [term 1 FOLLOWER]: Raft consensus is shut down! I0920 01:06:09.706791 233 raft_consensus.cc:2147] T TestTablet P 3f8dac7b0c654cc5bfca7d9e8c1b7211 [term 1 LEADER]: Raft consensus shutting down. I0920 01:06:09.707113 233 raft_consensus.cc:2174] T TestTablet P 3f8dac7b0c654cc5bfca7d9e8c1b7211 [term 1 FOLLOWER]: Raft consensus is shut down! I0920 01:06:09.707267 233 raft_consensus.cc:2147] T TestTablet P 80c942e351ab4627a152d9339b91c49c [term 1 FOLLOWER]: Raft consensus shutting down. I0920 01:06:09.707532 233 raft_consensus.cc:2174] T TestTablet P 80c942e351ab4627a152d9339b91c49c [term 1 FOLLOWER]: Raft consensus is shut down! [ OK ] RaftConsensusQuorumTest.TestConsensusStopsIfAMajorityFallsBehind (644 ms) [ RUN ] RaftConsensusQuorumTest.TestReplicasHandleCommunicationErrors I0920 01:06:09.735635 233 fs_manager.cc:262] Metadata directory not provided I0920 01:06:09.735810 233 fs_manager.cc:268] Using write-ahead log directory (fs_wal_dir) as metadata directory I0920 01:06:09.736649 233 fs_manager.cc:608] Generated new instance metadata in path /tmp/dist-test-taskEr7w4V/test-tmp/raft_consensus_quorum-test.0.RaftConsensusQuorumTest.TestReplicasHandleCommunicationErrors.1568941567107070-233/peer-0-root/instance: uuid: "2485c0bc559f43ee9b8244b2a1125f0e" format_stamp: "Formatted at 2019-09-20 01:06:09 on dist-test-slave-dist-test-slave-9x72" I0920 01:06:09.744496 233 fs_manager.cc:509] Time spent creating directory manager: real 0.007s user 0.005s sys 0.003s I0920 01:06:09.745661 233 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0920 01:06:09.751515 233 fs_report.cc:352] FS layout report -------------------- wal directory: metadata directory: 1 data directories: /tmp/dist-test-taskEr7w4V/test-tmp/raft_consensus_quorum-test.0.RaftConsensusQuorumTest.TestReplicasHandleCommunicationErrors.1568941567107070-233/peer-0-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0920 01:06:09.751894 233 fs_manager.cc:425] Time spent opening block manager: real 0.002s user 0.000s sys 0.000s I0920 01:06:09.752027 233 fs_manager.cc:442] Opened local filesystem: /tmp/dist-test-taskEr7w4V/test-tmp/raft_consensus_quorum-test.0.RaftConsensusQuorumTest.TestReplicasHandleCommunicationErrors.1568941567107070-233/peer-0-root uuid: "2485c0bc559f43ee9b8244b2a1125f0e" format_stamp: "Formatted at 2019-09-20 01:06:09 on dist-test-slave-dist-test-slave-9x72" I0920 01:06:09.758939 233 fs_manager.cc:262] Metadata directory not provided I0920 01:06:09.759104 233 fs_manager.cc:268] Using write-ahead log directory (fs_wal_dir) as metadata directory I0920 01:06:09.760013 233 fs_manager.cc:608] Generated new instance metadata in path /tmp/dist-test-taskEr7w4V/test-tmp/raft_consensus_quorum-test.0.RaftConsensusQuorumTest.TestReplicasHandleCommunicationErrors.1568941567107070-233/peer-1-root/instance: uuid: "aad1d142980a42798f4dd051364cc06b" format_stamp: "Formatted at 2019-09-20 01:06:09 on dist-test-slave-dist-test-slave-9x72" I0920 01:06:09.767343 233 fs_manager.cc:509] Time spent creating directory manager: real 0.007s user 0.007s sys 0.000s I0920 01:06:09.768518 233 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0920 01:06:09.774351 233 fs_report.cc:352] FS layout report -------------------- wal directory: metadata directory: 1 data directories: /tmp/dist-test-taskEr7w4V/test-tmp/raft_consensus_quorum-test.0.RaftConsensusQuorumTest.TestReplicasHandleCommunicationErrors.1568941567107070-233/peer-1-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0920 01:06:09.774678 233 fs_manager.cc:425] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s I0920 01:06:09.774834 233 fs_manager.cc:442] Opened local filesystem: /tmp/dist-test-taskEr7w4V/test-tmp/raft_consensus_quorum-test.0.RaftConsensusQuorumTest.TestReplicasHandleCommunicationErrors.1568941567107070-233/peer-1-root uuid: "aad1d142980a42798f4dd051364cc06b" format_stamp: "Formatted at 2019-09-20 01:06:09 on dist-test-slave-dist-test-slave-9x72" I0920 01:06:09.782311 233 fs_manager.cc:262] Metadata directory not provided I0920 01:06:09.782471 233 fs_manager.cc:268] Using write-ahead log directory (fs_wal_dir) as metadata directory I0920 01:06:09.783318 233 fs_manager.cc:608] Generated new instance metadata in path /tmp/dist-test-taskEr7w4V/test-tmp/raft_consensus_quorum-test.0.RaftConsensusQuorumTest.TestReplicasHandleCommunicationErrors.1568941567107070-233/peer-2-root/instance: uuid: "13355128a3c84c399b0a2f382a12e69f" format_stamp: "Formatted at 2019-09-20 01:06:09 on dist-test-slave-dist-test-slave-9x72" I0920 01:06:09.790670 233 fs_manager.cc:509] Time spent creating directory manager: real 0.007s user 0.007s sys 0.000s I0920 01:06:09.791810 233 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0920 01:06:09.797972 233 fs_report.cc:352] FS layout report -------------------- wal directory: metadata directory: 1 data directories: /tmp/dist-test-taskEr7w4V/test-tmp/raft_consensus_quorum-test.0.RaftConsensusQuorumTest.TestReplicasHandleCommunicationErrors.1568941567107070-233/peer-2-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0920 01:06:09.798288 233 fs_manager.cc:425] Time spent opening block manager: real 0.002s user 0.000s sys 0.000s I0920 01:06:09.798434 233 fs_manager.cc:442] Opened local filesystem: /tmp/dist-test-taskEr7w4V/test-tmp/raft_consensus_quorum-test.0.RaftConsensusQuorumTest.TestReplicasHandleCommunicationErrors.1568941567107070-233/peer-2-root uuid: "13355128a3c84c399b0a2f382a12e69f" format_stamp: "Formatted at 2019-09-20 01:06:09 on dist-test-slave-dist-test-slave-9x72" I0920 01:06:09.813963 233 raft_consensus.cc:340] T TestTablet P 2485c0bc559f43ee9b8244b2a1125f0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 peers { permanent_uuid: "2485c0bc559f43ee9b8244b2a1125f0e" member_type: VOTER last_known_addr { host: "peer-0.fake-domain-for-tests" port: 0 } } peers { permanent_uuid: "aad1d142980a42798f4dd051364cc06b" member_type: VOTER last_known_addr { host: "peer-1.fake-domain-for-tests" port: 0 } } peers { permanent_uuid: "13355128a3c84c399b0a2f382a12e69f" member_type: VOTER last_known_addr { host: "peer-2.fake-domain-for-tests" port: 0 } } I0920 01:06:09.814414 233 raft_consensus.cc:705] T TestTablet P 2485c0bc559f43ee9b8244b2a1125f0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2485c0bc559f43ee9b8244b2a1125f0e, State: Initialized, Role: FOLLOWER I0920 01:06:09.814715 233 consensus_queue.cc:229] T TestTablet P 2485c0bc559f43ee9b8244b2a1125f0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "2485c0bc559f43ee9b8244b2a1125f0e" member_type: VOTER last_known_addr { host: "peer-0.fake-domain-for-tests" port: 0 } } peers { permanent_uuid: "aad1d142980a42798f4dd051364cc06b" member_type: VOTER last_known_addr { host: "peer-1.fake-domain-for-tests" port: 0 } } peers { permanent_uuid: "13355128a3c84c399b0a2f382a12e69f" member_type: VOTER last_known_addr { host: "peer-2.fake-domain-for-tests" port: 0 } } I0920 01:06:09.819923 233 raft_consensus.cc:340] T TestTablet P aad1d142980a42798f4dd051364cc06b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 peers { permanent_uuid: "2485c0bc559f43ee9b8244b2a1125f0e" member_type: VOTER last_known_addr { host: "peer-0.fake-domain-for-tests" port: 0 } } peers { permanent_uuid: "aad1d142980a42798f4dd051364cc06b" member_type: VOTER last_known_addr { host: "peer-1.fake-domain-for-tests" port: 0 } } peers { permanent_uuid: "13355128a3c84c399b0a2f382a12e69f" member_type: VOTER last_known_addr { host: "peer-2.fake-domain-for-tests" port: 0 } } I0920 01:06:09.820358 233 raft_consensus.cc:705] T TestTablet P aad1d142980a42798f4dd051364cc06b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aad1d142980a42798f4dd051364cc06b, State: Initialized, Role: FOLLOWER I0920 01:06:09.820657 233 consensus_queue.cc:229] T TestTablet P aad1d142980a42798f4dd051364cc06b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "2485c0bc559f43ee9b8244b2a1125f0e" member_type: VOTER last_known_addr { host: "peer-0.fake-domain-for-tests" port: 0 } } peers { permanent_uuid: "aad1d142980a42798f4dd051364cc06b" member_type: VOTER last_known_addr { host: "peer-1.fake-domain-for-tests" port: 0 } } peers { permanent_uuid: "13355128a3c84c399b0a2f382a12e69f" member_type: VOTER last_known_addr { host: "peer-2.fake-domain-for-tests" port: 0 } } I0920 01:06:09.825158 233 raft_consensus.cc:340] T TestTablet P 13355128a3c84c399b0a2f382a12e69f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 peers { permanent_uuid: "2485c0bc559f43ee9b8244b2a1125f0e" member_type: VOTER last_known_addr { host: "peer-0.fake-domain-for-tests" port: 0 } } peers { permanent_uuid: "aad1d142980a42798f4dd051364cc06b" member_type: VOTER last_known_addr { host: "peer-1.fake-domain-for-tests" port: 0 } } peers { permanent_uuid: "13355128a3c84c399b0a2f382a12e69f" member_type: VOTER last_known_addr { host: "peer-2.fake-domain-for-tests" port: 0 } } I0920 01:06:09.825578 233 raft_consensus.cc:705] T TestTablet P 13355128a3c84c399b0a2f382a12e69f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13355128a3c84c399b0a2f382a12e69f, State: Initialized, Role: FOLLOWER I0920 01:06:09.825888 233 consensus_queue.cc:229] T TestTablet P 13355128a3c84c399b0a2f382a12e69f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "2485c0bc559f43ee9b8244b2a1125f0e" member_type: VOTER last_known_addr { host: "peer-0.fake-domain-for-tests" port: 0 } } peers { permanent_uuid: "aad1d142980a42798f4dd051364cc06b" member_type: VOTER last_known_addr { host: "peer-1.fake-domain-for-tests" port: 0 } } peers { permanent_uuid: "13355128a3c84c399b0a2f382a12e69f" member_type: VOTER last_known_addr { host: "peer-2.fake-domain-for-tests" port: 0 } } I0920 01:06:09.826433 233 raft_consensus.cc:405] T TestTablet P 13355128a3c84c399b0a2f382a12e69f [term 0 FOLLOWER]: Emulating election... I0920 01:06:09.826570 233 raft_consensus.cc:2890] T TestTablet P 13355128a3c84c399b0a2f382a12e69f [term 0 FOLLOWER]: Advancing to term 1 I0920 01:06:09.829880 233 raft_consensus.cc:667] T TestTablet P 13355128a3c84c399b0a2f382a12e69f [term 1 LEADER]: Becoming Leader. State: Replica: 13355128a3c84c399b0a2f382a12e69f, State: Running, Role: LEADER I0920 01:06:09.830226 233 consensus_queue.cc:206] T TestTablet P 13355128a3c84c399b0a2f382a12e69f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "2485c0bc559f43ee9b8244b2a1125f0e" member_type: VOTER last_known_addr { host: "peer-0.fake-domain-for-tests" port: 0 } } peers { permanent_uuid: "aad1d142980a42798f4dd051364cc06b" member_type: VOTER last_known_addr { host: "peer-1.fake-domain-for-tests" port: 0 } } peers { permanent_uuid: "13355128a3c84c399b0a2f382a12e69f" member_type: VOTER last_known_addr { host: "peer-2.fake-domain-for-tests" port: 0 } } I0920 01:06:09.835116 387 raft_consensus.cc:2890] T TestTablet P aad1d142980a42798f4dd051364cc06b [term 0 FOLLOWER]: Advancing to term 1 I0920 01:06:09.836027 388 raft_consensus.cc:2890] T TestTablet P 2485c0bc559f43ee9b8244b2a1125f0e [term 0 FOLLOWER]: Advancing to term 1 I0920 01:06:09.839419 387 raft_consensus.cc:1184] T TestTablet P aad1d142980a42798f4dd051364cc06b [term 1 FOLLOWER]: Refusing update from remote peer 13355128a3c84c399b0a2f382a12e69f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0920 01:06:09.839926 388 raft_consensus.cc:1184] T TestTablet P 2485c0bc559f43ee9b8244b2a1125f0e [term 1 FOLLOWER]: Refusing update from remote peer 13355128a3c84c399b0a2f382a12e69f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) W0920 01:06:09.840638 388 consensus_peers.cc:458] T TestTablet P 13355128a3c84c399b0a2f382a12e69f -> Peer aad1d142980a42798f4dd051364cc06b (peer-1.fake-domain-for-tests:0): Couldn't send request to peer aad1d142980a42798f4dd051364cc06b. Error code: UNKNOWN_ERROR (1). Status: IO error: Artificial error caused by communication failure injection.. This is attempt 1: this message will repeat every 5th retry. W0920 01:06:09.840867 387 consensus_peers.cc:458] T TestTablet P 13355128a3c84c399b0a2f382a12e69f -> Peer 2485c0bc559f43ee9b8244b2a1125f0e (peer-0.fake-domain-for-tests:0): Couldn't send request to peer 2485c0bc559f43ee9b8244b2a1125f0e. Error code: UNKNOWN_ERROR (1). Status: IO error: Artificial error caused by communication failure injection.. This is attempt 1: this message will repeat every 5th retry. W0920 01:06:10.243510 387 consensus_peers.cc:458] T TestTablet P 13355128a3c84c399b0a2f382a12e69f -> Peer 2485c0bc559f43ee9b8244b2a1125f0e (peer-0.fake-domain-for-tests:0): Couldn't send request to peer 2485c0bc559f43ee9b8244b2a1125f0e. Error code: UNKNOWN_ERROR (1). Status: IO error: Artificial error caused by communication failure injection.. This is attempt 1: this message will repeat every 5th retry. I0920 01:06:10.827374 393 raft_consensus.cc:1126] T TestTablet P aad1d142980a42798f4dd051364cc06b [term 1 FOLLOWER]: Deduplicated request from leader. Original: 0.0->[1.1-1.3] Dedup: 1.2->[1.3-1.3] W0920 01:06:10.831831 393 consensus_peers.cc:458] T TestTablet P 13355128a3c84c399b0a2f382a12e69f -> Peer aad1d142980a42798f4dd051364cc06b (peer-1.fake-domain-for-tests:0): Couldn't send request to peer aad1d142980a42798f4dd051364cc06b. Error code: UNKNOWN_ERROR (1). Status: IO error: Artificial error caused by communication failure injection.. This is attempt 1: this message will repeat every 5th retry. W0920 01:06:11.407310 400 consensus_peers.cc:458] T TestTablet P 13355128a3c84c399b0a2f382a12e69f -> Peer aad1d142980a42798f4dd051364cc06b (peer-1.fake-domain-for-tests:0): Couldn't send request to peer aad1d142980a42798f4dd051364cc06b. Error code: UNKNOWN_ERROR (1). Status: IO error: Artificial error caused by communication failure injection.. This is attempt 1: this message will repeat every 5th retry. I0920 01:06:11.418543 400 raft_consensus.cc:1126] T TestTablet P 2485c0bc559f43ee9b8244b2a1125f0e [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.2->[1.3-1.6] Dedup: 1.3->[1.4-1.6] I0920 01:06:11.973773 403 raft_consensus.cc:1126] T TestTablet P aad1d142980a42798f4dd051364cc06b [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.6] Dedup: 1.6->[] W0920 01:06:11.980235 403 consensus_peers.cc:458] T TestTablet P 13355128a3c84c399b0a2f382a12e69f -> Peer aad1d142980a42798f4dd051364cc06b (peer-1.fake-domain-for-tests:0): Couldn't send request to peer aad1d142980a42798f4dd051364cc06b. Error code: UNKNOWN_ERROR (1). Status: IO error: Artificial error caused by communication failure injection.. This is attempt 1: this message will repeat every 5th retry. I0920 01:06:11.992170 403 raft_consensus.cc:1126] T TestTablet P 2485c0bc559f43ee9b8244b2a1125f0e [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.2->[1.3-1.8] Dedup: 1.6->[1.7-1.8] I0920 01:06:12.461248 403 raft_consensus.cc:1126] T TestTablet P 2485c0bc559f43ee9b8244b2a1125f0e [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.2->[1.3-1.8] Dedup: 1.8->[] W0920 01:06:12.464591 403 consensus_peers.cc:458] T TestTablet P 13355128a3c84c399b0a2f382a12e69f -> Peer 2485c0bc559f43ee9b8244b2a1125f0e (peer-0.fake-domain-for-tests:0): Couldn't send request to peer 2485c0bc559f43ee9b8244b2a1125f0e. Error code: UNKNOWN_ERROR (1). Status: IO error: Artificial error caused by communication failure injection.. This is attempt 1: this message will repeat every 5th retry. I0920 01:06:12.572137 403 raft_consensus.cc:1126] T TestTablet P aad1d142980a42798f4dd051364cc06b [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.7->[1.8-1.9] Dedup: 1.8->[1.9-1.9] W0920 01:06:12.575947 403 consensus_peers.cc:458] T TestTablet P 13355128a3c84c399b0a2f382a12e69f -> Peer aad1d142980a42798f4dd051364cc06b (peer-1.fake-domain-for-tests:0): Couldn't send request to peer aad1d142980a42798f4dd051364cc06b. Error code: UNKNOWN_ERROR (1). Status: IO error: Artificial error caused by communication failure injection.. This is attempt 1: this message will repeat every 5th retry. I0920 01:06:12.984278 403 raft_consensus.cc:1126] T TestTablet P 2485c0bc559f43ee9b8244b2a1125f0e [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.8->[1.9-1.10] Dedup: 1.9->[1.10-1.10] W0920 01:06:12.988364 405 consensus_peers.cc:458] T TestTablet P 13355128a3c84c399b0a2f382a12e69f -> Peer 2485c0bc559f43ee9b8244b2a1125f0e (peer-0.fake-domain-for-tests:0): Couldn't send request to peer 2485c0bc559f43ee9b8244b2a1125f0e. Error code: UNKNOWN_ERROR (1). Status: IO error: Artificial error caused by communication failure injection.. This is attempt 1: this message will repeat every 5th retry. I0920 01:06:13.091702 405 raft_consensus.cc:1126] T TestTablet P aad1d142980a42798f4dd051364cc06b [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.9->[1.10-1.11] Dedup: 1.10->[1.11-1.11] W0920 01:06:13.095599 405 consensus_peers.cc:458] T TestTablet P 13355128a3c84c399b0a2f382a12e69f -> Peer aad1d142980a42798f4dd051364cc06b (peer-1.fake-domain-for-tests:0): Couldn't send request to peer aad1d142980a42798f4dd051364cc06b. Error code: UNKNOWN_ERROR (1). Status: IO error: Artificial error caused by communication failure injection.. This is attempt 1: this message will repeat every 5th retry. I0920 01:06:13.562988 405 raft_consensus.cc:1126] T TestTablet P aad1d142980a42798f4dd051364cc06b [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.11->[1.12-1.12] Dedup: 1.12->[] W0920 01:06:13.569319 403 consensus_peers.cc:458] T TestTablet P 13355128a3c84c399b0a2f382a12e69f -> Peer aad1d142980a42798f4dd051364cc06b (peer-1.fake-domain-for-tests:0): Couldn't send request to peer aad1d142980a42798f4dd051364cc06b. Error code: UNKNOWN_ERROR (1). Status: IO error: Artificial error caused by communication failure injection.. This is attempt 1: this message will repeat every 5th retry. I0920 01:06:13.591136 403 raft_consensus.cc:1126] T TestTablet P 2485c0bc559f43ee9b8244b2a1125f0e [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.10->[1.11-1.14] Dedup: 1.11->[1.12-1.14] I0920 01:06:13.973109 405 raft_consensus.cc:1126] T TestTablet P aad1d142980a42798f4dd051364cc06b [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.13->[1.14-1.14] Dedup: 1.14->[] W0920 01:06:13.979071 405 consensus_peers.cc:458] T TestTablet P 13355128a3c84c399b0a2f382a12e69f -> Peer aad1d142980a42798f4dd051364cc06b (peer-1.fake-domain-for-tests:0): Couldn't send request to peer aad1d142980a42798f4dd051364cc06b. Error code: UNKNOWN_ERROR (1). Status: IO error: Artificial error caused by communication failure injection.. This is attempt 1: this message will repeat every 5th retry. I0920 01:06:14.213655 405 raft_consensus.cc:1126] T TestTablet P 2485c0bc559f43ee9b8244b2a1125f0e [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.10->[1.11-1.16] Dedup: 1.14->[1.15-1.16] I0920 01:06:14.578141 403 raft_consensus.cc:1126] T TestTablet P aad1d142980a42798f4dd051364cc06b [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.15->[1.16-1.16] Dedup: 1.16->[] W0920 01:06:14.584153 403 consensus_peers.cc:458] T TestTablet P 13355128a3c84c399b0a2f382a12e69f -> Peer aad1d142980a42798f4dd051364cc06b (peer-1.fake-domain-for-tests:0): Couldn't send request to peer aad1d142980a42798f4dd051364cc06b. Error code: UNKNOWN_ERROR (1). Status: IO error: Artificial error caused by communication failure injection.. This is attempt 1: this message will repeat every 5th retry. I0920 01:06:14.735862 403 raft_consensus.cc:1126] T TestTablet P 2485c0bc559f43ee9b8244b2a1125f0e [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.10->[1.11-1.18] Dedup: 1.16->[1.17-1.18] I0920 01:06:15.118465 405 raft_consensus.cc:1126] T TestTablet P 2485c0bc559f43ee9b8244b2a1125f0e [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.10->[1.11-1.18] Dedup: 1.18->[] W0920 01:06:15.121373 405 consensus_peers.cc:458] T TestTablet P 13355128a3c84c399b0a2f382a12e69f -> Peer 2485c0bc559f43ee9b8244b2a1125f0e (peer-0.fake-domain-for-tests:0): Couldn't send request to peer 2485c0bc559f43ee9b8244b2a1125f0e. Error code: UNKNOWN_ERROR (1). Status: IO error: Artificial error caused by communication failure injection.. This is attempt 1: this message will repeat every 5th retry. I0920 01:06:15.199206 405 raft_consensus.cc:1126] T TestTablet P aad1d142980a42798f4dd051364cc06b [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.17->[1.18-1.19] Dedup: 1.18->[1.19-1.19] W0920 01:06:15.202435 405 consensus_peers.cc:458] T TestTablet P 13355128a3c84c399b0a2f382a12e69f -> Peer aad1d142980a42798f4dd051364cc06b (peer-1.fake-domain-for-tests:0): Couldn't send request to peer aad1d142980a42798f4dd051364cc06b. Error code: UNKNOWN_ERROR (1). Status: IO error: Artificial error caused by communication failure injection.. This is attempt 1: this message will repeat every 5th retry. W0920 01:06:15.672135 403 consensus_peers.cc:458] T TestTablet P 13355128a3c84c399b0a2f382a12e69f -> Peer 2485c0bc559f43ee9b8244b2a1125f0e (peer-0.fake-domain-for-tests:0): Couldn't send request to peer 2485c0bc559f43ee9b8244b2a1125f0e. Error code: UNKNOWN_ERROR (1). Status: IO error: Artificial error caused by communication failure injection.. This is attempt 1: this message will repeat every 5th retry. W0920 01:06:15.706476 405 consensus_peers.cc:458] T TestTablet P 13355128a3c84c399b0a2f382a12e69f -> Peer aad1d142980a42798f4dd051364cc06b (peer-1.fake-domain-for-tests:0): Couldn't send request to peer aad1d142980a42798f4dd051364cc06b. Error code: UNKNOWN_ERROR (1). Status: IO error: Artificial error caused by communication failure injection.. This is attempt 1: this message will repeat every 5th retry. I0920 01:06:16.120081 405 raft_consensus.cc:1126] T TestTablet P 2485c0bc559f43ee9b8244b2a1125f0e [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.20->[1.21-1.22] Dedup: 1.21->[1.22-1.22] W0920 01:06:16.124248 405 consensus_peers.cc:458] T TestTablet P 13355128a3c84c399b0a2f382a12e69f -> Peer 2485c0bc559f43ee9b8244b2a1125f0e (peer-0.fake-domain-for-tests:0): Couldn't send request to peer 2485c0bc559f43ee9b8244b2a1125f0e. Error code: UNKNOWN_ERROR (1). Status: IO error: Artificial error caused by communication failure injection.. This is attempt 1: this message will repeat every 5th retry. I0920 01:06:16.295159 405 raft_consensus.cc:1126] T TestTablet P aad1d142980a42798f4dd051364cc06b [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.21->[1.22-1.23] Dedup: 1.22->[1.23-1.23] W0920 01:06:16.299361 405 consensus_peers.cc:458] T TestTablet P 13355128a3c84c399b0a2f382a12e69f -> Peer aad1d142980a42798f4dd051364cc06b (peer-1.fake-domain-for-tests:0): Couldn't send request to peer aad1d142980a42798f4dd051364cc06b. Error code: UNKNOWN_ERROR (1). Status: IO error: Artificial error caused by communication failure injection.. This is attempt 1: this message will repeat every 5th retry. I0920 01:06:16.721074 405 raft_consensus.cc:1126] T TestTablet P aad1d142980a42798f4dd051364cc06b [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.23->[1.24-1.24] Dedup: 1.24->[] W0920 01:06:16.726850 403 consensus_peers.cc:458] T TestTablet P 13355128a3c84c399b0a2f382a12e69f -> Peer aad1d142980a42798f4dd051364cc06b (peer-1.fake-domain-for-tests:0): Couldn't send request to peer aad1d142980a42798f4dd051364cc06b. Error code: UNKNOWN_ERROR (1). Status: IO error: Artificial error caused by communication failure injection.. This is attempt 1: this message will repeat every 5th retry. I0920 01:06:16.732477 403 raft_consensus.cc:1126] T TestTablet P 2485c0bc559f43ee9b8244b2a1125f0e [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.22->[1.23-1.26] Dedup: 1.23->[1.24-1.26] W0920 01:06:17.110142 403 consensus_peers.cc:458] T TestTablet P 13355128a3c84c399b0a2f382a12e69f -> Peer aad1d142980a42798f4dd051364cc06b (peer-1.fake-domain-for-tests:0): Couldn't send request to peer aad1d142980a42798f4dd051364cc06b. Error code: UNKNOWN_ERROR (1). Status: IO error: Artificial error caused by communication failure injection.. This is attempt 1: this message will repeat every 5th retry. I0920 01:06:17.130152 403 raft_consensus.cc:1126] T TestTablet P 2485c0bc559f43ee9b8244b2a1125f0e [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.22->[1.23-1.28] Dedup: 1.26->[1.27-1.28] I0920 01:06:17.557862 405 raft_consensus.cc:1126] T TestTablet P aad1d142980a42798f4dd051364cc06b [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.27->[1.28-1.28] Dedup: 1.28->[] W0920 01:06:17.565510 403 consensus_peers.cc:458] T TestTablet P 13355128a3c84c399b0a2f382a12e69f -> Peer aad1d142980a42798f4dd051364cc06b (peer-1.fake-domain-for-tests:0): Couldn't send request to peer aad1d142980a42798f4dd051364cc06b. Error code: UNKNOWN_ERROR (1). Status: IO error: Artificial error caused by communication failure injection.. This is attempt 1: this message will repeat every 5th retry. I0920 01:06:17.637506 403 raft_consensus.cc:1126] T TestTablet P 2485c0bc559f43ee9b8244b2a1125f0e [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.22->[1.23-1.30] Dedup: 1.28->[1.29-1.30] I0920 01:06:17.977051 405 raft_consensus.cc:1126] T TestTablet P aad1d142980a42798f4dd051364cc06b [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.29->[1.30-1.30] Dedup: 1.30->[] W0920 01:06:17.982496 405 consensus_peers.cc:458] T TestTablet P 13355128a3c84c399b0a2f382a12e69f -> Peer aad1d142980a42798f4dd051364cc06b (peer-1.fake-domain-for-tests:0): Couldn't send request to peer aad1d142980a42798f4dd051364cc06b. Error code: UNKNOWN_ERROR (1). Status: IO error: Artificial error caused by communication failure injection.. This is attempt 1: this message will repeat every 5th retry. I0920 01:06:18.159834 405 raft_consensus.cc:1126] T TestTablet P 2485c0bc559f43ee9b8244b2a1125f0e [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.22->[1.23-1.32] Dedup: 1.30->[1.31-1.32] I0920 01:06:18.562125 403 raft_consensus.cc:1126] T TestTablet P aad1d142980a42798f4dd051364cc06b [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.31->[1.32-1.32] Dedup: 1.32->[] W0920 01:06:18.567728 403 consensus_peers.cc:458] T TestTablet P 13355128a3c84c399b0a2f382a12e69f -> Peer aad1d142980a42798f4dd051364cc06b (peer-1.fake-domain-for-tests:0): Couldn't send request to peer aad1d142980a42798f4dd051364cc06b. Error code: UNKNOWN_ERROR (1). Status: IO error: Artificial error caused by communication failure injection.. This is attempt 1: this message will repeat every 5th retry. I0920 01:06:18.739205 403 raft_consensus.cc:1126] T TestTablet P 2485c0bc559f43ee9b8244b2a1125f0e [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.22->[1.23-1.34] Dedup: 1.32->[1.33-1.34] W0920 01:06:18.740839 405 consensus_peers.cc:458] T TestTablet P 13355128a3c84c399b0a2f382a12e69f -> Peer 2485c0bc559f43ee9b8244b2a1125f0e (peer-0.fake-domain-for-tests:0): Couldn't send request to peer 2485c0bc559f43ee9b8244b2a1125f0e. Error code: UNKNOWN_ERROR (1). Status: IO error: Artificial error caused by communication failure injection.. This is attempt 6: this message will repeat every 5th retry. I0920 01:06:19.161079 405 raft_consensus.cc:1126] T TestTablet P 2485c0bc559f43ee9b8244b2a1125f0e [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.22->[1.23-1.34] Dedup: 1.34->[] I0920 01:06:19.164418 403 raft_consensus.cc:1126] T TestTablet P aad1d142980a42798f4dd051364cc06b [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.33->[1.34-1.34] Dedup: 1.34->[] W0920 01:06:19.168448 405 consensus_peers.cc:458] T TestTablet P 13355128a3c84c399b0a2f382a12e69f -> Peer 2485c0bc559f43ee9b8244b2a1125f0e (peer-0.fake-domain-for-tests:0): Couldn't send request to peer 2485c0bc559f43ee9b8244b2a1125f0e. Error code: UNKNOWN_ERROR (1). Status: IO error: Artificial error caused by communication failure injection.. This is attempt 1: this message will repeat every 5th retry. W0920 01:06:19.170900 403 consensus_peers.cc:458] T TestTablet P 13355128a3c84c399b0a2f382a12e69f -> Peer aad1d142980a42798f4dd051364cc06b (peer-1.fake-domain-for-tests:0): Couldn't send request to peer aad1d142980a42798f4dd051364cc06b. Error code: UNKNOWN_ERROR (1). Status: IO error: Artificial error caused by communication failure injection.. This is attempt 1: this message will repeat every 5th retry. I0920 01:06:19.581285 403 raft_consensus.cc:1126] T TestTablet P 2485c0bc559f43ee9b8244b2a1125f0e [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.34->[1.35-1.36] Dedup: 1.35->[1.36-1.36] W0920 01:06:19.585168 405 consensus_peers.cc:458] T TestTablet P 13355128a3c84c399b0a2f382a12e69f -> Peer 2485c0bc559f43ee9b8244b2a1125f0e (peer-0.fake-domain-for-tests:0): Couldn't send request to peer 2485c0bc559f43ee9b8244b2a1125f0e. Error code: UNKNOWN_ERROR (1). Status: IO error: Artificial error caused by communication failure injection.. This is attempt 1: this message will repeat every 5th retry. I0920 01:06:19.729812 405 raft_consensus.cc:1126] T TestTablet P aad1d142980a42798f4dd051364cc06b [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.35->[1.36-1.37] Dedup: 1.36->[1.37-1.37] W0920 01:06:19.733619 405 consensus_peers.cc:458] T TestTablet P 13355128a3c84c399b0a2f382a12e69f -> Peer aad1d142980a42798f4dd051364cc06b (peer-1.fake-domain-for-tests:0): Couldn't send request to peer aad1d142980a42798f4dd051364cc06b. Error code: UNKNOWN_ERROR (1). Status: IO error: Artificial error caused by communication failure injection.. This is attempt 1: this message will repeat every 5th retry. I0920 01:06:20.041451 405 raft_consensus.cc:1126] T TestTablet P 2485c0bc559f43ee9b8244b2a1125f0e [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.36->[1.37-1.38] Dedup: 1.37->[1.38-1.38] W0920 01:06:20.045542 403 consensus_peers.cc:458] T TestTablet P 13355128a3c84c399b0a2f382a12e69f -> Peer 2485c0bc559f43ee9b8244b2a1125f0e (peer-0.fake-domain-for-tests:0): Couldn't send request to peer 2485c0bc559f43ee9b8244b2a1125f0e. Error code: UNKNOWN_ERROR (1). Status: IO error: Artificial error caused by communication failure injection.. This is attempt 1: this message will repeat every 5th retry. I0920 01:06:20.293141 403 raft_consensus.cc:1126] T TestTablet P aad1d142980a42798f4dd051364cc06b [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.37->[1.38-1.39] Dedup: 1.38->[1.39-1.39] W0920 01:06:20.297268 405 consensus_peers.cc:458] T TestTablet P 13355128a3c84c399b0a2f382a12e69f -> Peer aad1d142980a42798f4dd051364cc06b (peer-1.fake-domain-for-tests:0): Couldn't send request to peer aad1d142980a42798f4dd051364cc06b. Error code: UNKNOWN_ERROR (1). Status: IO error: Artificial error caused by communication failure injection.. This is attempt 1: this message will repeat every 5th retry. I0920 01:06:20.549623 405 raft_consensus.cc:1126] T TestTablet P 2485c0bc559f43ee9b8244b2a1125f0e [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.38->[1.39-1.40] Dedup: 1.39->[1.40-1.40] W0920 01:06:20.553819 403 consensus_peers.cc:458] T TestTablet P 13355128a3c84c399b0a2f382a12e69f -> Peer 2485c0bc559f43ee9b8244b2a1125f0e (peer-0.fake-domain-for-tests:0): Couldn't send request to peer 2485c0bc559f43ee9b8244b2a1125f0e. Error code: UNKNOWN_ERROR (1). Status: IO error: Artificial error caused by communication failure injection.. This is attempt 1: this message will repeat every 5th retry. I0920 01:06:20.908007 403 raft_consensus.cc:1126] T TestTablet P aad1d142980a42798f4dd051364cc06b [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.39->[1.40-1.41] Dedup: 1.40->[1.41-1.41] W0920 01:06:20.912314 405 consensus_peers.cc:458] T TestTablet P 13355128a3c84c399b0a2f382a12e69f -> Peer aad1d142980a42798f4dd051364cc06b (peer-1.fake-domain-for-tests:0): Couldn't send request to peer aad1d142980a42798f4dd051364cc06b. Error code: UNKNOWN_ERROR (1). Status: IO error: Artificial error caused by communication failure injection.. This is attempt 1: this message will repeat every 5th retry. I0920 01:06:21.087867 405 raft_consensus.cc:1126] T TestTablet P 2485c0bc559f43ee9b8244b2a1125f0e [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.40->[1.41-1.42] Dedup: 1.41->[1.42-1.42] W0920 01:06:21.090749 405 consensus_peers.cc:458] T TestTablet P 13355128a3c84c399b0a2f382a12e69f -> Peer 2485c0bc559f43ee9b8244b2a1125f0e (peer-0.fake-domain-for-tests:0): Couldn't send request to peer 2485c0bc559f43ee9b8244b2a1125f0e. Error code: UNKNOWN_ERROR (1). Status: IO error: Artificial error caused by communication failure injection.. This is attempt 1: this message will repeat every 5th retry. I0920 01:06:21.473383 405 raft_consensus.cc:1126] T TestTablet P aad1d142980a42798f4dd051364cc06b [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.41->[1.42-1.43] Dedup: 1.42->[1.43-1.43] W0920 01:06:21.477890 403 consensus_peers.cc:458] T TestTablet P 13355128a3c84c399b0a2f382a12e69f -> Peer aad1d142980a42798f4dd051364cc06b (peer-1.fake-domain-for-tests:0): Couldn't send request to peer aad1d142980a42798f4dd051364cc06b. Error code: UNKNOWN_ERROR (1). Status: IO error: Artificial error caused by communication failure injection.. This is attempt 1: this message will repeat every 5th retry. W0920 01:06:21.612645 405 consensus_peers.cc:458] T TestTablet P 13355128a3c84c399b0a2f382a12e69f -> Peer 2485c0bc559f43ee9b8244b2a1125f0e (peer-0.fake-domain-for-tests:0): Couldn't send request to peer 2485c0bc559f43ee9b8244b2a1125f0e. Error code: UNKNOWN_ERROR (1). Status: IO error: Artificial error caused by communication failure injection.. This is attempt 1: this message will repeat every 5th retry. I0920 01:06:21.996580 405 raft_consensus.cc:1126] T TestTablet P 2485c0bc559f43ee9b8244b2a1125f0e [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.44->[1.45-1.45] Dedup: 1.45->[] W0920 01:06:22.002377 405 consensus_peers.cc:458] T TestTablet P 13355128a3c84c399b0a2f382a12e69f -> Peer 2485c0bc559f43ee9b8244b2a1125f0e (peer-0.fake-domain-for-tests:0): Couldn't send request to peer 2485c0bc559f43ee9b8244b2a1125f0e. Error code: UNKNOWN_ERROR (1). Status: IO error: Artificial error caused by communication failure injection.. This is attempt 1: this message will repeat every 5th retry. I0920 01:06:22.052490 405 raft_consensus.cc:1126] T TestTablet P aad1d142980a42798f4dd051364cc06b [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.43->[1.44-1.47] Dedup: 1.44->[1.45-1.47] I0920 01:06:22.545743 403 raft_consensus.cc:1126] T TestTablet P 2485c0bc559f43ee9b8244b2a1125f0e [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.46->[1.47-1.47] Dedup: 1.47->[] W0920 01:06:22.552592 405 consensus_peers.cc:458] T TestTablet P 13355128a3c84c399b0a2f382a12e69f -> Peer 2485c0bc559f43ee9b8244b2a1125f0e (peer-0.fake-domain-for-tests:0): Couldn't send request to peer 2485c0bc559f43ee9b8244b2a1125f0e. Error code: UNKNOWN_ERROR (1). Status: IO error: Artificial error caused by communication failure injection.. This is attempt 1: this message will repeat every 5th retry. I0920 01:06:22.661335 405 raft_consensus.cc:1126] T TestTablet P aad1d142980a42798f4dd051364cc06b [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.43->[1.44-1.49] Dedup: 1.47->[1.48-1.49] I0920 01:06:23.037741 403 raft_consensus.cc:1126] T TestTablet P aad1d142980a42798f4dd051364cc06b [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.43->[1.44-1.49] Dedup: 1.49->[] W0920 01:06:23.040966 403 consensus_peers.cc:458] T TestTablet P 13355128a3c84c399b0a2f382a12e69f -> Peer aad1d142980a42798f4dd051364cc06b (peer-1.fake-domain-for-tests:0): Couldn't send request to peer aad1d142980a42798f4dd051364cc06b. Error code: UNKNOWN_ERROR (1). Status: IO error: Artificial error caused by communication failure injection.. This is attempt 1: this message will repeat every 5th retry. I0920 01:06:23.112041 403 raft_consensus.cc:1126] T TestTablet P 2485c0bc559f43ee9b8244b2a1125f0e [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.48->[1.49-1.50] Dedup: 1.49->[1.50-1.50] W0920 01:06:23.116428 403 consensus_peers.cc:458] T TestTablet P 13355128a3c84c399b0a2f382a12e69f -> Peer 2485c0bc559f43ee9b8244b2a1125f0e (peer-0.fake-domain-for-tests:0): Couldn't send request to peer 2485c0bc559f43ee9b8244b2a1125f0e. Error code: UNKNOWN_ERROR (1). Status: IO error: Artificial error caused by communication failure injection.. This is attempt 1: this message will repeat every 5th retry. I0920 01:06:23.493712 405 raft_consensus.cc:1126] T TestTablet P aad1d142980a42798f4dd051364cc06b [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.49->[1.50-1.51] Dedup: 1.50->[1.51-1.51] W0920 01:06:23.496467 405 consensus_peers.cc:458] T TestTablet P 13355128a3c84c399b0a2f382a12e69f -> Peer aad1d142980a42798f4dd051364cc06b (peer-1.fake-domain-for-tests:0): Couldn't send request to peer aad1d142980a42798f4dd051364cc06b. Error code: UNKNOWN_ERROR (1). Status: IO error: Artificial error caused by communication failure injection.. This is attempt 1: this message will repeat every 5th retry. I0920 01:06:23.546272 405 raft_consensus.cc:1126] T TestTablet P 2485c0bc559f43ee9b8244b2a1125f0e [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.50->[1.51-1.52] Dedup: 1.51->[1.52-1.52] W0920 01:06:23.550768 403 consensus_peers.cc:458] T TestTablet P 13355128a3c84c399b0a2f382a12e69f -> Peer 2485c0bc559f43ee9b8244b2a1125f0e (peer-0.fake-domain-for-tests:0): Couldn't send request to peer 2485c0bc559f43ee9b8244b2a1125f0e. Error code: UNKNOWN_ERROR (1). Status: IO error: Artificial error caused by communication failure injection.. This is attempt 1: this message will repeat every 5th retry. I0920 01:06:23.999759 403 raft_consensus.cc:1126] T TestTablet P 2485c0bc559f43ee9b8244b2a1125f0e [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.52->[1.53-1.53] Dedup: 1.53->[] W0920 01:06:24.000797 403 consensus_peers.cc:458] T TestTablet P 13355128a3c84c399b0a2f382a12e69f -> Peer aad1d142980a42798f4dd051364cc06b (peer-1.fake-domain-for-tests:0): Couldn't send request to peer aad1d142980a42798f4dd051364cc06b. Error code: UNKNOWN_ERROR (1). Status: IO error: Artificial error caused by communication failure injection.. This is attempt 1: this message will repeat every 5th retry. W0920 01:06:24.005589 423 consensus_peers.cc:458] T TestTablet P 13355128a3c84c399b0a2f382a12e69f -> Peer 2485c0bc559f43ee9b8244b2a1125f0e (peer-0.fake-domain-for-tests:0): Couldn't send request to peer 2485c0bc559f43ee9b8244b2a1125f0e. Error code: UNKNOWN_ERROR (1). Status: IO error: Artificial error caused by communication failure injection.. This is attempt 1: this message will repeat every 5th retry. W0920 01:06:24.548595 426 consensus_peers.cc:458] T TestTablet P 13355128a3c84c399b0a2f382a12e69f -> Peer aad1d142980a42798f4dd051364cc06b (peer-1.fake-domain-for-tests:0): Couldn't send request to peer aad1d142980a42798f4dd051364cc06b. Error code: UNKNOWN_ERROR (1). Status: IO error: Artificial error caused by communication failure injection.. This is attempt 1: this message will repeat every 5th retry. I0920 01:06:24.620451 426 raft_consensus.cc:1126] T TestTablet P 2485c0bc559f43ee9b8244b2a1125f0e [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.54->[1.55-1.56] Dedup: 1.55->[1.56-1.56] W0920 01:06:24.624859 426 consensus_peers.cc:458] T TestTablet P 13355128a3c84c399b0a2f382a12e69f -> Peer 2485c0bc559f43ee9b8244b2a1125f0e (peer-0.fake-domain-for-tests:0): Couldn't send request to peer 2485c0bc559f43ee9b8244b2a1125f0e. Error code: UNKNOWN_ERROR (1). Status: IO error: Artificial error caused by communication failure injection.. This is attempt 1: this message will repeat every 5th retry. I0920 01:06:25.125319 426 raft_consensus.cc:1126] T TestTablet P 2485c0bc559f43ee9b8244b2a1125f0e [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.56->[1.57-1.57] Dedup: 1.57->[] W0920 01:06:25.137100 426 consensus_peers.cc:458] T TestTablet P 13355128a3c84c399b0a2f382a12e69f -> Peer 2485c0bc559f43ee9b8244b2a1125f0e (peer-0.fake-domain-for-tests:0): Couldn't send request to peer 2485c0bc559f43ee9b8244b2a1125f0e. Error code: UNKNOWN_ERROR (1). Status: IO error: Artificial error caused by communication failure injection.. This is attempt 1: this message will repeat every 5th retry. I0920 01:06:25.162119 426 raft_consensus.cc:1126] T TestTablet P aad1d142980a42798f4dd051364cc06b [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.55->[1.56-1.59] Dedup: 1.56->[1.57-1.59] I0920 01:06:25.661396 431 raft_consensus.cc:1126] T TestTablet P 2485c0bc559f43ee9b8244b2a1125f0e [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.58->[1.59-1.59] Dedup: 1.59->[] W0920 01:06:25.667184 426 consensus_peers.cc:458] T TestTablet P 13355128a3c84c399b0a2f382a12e69f -> Peer 2485c0bc559f43ee9b8244b2a1125f0e (peer-0.fake-domain-for-tests:0): Couldn't send request to peer 2485c0bc559f43ee9b8244b2a1125f0e. Error code: UNKNOWN_ERROR (1). Status: IO error: Artificial error caused by communication failure injection.. This is attempt 1: this message will repeat every 5th retry. I0920 01:06:25.764672 426 raft_consensus.cc:1126] T TestTablet P aad1d142980a42798f4dd051364cc06b [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.55->[1.56-1.61] Dedup: 1.59->[1.60-1.61] I0920 01:06:26.061640 431 raft_consensus.cc:1126] T TestTablet P 2485c0bc559f43ee9b8244b2a1125f0e [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.60->[1.61-1.61] Dedup: 1.61->[] W0920 01:06:26.067492 426 consensus_peers.cc:458] T TestTablet P 13355128a3c84c399b0a2f382a12e69f -> Peer 2485c0bc559f43ee9b8244b2a1125f0e (peer-0.fake-domain-for-tests:0): Couldn't send request to peer 2485c0bc559f43ee9b8244b2a1125f0e. Error code: UNKNOWN_ERROR (1). Status: IO error: Artificial error caused by communication failure injection.. This is attempt 1: this message will repeat every 5th retry. I0920 01:06:26.345259 426 raft_consensus.cc:1126] T TestTablet P aad1d142980a42798f4dd051364cc06b [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.55->[1.56-1.63] Dedup: 1.61->[1.62-1.63] I0920 01:06:26.597409 431 raft_consensus.cc:1126] T TestTablet P 2485c0bc559f43ee9b8244b2a1125f0e [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.62->[1.63-1.63] Dedup: 1.63->[] W0920 01:06:26.604705 431 consensus_peers.cc:458] T TestTablet P 13355128a3c84c399b0a2f382a12e69f -> Peer 2485c0bc559f43ee9b8244b2a1125f0e (peer-0.fake-domain-for-tests:0): Couldn't send request to peer 2485c0bc559f43ee9b8244b2a1125f0e. Error code: UNKNOWN_ERROR (1). Status: IO error: Artificial error caused by communication failure injection.. This is attempt 1: this message will repeat every 5th retry. I0920 01:06:26.938637 431 raft_consensus.cc:1126] T TestTablet P aad1d142980a42798f4dd051364cc06b [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.55->[1.56-1.65] Dedup: 1.63->[1.64-1.65] I0920 01:06:27.047994 426 raft_consensus.cc:1126] T TestTablet P 2485c0bc559f43ee9b8244b2a1125f0e [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.64->[1.65-1.65] Dedup: 1.65->[] W0920 01:06:27.053874 426 consensus_peers.cc:458] T TestTablet P 13355128a3c84c399b0a2f382a12e69f -> Peer 2485c0bc559f43ee9b8244b2a1125f0e (peer-0.fake-domain-for-tests:0): Couldn't send request to peer 2485c0bc559f43ee9b8244b2a1125f0e. Error code: UNKNOWN_ERROR (1). Status: IO error: Artificial error caused by communication failure injection.. This is attempt 1: this message will repeat every 5th retry. I0920 01:06:27.392871 426 raft_consensus.cc:1126] T TestTablet P aad1d142980a42798f4dd051364cc06b [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.55->[1.56-1.67] Dedup: 1.65->[1.66-1.67] W0920 01:06:27.395519 431 consensus_peers.cc:458] T TestTablet P 13355128a3c84c399b0a2f382a12e69f -> Peer aad1d142980a42798f4dd051364cc06b (peer-1.fake-domain-for-tests:0): Couldn't send request to peer aad1d142980a42798f4dd051364cc06b. Error code: UNKNOWN_ERROR (1). Status: IO error: Artificial error caused by communication failure injection.. This is attempt 6: this message will repeat every 5th retry. I0920 01:06:27.518929 431 raft_consensus.cc:1126] T TestTablet P 2485c0bc559f43ee9b8244b2a1125f0e [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.66->[1.67-1.67] Dedup: 1.67->[] W0920 01:06:27.525772 431 consensus_peers.cc:458] T TestTablet P 13355128a3c84c399b0a2f382a12e69f -> Peer 2485c0bc559f43ee9b8244b2a1125f0e (peer-0.fake-domain-for-tests:0): Couldn't send request to peer 2485c0bc559f43ee9b8244b2a1125f0e. Error code: UNKNOWN_ERROR (1). Status: IO error: Artificial error caused by communication failure injection.. This is attempt 1: this message will repeat every 5th retry. I0920 01:06:27.857314 431 raft_consensus.cc:1126] T TestTablet P aad1d142980a42798f4dd051364cc06b [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.55->[1.56-1.69] Dedup: 1.67->[1.68-1.69] I0920 01:06:28.082386 426 raft_consensus.cc:1126] T TestTablet P 2485c0bc559f43ee9b8244b2a1125f0e [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.68->[1.69-1.69] Dedup: 1.69->[] I0920 01:06:28.479671 431 raft_consensus.cc:1126] T TestTablet P aad1d142980a42798f4dd051364cc06b [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.55->[1.56-1.70] Dedup: 1.69->[1.70-1.70] W0920 01:06:28.501171 426 consensus_peers.cc:458] T TestTablet P 13355128a3c84c399b0a2f382a12e69f -> Peer 2485c0bc559f43ee9b8244b2a1125f0e (peer-0.fake-domain-for-tests:0): Couldn't send request to peer 2485c0bc559f43ee9b8244b2a1125f0e. Error code: UNKNOWN_ERROR (1). Status: IO error: Artificial error caused by communication failure injection.. This is attempt 1: this message will repeat every 5th retry. I0920 01:06:28.904455 426 raft_consensus.cc:1126] T TestTablet P aad1d142980a42798f4dd051364cc06b [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.55->[1.56-1.71] Dedup: 1.70->[1.71-1.71] W0920 01:06:28.907981 426 consensus_peers.cc:458] T TestTablet P 13355128a3c84c399b0a2f382a12e69f -> Peer aad1d142980a42798f4dd051364cc06b (peer-1.fake-domain-for-tests:0): Couldn't send request to peer aad1d142980a42798f4dd051364cc06b. Error code: UNKNOWN_ERROR (1). Status: IO error: Artificial error caused by communication failure injection.. This is attempt 1: this message will repeat every 5th retry. I0920 01:06:29.062911 426 raft_consensus.cc:1126] T TestTablet P 2485c0bc559f43ee9b8244b2a1125f0e [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.70->[1.71-1.72] Dedup: 1.71->[1.72-1.72] W0920 01:06:29.067155 431 consensus_peers.cc:458] T TestTablet P 13355128a3c84c399b0a2f382a12e69f -> Peer 2485c0bc559f43ee9b8244b2a1125f0e (peer-0.fake-domain-for-tests:0): Couldn't send request to peer 2485c0bc559f43ee9b8244b2a1125f0e. Error code: UNKNOWN_ERROR (1). Status: IO error: Artificial error caused by communication failure injection.. This is attempt 1: this message will repeat every 5th retry. W0920 01:06:29.510706 431 consensus_peers.cc:458] T TestTablet P 13355128a3c84c399b0a2f382a12e69f -> Peer aad1d142980a42798f4dd051364cc06b (peer-1.fake-domain-for-tests:0): Couldn't send request to peer aad1d142980a42798f4dd051364cc06b. Error code: UNKNOWN_ERROR (1). Status: IO error: Artificial error caused by communication failure injection.. This is attempt 1: this message will repeat every 5th retry. I0920 01:06:29.598191 431 raft_consensus.cc:1126] T TestTablet P 2485c0bc559f43ee9b8244b2a1125f0e [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.72->[1.73-1.74] Dedup: 1.73->[1.74-1.74] W0920 01:06:29.602470 431 consensus_peers.cc:458] T TestTablet P 13355128a3c84c399b0a2f382a12e69f -> Peer 2485c0bc559f43ee9b8244b2a1125f0e (peer-0.fake-domain-for-tests:0): Couldn't send request to peer 2485c0bc559f43ee9b8244b2a1125f0e. Error code: UNKNOWN_ERROR (1). Status: IO error: Artificial error caused by communication failure injection.. This is attempt 1: this message will repeat every 5th retry. I0920 01:06:29.997153 431 raft_consensus.cc:1126] T TestTablet P 2485c0bc559f43ee9b8244b2a1125f0e [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.74->[1.75-1.75] Dedup: 1.75->[] W0920 01:06:30.003798 426 consensus_peers.cc:458] T TestTablet P 13355128a3c84c399b0a2f382a12e69f -> Peer 2485c0bc559f43ee9b8244b2a1125f0e (peer-0.fake-domain-for-tests:0): Couldn't send request to peer 2485c0bc559f43ee9b8244b2a1125f0e. Error code: UNKNOWN_ERROR (1). Status: IO error: Artificial error caused by communication failure injection.. This is attempt 1: this message will repeat every 5th retry. I0920 01:06:30.475873 426 raft_consensus.cc:1126] T TestTablet P aad1d142980a42798f4dd051364cc06b [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.73->[1.74-1.77] Dedup: 1.76->[1.77-1.77] W0920 01:06:30.480177 442 consensus_peers.cc:458] T TestTablet P 13355128a3c84c399b0a2f382a12e69f -> Peer aad1d142980a42798f4dd051364cc06b (peer-1.fake-domain-for-tests:0): Couldn't send request to peer aad1d142980a42798f4dd051364cc06b. Error code: UNKNOWN_ERROR (1). Status: IO error: Artificial error caused by communication failure injection.. This is attempt 1: this message will repeat every 5th retry. I0920 01:06:30.598624 442 raft_consensus.cc:1126] T TestTablet P 2485c0bc559f43ee9b8244b2a1125f0e [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.76->[1.77-1.78] Dedup: 1.77->[1.78-1.78] W0920 01:06:30.603046 442 consensus_peers.cc:458] T TestTablet P 13355128a3c84c399b0a2f382a12e69f -> Peer 2485c0bc559f43ee9b8244b2a1125f0e (peer-0.fake-domain-for-tests:0): Couldn't send request to peer 2485c0bc559f43ee9b8244b2a1125f0e. Error code: UNKNOWN_ERROR (1). Status: IO error: Artificial error caused by communication failure injection.. This is attempt 1: this message will repeat every 5th retry. I0920 01:06:31.045307 442 raft_consensus.cc:1126] T TestTablet P aad1d142980a42798f4dd051364cc06b [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.77->[1.78-1.79] Dedup: 1.78->[1.79-1.79] W0920 01:06:31.048538 442 consensus_peers.cc:458] T TestTablet P 13355128a3c84c399b0a2f382a12e69f -> Peer aad1d142980a42798f4dd051364cc06b (peer-1.fake-domain-for-tests:0): Couldn't send request to peer aad1d142980a42798f4dd051364cc06b. Error code: UNKNOWN_ERROR (1). Status: IO error: Artificial error caused by communication failure injection.. This is attempt 1: this message will repeat every 5th retry. I0920 01:06:31.186969 442 raft_consensus.cc:1126] T TestTablet P 2485c0bc559f43ee9b8244b2a1125f0e [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.78->[1.79-1.80] Dedup: 1.79->[1.80-1.80] W0920 01:06:31.191190 442 consensus_peers.cc:458] T TestTablet P 13355128a3c84c399b0a2f382a12e69f -> Peer 2485c0bc559f43ee9b8244b2a1125f0e (peer-0.fake-domain-for-tests:0): Couldn't send request to peer 2485c0bc559f43ee9b8244b2a1125f0e. Error code: UNKNOWN_ERROR (1). Status: IO error: Artificial error caused by communication failure injection.. This is attempt 1: this message will repeat every 5th retry. W0920 01:06:31.511934 426 consensus_peers.cc:458] T TestTablet P 13355128a3c84c399b0a2f382a12e69f -> Peer aad1d142980a42798f4dd051364cc06b (peer-1.fake-domain-for-tests:0): Couldn't send request to peer aad1d142980a42798f4dd051364cc06b. Error code: UNKNOWN_ERROR (1). Status: IO error: Artificial error caused by communication failure injection.. This is attempt 1: this message will repeat every 5th retry. I0920 01:06:31.664153 442 raft_consensus.cc:1126] T TestTablet P 2485c0bc559f43ee9b8244b2a1125f0e [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.80->[1.81-1.82] Dedup: 1.81->[1.82-1.82] W0920 01:06:31.668159 442 consensus_peers.cc:458] T TestTablet P 13355128a3c84c399b0a2f382a12e69f -> Peer 2485c0bc559f43ee9b8244b2a1125f0e (peer-0.fake-domain-for-tests:0): Couldn't send request to peer 2485c0bc559f43ee9b8244b2a1125f0e. Error code: UNKNOWN_ERROR (1). Status: IO error: Artificial error caused by communication failure injection.. This is attempt 1: this message will repeat every 5th retry. W0920 01:06:32.032488 442 consensus_peers.cc:458] T TestTablet P 13355128a3c84c399b0a2f382a12e69f -> Peer aad1d142980a42798f4dd051364cc06b (peer-1.fake-domain-for-tests:0): Couldn't send request to peer aad1d142980a42798f4dd051364cc06b. Error code: UNKNOWN_ERROR (1). Status: IO error: Artificial error caused by communication failure injection.. This is attempt 1: this message will repeat every 5th retry. I0920 01:06:32.061265 442 raft_consensus.cc:1126] T TestTablet P 2485c0bc559f43ee9b8244b2a1125f0e [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.82->[1.83-1.84] Dedup: 1.83->[1.84-1.84] W0920 01:06:32.065379 426 consensus_peers.cc:458] T TestTablet P 13355128a3c84c399b0a2f382a12e69f -> Peer 2485c0bc559f43ee9b8244b2a1125f0e (peer-0.fake-domain-for-tests:0): Couldn't send request to peer 2485c0bc559f43ee9b8244b2a1125f0e. Error code: UNKNOWN_ERROR (1). Status: IO error: Artificial error caused by communication failure injection.. This is attempt 1: this message will repeat every 5th retry. I0920 01:06:32.534648 426 raft_consensus.cc:1126] T TestTablet P 2485c0bc559f43ee9b8244b2a1125f0e [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.84->[1.85-1.85] Dedup: 1.85->[] W0920 01:06:32.540580 442 consensus_peers.cc:458] T TestTablet P 13355128a3c84c399b0a2f382a12e69f -> Peer 2485c0bc559f43ee9b8244b2a1125f0e (peer-0.fake-domain-for-tests:0): Couldn't send request to peer 2485c0bc559f43ee9b8244b2a1125f0e. Error code: UNKNOWN_ERROR (1). Status: IO error: Artificial error caused by communication failure injection.. This is attempt 1: this message will repeat every 5th retry. I0920 01:06:32.546519 442 raft_consensus.cc:1126] T TestTablet P aad1d142980a42798f4dd051364cc06b [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.83->[1.84-1.87] Dedup: 1.84->[1.85-1.87] I0920 01:06:33.065775 445 raft_consensus.cc:1126] T TestTablet P aad1d142980a42798f4dd051364cc06b [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.83->[1.84-1.87] Dedup: 1.87->[] W0920 01:06:33.071211 445 consensus_peers.cc:458] T TestTablet P 13355128a3c84c399b0a2f382a12e69f -> Peer aad1d142980a42798f4dd051364cc06b (peer-1.fake-domain-for-tests:0): Couldn't send request to peer aad1d142980a42798f4dd051364cc06b. Error code: UNKNOWN_ERROR (1). Status: IO error: Artificial error caused by communication failure injection.. This is attempt 1: this message will repeat every 5th retry. I0920 01:06:33.144373 445 raft_consensus.cc:1126] T TestTablet P 2485c0bc559f43ee9b8244b2a1125f0e [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.86->[1.87-1.88] Dedup: 1.87->[1.88-1.88] W0920 01:06:33.148393 446 consensus_peers.cc:458] T TestTablet P 13355128a3c84c399b0a2f382a12e69f -> Peer 2485c0bc559f43ee9b8244b2a1125f0e (peer-0.fake-domain-for-tests:0): Couldn't send request to peer 2485c0bc559f43ee9b8244b2a1125f0e. Error code: UNKNOWN_ERROR (1). Status: IO error: Artificial error caused by communication failure injection.. This is attempt 1: this message will repeat every 5th retry. I0920 01:06:33.602718 446 raft_consensus.cc:1126] T TestTablet P aad1d142980a42798f4dd051364cc06b [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.87->[1.88-1.89] Dedup: 1.88->[1.89-1.89] W0920 01:06:33.606557 446 consensus_peers.cc:458] T TestTablet P 13355128a3c84c399b0a2f382a12e69f -> Peer aad1d142980a42798f4dd051364cc06b (peer-1.fake-domain-for-tests:0): Couldn't send request to peer aad1d142980a42798f4dd051364cc06b. Error code: UNKNOWN_ERROR (1). Status: IO error: Artificial error caused by communication failure injection.. This is attempt 1: this message will repeat every 5th retry. I0920 01:06:33.672348 446 raft_consensus.cc:1126] T TestTablet P 2485c0bc559f43ee9b8244b2a1125f0e [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.88->[1.89-1.90] Dedup: 1.89->[1.90-1.90] W0920 01:06:33.676249 446 consensus_peers.cc:458] T TestTablet P 13355128a3c84c399b0a2f382a12e69f -> Peer 2485c0bc559f43ee9b8244b2a1125f0e (peer-0.fake-domain-for-tests:0): Couldn't send request to peer 2485c0bc559f43ee9b8244b2a1125f0e. Error code: UNKNOWN_ERROR (1). Status: IO error: Artificial error caused by communication failure injection.. This is attempt 1: this message will repeat every 5th retry. I0920 01:06:33.986779 446 raft_consensus.cc:1126] T TestTablet P aad1d142980a42798f4dd051364cc06b [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.89->[1.90-1.91] Dedup: 1.90->[1.91-1.91] I0920 01:06:34.244242 450 raft_consensus.cc:1126] T TestTablet P 2485c0bc559f43ee9b8244b2a1125f0e [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.90->[1.91-1.91] Dedup: 1.91->[] W0920 01:06:34.393306 385 log.cc:669] Time spent T TestTablet P 13355128a3c84c399b0a2f382a12e69f: Append to log took a long time: real 0.147s user 0.000s sys 0.000s W0920 01:06:34.393596 392 log.cc:669] Time spent T TestTablet P aad1d142980a42798f4dd051364cc06b: Append to log took a long time: real 0.406s user 0.000s sys 0.000s W0920 01:06:34.393446 390 log.cc:669] Time spent T TestTablet P 2485c0bc559f43ee9b8244b2a1125f0e: Append to log took a long time: real 0.147s user 0.000s sys 0.000s W0920 01:06:34.397266 452 consensus_peers.cc:458] T TestTablet P 13355128a3c84c399b0a2f382a12e69f -> Peer 2485c0bc559f43ee9b8244b2a1125f0e (peer-0.fake-domain-for-tests:0): Couldn't send request to peer 2485c0bc559f43ee9b8244b2a1125f0e. Error code: UNKNOWN_ERROR (1). Status: IO error: Artificial error caused by communication failure injection.. This is attempt 1: this message will repeat every 5th retry. I0920 01:06:34.548820 452 raft_consensus.cc:1126] T TestTablet P aad1d142980a42798f4dd051364cc06b [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.89->[1.90-1.93] Dedup: 1.91->[1.92-1.93] W0920 01:06:34.552845 450 consensus_peers.cc:458] T TestTablet P 13355128a3c84c399b0a2f382a12e69f -> Peer aad1d142980a42798f4dd051364cc06b (peer-1.fake-domain-for-tests:0): Couldn't send request to peer aad1d142980a42798f4dd051364cc06b. Error code: UNKNOWN_ERROR (1). Status: IO error: Artificial error caused by communication failure injection.. This is attempt 1: this message will repeat every 5th retry. I0920 01:06:34.964574 450 raft_consensus.cc:1126] T TestTablet P 2485c0bc559f43ee9b8244b2a1125f0e [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.92->[1.93-1.94] Dedup: 1.93->[1.94-1.94] W0920 01:06:34.968648 452 consensus_peers.cc:458] T TestTablet P 13355128a3c84c399b0a2f382a12e69f -> Peer 2485c0bc559f43ee9b8244b2a1125f0e (peer-0.fake-domain-for-tests:0): Couldn't send request to peer 2485c0bc559f43ee9b8244b2a1125f0e. Error code: UNKNOWN_ERROR (1). Status: IO error: Artificial error caused by communication failure injection.. This is attempt 1: this message will repeat every 5th retry. I0920 01:06:34.998701 452 raft_consensus.cc:1126] T TestTablet P aad1d142980a42798f4dd051364cc06b [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.93->[1.94-1.95] Dedup: 1.94->[1.95-1.95] W0920 01:06:35.002681 450 consensus_peers.cc:458] T TestTablet P 13355128a3c84c399b0a2f382a12e69f -> Peer aad1d142980a42798f4dd051364cc06b (peer-1.fake-domain-for-tests:0): Couldn't send request to peer aad1d142980a42798f4dd051364cc06b. Error code: UNKNOWN_ERROR (1). Status: IO error: Artificial error caused by communication failure injection.. This is attempt 1: this message will repeat every 5th retry. I0920 01:06:35.374573 450 raft_consensus.cc:1126] T TestTablet P 2485c0bc559f43ee9b8244b2a1125f0e [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.94->[1.95-1.96] Dedup: 1.95->[1.96-1.96] W0920 01:06:35.378396 452 consensus_peers.cc:458] T TestTablet P 13355128a3c84c399b0a2f382a12e69f -> Peer 2485c0bc559f43ee9b8244b2a1125f0e (peer-0.fake-domain-for-tests:0): Couldn't send request to peer 2485c0bc559f43ee9b8244b2a1125f0e. Error code: UNKNOWN_ERROR (1). Status: IO error: Artificial error caused by communication failure injection.. This is attempt 1: this message will repeat every 5th retry. I0920 01:06:35.455811 452 raft_consensus.cc:1126] T TestTablet P aad1d142980a42798f4dd051364cc06b [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.95->[1.96-1.97] Dedup: 1.96->[1.97-1.97] W0920 01:06:35.459872 452 consensus_peers.cc:458] T TestTablet P 13355128a3c84c399b0a2f382a12e69f -> Peer aad1d142980a42798f4dd051364cc06b (peer-1.fake-domain-for-tests:0): Couldn't send request to peer aad1d142980a42798f4dd051364cc06b. Error code: UNKNOWN_ERROR (1). Status: IO error: Artificial error caused by communication failure injection.. This is attempt 1: this message will repeat every 5th retry. I0920 01:06:35.797639 452 raft_consensus.cc:1126] T TestTablet P 2485c0bc559f43ee9b8244b2a1125f0e [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.96->[1.97-1.98] Dedup: 1.97->[1.98-1.98] W0920 01:06:35.801676 450 consensus_peers.cc:458] T TestTablet P 13355128a3c84c399b0a2f382a12e69f -> Peer 2485c0bc559f43ee9b8244b2a1125f0e (peer-0.fake-domain-for-tests:0): Couldn't send request to peer 2485c0bc559f43ee9b8244b2a1125f0e. Error code: UNKNOWN_ERROR (1). Status: IO error: Artificial error caused by communication failure injection.. This is attempt 1: this message will repeat every 5th retry. I0920 01:06:35.898466 450 raft_consensus.cc:1126] T TestTablet P aad1d142980a42798f4dd051364cc06b [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.97->[1.98-1.99] Dedup: 1.98->[1.99-1.99] W0920 01:06:35.902623 452 consensus_peers.cc:458] T TestTablet P 13355128a3c84c399b0a2f382a12e69f -> Peer aad1d142980a42798f4dd051364cc06b (peer-1.fake-domain-for-tests:0): Couldn't send request to peer aad1d142980a42798f4dd051364cc06b. Error code: UNKNOWN_ERROR (1). Status: IO error: Artificial error caused by communication failure injection.. This is attempt 1: this message will repeat every 5th retry. I0920 01:06:36.191543 452 raft_consensus.cc:1126] T TestTablet P 2485c0bc559f43ee9b8244b2a1125f0e [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.98->[1.99-1.100] Dedup: 1.99->[1.100-1.100] W0920 01:06:36.195760 452 consensus_peers.cc:458] T TestTablet P 13355128a3c84c399b0a2f382a12e69f -> Peer 2485c0bc559f43ee9b8244b2a1125f0e (peer-0.fake-domain-for-tests:0): Couldn't send request to peer 2485c0bc559f43ee9b8244b2a1125f0e. Error code: UNKNOWN_ERROR (1). Status: IO error: Artificial error caused by communication failure injection.. This is attempt 1: this message will repeat every 5th retry. I0920 01:06:36.375625 452 raft_consensus.cc:1126] T TestTablet P aad1d142980a42798f4dd051364cc06b [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.99->[1.100-1.101] Dedup: 1.100->[1.101-1.101] W0920 01:06:36.379644 452 consensus_peers.cc:458] T TestTablet P 13355128a3c84c399b0a2f382a12e69f -> Peer aad1d142980a42798f4dd051364cc06b (peer-1.fake-domain-for-tests:0): Couldn't send request to peer aad1d142980a42798f4dd051364cc06b. Error code: UNKNOWN_ERROR (1). Status: IO error: Artificial error caused by communication failure injection.. This is attempt 1: this message will repeat every 5th retry. I0920 01:06:36.577543 452 raft_consensus.cc:1126] T TestTablet P 2485c0bc559f43ee9b8244b2a1125f0e [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.100->[1.101-1.102] Dedup: 1.101->[1.102-1.102] I0920 01:06:36.809845 452 raft_consensus.cc:1126] T TestTablet P aad1d142980a42798f4dd051364cc06b [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.101->[1.102-1.102] Dedup: 1.102->[] I0920 01:06:36.836988 233 raft_consensus.cc:2147] T TestTablet P 2485c0bc559f43ee9b8244b2a1125f0e [term 1 FOLLOWER]: Raft consensus shutting down. I0920 01:06:36.837285 233 raft_consensus.cc:2174] T TestTablet P 2485c0bc559f43ee9b8244b2a1125f0e [term 1 FOLLOWER]: Raft consensus is shut down! I0920 01:06:36.837498 233 raft_consensus.cc:2147] T TestTablet P aad1d142980a42798f4dd051364cc06b [term 1 FOLLOWER]: Raft consensus shutting down. I0920 01:06:36.837656 233 raft_consensus.cc:2174] T TestTablet P aad1d142980a42798f4dd051364cc06b [term 1 FOLLOWER]: Raft consensus is shut down! I0920 01:06:36.837785 233 raft_consensus.cc:2147] T TestTablet P 13355128a3c84c399b0a2f382a12e69f [term 1 LEADER]: Raft consensus shutting down. I0920 01:06:36.838196 233 raft_consensus.cc:2174] T TestTablet P 13355128a3c84c399b0a2f382a12e69f [term 1 FOLLOWER]: Raft consensus is shut down! [ OK ] RaftConsensusQuorumTest.TestReplicasHandleCommunicationErrors (27182 ms) [ RUN ] RaftConsensusQuorumTest.TestLeaderHeartbeats I0920 01:06:36.917474 233 fs_manager.cc:262] Metadata directory not provided I0920 01:06:36.917635 233 fs_manager.cc:268] Using write-ahead log directory (fs_wal_dir) as metadata directory I0920 01:06:36.918620 233 fs_manager.cc:608] Generated new instance metadata in path /tmp/dist-test-taskEr7w4V/test-tmp/raft_consensus_quorum-test.0.RaftConsensusQuorumTest.TestLeaderHeartbeats.1568941567107070-233/peer-0-root/instance: uuid: "e8b4a21c739947fe8de8334a36258aac" format_stamp: "Formatted at 2019-09-20 01:06:36 on dist-test-slave-dist-test-slave-9x72" I0920 01:06:36.926149 233 fs_manager.cc:509] Time spent creating directory manager: real 0.007s user 0.008s sys 0.000s I0920 01:06:36.927475 233 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0920 01:06:36.933084 233 fs_report.cc:352] FS layout report -------------------- wal directory: metadata directory: 1 data directories: /tmp/dist-test-taskEr7w4V/test-tmp/raft_consensus_quorum-test.0.RaftConsensusQuorumTest.TestLeaderHeartbeats.1568941567107070-233/peer-0-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0920 01:06:36.933368 233 fs_manager.cc:425] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s I0920 01:06:36.933507 233 fs_manager.cc:442] Opened local filesystem: /tmp/dist-test-taskEr7w4V/test-tmp/raft_consensus_quorum-test.0.RaftConsensusQuorumTest.TestLeaderHeartbeats.1568941567107070-233/peer-0-root uuid: "e8b4a21c739947fe8de8334a36258aac" format_stamp: "Formatted at 2019-09-20 01:06:36 on dist-test-slave-dist-test-slave-9x72" I0920 01:06:36.940397 233 fs_manager.cc:262] Metadata directory not provided I0920 01:06:36.940541 233 fs_manager.cc:268] Using write-ahead log directory (fs_wal_dir) as metadata directory I0920 01:06:36.941327 233 fs_manager.cc:608] Generated new instance metadata in path /tmp/dist-test-taskEr7w4V/test-tmp/raft_consensus_quorum-test.0.RaftConsensusQuorumTest.TestLeaderHeartbeats.1568941567107070-233/peer-1-root/instance: uuid: "39885cb134094e528bb8a1255dc42bf4" format_stamp: "Formatted at 2019-09-20 01:06:36 on dist-test-slave-dist-test-slave-9x72" I0920 01:06:36.948861 233 fs_manager.cc:509] Time spent creating directory manager: real 0.007s user 0.008s sys 0.000s I0920 01:06:36.949976 233 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0920 01:06:36.955808 233 fs_report.cc:352] FS layout report -------------------- wal directory: metadata directory: 1 data directories: /tmp/dist-test-taskEr7w4V/test-tmp/raft_consensus_quorum-test.0.RaftConsensusQuorumTest.TestLeaderHeartbeats.1568941567107070-233/peer-1-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0920 01:06:36.956105 233 fs_manager.cc:425] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s I0920 01:06:36.956252 233 fs_manager.cc:442] Opened local filesystem: /tmp/dist-test-taskEr7w4V/test-tmp/raft_consensus_quorum-test.0.RaftConsensusQuorumTest.TestLeaderHeartbeats.1568941567107070-233/peer-1-root uuid: "39885cb134094e528bb8a1255dc42bf4" format_stamp: "Formatted at 2019-09-20 01:06:36 on dist-test-slave-dist-test-slave-9x72" I0920 01:06:36.963515 233 fs_manager.cc:262] Metadata directory not provided I0920 01:06:36.963659 233 fs_manager.cc:268] Using write-ahead log directory (fs_wal_dir) as metadata directory I0920 01:06:36.964489 233 fs_manager.cc:608] Generated new instance metadata in path /tmp/dist-test-taskEr7w4V/test-tmp/raft_consensus_quorum-test.0.RaftConsensusQuorumTest.TestLeaderHeartbeats.1568941567107070-233/peer-2-root/instance: uuid: "ef7bcb0e2f834d5aa67e1b0ba42ef06e" format_stamp: "Formatted at 2019-09-20 01:06:36 on dist-test-slave-dist-test-slave-9x72" I0920 01:06:36.972512 233 fs_manager.cc:509] Time spent creating directory manager: real 0.008s user 0.006s sys 0.002s I0920 01:06:36.973712 233 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0920 01:06:36.979508 233 fs_report.cc:352] FS layout report -------------------- wal directory: metadata directory: 1 data directories: /tmp/dist-test-taskEr7w4V/test-tmp/raft_consensus_quorum-test.0.RaftConsensusQuorumTest.TestLeaderHeartbeats.1568941567107070-233/peer-2-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0920 01:06:36.979794 233 fs_manager.cc:425] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s I0920 01:06:36.979928 233 fs_manager.cc:442] Opened local filesystem: /tmp/dist-test-taskEr7w4V/test-tmp/raft_consensus_quorum-test.0.RaftConsensusQuorumTest.TestLeaderHeartbeats.1568941567107070-233/peer-2-root uuid: "ef7bcb0e2f834d5aa67e1b0ba42ef06e" format_stamp: "Formatted at 2019-09-20 01:06:36 on dist-test-slave-dist-test-slave-9x72" I0920 01:06:36.994822 233 raft_consensus.cc:340] T TestTablet P e8b4a21c739947fe8de8334a36258aac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 peers { permanent_uuid: "e8b4a21c739947fe8de8334a36258aac" member_type: VOTER last_known_addr { host: "peer-0.fake-domain-for-tests" port: 0 } } peers { permanent_uuid: "39885cb134094e528bb8a1255dc42bf4" member_type: VOTER last_known_addr { host: "peer-1.fake-domain-for-tests" port: 0 } } peers { permanent_uuid: "ef7bcb0e2f834d5aa67e1b0ba42ef06e" member_type: VOTER last_known_addr { host: "peer-2.fake-domain-for-tests" port: 0 } } I0920 01:06:36.995357 233 raft_consensus.cc:705] T TestTablet P e8b4a21c739947fe8de8334a36258aac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8b4a21c739947fe8de8334a36258aac, State: Initialized, Role: FOLLOWER I0920 01:06:36.995690 233 consensus_queue.cc:229] T TestTablet P e8b4a21c739947fe8de8334a36258aac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "e8b4a21c739947fe8de8334a36258aac" member_type: VOTER last_known_addr { host: "peer-0.fake-domain-for-tests" port: 0 } } peers { permanent_uuid: "39885cb134094e528bb8a1255dc42bf4" member_type: VOTER last_known_addr { host: "peer-1.fake-domain-for-tests" port: 0 } } peers { permanent_uuid: "ef7bcb0e2f834d5aa67e1b0ba42ef06e" member_type: VOTER last_known_addr { host: "peer-2.fake-domain-for-tests" port: 0 } } I0920 01:06:37.000723 233 raft_consensus.cc:340] T TestTablet P 39885cb134094e528bb8a1255dc42bf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 peers { permanent_uuid: "e8b4a21c739947fe8de8334a36258aac" member_type: VOTER last_known_addr { host: "peer-0.fake-domain-for-tests" port: 0 } } peers { permanent_uuid: "39885cb134094e528bb8a1255dc42bf4" member_type: VOTER last_known_addr { host: "peer-1.fake-domain-for-tests" port: 0 } } peers { permanent_uuid: "ef7bcb0e2f834d5aa67e1b0ba42ef06e" member_type: VOTER last_known_addr { host: "peer-2.fake-domain-for-tests" port: 0 } } I0920 01:06:37.001147 233 raft_consensus.cc:705] T TestTablet P 39885cb134094e528bb8a1255dc42bf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39885cb134094e528bb8a1255dc42bf4, State: Initialized, Role: FOLLOWER I0920 01:06:37.001447 233 consensus_queue.cc:229] T TestTablet P 39885cb134094e528bb8a1255dc42bf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "e8b4a21c739947fe8de8334a36258aac" member_type: VOTER last_known_addr { host: "peer-0.fake-domain-for-tests" port: 0 } } peers { permanent_uuid: "39885cb134094e528bb8a1255dc42bf4" member_type: VOTER last_known_addr { host: "peer-1.fake-domain-for-tests" port: 0 } } peers { permanent_uuid: "ef7bcb0e2f834d5aa67e1b0ba42ef06e" member_type: VOTER last_known_addr { host: "peer-2.fake-domain-for-tests" port: 0 } } I0920 01:06:37.005708 233 raft_consensus.cc:340] T TestTablet P ef7bcb0e2f834d5aa67e1b0ba42ef06e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 peers { permanent_uuid: "e8b4a21c739947fe8de8334a36258aac" member_type: VOTER last_known_addr { host: "peer-0.fake-domain-for-tests" port: 0 } } peers { permanent_uuid: "39885cb134094e528bb8a1255dc42bf4" member_type: VOTER last_known_addr { host: "peer-1.fake-domain-for-tests" port: 0 } } peers { permanent_uuid: "ef7bcb0e2f834d5aa67e1b0ba42ef06e" member_type: VOTER last_known_addr { host: "peer-2.fake-domain-for-tests" port: 0 } } I0920 01:06:37.006124 233 raft_consensus.cc:705] T TestTablet P ef7bcb0e2f834d5aa67e1b0ba42ef06e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef7bcb0e2f834d5aa67e1b0ba42ef06e, State: Initialized, Role: FOLLOWER I0920 01:06:37.006430 233 consensus_queue.cc:229] T TestTablet P ef7bcb0e2f834d5aa67e1b0ba42ef06e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "e8b4a21c739947fe8de8334a36258aac" member_type: VOTER last_known_addr { host: "peer-0.fake-domain-for-tests" port: 0 } } peers { permanent_uuid: "39885cb134094e528bb8a1255dc42bf4" member_type: VOTER last_known_addr { host: "peer-1.fake-domain-for-tests" port: 0 } } peers { permanent_uuid: "ef7bcb0e2f834d5aa67e1b0ba42ef06e" member_type: VOTER last_known_addr { host: "peer-2.fake-domain-for-tests" port: 0 } } I0920 01:06:37.007030 233 raft_consensus.cc:405] T TestTablet P ef7bcb0e2f834d5aa67e1b0ba42ef06e [term 0 FOLLOWER]: Emulating election... I0920 01:06:37.007177 233 raft_consensus.cc:2890] T TestTablet P ef7bcb0e2f834d5aa67e1b0ba42ef06e [term 0 FOLLOWER]: Advancing to term 1 I0920 01:06:37.397995 233 raft_consensus.cc:667] T TestTablet P ef7bcb0e2f834d5aa67e1b0ba42ef06e [term 1 LEADER]: Becoming Leader. State: Replica: ef7bcb0e2f834d5aa67e1b0ba42ef06e, State: Running, Role: LEADER I0920 01:06:37.398751 233 consensus_queue.cc:206] T TestTablet P ef7bcb0e2f834d5aa67e1b0ba42ef06e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "e8b4a21c739947fe8de8334a36258aac" member_type: VOTER last_known_addr { host: "peer-0.fake-domain-for-tests" port: 0 } } peers { permanent_uuid: "39885cb134094e528bb8a1255dc42bf4" member_type: VOTER last_known_addr { host: "peer-1.fake-domain-for-tests" port: 0 } } peers { permanent_uuid: "ef7bcb0e2f834d5aa67e1b0ba42ef06e" member_type: VOTER last_known_addr { host: "peer-2.fake-domain-for-tests" port: 0 } } I0920 01:06:37.802307 477 raft_consensus.cc:2890] T TestTablet P e8b4a21c739947fe8de8334a36258aac [term 0 FOLLOWER]: Advancing to term 1 I0920 01:06:37.930691 478 raft_consensus.cc:2890] T TestTablet P 39885cb134094e528bb8a1255dc42bf4 [term 0 FOLLOWER]: Advancing to term 1 W0920 01:06:38.338713 477 consensus_meta.cc:220] T TestTablet P e8b4a21c739947fe8de8334a36258aac: Time spent flushing consensus metadata: real 0.536s user 0.005s sys 0.000s I0920 01:06:38.340266 466 consensus_queue.cc:984] T TestTablet P ef7bcb0e2f834d5aa67e1b0ba42ef06e [LEADER]: Connected to new peer: Peer: permanent_uuid: "39885cb134094e528bb8a1255dc42bf4" member_type: VOTER last_known_addr { host: "peer-1.fake-domain-for-tests" port: 0 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s I0920 01:06:38.341128 466 consensus_queue.cc:984] T TestTablet P ef7bcb0e2f834d5aa67e1b0ba42ef06e [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8b4a21c739947fe8de8334a36258aac" member_type: VOTER last_known_addr { host: "peer-0.fake-domain-for-tests" port: 0 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0920 01:06:40.596407 233 raft_consensus.cc:2147] T TestTablet P e8b4a21c739947fe8de8334a36258aac [term 1 FOLLOWER]: Raft consensus shutting down. I0920 01:06:40.596735 233 raft_consensus.cc:2174] T TestTablet P e8b4a21c739947fe8de8334a36258aac [term 1 FOLLOWER]: Raft consensus is shut down! I0920 01:06:40.596899 233 raft_consensus.cc:2147] T TestTablet P ef7bcb0e2f834d5aa67e1b0ba42ef06e [term 1 LEADER]: Raft consensus shutting down. I0920 01:06:40.597347 233 raft_consensus.cc:2174] T TestTablet P ef7bcb0e2f834d5aa67e1b0ba42ef06e [term 1 FOLLOWER]: Raft consensus is shut down! I0920 01:06:40.597518 233 raft_consensus.cc:2147] T TestTablet P 39885cb134094e528bb8a1255dc42bf4 [term 1 FOLLOWER]: Raft consensus shutting down. I0920 01:06:40.597713 233 raft_consensus.cc:2174] T TestTablet P 39885cb134094e528bb8a1255dc42bf4 [term 1 FOLLOWER]: Raft consensus is shut down! [ OK ] RaftConsensusQuorumTest.TestLeaderHeartbeats (3709 ms) [ RUN ] RaftConsensusQuorumTest.TestLeaderElectionWithQuiescedQuorum I0920 01:06:40.626691 233 fs_manager.cc:262] Metadata directory not provided I0920 01:06:40.626853 233 fs_manager.cc:268] Using write-ahead log directory (fs_wal_dir) as metadata directory I0920 01:06:40.627764 233 fs_manager.cc:608] Generated new instance metadata in path /tmp/dist-test-taskEr7w4V/test-tmp/raft_consensus_quorum-test.0.RaftConsensusQuorumTest.TestLeaderElectionWithQuiescedQuorum.1568941567107070-233/peer-0-root/instance: uuid: "30aa61a2b4504f5aa42b3a14c0bb85b0" format_stamp: "Formatted at 2019-09-20 01:06:40 on dist-test-slave-dist-test-slave-9x72" I0920 01:06:40.635452 233 fs_manager.cc:509] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s I0920 01:06:40.636688 233 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0920 01:06:40.642295 233 fs_report.cc:352] FS layout report -------------------- wal directory: metadata directory: 1 data directories: /tmp/dist-test-taskEr7w4V/test-tmp/raft_consensus_quorum-test.0.RaftConsensusQuorumTest.TestLeaderElectionWithQuiescedQuorum.1568941567107070-233/peer-0-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0920 01:06:40.642612 233 fs_manager.cc:425] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s I0920 01:06:40.642717 233 fs_manager.cc:442] Opened local filesystem: /tmp/dist-test-taskEr7w4V/test-tmp/raft_consensus_quorum-test.0.RaftConsensusQuorumTest.TestLeaderElectionWithQuiescedQuorum.1568941567107070-233/peer-0-root uuid: "30aa61a2b4504f5aa42b3a14c0bb85b0" format_stamp: "Formatted at 2019-09-20 01:06:40 on dist-test-slave-dist-test-slave-9x72" I0920 01:06:40.649663 233 fs_manager.cc:262] Metadata directory not provided I0920 01:06:40.649802 233 fs_manager.cc:268] Using write-ahead log directory (fs_wal_dir) as metadata directory I0920 01:06:40.650555 233 fs_manager.cc:608] Generated new instance metadata in path /tmp/dist-test-taskEr7w4V/test-tmp/raft_consensus_quorum-test.0.RaftConsensusQuorumTest.TestLeaderElectionWithQuiescedQuorum.1568941567107070-233/peer-1-root/instance: uuid: "599aa566ed90440a82933cb19c3410d4" format_stamp: "Formatted at 2019-09-20 01:06:40 on dist-test-slave-dist-test-slave-9x72" I0920 01:06:40.657723 233 fs_manager.cc:509] Time spent creating directory manager: real 0.007s user 0.007s sys 0.000s I0920 01:06:40.658797 233 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0920 01:06:40.665382 233 fs_report.cc:352] FS layout report -------------------- wal directory: metadata directory: 1 data directories: /tmp/dist-test-taskEr7w4V/test-tmp/raft_consensus_quorum-test.0.RaftConsensusQuorumTest.TestLeaderElectionWithQuiescedQuorum.1568941567107070-233/peer-1-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0920 01:06:40.665699 233 fs_manager.cc:425] Time spent opening block manager: real 0.002s user 0.000s sys 0.000s I0920 01:06:40.665802 233 fs_manager.cc:442] Opened local filesystem: /tmp/dist-test-taskEr7w4V/test-tmp/raft_consensus_quorum-test.0.RaftConsensusQuorumTest.TestLeaderElectionWithQuiescedQuorum.1568941567107070-233/peer-1-root uuid: "599aa566ed90440a82933cb19c3410d4" format_stamp: "Formatted at 2019-09-20 01:06:40 on dist-test-slave-dist-test-slave-9x72" I0920 01:06:40.672644 233 fs_manager.cc:262] Metadata directory not provided I0920 01:06:40.672785 233 fs_manager.cc:268] Using write-ahead log directory (fs_wal_dir) as metadata directory I0920 01:06:40.673548 233 fs_manager.cc:608] Generated new instance metadata in path /tmp/dist-test-taskEr7w4V/test-tmp/raft_consensus_quorum-test.0.RaftConsensusQuorumTest.TestLeaderElectionWithQuiescedQuorum.1568941567107070-233/peer-2-root/instance: uuid: "f8f4f847142443b6b91e9e4776921d17" format_stamp: "Formatted at 2019-09-20 01:06:40 on dist-test-slave-dist-test-slave-9x72" I0920 01:06:40.680959 233 fs_manager.cc:509] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s I0920 01:06:40.682010 233 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0920 01:06:40.687906 233 fs_report.cc:352] FS layout report -------------------- wal directory: metadata directory: 1 data directories: /tmp/dist-test-taskEr7w4V/test-tmp/raft_consensus_quorum-test.0.RaftConsensusQuorumTest.TestLeaderElectionWithQuiescedQuorum.1568941567107070-233/peer-2-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0920 01:06:40.688194 233 fs_manager.cc:425] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s I0920 01:06:40.688294 233 fs_manager.cc:442] Opened local filesystem: /tmp/dist-test-taskEr7w4V/test-tmp/raft_consensus_quorum-test.0.RaftConsensusQuorumTest.TestLeaderElectionWithQuiescedQuorum.1568941567107070-233/peer-2-root uuid: "f8f4f847142443b6b91e9e4776921d17" format_stamp: "Formatted at 2019-09-20 01:06:40 on dist-test-slave-dist-test-slave-9x72" I0920 01:06:40.695521 233 fs_manager.cc:262] Metadata directory not provided I0920 01:06:40.695664 233 fs_manager.cc:268] Using write-ahead log directory (fs_wal_dir) as metadata directory I0920 01:06:40.696406 233 fs_manager.cc:608] Generated new instance metadata in path /tmp/dist-test-taskEr7w4V/test-tmp/raft_consensus_quorum-test.0.RaftConsensusQuorumTest.TestLeaderElectionWithQuiescedQuorum.1568941567107070-233/peer-3-root/instance: uuid: "fb781f0d4c9d4c8a9d9dda1015ac4d33" format_stamp: "Formatted at 2019-09-20 01:06:40 on dist-test-slave-dist-test-slave-9x72" I0920 01:06:40.703914 233 fs_manager.cc:509] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s I0920 01:06:40.704978 233 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0920 01:06:40.710999 233 fs_report.cc:352] FS layout report -------------------- wal directory: metadata directory: 1 data directories: /tmp/dist-test-taskEr7w4V/test-tmp/raft_consensus_quorum-test.0.RaftConsensusQuorumTest.TestLeaderElectionWithQuiescedQuorum.1568941567107070-233/peer-3-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0920 01:06:40.711534 233 fs_manager.cc:425] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s I0920 01:06:40.711691 233 fs_manager.cc:442] Opened local filesystem: /tmp/dist-test-taskEr7w4V/test-tmp/raft_consensus_quorum-test.0.RaftConsensusQuorumTest.TestLeaderElectionWithQuiescedQuorum.1568941567107070-233/peer-3-root uuid: "fb781f0d4c9d4c8a9d9dda1015ac4d33" format_stamp: "Formatted at 2019-09-20 01:06:40 on dist-test-slave-dist-test-slave-9x72" I0920 01:06:40.720947 233 fs_manager.cc:262] Metadata directory not provided I0920 01:06:40.721107 233 fs_manager.cc:268] Using write-ahead log directory (fs_wal_dir) as metadata directory I0920 01:06:40.721920 233 fs_manager.cc:608] Generated new instance metadata in path /tmp/dist-test-taskEr7w4V/test-tmp/raft_consensus_quorum-test.0.RaftConsensusQuorumTest.TestLeaderElectionWithQuiescedQuorum.1568941567107070-233/peer-4-root/instance: uuid: "7e5999ff17b84da2936f572df16919de" format_stamp: "Formatted at 2019-09-20 01:06:40 on dist-test-slave-dist-test-slave-9x72" I0920 01:06:40.729457 233 fs_manager.cc:509] Time spent creating directory manager: real 0.007s user 0.008s sys 0.000s I0920 01:06:40.730520 233 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0920 01:06:40.736521 233 fs_report.cc:352] FS layout report -------------------- wal directory: metadata directory: 1 data directories: /tmp/dist-test-taskEr7w4V/test-tmp/raft_consensus_quorum-test.0.RaftConsensusQuorumTest.TestLeaderElectionWithQuiescedQuorum.1568941567107070-233/peer-4-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0920 01:06:40.736853 233 fs_manager.cc:425] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s I0920 01:06:40.736950 233 fs_manager.cc:442] Opened local filesystem: /tmp/dist-test-taskEr7w4V/test-tmp/raft_consensus_quorum-test.0.RaftConsensusQuorumTest.TestLeaderElectionWithQuiescedQuorum.1568941567107070-233/peer-4-root uuid: "7e5999ff17b84da2936f572df16919de" format_stamp: "Formatted at 2019-09-20 01:06:40 on dist-test-slave-dist-test-slave-9x72" I0920 01:06:40.756966 233 raft_consensus.cc:340] T TestTablet P 30aa61a2b4504f5aa42b3a14c0bb85b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 peers { permanent_uuid: "30aa61a2b4504f5aa42b3a14c0bb85b0" member_type: VOTER last_known_addr { host: "peer-0.fake-domain-for-tests" port: 0 } } peers { permanent_uuid: "599aa566ed90440a82933cb19c3410d4" member_type: VOTER last_known_addr { host: "peer-1.fake-domain-for-tests" port: 0 } } peers { permanent_uuid: "f8f4f847142443b6b91e9e4776921d17" member_type: VOTER last_known_addr { host: "peer-2.fake-domain-for-tests" port: 0 } } peers { permanent_uuid: "fb781f0d4c9d4c8a9d9dda1015ac4d33" member_type: VOTER last_known_addr { host: "peer-3.fake-domain-for-tests" port: 0 } } peers { permanent_uuid: "7e5999ff17b84da2936f572df16919de" member_type: VOTER last_known_addr { host: "peer-4.fake-domain-for-tests" port: 0 } } I0920 01:06:40.757504 233 raft_consensus.cc:705] T TestTablet P 30aa61a2b4504f5aa42b3a14c0bb85b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30aa61a2b4504f5aa42b3a14c0bb85b0, State: Initialized, Role: FOLLOWER I0920 01:06:40.757808 233 consensus_queue.cc:229] T TestTablet P 30aa61a2b4504f5aa42b3a14c0bb85b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "30aa61a2b4504f5aa42b3a14c0bb85b0" member_type: VOTER last_known_addr { host: "peer-0.fake-domain-for-tests" port: 0 } } peers { permanent_uuid: "599aa566ed90440a82933cb19c3410d4" member_type: VOTER last_known_addr { host: "peer-1.fake-domain-for-tests" port: 0 } } peers { permanent_uuid: "f8f4f847142443b6b91e9e4776921d17" member_type: VOTER last_known_addr { host: "peer-2.fake-domain-for-tests" port: 0 } } peers { permanent_uuid: "fb781f0d4c9d4c8a9d9dda1015ac4d33" member_type: VOTER last_known_addr { host: "peer-3.fake-domain-for-tests" port: 0 } } peers { permanent_uuid: "7e5999ff17b84da2936f572df16919de" member_type: VOTER last_known_addr { host: "peer-4.fake-domain-for-tests" port: 0 } } I0920 01:06:40.763238 233 raft_consensus.cc:340] T TestTablet P 599aa566ed90440a82933cb19c3410d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 peers { permanent_uuid: "30aa61a2b4504f5aa42b3a14c0bb85b0" member_type: VOTER last_known_addr { host: "peer-0.fake-domain-for-tests" port: 0 } } peers { permanent_uuid: "599aa566ed90440a82933cb19c3410d4" member_type: VOTER last_known_addr { host: "peer-1.fake-domain-for-tests" port: 0 } } peers { permanent_uuid: "f8f4f847142443b6b91e9e4776921d17" member_type: VOTER last_known_addr { host: "peer-2.fake-domain-for-tests" port: 0 } } peers { permanent_uuid: "fb781f0d4c9d4c8a9d9dda1015ac4d33" member_type: VOTER last_known_addr { host: "peer-3.fake-domain-for-tests" port: 0 } } peers { permanent_uuid: "7e5999ff17b84da2936f572df16919de" member_type: VOTER last_known_addr { host: "peer-4.fake-domain-for-tests" port: 0 } } I0920 01:06:40.763816 233 raft_consensus.cc:705] T TestTablet P 599aa566ed90440a82933cb19c3410d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 599aa566ed90440a82933cb19c3410d4, State: Initialized, Role: FOLLOWER I0920 01:06:40.764082 233 consensus_queue.cc:229] T TestTablet P 599aa566ed90440a82933cb19c3410d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "30aa61a2b4504f5aa42b3a14c0bb85b0" member_type: VOTER last_known_addr { host: "peer-0.fake-domain-for-tests" port: 0 } } peers { permanent_uuid: "599aa566ed90440a82933cb19c3410d4" member_type: VOTER last_known_addr { host: "peer-1.fake-domain-for-tests" port: 0 } } peers { permanent_uuid: "f8f4f847142443b6b91e9e4776921d17" member_type: VOTER last_known_addr { host: "peer-2.fake-domain-for-tests" port: 0 } } peers { permanent_uuid: "fb781f0d4c9d4c8a9d9dda1015ac4d33" member_type: VOTER last_known_addr { host: "peer-3.fake-domain-for-tests" port: 0 } } peers { permanent_uuid: "7e5999ff17b84da2936f572df16919de" member_type: VOTER last_known_addr { host: "peer-4.fake-domain-for-tests" port: 0 } } I0920 01:06:40.768577 233 raft_consensus.cc:340] T TestTablet P f8f4f847142443b6b91e9e4776921d17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 peers { permanent_uuid: "30aa61a2b4504f5aa42b3a14c0bb85b0" member_type: VOTER last_known_addr { host: "peer-0.fake-domain-for-tests" port: 0 } } peers { permanent_uuid: "599aa566ed90440a82933cb19c3410d4" member_type: VOTER last_known_addr { host: "peer-1.fake-domain-for-tests" port: 0 } } peers { permanent_uuid: "f8f4f847142443b6b91e9e4776921d17" member_type: VOTER last_known_addr { host: "peer-2.fake-domain-for-tests" port: 0 } } peers { permanent_uuid: "fb781f0d4c9d4c8a9d9dda1015ac4d33" member_type: VOTER last_known_addr { host: "peer-3.fake-domain-for-tests" port: 0 } } peers { permanent_uuid: "7e5999ff17b84da2936f572df16919de" member_type: VOTER last_known_addr { host: "peer-4.fake-domain-for-tests" port: 0 } } I0920 01:06:40.769085 233 raft_consensus.cc:705] T TestTablet P f8f4f847142443b6b91e9e4776921d17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8f4f847142443b6b91e9e4776921d17, State: Initialized, Role: FOLLOWER I0920 01:06:40.769354 233 consensus_queue.cc:229] T TestTablet P f8f4f847142443b6b91e9e4776921d17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "30aa61a2b4504f5aa42b3a14c0bb85b0" member_type: VOTER last_known_addr { host: "peer-0.fake-domain-for-tests" port: 0 } } peers { permanent_uuid: "599aa566ed90440a82933cb19c3410d4" member_type: VOTER last_known_addr { host: "peer-1.fake-domain-for-tests" port: 0 } } peers { permanent_uuid: "f8f4f847142443b6b91e9e4776921d17" member_type: VOTER last_known_addr { host: "peer-2.fake-domain-for-tests" port: 0 } } peers { permanent_uuid: "fb781f0d4c9d4c8a9d9dda1015ac4d33" member_type: VOTER last_known_addr { host: "peer-3.fake-domain-for-tests" port: 0 } } peers { permanent_uuid: "7e5999ff17b84da2936f572df16919de" member_type: VOTER last_known_addr { host: "peer-4.fake-domain-for-tests" port: 0 } } I0920 01:06:40.774335 233 raft_consensus.cc:340] T TestTablet P fb781f0d4c9d4c8a9d9dda1015ac4d33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 peers { permanent_uuid: "30aa61a2b4504f5aa42b3a14c0bb85b0" member_type: VOTER last_known_addr { host: "peer-0.fake-domain-for-tests" port: 0 } } peers { permanent_uuid: "599aa566ed90440a82933cb19c3410d4" member_type: VOTER last_known_addr { host: "peer-1.fake-domain-for-tests" port: 0 } } peers { permanent_uuid: "f8f4f847142443b6b91e9e4776921d17" member_type: VOTER last_known_addr { host: "peer-2.fake-domain-for-tests" port: 0 } } peers { permanent_uuid: "fb781f0d4c9d4c8a9d9dda1015ac4d33" member_type: VOTER last_known_addr { host: "peer-3.fake-domain-for-tests" port: 0 } } peers { permanent_uuid: "7e5999ff17b84da2936f572df16919de" member_type: VOTER last_known_addr { host: "peer-4.fake-domain-for-tests" port: 0 } } I0920 01:06:40.774847 233 raft_consensus.cc:705] T TestTablet P fb781f0d4c9d4c8a9d9dda1015ac4d33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb781f0d4c9d4c8a9d9dda1015ac4d33, State: Initialized, Role: FOLLOWER I0920 01:06:40.775117 233 consensus_queue.cc:229] T TestTablet P fb781f0d4c9d4c8a9d9dda1015ac4d33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "30aa61a2b4504f5aa42b3a14c0bb85b0" member_type: VOTER last_known_addr { host: "peer-0.fake-domain-for-tests" port: 0 } } peers { permanent_uuid: "599aa566ed90440a82933cb19c3410d4" member_type: VOTER last_known_addr { host: "peer-1.fake-domain-for-tests" port: 0 } } peers { permanent_uuid: "f8f4f847142443b6b91e9e4776921d17" member_type: VOTER last_known_addr { host: "peer-2.fake-domain-for-tests" port: 0 } } peers { permanent_uuid: "fb781f0d4c9d4c8a9d9dda1015ac4d33" member_type: VOTER last_known_addr { host: "peer-3.fake-domain-for-tests" port: 0 } } peers { permanent_uuid: "7e5999ff17b84da2936f572df16919de" member_type: VOTER last_known_addr { host: "peer-4.fake-domain-for-tests" port: 0 } } I0920 01:06:40.781765 233 raft_consensus.cc:340] T TestTablet P 7e5999ff17b84da2936f572df16919de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 peers { permanent_uuid: "30aa61a2b4504f5aa42b3a14c0bb85b0" member_type: VOTER last_known_addr { host: "peer-0.fake-domain-for-tests" port: 0 } } peers { permanent_uuid: "599aa566ed90440a82933cb19c3410d4" member_type: VOTER last_known_addr { host: "peer-1.fake-domain-for-tests" port: 0 } } peers { permanent_uuid: "f8f4f847142443b6b91e9e4776921d17" member_type: VOTER last_known_addr { host: "peer-2.fake-domain-for-tests" port: 0 } } peers { permanent_uuid: "fb781f0d4c9d4c8a9d9dda1015ac4d33" member_type: VOTER last_known_addr { host: "peer-3.fake-domain-for-tests" port: 0 } } peers { permanent_uuid: "7e5999ff17b84da2936f572df16919de" member_type: VOTER last_known_addr { host: "peer-4.fake-domain-for-tests" port: 0 } } I0920 01:06:40.782276 233 raft_consensus.cc:705] T TestTablet P 7e5999ff17b84da2936f572df16919de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e5999ff17b84da2936f572df16919de, State: Initialized, Role: FOLLOWER I0920 01:06:40.782531 233 consensus_queue.cc:229] T TestTablet P 7e5999ff17b84da2936f572df16919de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "30aa61a2b4504f5aa42b3a14c0bb85b0" member_type: VOTER last_known_addr { host: "peer-0.fake-domain-for-tests" port: 0 } } peers { permanent_uuid: "599aa566ed90440a82933cb19c3410d4" member_type: VOTER last_known_addr { host: "peer-1.fake-domain-for-tests" port: 0 } } peers { permanent_uuid: "f8f4f847142443b6b91e9e4776921d17" member_type: VOTER last_known_addr { host: "peer-2.fake-domain-for-tests" port: 0 } } peers { permanent_uuid: "fb781f0d4c9d4c8a9d9dda1015ac4d33" member_type: VOTER last_known_addr { host: "peer-3.fake-domain-for-tests" port: 0 } } peers { permanent_uuid: "7e5999ff17b84da2936f572df16919de" member_type: VOTER last_known_addr { host: "peer-4.fake-domain-for-tests" port: 0 } } I0920 01:06:40.783160 233 raft_consensus.cc:405] T TestTablet P 7e5999ff17b84da2936f572df16919de [term 0 FOLLOWER]: Emulating election... I0920 01:06:40.783329 233 raft_consensus.cc:2890] T TestTablet P 7e5999ff17b84da2936f572df16919de [term 0 FOLLOWER]: Advancing to term 1 I0920 01:06:41.276705 233 raft_consensus.cc:667] T TestTablet P 7e5999ff17b84da2936f572df16919de [term 1 LEADER]: Becoming Leader. State: Replica: 7e5999ff17b84da2936f572df16919de, State: Running, Role: LEADER I0920 01:06:41.277272 233 consensus_queue.cc:206] T TestTablet P 7e5999ff17b84da2936f572df16919de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "30aa61a2b4504f5aa42b3a14c0bb85b0" member_type: VOTER last_known_addr { host: "peer-0.fake-domain-for-tests" port: 0 } } peers { permanent_uuid: "599aa566ed90440a82933cb19c3410d4" member_type: VOTER last_known_addr { host: "peer-1.fake-domain-for-tests" port: 0 } } peers { permanent_uuid: "f8f4f847142443b6b91e9e4776921d17" member_type: VOTER last_known_addr { host: "peer-2.fake-domain-for-tests" port: 0 } } peers { permanent_uuid: "fb781f0d4c9d4c8a9d9dda1015ac4d33" member_type: VOTER last_known_addr { host: "peer-3.fake-domain-for-tests" port: 0 } } peers { permanent_uuid: "7e5999ff17b84da2936f572df16919de" member_type: VOTER last_known_addr { host: "peer-4.fake-domain-for-tests" port: 0 } } I0920 01:06:41.282502 531 raft_consensus.cc:2890] T TestTablet P fb781f0d4c9d4c8a9d9dda1015ac4d33 [term 0 FOLLOWER]: Advancing to term 1 I0920 01:06:41.288347 534 raft_consensus.cc:2890] T TestTablet P f8f4f847142443b6b91e9e4776921d17 [term 0 FOLLOWER]: Advancing to term 1 I0920 01:06:41.291872 533 raft_consensus.cc:2890] T TestTablet P 599aa566ed90440a82933cb19c3410d4 [term 0 FOLLOWER]: Advancing to term 1 W0920 01:06:42.360664 533 consensus_meta.cc:220] T TestTablet P 599aa566ed90440a82933cb19c3410d4: Time spent flushing consensus metadata: real 1.069s user 0.003s sys 0.000s W0920 01:06:42.360669 534 consensus_meta.cc:220] T TestTablet P f8f4f847142443b6b91e9e4776921d17: Time spent flushing consensus metadata: real 1.072s user 0.004s sys 0.000s W0920 01:06:42.360671 531 consensus_meta.cc:220] T TestTablet P fb781f0d4c9d4c8a9d9dda1015ac4d33: Time spent flushing consensus metadata: real 1.078s user 0.005s sys 0.000s I0920 01:06:42.361326 531 raft_consensus.cc:1184] T TestTablet P fb781f0d4c9d4c8a9d9dda1015ac4d33 [term 1 FOLLOWER]: Refusing update from remote peer 7e5999ff17b84da2936f572df16919de: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0920 01:06:42.361326 534 raft_consensus.cc:1184] T TestTablet P f8f4f847142443b6b91e9e4776921d17 [term 1 FOLLOWER]: Refusing update from remote peer 7e5999ff17b84da2936f572df16919de: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0920 01:06:42.361428 533 raft_consensus.cc:1184] T TestTablet P 599aa566ed90440a82933cb19c3410d4 [term 1 FOLLOWER]: Refusing update from remote peer 7e5999ff17b84da2936f572df16919de: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0920 01:06:42.362087 531 raft_consensus.cc:2890] T TestTablet P 30aa61a2b4504f5aa42b3a14c0bb85b0 [term 0 FOLLOWER]: Advancing to term 1 I0920 01:06:42.368872 535 consensus_queue.cc:984] T TestTablet P 7e5999ff17b84da2936f572df16919de [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb781f0d4c9d4c8a9d9dda1015ac4d33" member_type: VOTER last_known_addr { host: "peer-3.fake-domain-for-tests" port: 0 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0920 01:06:42.369565 536 consensus_queue.cc:984] T TestTablet P 7e5999ff17b84da2936f572df16919de [LEADER]: Connected to new peer: Peer: permanent_uuid: "599aa566ed90440a82933cb19c3410d4" member_type: VOTER last_known_addr { host: "peer-1.fake-domain-for-tests" port: 0 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0920 01:06:42.370625 537 consensus_queue.cc:984] T TestTablet P 7e5999ff17b84da2936f572df16919de [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8f4f847142443b6b91e9e4776921d17" member_type: VOTER last_known_addr { host: "peer-2.fake-domain-for-tests" port: 0 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s W0920 01:06:43.302954 531 consensus_meta.cc:220] T TestTablet P 30aa61a2b4504f5aa42b3a14c0bb85b0: Time spent flushing consensus metadata: real 0.940s user 0.005s sys 0.000s I0920 01:06:43.303547 531 raft_consensus.cc:1184] T TestTablet P 30aa61a2b4504f5aa42b3a14c0bb85b0 [term 1 FOLLOWER]: Refusing update from remote peer 7e5999ff17b84da2936f572df16919de: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0920 01:06:43.304278 542 consensus_queue.cc:984] T TestTablet P 7e5999ff17b84da2936f572df16919de [LEADER]: Connected to new peer: Peer: permanent_uuid: "30aa61a2b4504f5aa42b3a14c0bb85b0" member_type: VOTER last_known_addr { host: "peer-0.fake-domain-for-tests" port: 0 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0920 01:06:43.309478 233 raft_consensus_quorum-test.cc:890] Shutting down current leader with index 4 I0920 01:06:43.309710 233 raft_consensus.cc:2147] T TestTablet P 7e5999ff17b84da2936f572df16919de [term 1 LEADER]: Raft consensus shutting down. I0920 01:06:43.310221 233 raft_consensus.cc:2174] T TestTablet P 7e5999ff17b84da2936f572df16919de [term 1 FOLLOWER]: Raft consensus is shut down! I0920 01:06:43.310405 233 raft_consensus_quorum-test.cc:904] Running election for future leader with index 4 I0920 01:06:43.310560 233 raft_consensus.cc:472] T TestTablet P fb781f0d4c9d4c8a9d9dda1015ac4d33 [term 1 FOLLOWER]: Starting forced leader election (received explicit request) I0920 01:06:43.310767 233 raft_consensus.cc:2890] T TestTablet P fb781f0d4c9d4c8a9d9dda1015ac4d33 [term 1 FOLLOWER]: Advancing to term 2 W0920 01:06:44.141324 233 consensus_meta.cc:220] T TestTablet P fb781f0d4c9d4c8a9d9dda1015ac4d33: Time spent flushing consensus metadata: real 0.830s user 0.005s sys 0.000s I0920 01:06:44.141625 233 raft_consensus.cc:494] T TestTablet P fb781f0d4c9d4c8a9d9dda1015ac4d33 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 peers { permanent_uuid: "30aa61a2b4504f5aa42b3a14c0bb85b0" member_type: VOTER last_known_addr { host: "peer-0.fake-domain-for-tests" port: 0 } } peers { permanent_uuid: "599aa566ed90440a82933cb19c3410d4" member_type: VOTER last_known_addr { host: "peer-1.fake-domain-for-tests" port: 0 } } peers { permanent_uuid: "f8f4f847142443b6b91e9e4776921d17" member_type: VOTER last_known_addr { host: "peer-2.fake-domain-for-tests" port: 0 } } peers { permanent_uuid: "fb781f0d4c9d4c8a9d9dda1015ac4d33" member_type: VOTER last_known_addr { host: "peer-3.fake-domain-for-tests" port: 0 } } peers { permanent_uuid: "7e5999ff17b84da2936f572df16919de" member_type: VOTER last_known_addr { host: "peer-4.fake-domain-for-tests" port: 0 } } I0920 01:06:44.143702 545 raft_consensus.cc:2890] T TestTablet P 30aa61a2b4504f5aa42b3a14c0bb85b0 [term 1 FOLLOWER]: Advancing to term 2 I0920 01:06:44.144616 546 raft_consensus.cc:2890] T TestTablet P 599aa566ed90440a82933cb19c3410d4 [term 1 FOLLOWER]: Advancing to term 2 I0920 01:06:44.145720 233 leader_election.cc:296] T TestTablet P fb781f0d4c9d4c8a9d9dda1015ac4d33 [CANDIDATE]: Term 2 election: Requested vote from peers 30aa61a2b4504f5aa42b3a14c0bb85b0 (TestPeerProxy), 599aa566ed90440a82933cb19c3410d4 (TestPeerProxy), f8f4f847142443b6b91e9e4776921d17 (TestPeerProxy), 7e5999ff17b84da2936f572df16919de (TestPeerProxy) I0920 01:06:44.145949 547 raft_consensus.cc:2890] T TestTablet P f8f4f847142443b6b91e9e4776921d17 [term 1 FOLLOWER]: Advancing to term 2 W0920 01:06:45.107913 547 consensus_meta.cc:220] T TestTablet P f8f4f847142443b6b91e9e4776921d17: Time spent flushing consensus metadata: real 0.961s user 0.004s sys 0.000s W0920 01:06:45.107916 546 consensus_meta.cc:220] T TestTablet P 599aa566ed90440a82933cb19c3410d4: Time spent flushing consensus metadata: real 0.963s user 0.004s sys 0.000s W0920 01:06:45.107914 545 consensus_meta.cc:220] T TestTablet P 30aa61a2b4504f5aa42b3a14c0bb85b0: Time spent flushing consensus metadata: real 0.964s user 0.003s sys 0.002s I0920 01:06:45.108335 547 raft_consensus.cc:2350] T TestTablet P f8f4f847142443b6b91e9e4776921d17 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb781f0d4c9d4c8a9d9dda1015ac4d33 in term 2. I0920 01:06:45.108356 546 raft_consensus.cc:2350] T TestTablet P 599aa566ed90440a82933cb19c3410d4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb781f0d4c9d4c8a9d9dda1015ac4d33 in term 2. I0920 01:06:45.108479 545 raft_consensus.cc:2350] T TestTablet P 30aa61a2b4504f5aa42b3a14c0bb85b0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb781f0d4c9d4c8a9d9dda1015ac4d33 in term 2. I0920 01:06:45.109232 545 leader_election.cc:310] T TestTablet P fb781f0d4c9d4c8a9d9dda1015ac4d33 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 599aa566ed90440a82933cb19c3410d4, f8f4f847142443b6b91e9e4776921d17, fb781f0d4c9d4c8a9d9dda1015ac4d33; no voters: W0920 01:06:45.109068 547 consensus-test-util.h:580] Could not RequestVote from replica with request: tablet_id: "TestTablet" candidate_uuid: "fb781f0d4c9d4c8a9d9dda1015ac4d33" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: true dest_uuid: "7e5999ff17b84da2936f572df16919de" Status: Not found: Other consensus instance was destroyed W0920 01:06:45.109953 546 leader_election.cc:348] T TestTablet P fb781f0d4c9d4c8a9d9dda1015ac4d33 [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer 7e5999ff17b84da2936f572df16919de (TestPeerProxy): Not found: Other consensus instance was destroyed I0920 01:06:45.110800 548 raft_consensus.cc:2644] T TestTablet P fb781f0d4c9d4c8a9d9dda1015ac4d33 [term 2 FOLLOWER]: Leader election won for term 2 I0920 01:06:45.111711 548 raft_consensus.cc:667] T TestTablet P fb781f0d4c9d4c8a9d9dda1015ac4d33 [term 2 LEADER]: Becoming Leader. State: Replica: fb781f0d4c9d4c8a9d9dda1015ac4d33, State: Running, Role: LEADER I0920 01:06:45.112339 548 consensus_queue.cc:206] T TestTablet P fb781f0d4c9d4c8a9d9dda1015ac4d33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 11, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "30aa61a2b4504f5aa42b3a14c0bb85b0" member_type: VOTER last_known_addr { host: "peer-0.fake-domain-for-tests" port: 0 } } peers { permanent_uuid: "599aa566ed90440a82933cb19c3410d4" member_type: VOTER last_known_addr { host: "peer-1.fake-domain-for-tests" port: 0 } } peers { permanent_uuid: "f8f4f847142443b6b91e9e4776921d17" member_type: VOTER last_known_addr { host: "peer-2.fake-domain-for-tests" port: 0 } } peers { permanent_uuid: "fb781f0d4c9d4c8a9d9dda1015ac4d33" member_type: VOTER last_known_addr { host: "peer-3.fake-domain-for-tests" port: 0 } } peers { permanent_uuid: "7e5999ff17b84da2936f572df16919de" member_type: VOTER last_known_addr { host: "peer-4.fake-domain-for-tests" port: 0 } } I0920 01:06:45.115455 233 raft_consensus_quorum-test.cc:908] Election won W0920 01:06:45.116506 545 consensus-test-util.h:549] Could not Update replica with request: tablet_id: "TestTablet" caller_uuid: "fb781f0d4c9d4c8a9d9dda1015ac4d33" caller_term: 2 preceding_id { term: 2 index: 13 } dest_uuid: "7e5999ff17b84da2936f572df16919de" committed_index: 11 all_replicated_index: 0 safe_timestamp: 50 last_idx_appended_to_leader: 13 Status: Not found: Other consensus instance was destroyed I0920 01:06:45.116904 546 raft_consensus.cc:1184] T TestTablet P f8f4f847142443b6b91e9e4776921d17 [term 2 FOLLOWER]: Refusing update from remote peer fb781f0d4c9d4c8a9d9dda1015ac4d33: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 2 index: 13. (index mismatch) W0920 01:06:45.117663 547 consensus_peers.cc:458] T TestTablet P fb781f0d4c9d4c8a9d9dda1015ac4d33 -> Peer 7e5999ff17b84da2936f572df16919de (peer-4.fake-domain-for-tests:0): Couldn't send request to peer 7e5999ff17b84da2936f572df16919de. Error code: UNKNOWN_ERROR (1). Status: Not found: Other consensus instance was destroyed. This is attempt 1: this message will repeat every 5th retry. I0920 01:06:45.118158 547 raft_consensus.cc:1184] T TestTablet P 30aa61a2b4504f5aa42b3a14c0bb85b0 [term 2 FOLLOWER]: Refusing update from remote peer fb781f0d4c9d4c8a9d9dda1015ac4d33: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 2 index: 13. (index mismatch) I0920 01:06:45.118191 546 raft_consensus.cc:1184] T TestTablet P 599aa566ed90440a82933cb19c3410d4 [term 2 FOLLOWER]: Refusing update from remote peer fb781f0d4c9d4c8a9d9dda1015ac4d33: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 2 index: 13. (index mismatch) I0920 01:06:45.118628 548 consensus_queue.cc:984] T TestTablet P fb781f0d4c9d4c8a9d9dda1015ac4d33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8f4f847142443b6b91e9e4776921d17" member_type: VOTER last_known_addr { host: "peer-2.fake-domain-for-tests" port: 0 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s I0920 01:06:45.119112 549 consensus_queue.cc:984] T TestTablet P fb781f0d4c9d4c8a9d9dda1015ac4d33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30aa61a2b4504f5aa42b3a14c0bb85b0" member_type: VOTER last_known_addr { host: "peer-0.fake-domain-for-tests" port: 0 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s I0920 01:06:45.119614 551 consensus_queue.cc:984] T TestTablet P fb781f0d4c9d4c8a9d9dda1015ac4d33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "599aa566ed90440a82933cb19c3410d4" member_type: VOTER last_known_addr { host: "peer-1.fake-domain-for-tests" port: 0 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s I0920 01:06:45.188470 233 raft_consensus_quorum-test.cc:890] Shutting down current leader with index 3 I0920 01:06:45.188654 233 raft_consensus.cc:2147] T TestTablet P fb781f0d4c9d4c8a9d9dda1015ac4d33 [term 2 LEADER]: Raft consensus shutting down. I0920 01:06:45.189098 233 raft_consensus.cc:2174] T TestTablet P fb781f0d4c9d4c8a9d9dda1015ac4d33 [term 2 FOLLOWER]: Raft consensus is shut down! I0920 01:06:45.189275 233 raft_consensus_quorum-test.cc:904] Running election for future leader with index 3 I0920 01:06:45.189380 233 raft_consensus.cc:472] T TestTablet P f8f4f847142443b6b91e9e4776921d17 [term 2 FOLLOWER]: Starting forced leader election (received explicit request) I0920 01:06:45.189541 233 raft_consensus.cc:2890] T TestTablet P f8f4f847142443b6b91e9e4776921d17 [term 2 FOLLOWER]: Advancing to term 3 W0920 01:06:46.324298 233 consensus_meta.cc:220] T TestTablet P f8f4f847142443b6b91e9e4776921d17: Time spent flushing consensus metadata: real 1.135s user 0.004s sys 0.000s I0920 01:06:46.324589 233 raft_consensus.cc:494] T TestTablet P f8f4f847142443b6b91e9e4776921d17 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 peers { permanent_uuid: "30aa61a2b4504f5aa42b3a14c0bb85b0" member_type: VOTER last_known_addr { host: "peer-0.fake-domain-for-tests" port: 0 } } peers { permanent_uuid: "599aa566ed90440a82933cb19c3410d4" member_type: VOTER last_known_addr { host: "peer-1.fake-domain-for-tests" port: 0 } } peers { permanent_uuid: "f8f4f847142443b6b91e9e4776921d17" member_type: VOTER last_known_addr { host: "peer-2.fake-domain-for-tests" port: 0 } } peers { permanent_uuid: "fb781f0d4c9d4c8a9d9dda1015ac4d33" member_type: VOTER last_known_addr { host: "peer-3.fake-domain-for-tests" port: 0 } } peers { permanent_uuid: "7e5999ff17b84da2936f572df16919de" member_type: VOTER last_known_addr { host: "peer-4.fake-domain-for-tests" port: 0 } } I0920 01:06:46.326229 556 raft_consensus.cc:2890] T TestTablet P 30aa61a2b4504f5aa42b3a14c0bb85b0 [term 2 FOLLOWER]: Advancing to term 3 I0920 01:06:46.327065 557 raft_consensus.cc:2890] T TestTablet P 599aa566ed90440a82933cb19c3410d4 [term 2 FOLLOWER]: Advancing to term 3 I0920 01:06:46.328076 233 leader_election.cc:296] T TestTablet P f8f4f847142443b6b91e9e4776921d17 [CANDIDATE]: Term 3 election: Requested vote from peers 30aa61a2b4504f5aa42b3a14c0bb85b0 (TestPeerProxy), 599aa566ed90440a82933cb19c3410d4 (TestPeerProxy), fb781f0d4c9d4c8a9d9dda1015ac4d33 (TestPeerProxy), 7e5999ff17b84da2936f572df16919de (TestPeerProxy) W0920 01:06:46.328230 558 consensus-test-util.h:580] Could not RequestVote from replica with request: tablet_id: "TestTablet" candidate_uuid: "f8f4f847142443b6b91e9e4776921d17" candidate_term: 3 candidate_status { last_received { term: 2 index: 32 } } ignore_live_leader: true dest_uuid: "fb781f0d4c9d4c8a9d9dda1015ac4d33" Status: Not found: Other consensus instance was destroyed W0920 01:06:46.328850 558 consensus-test-util.h:580] Could not RequestVote from replica with request: tablet_id: "TestTablet" candidate_uuid: "f8f4f847142443b6b91e9e4776921d17" candidate_term: 3 candidate_status { last_received { term: 2 index: 32 } } ignore_live_leader: true dest_uuid: "7e5999ff17b84da2936f572df16919de" Status: Not found: Other consensus instance was destroyed W0920 01:06:46.329377 558 leader_election.cc:348] T TestTablet P f8f4f847142443b6b91e9e4776921d17 [CANDIDATE]: Term 3 election: Tablet error from VoteRequest() call to peer fb781f0d4c9d4c8a9d9dda1015ac4d33 (TestPeerProxy): Not found: Other consensus instance was destroyed W0920 01:06:46.329669 558 leader_election.cc:348] T TestTablet P f8f4f847142443b6b91e9e4776921d17 [CANDIDATE]: Term 3 election: Tablet error from VoteRequest() call to peer 7e5999ff17b84da2936f572df16919de (TestPeerProxy): Not found: Other consensus instance was destroyed I0920 01:06:46.331146 556 raft_consensus.cc:2350] T TestTablet P 30aa61a2b4504f5aa42b3a14c0bb85b0 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8f4f847142443b6b91e9e4776921d17 in term 3. I0920 01:06:46.331352 557 raft_consensus.cc:2350] T TestTablet P 599aa566ed90440a82933cb19c3410d4 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8f4f847142443b6b91e9e4776921d17 in term 3. I0920 01:06:46.332036 558 leader_election.cc:310] T TestTablet P f8f4f847142443b6b91e9e4776921d17 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 5 responses out of 5 voters: 3 yes votes; 2 no votes. yes voters: 30aa61a2b4504f5aa42b3a14c0bb85b0, 599aa566ed90440a82933cb19c3410d4, f8f4f847142443b6b91e9e4776921d17; no voters: 7e5999ff17b84da2936f572df16919de, fb781f0d4c9d4c8a9d9dda1015ac4d33 I0920 01:06:46.333365 559 raft_consensus.cc:2644] T TestTablet P f8f4f847142443b6b91e9e4776921d17 [term 3 FOLLOWER]: Leader election won for term 3 I0920 01:06:46.334412 559 raft_consensus.cc:667] T TestTablet P f8f4f847142443b6b91e9e4776921d17 [term 3 LEADER]: Becoming Leader. State: Replica: f8f4f847142443b6b91e9e4776921d17, State: Running, Role: LEADER I0920 01:06:46.334897 559 consensus_queue.cc:206] T TestTablet P f8f4f847142443b6b91e9e4776921d17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 2.32, Last appended by leader: 32, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "30aa61a2b4504f5aa42b3a14c0bb85b0" member_type: VOTER last_known_addr { host: "peer-0.fake-domain-for-tests" port: 0 } } peers { permanent_uuid: "599aa566ed90440a82933cb19c3410d4" member_type: VOTER last_known_addr { host: "peer-1.fake-domain-for-tests" port: 0 } } peers { permanent_uuid: "f8f4f847142443b6b91e9e4776921d17" member_type: VOTER last_known_addr { host: "peer-2.fake-domain-for-tests" port: 0 } } peers { permanent_uuid: "fb781f0d4c9d4c8a9d9dda1015ac4d33" member_type: VOTER last_known_addr { host: "peer-3.fake-domain-for-tests" port: 0 } } peers { permanent_uuid: "7e5999ff17b84da2936f572df16919de" member_type: VOTER last_known_addr { host: "peer-4.fake-domain-for-tests" port: 0 } } I0920 01:06:46.338681 233 raft_consensus_quorum-test.cc:908] Election won W0920 01:06:46.339722 558 consensus-test-util.h:549] Could not Update replica with request: tablet_id: "TestTablet" caller_uuid: "f8f4f847142443b6b91e9e4776921d17" caller_term: 3 preceding_id { term: 3 index: 34 } dest_uuid: "7e5999ff17b84da2936f572df16919de" committed_index: 32 all_replicated_index: 0 safe_timestamp: 132 last_idx_appended_to_leader: 34 Status: Not found: Other consensus instance was destroyed W0920 01:06:46.339874 557 consensus-test-util.h:549] Could not Update replica with request: tablet_id: "TestTablet" caller_uuid: "f8f4f847142443b6b91e9e4776921d17" caller_term: 3 preceding_id { term: 3 index: 34 } dest_uuid: "fb781f0d4c9d4c8a9d9dda1015ac4d33" committed_index: 32 all_replicated_index: 0 safe_timestamp: 133 last_idx_appended_to_leader: 34 Status: Not found: Other consensus instance was destroyed W0920 01:06:46.340796 557 consensus_peers.cc:458] T TestTablet P f8f4f847142443b6b91e9e4776921d17 -> Peer fb781f0d4c9d4c8a9d9dda1015ac4d33 (peer-3.fake-domain-for-tests:0): Couldn't send request to peer fb781f0d4c9d4c8a9d9dda1015ac4d33. Error code: UNKNOWN_ERROR (1). Status: Not found: Other consensus instance was destroyed. This is attempt 1: this message will repeat every 5th retry. I0920 01:06:46.341181 558 raft_consensus.cc:1184] T TestTablet P 30aa61a2b4504f5aa42b3a14c0bb85b0 [term 3 FOLLOWER]: Refusing update from remote peer f8f4f847142443b6b91e9e4776921d17: Log matching property violated. Preceding OpId in replica: term: 2 index: 32. Preceding OpId from leader: term: 3 index: 34. (index mismatch) I0920 01:06:46.341231 557 raft_consensus.cc:1184] T TestTablet P 599aa566ed90440a82933cb19c3410d4 [term 3 FOLLOWER]: Refusing update from remote peer f8f4f847142443b6b91e9e4776921d17: Log matching property violated. Preceding OpId in replica: term: 2 index: 32. Preceding OpId from leader: term: 3 index: 34. (index mismatch) I0920 01:06:46.341667 560 consensus_queue.cc:984] T TestTablet P f8f4f847142443b6b91e9e4776921d17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "599aa566ed90440a82933cb19c3410d4" member_type: VOTER last_known_addr { host: "peer-1.fake-domain-for-tests" port: 0 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s I0920 01:06:46.342147 559 consensus_queue.cc:984] T TestTablet P f8f4f847142443b6b91e9e4776921d17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30aa61a2b4504f5aa42b3a14c0bb85b0" member_type: VOTER last_known_addr { host: "peer-0.fake-domain-for-tests" port: 0 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s W0920 01:06:46.342602 556 consensus_peers.cc:458] T TestTablet P f8f4f847142443b6b91e9e4776921d17 -> Peer 7e5999ff17b84da2936f572df16919de (peer-4.fake-domain-for-tests:0): Couldn't send request to peer 7e5999ff17b84da2936f572df16919de. Error code: UNKNOWN_ERROR (1). Status: Not found: Other consensus instance was destroyed. This is attempt 1: this message will repeat every 5th retry. I0920 01:06:46.377786 233 raft_consensus.cc:2147] T TestTablet P 30aa61a2b4504f5aa42b3a14c0bb85b0 [term 3 FOLLOWER]: Raft consensus shutting down. I0920 01:06:46.378032 233 raft_consensus.cc:2174] T TestTablet P 30aa61a2b4504f5aa42b3a14c0bb85b0 [term 3 FOLLOWER]: Raft consensus is shut down! I0920 01:06:46.378182 233 raft_consensus.cc:2147] T TestTablet P 599aa566ed90440a82933cb19c3410d4 [term 3 FOLLOWER]: Raft consensus shutting down. I0920 01:06:46.378347 233 raft_consensus.cc:2174] T TestTablet P 599aa566ed90440a82933cb19c3410d4 [term 3 FOLLOWER]: Raft consensus is shut down! I0920 01:06:46.378487 233 raft_consensus.cc:2147] T TestTablet P f8f4f847142443b6b91e9e4776921d17 [term 3 LEADER]: Raft consensus shutting down. I0920 01:06:46.378865 233 raft_consensus.cc:2174] T TestTablet P f8f4f847142443b6b91e9e4776921d17 [term 3 FOLLOWER]: Raft consensus is shut down! [ OK ] RaftConsensusQuorumTest.TestLeaderElectionWithQuiescedQuorum (5805 ms) [ RUN ] RaftConsensusQuorumTest.TestReplicasEnforceTheLogMatchingProperty I0920 01:06:46.432271 233 fs_manager.cc:262] Metadata directory not provided I0920 01:06:46.432495 233 fs_manager.cc:268] Using write-ahead log directory (fs_wal_dir) as metadata directory I0920 01:06:46.433344 233 fs_manager.cc:608] Generated new instance metadata in path /tmp/dist-test-taskEr7w4V/test-tmp/raft_consensus_quorum-test.0.RaftConsensusQuorumTest.TestReplicasEnforceTheLogMatchingProperty.1568941567107070-233/peer-0-root/instance: uuid: "3803c4211ba8401195e5a0c33aefbf99" format_stamp: "Formatted at 2019-09-20 01:06:46 on dist-test-slave-dist-test-slave-9x72" I0920 01:06:46.440779 233 fs_manager.cc:509] Time spent creating directory manager: real 0.007s user 0.008s sys 0.000s I0920 01:06:46.441876 233 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0920 01:06:46.448910 233 fs_report.cc:352] FS layout report -------------------- wal directory: metadata directory: 1 data directories: /tmp/dist-test-taskEr7w4V/test-tmp/raft_consensus_quorum-test.0.RaftConsensusQuorumTest.TestReplicasEnforceTheLogMatchingProperty.1568941567107070-233/peer-0-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0920 01:06:46.449231 233 fs_manager.cc:425] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s I0920 01:06:46.449370 233 fs_manager.cc:442] Opened local filesystem: /tmp/dist-test-taskEr7w4V/test-tmp/raft_consensus_quorum-test.0.RaftConsensusQuorumTest.TestReplicasEnforceTheLogMatchingProperty.1568941567107070-233/peer-0-root uuid: "3803c4211ba8401195e5a0c33aefbf99" format_stamp: "Formatted at 2019-09-20 01:06:46 on dist-test-slave-dist-test-slave-9x72" I0920 01:06:46.456887 233 fs_manager.cc:262] Metadata directory not provided I0920 01:06:46.457060 233 fs_manager.cc:268] Using write-ahead log directory (fs_wal_dir) as metadata directory I0920 01:06:46.457938 233 fs_manager.cc:608] Generated new instance metadata in path /tmp/dist-test-taskEr7w4V/test-tmp/raft_consensus_quorum-test.0.RaftConsensusQuorumTest.TestReplicasEnforceTheLogMatchingProperty.1568941567107070-233/peer-1-root/instance: uuid: "45f91d7a874b41e4bb3ffec9847435f1" format_stamp: "Formatted at 2019-09-20 01:06:46 on dist-test-slave-dist-test-slave-9x72" I0920 01:06:46.466230 233 fs_manager.cc:509] Time spent creating directory manager: real 0.008s user 0.009s sys 0.000s I0920 01:06:46.467473 233 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0920 01:06:46.474175 233 fs_report.cc:352] FS layout report -------------------- wal directory: metadata directory: 1 data directories: /tmp/dist-test-taskEr7w4V/test-tmp/raft_consensus_quorum-test.0.RaftConsensusQuorumTest.TestReplicasEnforceTheLogMatchingProperty.1568941567107070-233/peer-1-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0920 01:06:46.474551 233 fs_manager.cc:425] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s I0920 01:06:46.474681 233 fs_manager.cc:442] Opened local filesystem: /tmp/dist-test-taskEr7w4V/test-tmp/raft_consensus_quorum-test.0.RaftConsensusQuorumTest.TestReplicasEnforceTheLogMatchingProperty.1568941567107070-233/peer-1-root uuid: "45f91d7a874b41e4bb3ffec9847435f1" format_stamp: "Formatted at 2019-09-20 01:06:46 on dist-test-slave-dist-test-slave-9x72" I0920 01:06:46.482580 233 fs_manager.cc:262] Metadata directory not provided I0920 01:06:46.482750 233 fs_manager.cc:268] Using write-ahead log directory (fs_wal_dir) as metadata directory I0920 01:06:46.483644 233 fs_manager.cc:608] Generated new instance metadata in path /tmp/dist-test-taskEr7w4V/test-tmp/raft_consensus_quorum-test.0.RaftConsensusQuorumTest.TestReplicasEnforceTheLogMatchingProperty.1568941567107070-233/peer-2-root/instance: uuid: "8e0a460f53bb477d9a43f8bd1a730cdb" format_stamp: "Formatted at 2019-09-20 01:06:46 on dist-test-slave-dist-test-slave-9x72" I0920 01:06:46.492123 233 fs_manager.cc:509] Time spent creating directory manager: real 0.008s user 0.003s sys 0.005s I0920 01:06:46.493366 233 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0920 01:06:46.500146 233 fs_report.cc:352] FS layout report -------------------- wal directory: metadata directory: 1 data directories: /tmp/dist-test-taskEr7w4V/test-tmp/raft_consensus_quorum-test.0.RaftConsensusQuorumTest.TestReplicasEnforceTheLogMatchingProperty.1568941567107070-233/peer-2-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0920 01:06:46.500538 233 fs_manager.cc:425] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s I0920 01:06:46.500669 233 fs_manager.cc:442] Opened local filesystem: /tmp/dist-test-taskEr7w4V/test-tmp/raft_consensus_quorum-test.0.RaftConsensusQuorumTest.TestReplicasEnforceTheLogMatchingProperty.1568941567107070-233/peer-2-root uuid: "8e0a460f53bb477d9a43f8bd1a730cdb" format_stamp: "Formatted at 2019-09-20 01:06:46 on dist-test-slave-dist-test-slave-9x72" I0920 01:06:46.516629 233 raft_consensus.cc:340] T TestTablet P 3803c4211ba8401195e5a0c33aefbf99 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 peers { permanent_uuid: "3803c4211ba8401195e5a0c33aefbf99" member_type: VOTER last_known_addr { host: "peer-0.fake-domain-for-tests" port: 0 } } peers { permanent_uuid: "45f91d7a874b41e4bb3ffec9847435f1" member_type: VOTER last_known_addr { host: "peer-1.fake-domain-for-tests" port: 0 } } peers { permanent_uuid: "8e0a460f53bb477d9a43f8bd1a730cdb" member_type: VOTER last_known_addr { host: "peer-2.fake-domain-for-tests" port: 0 } } I0920 01:06:46.517081 233 raft_consensus.cc:705] T TestTablet P 3803c4211ba8401195e5a0c33aefbf99 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3803c4211ba8401195e5a0c33aefbf99, State: Initialized, Role: FOLLOWER I0920 01:06:46.517402 233 consensus_queue.cc:229] T TestTablet P 3803c4211ba8401195e5a0c33aefbf99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "3803c4211ba8401195e5a0c33aefbf99" member_type: VOTER last_known_addr { host: "peer-0.fake-domain-for-tests" port: 0 } } peers { permanent_uuid: "45f91d7a874b41e4bb3ffec9847435f1" member_type: VOTER last_known_addr { host: "peer-1.fake-domain-for-tests" port: 0 } } peers { permanent_uuid: "8e0a460f53bb477d9a43f8bd1a730cdb" member_type: VOTER last_known_addr { host: "peer-2.fake-domain-for-tests" port: 0 } } I0920 01:06:46.522866 233 raft_consensus.cc:340] T TestTablet P 45f91d7a874b41e4bb3ffec9847435f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 peers { permanent_uuid: "3803c4211ba8401195e5a0c33aefbf99" member_type: VOTER last_known_addr { host: "peer-0.fake-domain-for-tests" port: 0 } } peers { permanent_uuid: "45f91d7a874b41e4bb3ffec9847435f1" member_type: VOTER last_known_addr { host: "peer-1.fake-domain-for-tests" port: 0 } } peers { permanent_uuid: "8e0a460f53bb477d9a43f8bd1a730cdb" member_type: VOTER last_known_addr { host: "peer-2.fake-domain-for-tests" port: 0 } } I0920 01:06:46.523365 233 raft_consensus.cc:705] T TestTablet P 45f91d7a874b41e4bb3ffec9847435f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45f91d7a874b41e4bb3ffec9847435f1, State: Initialized, Role: FOLLOWER I0920 01:06:46.523658 233 consensus_queue.cc:229] T TestTablet P 45f91d7a874b41e4bb3ffec9847435f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "3803c4211ba8401195e5a0c33aefbf99" member_type: VOTER last_known_addr { host: "peer-0.fake-domain-for-tests" port: 0 } } peers { permanent_uuid: "45f91d7a874b41e4bb3ffec9847435f1" member_type: VOTER last_known_addr { host: "peer-1.fake-domain-for-tests" port: 0 } } peers { permanent_uuid: "8e0a460f53bb477d9a43f8bd1a730cdb" member_type: VOTER last_known_addr { host: "peer-2.fake-domain-for-tests" port: 0 } } I0920 01:06:46.528822 233 raft_consensus.cc:340] T TestTablet P 8e0a460f53bb477d9a43f8bd1a730cdb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 peers { permanent_uuid: "3803c4211ba8401195e5a0c33aefbf99" member_type: VOTER last_known_addr { host: "peer-0.fake-domain-for-tests" port: 0 } } peers { permanent_uuid: "45f91d7a874b41e4bb3ffec9847435f1" member_type: VOTER last_known_addr { host: "peer-1.fake-domain-for-tests" port: 0 } } peers { permanent_uuid: "8e0a460f53bb477d9a43f8bd1a730cdb" member_type: VOTER last_known_addr { host: "peer-2.fake-domain-for-tests" port: 0 } } I0920 01:06:46.529234 233 raft_consensus.cc:705] T TestTablet P 8e0a460f53bb477d9a43f8bd1a730cdb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e0a460f53bb477d9a43f8bd1a730cdb, State: Initialized, Role: FOLLOWER I0920 01:06:46.529526 233 consensus_queue.cc:229] T TestTablet P 8e0a460f53bb477d9a43f8bd1a730cdb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "3803c4211ba8401195e5a0c33aefbf99" member_type: VOTER last_known_addr { host: "peer-0.fake-domain-for-tests" port: 0 } } peers { permanent_uuid: "45f91d7a874b41e4bb3ffec9847435f1" member_type: VOTER last_known_addr { host: "peer-1.fake-domain-for-tests" port: 0 } } peers { permanent_uuid: "8e0a460f53bb477d9a43f8bd1a730cdb" member_type: VOTER last_known_addr { host: "peer-2.fake-domain-for-tests" port: 0 } } I0920 01:06:46.530087 233 raft_consensus.cc:405] T TestTablet P 8e0a460f53bb477d9a43f8bd1a730cdb [term 0 FOLLOWER]: Emulating election... I0920 01:06:46.530252 233 raft_consensus.cc:2890] T TestTablet P 8e0a460f53bb477d9a43f8bd1a730cdb [term 0 FOLLOWER]: Advancing to term 1 W0920 01:06:47.272233 233 consensus_meta.cc:220] T TestTablet P 8e0a460f53bb477d9a43f8bd1a730cdb: Time spent flushing consensus metadata: real 0.742s user 0.003s sys 0.002s I0920 01:06:47.273285 233 raft_consensus.cc:667] T TestTablet P 8e0a460f53bb477d9a43f8bd1a730cdb [term 1 LEADER]: Becoming Leader. State: Replica: 8e0a460f53bb477d9a43f8bd1a730cdb, State: Running, Role: LEADER I0920 01:06:47.273739 233 consensus_queue.cc:206] T TestTablet P 8e0a460f53bb477d9a43f8bd1a730cdb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "3803c4211ba8401195e5a0c33aefbf99" member_type: VOTER last_known_addr { host: "peer-0.fake-domain-for-tests" port: 0 } } peers { permanent_uuid: "45f91d7a874b41e4bb3ffec9847435f1" member_type: VOTER last_known_addr { host: "peer-1.fake-domain-for-tests" port: 0 } } peers { permanent_uuid: "8e0a460f53bb477d9a43f8bd1a730cdb" member_type: VOTER last_known_addr { host: "peer-2.fake-domain-for-tests" port: 0 } } I0920 01:06:47.278367 591 raft_consensus.cc:2890] T TestTablet P 45f91d7a874b41e4bb3ffec9847435f1 [term 0 FOLLOWER]: Advancing to term 1 I0920 01:06:47.279902 592 raft_consensus.cc:2890] T TestTablet P 3803c4211ba8401195e5a0c33aefbf99 [term 0 FOLLOWER]: Advancing to term 1 W0920 01:06:48.269665 592 consensus_meta.cc:220] T TestTablet P 3803c4211ba8401195e5a0c33aefbf99: Time spent flushing consensus metadata: real 0.989s user 0.004s sys 0.000s W0920 01:06:48.269485 591 consensus_meta.cc:220] T TestTablet P 45f91d7a874b41e4bb3ffec9847435f1: Time spent flushing consensus metadata: real 0.991s user 0.004s sys 0.000s I0920 01:06:48.271267 591 raft_consensus.cc:1184] T TestTablet P 45f91d7a874b41e4bb3ffec9847435f1 [term 1 FOLLOWER]: Refusing update from remote peer 8e0a460f53bb477d9a43f8bd1a730cdb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0920 01:06:48.271266 592 raft_consensus.cc:1184] T TestTablet P 3803c4211ba8401195e5a0c33aefbf99 [term 1 FOLLOWER]: Refusing update from remote peer 8e0a460f53bb477d9a43f8bd1a730cdb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0920 01:06:48.273797 595 consensus_queue.cc:984] T TestTablet P 8e0a460f53bb477d9a43f8bd1a730cdb [LEADER]: Connected to new peer: Peer: permanent_uuid: "45f91d7a874b41e4bb3ffec9847435f1" member_type: VOTER last_known_addr { host: "peer-1.fake-domain-for-tests" port: 0 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s I0920 01:06:48.274796 596 consensus_queue.cc:984] T TestTablet P 8e0a460f53bb477d9a43f8bd1a730cdb [LEADER]: Connected to new peer: Peer: permanent_uuid: "3803c4211ba8401195e5a0c33aefbf99" member_type: VOTER last_known_addr { host: "peer-0.fake-domain-for-tests" port: 0 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0920 01:06:48.311077 233 raft_consensus.cc:1184] T TestTablet P 3803c4211ba8401195e5a0c33aefbf99 [term 1 FOLLOWER]: Refusing update from remote peer 8e0a460f53bb477d9a43f8bd1a730cdb: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch) I0920 01:06:48.311756 233 raft_consensus.cc:2147] T TestTablet P 3803c4211ba8401195e5a0c33aefbf99 [term 1 FOLLOWER]: Raft consensus shutting down. I0920 01:06:48.311946 233 pending_rounds.cc:64] T TestTablet P 3803c4211ba8401195e5a0c33aefbf99: Trying to abort 1 pending transactions. I0920 01:06:48.312045 233 pending_rounds.cc:70] T TestTablet P 3803c4211ba8401195e5a0c33aefbf99: Aborting transaction as it isn't in flight: id { term: 1 index: 14 } timestamp: 39 op_type: NO_OP I0920 01:06:48.312235 233 raft_consensus.cc:2727] T TestTablet P 3803c4211ba8401195e5a0c33aefbf99 [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Transaction aborted I0920 01:06:48.312400 233 raft_consensus.cc:2174] T TestTablet P 3803c4211ba8401195e5a0c33aefbf99 [term 1 FOLLOWER]: Raft consensus is shut down! I0920 01:06:48.313963 233 raft_consensus.cc:2147] T TestTablet P 45f91d7a874b41e4bb3ffec9847435f1 [term 1 FOLLOWER]: Raft consensus shutting down. I0920 01:06:48.314268 233 raft_consensus.cc:2174] T TestTablet P 45f91d7a874b41e4bb3ffec9847435f1 [term 1 FOLLOWER]: Raft consensus is shut down! I0920 01:06:48.315865 233 raft_consensus.cc:2147] T TestTablet P 8e0a460f53bb477d9a43f8bd1a730cdb [term 1 LEADER]: Raft consensus shutting down. I0920 01:06:48.316278 233 raft_consensus.cc:2174] T TestTablet P 8e0a460f53bb477d9a43f8bd1a730cdb [term 1 FOLLOWER]: Raft consensus is shut down! [ OK ] RaftConsensusQuorumTest.TestReplicasEnforceTheLogMatchingProperty (2015 ms) [ RUN ] RaftConsensusQuorumTest.TestRequestVote I0920 01:06:48.446656 233 fs_manager.cc:262] Metadata directory not provided I0920 01:06:48.446821 233 fs_manager.cc:268] Using write-ahead log directory (fs_wal_dir) as metadata directory I0920 01:06:48.447814 233 fs_manager.cc:608] Generated new instance metadata in path /tmp/dist-test-taskEr7w4V/test-tmp/raft_consensus_quorum-test.0.RaftConsensusQuorumTest.TestRequestVote.1568941567107070-233/peer-0-root/instance: uuid: "938a40f5da7f4e62b2c317ecb1ba184c" format_stamp: "Formatted at 2019-09-20 01:06:48 on dist-test-slave-dist-test-slave-9x72" I0920 01:06:48.455468 233 fs_manager.cc:509] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s I0920 01:06:48.456614 233 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0920 01:06:48.462612 233 fs_report.cc:352] FS layout report -------------------- wal directory: metadata directory: 1 data directories: /tmp/dist-test-taskEr7w4V/test-tmp/raft_consensus_quorum-test.0.RaftConsensusQuorumTest.TestRequestVote.1568941567107070-233/peer-0-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0920 01:06:48.462925 233 fs_manager.cc:425] Time spent opening block manager: real 0.002s user 0.000s sys 0.000s I0920 01:06:48.463099 233 fs_manager.cc:442] Opened local filesystem: /tmp/dist-test-taskEr7w4V/test-tmp/raft_consensus_quorum-test.0.RaftConsensusQuorumTest.TestRequestVote.1568941567107070-233/peer-0-root uuid: "938a40f5da7f4e62b2c317ecb1ba184c" format_stamp: "Formatted at 2019-09-20 01:06:48 on dist-test-slave-dist-test-slave-9x72" I0920 01:06:48.470129 233 fs_manager.cc:262] Metadata directory not provided I0920 01:06:48.470274 233 fs_manager.cc:268] Using write-ahead log directory (fs_wal_dir) as metadata directory I0920 01:06:48.471119 233 fs_manager.cc:608] Generated new instance metadata in path /tmp/dist-test-taskEr7w4V/test-tmp/raft_consensus_quorum-test.0.RaftConsensusQuorumTest.TestRequestVote.1568941567107070-233/peer-1-root/instance: uuid: "fa4335770e644e66a6ce962fb388c6f6" format_stamp: "Formatted at 2019-09-20 01:06:48 on dist-test-slave-dist-test-slave-9x72" I0920 01:06:48.478775 233 fs_manager.cc:509] Time spent creating directory manager: real 0.007s user 0.005s sys 0.002s I0920 01:06:48.480000 233 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0920 01:06:48.485888 233 fs_report.cc:352] FS layout report -------------------- wal directory: metadata directory: 1 data directories: /tmp/dist-test-taskEr7w4V/test-tmp/raft_consensus_quorum-test.0.RaftConsensusQuorumTest.TestRequestVote.1568941567107070-233/peer-1-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0920 01:06:48.486183 233 fs_manager.cc:425] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s I0920 01:06:48.486316 233 fs_manager.cc:442] Opened local filesystem: /tmp/dist-test-taskEr7w4V/test-tmp/raft_consensus_quorum-test.0.RaftConsensusQuorumTest.TestRequestVote.1568941567107070-233/peer-1-root uuid: "fa4335770e644e66a6ce962fb388c6f6" format_stamp: "Formatted at 2019-09-20 01:06:48 on dist-test-slave-dist-test-slave-9x72" I0920 01:06:48.493698 233 fs_manager.cc:262] Metadata directory not provided I0920 01:06:48.493862 233 fs_manager.cc:268] Using write-ahead log directory (fs_wal_dir) as metadata directory I0920 01:06:48.494696 233 fs_manager.cc:608] Generated new instance metadata in path /tmp/dist-test-taskEr7w4V/test-tmp/raft_consensus_quorum-test.0.RaftConsensusQuorumTest.TestRequestVote.1568941567107070-233/peer-2-root/instance: uuid: "6fec94ba5fb343fbad900ca4ef38b251" format_stamp: "Formatted at 2019-09-20 01:06:48 on dist-test-slave-dist-test-slave-9x72" I0920 01:06:48.502457 233 fs_manager.cc:509] Time spent creating directory manager: real 0.007s user 0.008s sys 0.000s I0920 01:06:48.503664 233 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0920 01:06:48.509366 233 fs_report.cc:352] FS layout report -------------------- wal directory: metadata directory: 1 data directories: /tmp/dist-test-taskEr7w4V/test-tmp/raft_consensus_quorum-test.0.RaftConsensusQuorumTest.TestRequestVote.1568941567107070-233/peer-2-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0920 01:06:48.509649 233 fs_manager.cc:425] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s I0920 01:06:48.509788 233 fs_manager.cc:442] Opened local filesystem: /tmp/dist-test-taskEr7w4V/test-tmp/raft_consensus_quorum-test.0.RaftConsensusQuorumTest.TestRequestVote.1568941567107070-233/peer-2-root uuid: "6fec94ba5fb343fbad900ca4ef38b251" format_stamp: "Formatted at 2019-09-20 01:06:48 on dist-test-slave-dist-test-slave-9x72" I0920 01:06:48.525866 233 raft_consensus.cc:340] T TestTablet P 938a40f5da7f4e62b2c317ecb1ba184c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 peers { permanent_uuid: "938a40f5da7f4e62b2c317ecb1ba184c" member_type: VOTER last_known_addr { host: "peer-0.fake-domain-for-tests" port: 0 } } peers { permanent_uuid: "fa4335770e644e66a6ce962fb388c6f6" member_type: VOTER last_known_addr { host: "peer-1.fake-domain-for-tests" port: 0 } } peers { permanent_uuid: "6fec94ba5fb343fbad900ca4ef38b251" member_type: VOTER last_known_addr { host: "peer-2.fake-domain-for-tests" port: 0 } } I0920 01:06:48.526325 233 raft_consensus.cc:705] T TestTablet P 938a40f5da7f4e62b2c317ecb1ba184c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 938a40f5da7f4e62b2c317ecb1ba184c, State: Initialized, Role: FOLLOWER I0920 01:06:48.526655 233 consensus_queue.cc:229] T TestTablet P 938a40f5da7f4e62b2c317ecb1ba184c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "938a40f5da7f4e62b2c317ecb1ba184c" member_type: VOTER last_known_addr { host: "peer-0.fake-domain-for-tests" port: 0 } } peers { permanent_uuid: "fa4335770e644e66a6ce962fb388c6f6" member_type: VOTER last_known_addr { host: "peer-1.fake-domain-for-tests" port: 0 } } peers { permanent_uuid: "6fec94ba5fb343fbad900ca4ef38b251" member_type: VOTER last_known_addr { host: "peer-2.fake-domain-for-tests" port: 0 } } I0920 01:06:48.531771 233 raft_consensus.cc:340] T TestTablet P fa4335770e644e66a6ce962fb388c6f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 peers { permanent_uuid: "938a40f5da7f4e62b2c317ecb1ba184c" member_type: VOTER last_known_addr { host: "peer-0.fake-domain-for-tests" port: 0 } } peers { permanent_uuid: "fa4335770e644e66a6ce962fb388c6f6" member_type: VOTER last_known_addr { host: "peer-1.fake-domain-for-tests" port: 0 } } peers { permanent_uuid: "6fec94ba5fb343fbad900ca4ef38b251" member_type: VOTER last_known_addr { host: "peer-2.fake-domain-for-tests" port: 0 } } I0920 01:06:48.532192 233 raft_consensus.cc:705] T TestTablet P fa4335770e644e66a6ce962fb388c6f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa4335770e644e66a6ce962fb388c6f6, State: Initialized, Role: FOLLOWER I0920 01:06:48.532505 233 consensus_queue.cc:229] T TestTablet P fa4335770e644e66a6ce962fb388c6f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "938a40f5da7f4e62b2c317ecb1ba184c" member_type: VOTER last_known_addr { host: "peer-0.fake-domain-for-tests" port: 0 } } peers { permanent_uuid: "fa4335770e644e66a6ce962fb388c6f6" member_type: VOTER last_known_addr { host: "peer-1.fake-domain-for-tests" port: 0 } } peers { permanent_uuid: "6fec94ba5fb343fbad900ca4ef38b251" member_type: VOTER last_known_addr { host: "peer-2.fake-domain-for-tests" port: 0 } } I0920 01:06:48.537180 233 raft_consensus.cc:340] T TestTablet P 6fec94ba5fb343fbad900ca4ef38b251 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 peers { permanent_uuid: "938a40f5da7f4e62b2c317ecb1ba184c" member_type: VOTER last_known_addr { host: "peer-0.fake-domain-for-tests" port: 0 } } peers { permanent_uuid: "fa4335770e644e66a6ce962fb388c6f6" member_type: VOTER last_known_addr { host: "peer-1.fake-domain-for-tests" port: 0 } } peers { permanent_uuid: "6fec94ba5fb343fbad900ca4ef38b251" member_type: VOTER last_known_addr { host: "peer-2.fake-domain-for-tests" port: 0 } } I0920 01:06:48.537595 233 raft_consensus.cc:705] T TestTablet P 6fec94ba5fb343fbad900ca4ef38b251 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6fec94ba5fb343fbad900ca4ef38b251, State: Initialized, Role: FOLLOWER I0920 01:06:48.537946 233 consensus_queue.cc:229] T TestTablet P 6fec94ba5fb343fbad900ca4ef38b251 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "938a40f5da7f4e62b2c317ecb1ba184c" member_type: VOTER last_known_addr { host: "peer-0.fake-domain-for-tests" port: 0 } } peers { permanent_uuid: "fa4335770e644e66a6ce962fb388c6f6" member_type: VOTER last_known_addr { host: "peer-1.fake-domain-for-tests" port: 0 } } peers { permanent_uuid: "6fec94ba5fb343fbad900ca4ef38b251" member_type: VOTER last_known_addr { host: "peer-2.fake-domain-for-tests" port: 0 } } I0920 01:06:48.538498 233 raft_consensus.cc:405] T TestTablet P 6fec94ba5fb343fbad900ca4ef38b251 [term 0 FOLLOWER]: Emulating election... I0920 01:06:48.538648 233 raft_consensus.cc:2890] T TestTablet P 6fec94ba5fb343fbad900ca4ef38b251 [term 0 FOLLOWER]: Advancing to term 1 W0920 01:06:49.314579 233 consensus_meta.cc:220] T TestTablet P 6fec94ba5fb343fbad900ca4ef38b251: Time spent flushing consensus metadata: real 0.776s user 0.005s sys 0.000s I0920 01:06:49.315672 233 raft_consensus.cc:667] T TestTablet P 6fec94ba5fb343fbad900ca4ef38b251 [term 1 LEADER]: Becoming Leader. State: Replica: 6fec94ba5fb343fbad900ca4ef38b251, State: Running, Role: LEADER I0920 01:06:49.316112 233 consensus_queue.cc:206] T TestTablet P 6fec94ba5fb343fbad900ca4ef38b251 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "938a40f5da7f4e62b2c317ecb1ba184c" member_type: VOTER last_known_addr { host: "peer-0.fake-domain-for-tests" port: 0 } } peers { permanent_uuid: "fa4335770e644e66a6ce962fb388c6f6" member_type: VOTER last_known_addr { host: "peer-1.fake-domain-for-tests" port: 0 } } peers { permanent_uuid: "6fec94ba5fb343fbad900ca4ef38b251" member_type: VOTER last_known_addr { host: "peer-2.fake-domain-for-tests" port: 0 } } I0920 01:06:49.320704 627 raft_consensus.cc:2890] T TestTablet P fa4335770e644e66a6ce962fb388c6f6 [term 0 FOLLOWER]: Advancing to term 1 I0920 01:06:49.321324 628 raft_consensus.cc:2890] T TestTablet P 938a40f5da7f4e62b2c317ecb1ba184c [term 0 FOLLOWER]: Advancing to term 1 I0920 01:06:49.325456 628 raft_consensus.cc:1184] T TestTablet P 938a40f5da7f4e62b2c317ecb1ba184c [term 1 FOLLOWER]: Refusing update from remote peer 6fec94ba5fb343fbad900ca4ef38b251: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0920 01:06:49.325459 627 raft_consensus.cc:1184] T TestTablet P fa4335770e644e66a6ce962fb388c6f6 [term 1 FOLLOWER]: Refusing update from remote peer 6fec94ba5fb343fbad900ca4ef38b251: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0920 01:06:49.326063 626 consensus_queue.cc:984] T TestTablet P 6fec94ba5fb343fbad900ca4ef38b251 [LEADER]: Connected to new peer: Peer: permanent_uuid: "938a40f5da7f4e62b2c317ecb1ba184c" member_type: VOTER last_known_addr { host: "peer-0.fake-domain-for-tests" port: 0 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0920 01:06:49.326789 624 consensus_queue.cc:984] T TestTablet P 6fec94ba5fb343fbad900ca4ef38b251 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa4335770e644e66a6ce962fb388c6f6" member_type: VOTER last_known_addr { host: "peer-1.fake-domain-for-tests" port: 0 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0920 01:06:49.357350 233 raft_consensus.cc:2311] T TestTablet P fa4335770e644e66a6ce962fb388c6f6 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 938a40f5da7f4e62b2c317ecb1ba184c for term 2 because replica is either leader or believes a valid leader to be alive. I0920 01:06:49.357631 233 raft_consensus.cc:2890] T TestTablet P fa4335770e644e66a6ce962fb388c6f6 [term 1 FOLLOWER]: Advancing to term 2 W0920 01:06:50.192647 233 consensus_meta.cc:220] T TestTablet P fa4335770e644e66a6ce962fb388c6f6: Time spent flushing consensus metadata: real 0.835s user 0.004s sys 0.000s I0920 01:06:50.192862 233 raft_consensus.cc:2350] T TestTablet P fa4335770e644e66a6ce962fb388c6f6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 938a40f5da7f4e62b2c317ecb1ba184c in term 2. F0920 01:06:50.193255 233 thread_collision_warner.cc:23] Thread Collision! Previous thread id: 635, current thread id: 233 *** Check failure stack trace: *** I0920 01:06:50.193334 635 raft_consensus.cc:1149] T TestTablet P fa4335770e644e66a6ce962fb388c6f6 [term 2 FOLLOWER]: Rejecting Update request from peer 6fec94ba5fb343fbad900ca4ef38b251 for earlier term 1. Current term is 2. Ops: [] *** Aborted at 1568941610 (unix time) try "date -d @1568941610" if you are using GNU date *** I0920 01:06:50.195124 634 consensus_queue.cc:995] T TestTablet P 6fec94ba5fb343fbad900ca4ef38b251 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "fa4335770e644e66a6ce962fb388c6f6" member_type: VOTER last_known_addr { host: "peer-1.fake-domain-for-tests" port: 0 }, Status: INVALID_TERM, Last received: 1.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s I0920 01:06:50.197060 634 raft_consensus.cc:2885] T TestTablet P 6fec94ba5fb343fbad900ca4ef38b251 [term 1 LEADER]: Stepping down as leader of term 1 I0920 01:06:50.197245 634 raft_consensus.cc:705] T TestTablet P 6fec94ba5fb343fbad900ca4ef38b251 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6fec94ba5fb343fbad900ca4ef38b251, State: Running, Role: LEADER I0920 01:06:50.197593 634 consensus_queue.cc:229] T TestTablet P 6fec94ba5fb343fbad900ca4ef38b251 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 11, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "938a40f5da7f4e62b2c317ecb1ba184c" member_type: VOTER last_known_addr { host: "peer-0.fake-domain-for-tests" port: 0 } } peers { permanent_uuid: "fa4335770e644e66a6ce962fb388c6f6" member_type: VOTER last_known_addr { host: "peer-1.fake-domain-for-tests" port: 0 } } peers { permanent_uuid: "6fec94ba5fb343fbad900ca4ef38b251" member_type: VOTER last_known_addr { host: "peer-2.fake-domain-for-tests" port: 0 } } PC: @ 0x7ff647119c37 gsignal I0920 01:06:50.198277 634 raft_consensus.cc:2890] T TestTablet P 6fec94ba5fb343fbad900ca4ef38b251 [term 1 FOLLOWER]: Advancing to term 2 *** SIGABRT (@0x3e8000000e9) received by PID 233 (TID 0x7ff653518bc0) from PID 233; stack trace: *** @ 0x45b7bd __tsan::CallUserSignalHandler() at /home/jenkins-slave/workspace/kudu-master/1/thirdparty/src/llvm-6.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:1908 @ 0x45c7ab rtl_sigaction() at /home/jenkins-slave/workspace/kudu-master/1/thirdparty/src/llvm-6.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:1997 @ 0x7ff647f2b330 (unknown) at ??:0 @ 0x7ff647119c37 gsignal at ??:0 @ 0x7ff64711d028 abort at ??:0 @ 0x45e890 __interceptor_abort at /home/jenkins-slave/workspace/kudu-master/1/thirdparty/src/llvm-6.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:1763 @ 0x7ff649bfdac2 google::logging_fail() at ??:0 @ 0x7ff649bfc905 google::LogMessage::SendToLog() at ??:0 @ 0x7ff649bfd3d2 google::LogMessage::Flush() at ??:0 @ 0x7ff649c01c0b google::LogMessageFatal::~LogMessageFatal() at ??:0 @ 0x7ff649e8ed25 base::DCheckAsserter::warn() at ??:0 @ 0x7ff649e8ed9e base::ThreadCollisionWarner::EnterSelf() at ??:0 @ 0x7ff6524ad7b9 base::ThreadCollisionWarner::ScopedRecursiveCheck::ScopedRecursiveCheck() at ??:0 @ 0x7ff6524ab582 kudu::consensus::ConsensusMetadata::current_term() at ??:0 @ 0x4ffcc1 kudu::consensus::RaftConsensusQuorumTest::AssertDurableTermAndVote() at /home/jenkins-slave/workspace/kudu-master/1/src/kudu/consensus/raft_consensus_quorum-test.cc:554 @ 0x4f6e9b kudu::consensus::RaftConsensusQuorumTest_TestRequestVote_Test::TestBody() at /home/jenkins-slave/workspace/kudu-master/1/src/kudu/consensus/raft_consensus_quorum-test.cc:? @ 0x7ff6527c72f0 testing::internal::HandleExceptionsInMethodIfSupported<>() at ??:0 @ 0x7ff6527a64b9 testing::Test::Run() at ??:0 @ 0x7ff6527a774d testing::TestInfo::Run() at ??:0 @ 0x7ff6527a8227 testing::TestCase::Run() at ??:0 @ 0x7ff6527b45fb testing::internal::UnitTestImpl::RunAllTests() at ??:0 @ 0x7ff6527c8260 testing::internal::HandleExceptionsInMethodIfSupported<>() at ??:0 @ 0x7ff6527b3ee3 testing::UnitTest::Run() at ??:0 @ 0x7ff65311c51c RUN_ALL_TESTS() at ??:0 @ 0x7ff65311bcc7 main at ??:0 @ 0x7ff647104f45 __libc_start_main at ??:0 @ 0x4540f6 (unknown) at ??:?