[==========] Running 10 tests from 1 test case. [----------] Global test environment set-up. [----------] 10 tests from RaftConsensusElectionITest [ RUN ] RaftConsensusElectionITest.RunLeaderElection I0312 20:10:01.809968 212 test_util.cc:192] Using random seed: -1172409378 I0312 20:10:01.972879 212 ts_itest-base.cc:113] Starting cluster with: I0312 20:10:01.973021 212 ts_itest-base.cc:114] -------------- I0312 20:10:01.973114 212 ts_itest-base.cc:115] 3 tablet servers I0312 20:10:01.973197 212 ts_itest-base.cc:116] 3 replicas per TS I0312 20:10:01.973274 212 ts_itest-base.cc:117] -------------- I0312 20:10:01.976891 212 external_mini_cluster.cc:805] Running /tmp/dist-test-task3bwPeE/build/tsan/bin/kudu-master /tmp/dist-test-task3bwPeE/build/tsan/bin/kudu-master --fs_wal_dir=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.RunLeaderElection.1520885401735786-212/raft_consensus-itest-cluster/master-0/wal --fs_data_dirs=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.RunLeaderElection.1520885401735786-212/raft_consensus-itest-cluster/master-0/data --block_manager=log --webserver_interface=localhost --ipki_ca_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.0.53.62:0 --webserver_interface=127.0.53.62 --webserver_port=0 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.RunLeaderElection.1520885401735786-212/raft_consensus-itest-cluster/master-0/logs --server_dump_info_path=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.RunLeaderElection.1520885401735786-212/raft_consensus-itest-cluster/master-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W0312 20:10:02.099366 224 flags.cc:412] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0312 20:10:02.099663 224 flags.cc:412] Enabled unsafe flag: --never_fsync=true W0312 20:10:02.115250 224 flags.cc:412] Enabled experimental flag: --ipki_ca_key_size=1024 W0312 20:10:02.115375 224 flags.cc:412] Enabled experimental flag: --ipki_server_key_size=1024 W0312 20:10:02.115468 224 flags.cc:412] Enabled experimental flag: --tsk_num_rsa_bits=512 I0312 20:10:02.136819 224 master_main.cc:66] Master server non-default flags: --fs_data_dirs=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.RunLeaderElection.1520885401735786-212/raft_consensus-itest-cluster/master-0/data --fs_wal_dir=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.RunLeaderElection.1520885401735786-212/raft_consensus-itest-cluster/master-0/wal --ipki_ca_key_size=1024 --ipki_server_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.0.53.62:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.RunLeaderElection.1520885401735786-212/raft_consensus-itest-cluster/master-0/data/info.pb --webserver_interface=127.0.53.62 --webserver_port=0 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --log_dir=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.RunLeaderElection.1520885401735786-212/raft_consensus-itest-cluster/master-0/logs --logbuflevel=-1 --logtostderr=true Master server version: kudu 1.7.0-SNAPSHOT revision 13e10c63c46de30ac85f34b543cdd6fb44881f61 build type FASTDEBUG built by None at 12 Mar 2018 20:06:40 UTC on jenkins-slave-vn0g build id 12388 TSAN enabled I0312 20:10:02.147253 224 master_main.cc:73] Initializing master server... I0312 20:10:02.148623 224 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 1068276us I0312 20:10:02.149322 224 fs_manager.cc:260] Metadata directory not provided I0312 20:10:02.149441 224 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0312 20:10:02.149634 224 server_base.cc:434] Could not load existing FS layout: Not found: /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.RunLeaderElection.1520885401735786-212/raft_consensus-itest-cluster/master-0/data/instance: No such file or directory (error 2) I0312 20:10:02.149739 224 server_base.cc:435] Attempting to create new FS layout instead I0312 20:10:02.157058 224 fs_manager.cc:596] Generated new instance metadata in path /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.RunLeaderElection.1520885401735786-212/raft_consensus-itest-cluster/master-0/data/instance: uuid: "1b72f499f005484ebfd8e0762cf59a2b" format_stamp: "Formatted at 2018-03-12 20:10:02 on dist-test-slave-dist-test-slave-f27p" I0312 20:10:02.157743 224 fs_manager.cc:596] Generated new instance metadata in path /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.RunLeaderElection.1520885401735786-212/raft_consensus-itest-cluster/master-0/wal/instance: uuid: "1b72f499f005484ebfd8e0762cf59a2b" format_stamp: "Formatted at 2018-03-12 20:10:02 on dist-test-slave-dist-test-slave-f27p" I0312 20:10:02.166481 224 fs_manager.cc:496] Time spent creating directory manager: real 0.008s user 0.004s sys 0.004s I0312 20:10:02.167590 224 env_posix.cc:1620] Not raising this process' open files per process limit of 65536; it is already as high as it can go I0312 20:10:02.167851 224 file_cache.cc:470] Constructed file cache lbm with capacity 26214 I0312 20:10:02.174710 224 fs_manager.cc:418] Time spent opening block manager: real 0.002s user 0.004s sys 0.000s I0312 20:10:02.174829 224 fs_manager.cc:429] Opened local filesystem: /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.RunLeaderElection.1520885401735786-212/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.RunLeaderElection.1520885401735786-212/raft_consensus-itest-cluster/master-0/wal uuid: "1b72f499f005484ebfd8e0762cf59a2b" format_stamp: "Formatted at 2018-03-12 20:10:02 on dist-test-slave-dist-test-slave-f27p" I0312 20:10:02.175011 224 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.RunLeaderElection.1520885401735786-212/raft_consensus-itest-cluster/master-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0312 20:10:02.342777 224 env_posix.cc:1620] Not raising this process' running threads per effective uid limit of 60284; it is already as high as it can go I0312 20:10:02.343379 224 master_main.cc:76] Starting Master server... I0312 20:10:02.345101 241 process_memory.cc:182] Process hard memory limit is 11.777631 GB I0312 20:10:02.345238 241 process_memory.cc:184] Process soft memory limit is 9.422105 GB I0312 20:10:02.345335 241 process_memory.cc:187] Process memory pressure threshold is 7.066578 GB I0312 20:10:02.395141 224 rpc_server.cc:201] RPC server started. Bound to: 127.0.53.62:54194 I0312 20:10:02.421510 224 webserver.cc:173] Starting webserver on 127.0.53.62:0 I0312 20:10:02.421617 224 webserver.cc:184] Document root disabled I0312 20:10:02.422760 224 webserver.cc:311] Webserver started. Bound to: http://127.0.53.62:34885/ I0312 20:10:02.423511 224 server_base.cc:607] Dumped server information to /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.RunLeaderElection.1520885401735786-212/raft_consensus-itest-cluster/master-0/data/info.pb I0312 20:10:02.426661 285 data_dirs.cc:931] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0312 20:10:02.427304 212 external_mini_cluster.cc:867] Started /tmp/dist-test-task3bwPeE/build/tsan/bin/kudu-master as pid 224 I0312 20:10:02.428273 212 external_mini_cluster.cc:805] Running /tmp/dist-test-task3bwPeE/build/tsan/bin/kudu-tserver /tmp/dist-test-task3bwPeE/build/tsan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.RunLeaderElection.1520885401735786-212/raft_consensus-itest-cluster/ts-0/wal --fs_data_dirs=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.RunLeaderElection.1520885401735786-212/raft_consensus-itest-cluster/ts-0/data --block_manager=log --rpc_bind_addresses=127.0.53.1:0 --local_ip_for_outbound_sockets=127.0.53.1 --webserver_interface=127.0.53.1 --webserver_port=0 --tserver_master_addrs=127.0.53.62:54194 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.RunLeaderElection.1520885401735786-212/raft_consensus-itest-cluster/ts-0/logs --server_dump_info_path=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.RunLeaderElection.1520885401735786-212/raft_consensus-itest-cluster/ts-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --log_cache_size_limit_mb=10 --consensus_rpc_timeout_ms=1000 with env {} I0312 20:10:02.441838 285 tablet_bootstrap.cc:436] T 00000000000000000000000000000000 P 1b72f499f005484ebfd8e0762cf59a2b: Bootstrap starting. I0312 20:10:02.443977 285 tablet_bootstrap.cc:581] T 00000000000000000000000000000000 P 1b72f499f005484ebfd8e0762cf59a2b: No blocks or log segments found. Creating new log. I0312 20:10:02.445050 285 log.cc:520] T 00000000000000000000000000000000 P 1b72f499f005484ebfd8e0762cf59a2b: Log is configured to *not* fsync() on all Append() calls I0312 20:10:02.448297 285 tablet_bootstrap.cc:436] T 00000000000000000000000000000000 P 1b72f499f005484ebfd8e0762cf59a2b: No bootstrap required, opened a new log I0312 20:10:02.465265 285 raft_consensus.cc:304] T 00000000000000000000000000000000 P 1b72f499f005484ebfd8e0762cf59a2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b72f499f005484ebfd8e0762cf59a2b" member_type: VOTER } I0312 20:10:02.465591 285 raft_consensus.cc:330] T 00000000000000000000000000000000 P 1b72f499f005484ebfd8e0762cf59a2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0312 20:10:02.465746 285 raft_consensus.cc:605] T 00000000000000000000000000000000 P 1b72f499f005484ebfd8e0762cf59a2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b72f499f005484ebfd8e0762cf59a2b, State: Initialized, Role: FOLLOWER I0312 20:10:02.466161 285 consensus_queue.cc:219] T 00000000000000000000000000000000 P 1b72f499f005484ebfd8e0762cf59a2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b72f499f005484ebfd8e0762cf59a2b" member_type: VOTER } I0312 20:10:02.466485 285 raft_consensus.cc:344] T 00000000000000000000000000000000 P 1b72f499f005484ebfd8e0762cf59a2b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0312 20:10:02.466645 285 raft_consensus.cc:436] T 00000000000000000000000000000000 P 1b72f499f005484ebfd8e0762cf59a2b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0312 20:10:02.466799 285 raft_consensus.cc:2700] T 00000000000000000000000000000000 P 1b72f499f005484ebfd8e0762cf59a2b: Snoozing failure detection for 2.249s (starting election) I0312 20:10:02.466950 285 raft_consensus.cc:2752] T 00000000000000000000000000000000 P 1b72f499f005484ebfd8e0762cf59a2b [term 0 FOLLOWER]: Advancing to term 1 I0312 20:10:02.470271 285 raft_consensus.cc:458] T 00000000000000000000000000000000 P 1b72f499f005484ebfd8e0762cf59a2b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b72f499f005484ebfd8e0762cf59a2b" member_type: VOTER } I0312 20:10:02.470695 285 leader_election.cc:258] T 00000000000000000000000000000000 P 1b72f499f005484ebfd8e0762cf59a2b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0312 20:10:02.473026 288 raft_consensus.cc:2700] T 00000000000000000000000000000000 P 1b72f499f005484ebfd8e0762cf59a2b: Snoozing failure detection for 1.638s (election complete) I0312 20:10:02.473286 288 raft_consensus.cc:2507] T 00000000000000000000000000000000 P 1b72f499f005484ebfd8e0762cf59a2b [term 1 FOLLOWER]: Leader election won for term 1 I0312 20:10:02.481528 288 raft_consensus.cc:572] T 00000000000000000000000000000000 P 1b72f499f005484ebfd8e0762cf59a2b [term 1 LEADER]: Becoming Leader. State: Replica: 1b72f499f005484ebfd8e0762cf59a2b, State: Running, Role: LEADER I0312 20:10:02.481914 288 consensus_queue.cc:196] T 00000000000000000000000000000000 P 1b72f499f005484ebfd8e0762cf59a2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b72f499f005484ebfd8e0762cf59a2b" member_type: VOTER } I0312 20:10:02.486745 285 sys_catalog.cc:335] T 00000000000000000000000000000000 P 1b72f499f005484ebfd8e0762cf59a2b [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 1 leader_uuid: "1b72f499f005484ebfd8e0762cf59a2b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b72f499f005484ebfd8e0762cf59a2b" member_type: VOTER } } I0312 20:10:02.487145 285 sys_catalog.cc:338] T 00000000000000000000000000000000 P 1b72f499f005484ebfd8e0762cf59a2b [sys.catalog]: This master's current role is: LEADER I0312 20:10:02.493225 291 sys_catalog.cc:335] T 00000000000000000000000000000000 P 1b72f499f005484ebfd8e0762cf59a2b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1b72f499f005484ebfd8e0762cf59a2b. Latest consensus state: current_term: 1 leader_uuid: "1b72f499f005484ebfd8e0762cf59a2b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b72f499f005484ebfd8e0762cf59a2b" member_type: VOTER } } I0312 20:10:02.493211 289 sys_catalog.cc:335] T 00000000000000000000000000000000 P 1b72f499f005484ebfd8e0762cf59a2b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1b72f499f005484ebfd8e0762cf59a2b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b72f499f005484ebfd8e0762cf59a2b" member_type: VOTER } } I0312 20:10:02.493643 291 sys_catalog.cc:338] T 00000000000000000000000000000000 P 1b72f499f005484ebfd8e0762cf59a2b [sys.catalog]: This master's current role is: LEADER I0312 20:10:02.493681 289 sys_catalog.cc:338] T 00000000000000000000000000000000 P 1b72f499f005484ebfd8e0762cf59a2b [sys.catalog]: This master's current role is: LEADER I0312 20:10:02.494222 285 sys_catalog.cc:422] T 00000000000000000000000000000000 P 1b72f499f005484ebfd8e0762cf59a2b [sys.catalog]: configured and running, proceeding with master startup. I0312 20:10:02.496579 224 master_main.cc:79] Master server successfully started. W0312 20:10:02.499788 294 catalog_manager.cc:1018] T 00000000000000000000000000000000 P 1b72f499f005484ebfd8e0762cf59a2b: acquiring CA information for follower catalog manager: Not found: root CA entry not found W0312 20:10:02.499944 294 catalog_manager.cc:560] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry I0312 20:10:02.501490 293 catalog_manager.cc:972] Loading table and tablet metadata into memory... I0312 20:10:02.503348 293 catalog_manager.cc:983] Initializing Kudu internal certificate authority... I0312 20:10:02.579185 293 catalog_manager.cc:870] Generated new certificate authority record I0312 20:10:02.581630 293 catalog_manager.cc:992] Loading token signing keys... I0312 20:10:02.592627 293 catalog_manager.cc:3868] T 00000000000000000000000000000000 P 1b72f499f005484ebfd8e0762cf59a2b: Generated new TSK 0 WARNING: Logging before InitGoogleLogging() is written to STDERR W0312 20:10:02.608407 286 flags.cc:412] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0312 20:10:02.608885 286 flags.cc:412] Enabled unsafe flag: --never_fsync=true W0312 20:10:02.635424 286 flags.cc:412] Enabled experimental flag: --ipki_server_key_size=1024 W0312 20:10:02.635895 286 flags.cc:412] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.53.1 I0312 20:10:02.669564 286 tablet_server_main.cc:72] Tablet server non-default flags: --consensus_rpc_timeout_ms=1000 --log_cache_size_limit_mb=10 --fs_data_dirs=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.RunLeaderElection.1520885401735786-212/raft_consensus-itest-cluster/ts-0/data --fs_wal_dir=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.RunLeaderElection.1520885401735786-212/raft_consensus-itest-cluster/ts-0/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.0.53.1:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.RunLeaderElection.1520885401735786-212/raft_consensus-itest-cluster/ts-0/data/info.pb --webserver_interface=127.0.53.1 --webserver_port=0 --tserver_master_addrs=127.0.53.62:54194 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.0.53.1 --log_dir=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.RunLeaderElection.1520885401735786-212/raft_consensus-itest-cluster/ts-0/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.7.0-SNAPSHOT revision 13e10c63c46de30ac85f34b543cdd6fb44881f61 build type FASTDEBUG built by None at 12 Mar 2018 20:06:40 UTC on jenkins-slave-vn0g build id 12388 TSAN enabled I0312 20:10:02.686862 286 tablet_server_main.cc:79] Initializing tablet server... I0312 20:10:02.689888 286 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 1068276us I0312 20:10:02.690487 286 fs_manager.cc:260] Metadata directory not provided I0312 20:10:02.690662 286 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0312 20:10:02.690932 286 server_base.cc:434] Could not load existing FS layout: Not found: /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.RunLeaderElection.1520885401735786-212/raft_consensus-itest-cluster/ts-0/data/instance: No such file or directory (error 2) I0312 20:10:02.691102 286 server_base.cc:435] Attempting to create new FS layout instead I0312 20:10:02.703682 286 fs_manager.cc:596] Generated new instance metadata in path /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.RunLeaderElection.1520885401735786-212/raft_consensus-itest-cluster/ts-0/data/instance: uuid: "90c2a1b6af6d40fa8b9148fb40ab6f90" format_stamp: "Formatted at 2018-03-12 20:10:02 on dist-test-slave-dist-test-slave-f27p" I0312 20:10:02.704697 286 fs_manager.cc:596] Generated new instance metadata in path /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.RunLeaderElection.1520885401735786-212/raft_consensus-itest-cluster/ts-0/wal/instance: uuid: "90c2a1b6af6d40fa8b9148fb40ab6f90" format_stamp: "Formatted at 2018-03-12 20:10:02 on dist-test-slave-dist-test-slave-f27p" I0312 20:10:02.718914 286 fs_manager.cc:496] Time spent creating directory manager: real 0.014s user 0.008s sys 0.004s I0312 20:10:02.720649 286 env_posix.cc:1620] Not raising this process' open files per process limit of 65536; it is already as high as it can go I0312 20:10:02.721055 286 file_cache.cc:470] Constructed file cache lbm with capacity 26214 I0312 20:10:02.732425 286 fs_manager.cc:418] Time spent opening block manager: real 0.002s user 0.000s sys 0.000s I0312 20:10:02.732583 286 fs_manager.cc:429] Opened local filesystem: /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.RunLeaderElection.1520885401735786-212/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.RunLeaderElection.1520885401735786-212/raft_consensus-itest-cluster/ts-0/wal uuid: "90c2a1b6af6d40fa8b9148fb40ab6f90" format_stamp: "Formatted at 2018-03-12 20:10:02 on dist-test-slave-dist-test-slave-f27p" I0312 20:10:02.732861 286 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.RunLeaderElection.1520885401735786-212/raft_consensus-itest-cluster/ts-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0312 20:10:02.996192 286 env_posix.cc:1620] Not raising this process' running threads per effective uid limit of 60284; it is already as high as it can go I0312 20:10:02.997750 286 ts_tablet_manager.cc:349] Loaded tablet metadata (0 live tablets) I0312 20:10:02.998867 286 tablet_server_main.cc:84] Starting tablet server... I0312 20:10:03.134037 286 rpc_server.cc:201] RPC server started. Bound to: 127.0.53.1:43100 I0312 20:10:03.134938 286 webserver.cc:173] Starting webserver on 127.0.53.1:0 I0312 20:10:03.135046 286 webserver.cc:184] Document root disabled I0312 20:10:03.137121 286 webserver.cc:311] Webserver started. Bound to: http://127.0.53.1:48130/ I0312 20:10:03.137895 286 server_base.cc:607] Dumped server information to /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.RunLeaderElection.1520885401735786-212/raft_consensus-itest-cluster/ts-0/data/info.pb I0312 20:10:03.143905 212 external_mini_cluster.cc:867] Started /tmp/dist-test-task3bwPeE/build/tsan/bin/kudu-tserver as pid 286 I0312 20:10:03.145081 212 external_mini_cluster.cc:805] Running /tmp/dist-test-task3bwPeE/build/tsan/bin/kudu-tserver /tmp/dist-test-task3bwPeE/build/tsan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.RunLeaderElection.1520885401735786-212/raft_consensus-itest-cluster/ts-1/wal --fs_data_dirs=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.RunLeaderElection.1520885401735786-212/raft_consensus-itest-cluster/ts-1/data --block_manager=log --rpc_bind_addresses=127.0.53.2:0 --local_ip_for_outbound_sockets=127.0.53.2 --webserver_interface=127.0.53.2 --webserver_port=0 --tserver_master_addrs=127.0.53.62:54194 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.RunLeaderElection.1520885401735786-212/raft_consensus-itest-cluster/ts-1/logs --server_dump_info_path=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.RunLeaderElection.1520885401735786-212/raft_consensus-itest-cluster/ts-1/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --log_cache_size_limit_mb=10 --consensus_rpc_timeout_ms=1000 with env {} I0312 20:10:03.147632 418 process_memory.cc:182] Process hard memory limit is 11.777631 GB I0312 20:10:03.148321 418 process_memory.cc:184] Process soft memory limit is 9.422105 GB I0312 20:10:03.148605 418 process_memory.cc:187] Process memory pressure threshold is 7.066578 GB I0312 20:10:03.147678 286 tablet_server_main.cc:87] Tablet server successfully started. I0312 20:10:03.203737 417 heartbeater.cc:346] Connected to a master server at 127.0.53.62:54194 I0312 20:10:03.204047 417 heartbeater.cc:426] Registering TS with master... I0312 20:10:03.204618 417 heartbeater.cc:475] Master 127.0.53.62:54194 requested a full tablet report, sending... I0312 20:10:03.207329 252 ts_manager.cc:81] Registered new tserver with Master: 90c2a1b6af6d40fa8b9148fb40ab6f90 (127.0.53.1:43100) I0312 20:10:03.210052 252 master_service.cc:243] Signed X509 certificate for tserver {username='slave'} at 127.0.53.1:39405 WARNING: Logging before InitGoogleLogging() is written to STDERR W0312 20:10:03.295534 421 flags.cc:412] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0312 20:10:03.295879 421 flags.cc:412] Enabled unsafe flag: --never_fsync=true W0312 20:10:03.310406 421 flags.cc:412] Enabled experimental flag: --ipki_server_key_size=1024 W0312 20:10:03.310708 421 flags.cc:412] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.53.2 I0312 20:10:03.330991 421 tablet_server_main.cc:72] Tablet server non-default flags: --consensus_rpc_timeout_ms=1000 --log_cache_size_limit_mb=10 --fs_data_dirs=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.RunLeaderElection.1520885401735786-212/raft_consensus-itest-cluster/ts-1/data --fs_wal_dir=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.RunLeaderElection.1520885401735786-212/raft_consensus-itest-cluster/ts-1/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.0.53.2:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.RunLeaderElection.1520885401735786-212/raft_consensus-itest-cluster/ts-1/data/info.pb --webserver_interface=127.0.53.2 --webserver_port=0 --tserver_master_addrs=127.0.53.62:54194 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.0.53.2 --log_dir=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.RunLeaderElection.1520885401735786-212/raft_consensus-itest-cluster/ts-1/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.7.0-SNAPSHOT revision 13e10c63c46de30ac85f34b543cdd6fb44881f61 build type FASTDEBUG built by None at 12 Mar 2018 20:06:40 UTC on jenkins-slave-vn0g build id 12388 TSAN enabled I0312 20:10:03.347569 421 tablet_server_main.cc:79] Initializing tablet server... I0312 20:10:03.349920 421 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 1068776us I0312 20:10:03.350353 421 fs_manager.cc:260] Metadata directory not provided I0312 20:10:03.350474 421 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0312 20:10:03.350692 421 server_base.cc:434] Could not load existing FS layout: Not found: /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.RunLeaderElection.1520885401735786-212/raft_consensus-itest-cluster/ts-1/data/instance: No such file or directory (error 2) I0312 20:10:03.350811 421 server_base.cc:435] Attempting to create new FS layout instead I0312 20:10:03.359201 421 fs_manager.cc:596] Generated new instance metadata in path /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.RunLeaderElection.1520885401735786-212/raft_consensus-itest-cluster/ts-1/data/instance: uuid: "2777b13979e24d25b4f50048a0a1ecb4" format_stamp: "Formatted at 2018-03-12 20:10:03 on dist-test-slave-dist-test-slave-f27p" I0312 20:10:03.359933 421 fs_manager.cc:596] Generated new instance metadata in path /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.RunLeaderElection.1520885401735786-212/raft_consensus-itest-cluster/ts-1/wal/instance: uuid: "2777b13979e24d25b4f50048a0a1ecb4" format_stamp: "Formatted at 2018-03-12 20:10:03 on dist-test-slave-dist-test-slave-f27p" I0312 20:10:03.376263 421 fs_manager.cc:496] Time spent creating directory manager: real 0.016s user 0.016s sys 0.000s I0312 20:10:03.377429 421 env_posix.cc:1620] Not raising this process' open files per process limit of 65536; it is already as high as it can go I0312 20:10:03.377701 421 file_cache.cc:470] Constructed file cache lbm with capacity 26214 I0312 20:10:03.391644 421 fs_manager.cc:418] Time spent opening block manager: real 0.009s user 0.000s sys 0.008s I0312 20:10:03.391762 421 fs_manager.cc:429] Opened local filesystem: /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.RunLeaderElection.1520885401735786-212/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.RunLeaderElection.1520885401735786-212/raft_consensus-itest-cluster/ts-1/wal uuid: "2777b13979e24d25b4f50048a0a1ecb4" format_stamp: "Formatted at 2018-03-12 20:10:03 on dist-test-slave-dist-test-slave-f27p" I0312 20:10:03.391978 421 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.RunLeaderElection.1520885401735786-212/raft_consensus-itest-cluster/ts-1/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0312 20:10:03.618702 421 env_posix.cc:1620] Not raising this process' running threads per effective uid limit of 60284; it is already as high as it can go I0312 20:10:03.619971 421 ts_tablet_manager.cc:349] Loaded tablet metadata (0 live tablets) I0312 20:10:03.621654 421 tablet_server_main.cc:84] Starting tablet server... I0312 20:10:03.756278 421 rpc_server.cc:201] RPC server started. Bound to: 127.0.53.2:59907 I0312 20:10:03.757297 421 webserver.cc:173] Starting webserver on 127.0.53.2:0 I0312 20:10:03.757413 421 webserver.cc:184] Document root disabled I0312 20:10:03.758680 421 webserver.cc:311] Webserver started. Bound to: http://127.0.53.2:40677/ I0312 20:10:03.759419 421 server_base.cc:607] Dumped server information to /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.RunLeaderElection.1520885401735786-212/raft_consensus-itest-cluster/ts-1/data/info.pb I0312 20:10:03.762658 421 tablet_server_main.cc:87] Tablet server successfully started. I0312 20:10:03.765667 212 external_mini_cluster.cc:867] Started /tmp/dist-test-task3bwPeE/build/tsan/bin/kudu-tserver as pid 421 I0312 20:10:03.766537 212 external_mini_cluster.cc:805] Running /tmp/dist-test-task3bwPeE/build/tsan/bin/kudu-tserver /tmp/dist-test-task3bwPeE/build/tsan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.RunLeaderElection.1520885401735786-212/raft_consensus-itest-cluster/ts-2/wal --fs_data_dirs=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.RunLeaderElection.1520885401735786-212/raft_consensus-itest-cluster/ts-2/data --block_manager=log --rpc_bind_addresses=127.0.53.3:0 --local_ip_for_outbound_sockets=127.0.53.3 --webserver_interface=127.0.53.3 --webserver_port=0 --tserver_master_addrs=127.0.53.62:54194 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.RunLeaderElection.1520885401735786-212/raft_consensus-itest-cluster/ts-2/logs --server_dump_info_path=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.RunLeaderElection.1520885401735786-212/raft_consensus-itest-cluster/ts-2/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --log_cache_size_limit_mb=10 --consensus_rpc_timeout_ms=1000 with env {} I0312 20:10:03.763916 543 process_memory.cc:182] Process hard memory limit is 11.777631 GB I0312 20:10:03.795109 543 process_memory.cc:184] Process soft memory limit is 9.422105 GB I0312 20:10:03.795217 543 process_memory.cc:187] Process memory pressure threshold is 7.066578 GB I0312 20:10:03.828562 542 heartbeater.cc:346] Connected to a master server at 127.0.53.62:54194 I0312 20:10:03.828871 542 heartbeater.cc:426] Registering TS with master... I0312 20:10:03.829447 542 heartbeater.cc:475] Master 127.0.53.62:54194 requested a full tablet report, sending... I0312 20:10:03.832242 252 ts_manager.cc:81] Registered new tserver with Master: 2777b13979e24d25b4f50048a0a1ecb4 (127.0.53.2:59907) I0312 20:10:03.833969 252 master_service.cc:243] Signed X509 certificate for tserver {username='slave'} at 127.0.53.2:42574 WARNING: Logging before InitGoogleLogging() is written to STDERR W0312 20:10:03.904207 546 flags.cc:412] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0312 20:10:03.904568 546 flags.cc:412] Enabled unsafe flag: --never_fsync=true W0312 20:10:03.920019 546 flags.cc:412] Enabled experimental flag: --ipki_server_key_size=1024 W0312 20:10:03.920346 546 flags.cc:412] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.53.3 I0312 20:10:03.941238 546 tablet_server_main.cc:72] Tablet server non-default flags: --consensus_rpc_timeout_ms=1000 --log_cache_size_limit_mb=10 --fs_data_dirs=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.RunLeaderElection.1520885401735786-212/raft_consensus-itest-cluster/ts-2/data --fs_wal_dir=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.RunLeaderElection.1520885401735786-212/raft_consensus-itest-cluster/ts-2/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.0.53.3:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.RunLeaderElection.1520885401735786-212/raft_consensus-itest-cluster/ts-2/data/info.pb --webserver_interface=127.0.53.3 --webserver_port=0 --tserver_master_addrs=127.0.53.62:54194 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.0.53.3 --log_dir=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.RunLeaderElection.1520885401735786-212/raft_consensus-itest-cluster/ts-2/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.7.0-SNAPSHOT revision 13e10c63c46de30ac85f34b543cdd6fb44881f61 build type FASTDEBUG built by None at 12 Mar 2018 20:06:40 UTC on jenkins-slave-vn0g build id 12388 TSAN enabled I0312 20:10:03.952765 546 tablet_server_main.cc:79] Initializing tablet server... I0312 20:10:03.954413 546 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 1068776us I0312 20:10:03.954845 546 fs_manager.cc:260] Metadata directory not provided I0312 20:10:03.954969 546 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0312 20:10:03.955196 546 server_base.cc:434] Could not load existing FS layout: Not found: /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.RunLeaderElection.1520885401735786-212/raft_consensus-itest-cluster/ts-2/data/instance: No such file or directory (error 2) I0312 20:10:03.955305 546 server_base.cc:435] Attempting to create new FS layout instead I0312 20:10:03.962666 546 fs_manager.cc:596] Generated new instance metadata in path /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.RunLeaderElection.1520885401735786-212/raft_consensus-itest-cluster/ts-2/data/instance: uuid: "15558bb7548e42789ec21ac818f1b4b8" format_stamp: "Formatted at 2018-03-12 20:10:03 on dist-test-slave-dist-test-slave-f27p" I0312 20:10:03.963356 546 fs_manager.cc:596] Generated new instance metadata in path /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.RunLeaderElection.1520885401735786-212/raft_consensus-itest-cluster/ts-2/wal/instance: uuid: "15558bb7548e42789ec21ac818f1b4b8" format_stamp: "Formatted at 2018-03-12 20:10:03 on dist-test-slave-dist-test-slave-f27p" I0312 20:10:03.972121 546 fs_manager.cc:496] Time spent creating directory manager: real 0.008s user 0.008s sys 0.000s I0312 20:10:03.973280 546 env_posix.cc:1620] Not raising this process' open files per process limit of 65536; it is already as high as it can go I0312 20:10:03.973552 546 file_cache.cc:470] Constructed file cache lbm with capacity 26214 I0312 20:10:03.979724 546 fs_manager.cc:418] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s I0312 20:10:03.979835 546 fs_manager.cc:429] Opened local filesystem: /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.RunLeaderElection.1520885401735786-212/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.RunLeaderElection.1520885401735786-212/raft_consensus-itest-cluster/ts-2/wal uuid: "15558bb7548e42789ec21ac818f1b4b8" format_stamp: "Formatted at 2018-03-12 20:10:03 on dist-test-slave-dist-test-slave-f27p" I0312 20:10:03.980023 546 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.RunLeaderElection.1520885401735786-212/raft_consensus-itest-cluster/ts-2/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0312 20:10:04.163900 546 env_posix.cc:1620] Not raising this process' running threads per effective uid limit of 60284; it is already as high as it can go I0312 20:10:04.165180 546 ts_tablet_manager.cc:349] Loaded tablet metadata (0 live tablets) I0312 20:10:04.166059 546 tablet_server_main.cc:84] Starting tablet server... I0312 20:10:04.214340 417 heartbeater.cc:467] Master 127.0.53.62:54194 was elected leader, sending a full tablet report... I0312 20:10:04.282672 546 rpc_server.cc:201] RPC server started. Bound to: 127.0.53.3:33977 I0312 20:10:04.283541 546 webserver.cc:173] Starting webserver on 127.0.53.3:0 I0312 20:10:04.283640 546 webserver.cc:184] Document root disabled I0312 20:10:04.285398 546 webserver.cc:311] Webserver started. Bound to: http://127.0.53.3:39202/ I0312 20:10:04.286118 546 server_base.cc:607] Dumped server information to /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.RunLeaderElection.1520885401735786-212/raft_consensus-itest-cluster/ts-2/data/info.pb I0312 20:10:04.289244 668 process_memory.cc:182] Process hard memory limit is 11.777631 GB I0312 20:10:04.289666 668 process_memory.cc:184] Process soft memory limit is 9.422105 GB I0312 20:10:04.289793 668 process_memory.cc:187] Process memory pressure threshold is 7.066578 GB I0312 20:10:04.290984 212 external_mini_cluster.cc:867] Started /tmp/dist-test-task3bwPeE/build/tsan/bin/kudu-tserver as pid 546 I0312 20:10:04.289610 546 tablet_server_main.cc:87] Tablet server successfully started. I0312 20:10:04.343276 667 heartbeater.cc:346] Connected to a master server at 127.0.53.62:54194 I0312 20:10:04.343555 667 heartbeater.cc:426] Registering TS with master... I0312 20:10:04.344041 667 heartbeater.cc:475] Master 127.0.53.62:54194 requested a full tablet report, sending... I0312 20:10:04.345799 252 ts_manager.cc:81] Registered new tserver with Master: 15558bb7548e42789ec21ac818f1b4b8 (127.0.53.3:33977) I0312 20:10:04.347440 252 master_service.cc:243] Signed X509 certificate for tserver {username='slave'} at 127.0.53.3:43785 I0312 20:10:04.347651 212 external_mini_cluster.cc:498] 3 TS(s) registered with all masters I0312 20:10:04.381302 252 catalog_manager.cc:1294] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50360: name: "TestTable" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 3 split_rows_range_bounds { } partition_schema { range_schema { columns { name: "key" } } } W0312 20:10:04.389962 252 catalog_manager.cc:1450] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a replica failure: 4 tablet servers are needed, 3 are alive. Consider bringing up additional tablet server(s) or running both the masters and all tablet servers with --raft_prepare_replacement_before_eviction=false flag (not recommended). I0312 20:10:04.434186 479 tablet_service.cc:755] Processing CreateTablet for tablet 041f375b4ec14b1c9cec5fe2a1503b98 (table=TestTable [id=3fc586f2c01f4a3c86efc77a1fba327b]), partition=RANGE (key) PARTITION UNBOUNDED I0312 20:10:04.434906 479 data_dirs.cc:931] Could only allocate 1 dirs of requested 3 for tablet 041f375b4ec14b1c9cec5fe2a1503b98. 1 dirs total, 0 dirs full, 0 dirs failed I0312 20:10:04.442200 354 tablet_service.cc:755] Processing CreateTablet for tablet 041f375b4ec14b1c9cec5fe2a1503b98 (table=TestTable [id=3fc586f2c01f4a3c86efc77a1fba327b]), partition=RANGE (key) PARTITION UNBOUNDED I0312 20:10:04.442245 604 tablet_service.cc:755] Processing CreateTablet for tablet 041f375b4ec14b1c9cec5fe2a1503b98 (table=TestTable [id=3fc586f2c01f4a3c86efc77a1fba327b]), partition=RANGE (key) PARTITION UNBOUNDED I0312 20:10:04.443138 604 data_dirs.cc:931] Could only allocate 1 dirs of requested 3 for tablet 041f375b4ec14b1c9cec5fe2a1503b98. 1 dirs total, 0 dirs full, 0 dirs failed I0312 20:10:04.443195 354 data_dirs.cc:931] Could only allocate 1 dirs of requested 3 for tablet 041f375b4ec14b1c9cec5fe2a1503b98. 1 dirs total, 0 dirs full, 0 dirs failed I0312 20:10:04.453980 479 ts_tablet_manager.cc:1163] T 041f375b4ec14b1c9cec5fe2a1503b98 P 2777b13979e24d25b4f50048a0a1ecb4: Registered tablet (data state: TABLET_DATA_READY) I0312 20:10:04.460080 687 ts_tablet_manager.cc:1018] T 041f375b4ec14b1c9cec5fe2a1503b98 P 2777b13979e24d25b4f50048a0a1ecb4: Bootstrapping tablet I0312 20:10:04.460602 687 tablet_bootstrap.cc:436] T 041f375b4ec14b1c9cec5fe2a1503b98 P 2777b13979e24d25b4f50048a0a1ecb4: Bootstrap starting. I0312 20:10:04.461001 604 ts_tablet_manager.cc:1163] T 041f375b4ec14b1c9cec5fe2a1503b98 P 15558bb7548e42789ec21ac818f1b4b8: Registered tablet (data state: TABLET_DATA_READY) I0312 20:10:04.463526 688 ts_tablet_manager.cc:1018] T 041f375b4ec14b1c9cec5fe2a1503b98 P 15558bb7548e42789ec21ac818f1b4b8: Bootstrapping tablet I0312 20:10:04.464175 688 tablet_bootstrap.cc:436] T 041f375b4ec14b1c9cec5fe2a1503b98 P 15558bb7548e42789ec21ac818f1b4b8: Bootstrap starting. I0312 20:10:04.464901 687 tablet_bootstrap.cc:581] T 041f375b4ec14b1c9cec5fe2a1503b98 P 2777b13979e24d25b4f50048a0a1ecb4: No blocks or log segments found. Creating new log. I0312 20:10:04.465829 354 ts_tablet_manager.cc:1163] T 041f375b4ec14b1c9cec5fe2a1503b98 P 90c2a1b6af6d40fa8b9148fb40ab6f90: Registered tablet (data state: TABLET_DATA_READY) I0312 20:10:04.467360 687 log.cc:520] T 041f375b4ec14b1c9cec5fe2a1503b98 P 2777b13979e24d25b4f50048a0a1ecb4: Log is configured to *not* fsync() on all Append() calls I0312 20:10:04.468870 688 tablet_bootstrap.cc:581] T 041f375b4ec14b1c9cec5fe2a1503b98 P 15558bb7548e42789ec21ac818f1b4b8: No blocks or log segments found. Creating new log. I0312 20:10:04.471092 688 log.cc:520] T 041f375b4ec14b1c9cec5fe2a1503b98 P 15558bb7548e42789ec21ac818f1b4b8: Log is configured to *not* fsync() on all Append() calls I0312 20:10:04.471565 689 ts_tablet_manager.cc:1018] T 041f375b4ec14b1c9cec5fe2a1503b98 P 90c2a1b6af6d40fa8b9148fb40ab6f90: Bootstrapping tablet I0312 20:10:04.471938 689 tablet_bootstrap.cc:436] T 041f375b4ec14b1c9cec5fe2a1503b98 P 90c2a1b6af6d40fa8b9148fb40ab6f90: Bootstrap starting. I0312 20:10:04.473517 687 tablet_bootstrap.cc:436] T 041f375b4ec14b1c9cec5fe2a1503b98 P 2777b13979e24d25b4f50048a0a1ecb4: No bootstrap required, opened a new log I0312 20:10:04.473966 687 ts_tablet_manager.cc:1035] T 041f375b4ec14b1c9cec5fe2a1503b98 P 2777b13979e24d25b4f50048a0a1ecb4: Time spent bootstrapping tablet: real 0.014s user 0.008s sys 0.004s I0312 20:10:04.477180 689 tablet_bootstrap.cc:581] T 041f375b4ec14b1c9cec5fe2a1503b98 P 90c2a1b6af6d40fa8b9148fb40ab6f90: No blocks or log segments found. Creating new log. I0312 20:10:04.478694 689 log.cc:520] T 041f375b4ec14b1c9cec5fe2a1503b98 P 90c2a1b6af6d40fa8b9148fb40ab6f90: Log is configured to *not* fsync() on all Append() calls I0312 20:10:04.481539 688 tablet_bootstrap.cc:436] T 041f375b4ec14b1c9cec5fe2a1503b98 P 15558bb7548e42789ec21ac818f1b4b8: No bootstrap required, opened a new log I0312 20:10:04.481835 688 ts_tablet_manager.cc:1035] T 041f375b4ec14b1c9cec5fe2a1503b98 P 15558bb7548e42789ec21ac818f1b4b8: Time spent bootstrapping tablet: real 0.018s user 0.008s sys 0.008s I0312 20:10:04.494706 688 raft_consensus.cc:304] T 041f375b4ec14b1c9cec5fe2a1503b98 P 15558bb7548e42789ec21ac818f1b4b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90c2a1b6af6d40fa8b9148fb40ab6f90" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 43100 } } peers { permanent_uuid: "15558bb7548e42789ec21ac818f1b4b8" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 33977 } } peers { permanent_uuid: "2777b13979e24d25b4f50048a0a1ecb4" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 59907 } } I0312 20:10:04.495256 688 raft_consensus.cc:330] T 041f375b4ec14b1c9cec5fe2a1503b98 P 15558bb7548e42789ec21ac818f1b4b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0312 20:10:04.495414 688 raft_consensus.cc:605] T 041f375b4ec14b1c9cec5fe2a1503b98 P 15558bb7548e42789ec21ac818f1b4b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15558bb7548e42789ec21ac818f1b4b8, State: Initialized, Role: FOLLOWER I0312 20:10:04.495471 689 tablet_bootstrap.cc:436] T 041f375b4ec14b1c9cec5fe2a1503b98 P 90c2a1b6af6d40fa8b9148fb40ab6f90: No bootstrap required, opened a new log I0312 20:10:04.495836 689 ts_tablet_manager.cc:1035] T 041f375b4ec14b1c9cec5fe2a1503b98 P 90c2a1b6af6d40fa8b9148fb40ab6f90: Time spent bootstrapping tablet: real 0.024s user 0.008s sys 0.012s I0312 20:10:04.495868 688 consensus_queue.cc:219] T 041f375b4ec14b1c9cec5fe2a1503b98 P 15558bb7548e42789ec21ac818f1b4b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90c2a1b6af6d40fa8b9148fb40ab6f90" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 43100 } } peers { permanent_uuid: "15558bb7548e42789ec21ac818f1b4b8" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 33977 } } peers { permanent_uuid: "2777b13979e24d25b4f50048a0a1ecb4" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 59907 } } I0312 20:10:04.498024 687 raft_consensus.cc:304] T 041f375b4ec14b1c9cec5fe2a1503b98 P 2777b13979e24d25b4f50048a0a1ecb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90c2a1b6af6d40fa8b9148fb40ab6f90" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 43100 } } peers { permanent_uuid: "15558bb7548e42789ec21ac818f1b4b8" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 33977 } } peers { permanent_uuid: "2777b13979e24d25b4f50048a0a1ecb4" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 59907 } } I0312 20:10:04.498770 687 raft_consensus.cc:330] T 041f375b4ec14b1c9cec5fe2a1503b98 P 2777b13979e24d25b4f50048a0a1ecb4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0312 20:10:04.498965 687 raft_consensus.cc:605] T 041f375b4ec14b1c9cec5fe2a1503b98 P 2777b13979e24d25b4f50048a0a1ecb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2777b13979e24d25b4f50048a0a1ecb4, State: Initialized, Role: FOLLOWER I0312 20:10:04.499594 687 consensus_queue.cc:219] T 041f375b4ec14b1c9cec5fe2a1503b98 P 2777b13979e24d25b4f50048a0a1ecb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90c2a1b6af6d40fa8b9148fb40ab6f90" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 43100 } } peers { permanent_uuid: "15558bb7548e42789ec21ac818f1b4b8" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 33977 } } peers { permanent_uuid: "2777b13979e24d25b4f50048a0a1ecb4" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 59907 } } I0312 20:10:04.505949 667 heartbeater.cc:467] Master 127.0.53.62:54194 was elected leader, sending a full tablet report... I0312 20:10:04.505944 542 heartbeater.cc:467] Master 127.0.53.62:54194 was elected leader, sending a full tablet report... I0312 20:10:04.507004 687 ts_tablet_manager.cc:1063] T 041f375b4ec14b1c9cec5fe2a1503b98 P 2777b13979e24d25b4f50048a0a1ecb4: Time spent starting tablet: real 0.033s user 0.016s sys 0.016s I0312 20:10:04.508463 688 ts_tablet_manager.cc:1063] T 041f375b4ec14b1c9cec5fe2a1503b98 P 15558bb7548e42789ec21ac818f1b4b8: Time spent starting tablet: real 0.026s user 0.012s sys 0.012s I0312 20:10:04.514456 689 raft_consensus.cc:304] T 041f375b4ec14b1c9cec5fe2a1503b98 P 90c2a1b6af6d40fa8b9148fb40ab6f90 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90c2a1b6af6d40fa8b9148fb40ab6f90" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 43100 } } peers { permanent_uuid: "15558bb7548e42789ec21ac818f1b4b8" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 33977 } } peers { permanent_uuid: "2777b13979e24d25b4f50048a0a1ecb4" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 59907 } } I0312 20:10:04.514983 689 raft_consensus.cc:330] T 041f375b4ec14b1c9cec5fe2a1503b98 P 90c2a1b6af6d40fa8b9148fb40ab6f90 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0312 20:10:04.515111 689 raft_consensus.cc:605] T 041f375b4ec14b1c9cec5fe2a1503b98 P 90c2a1b6af6d40fa8b9148fb40ab6f90 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90c2a1b6af6d40fa8b9148fb40ab6f90, State: Initialized, Role: FOLLOWER I0312 20:10:04.515483 689 consensus_queue.cc:219] T 041f375b4ec14b1c9cec5fe2a1503b98 P 90c2a1b6af6d40fa8b9148fb40ab6f90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90c2a1b6af6d40fa8b9148fb40ab6f90" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 43100 } } peers { permanent_uuid: "15558bb7548e42789ec21ac818f1b4b8" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 33977 } } peers { permanent_uuid: "2777b13979e24d25b4f50048a0a1ecb4" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 59907 } } I0312 20:10:04.517956 689 ts_tablet_manager.cc:1063] T 041f375b4ec14b1c9cec5fe2a1503b98 P 90c2a1b6af6d40fa8b9148fb40ab6f90: Time spent starting tablet: real 0.022s user 0.008s sys 0.016s W0312 20:10:04.540125 668 tablet.cc:1679] T 041f375b4ec14b1c9cec5fe2a1503b98 P 15558bb7548e42789ec21ac818f1b4b8: Can't schedule compaction. Clean time has not been advanced past its initial value. W0312 20:10:04.545809 543 tablet.cc:1679] T 041f375b4ec14b1c9cec5fe2a1503b98 P 2777b13979e24d25b4f50048a0a1ecb4: Can't schedule compaction. Clean time has not been advanced past its initial value. I0312 20:10:04.608572 693 raft_consensus.cc:436] T 041f375b4ec14b1c9cec5fe2a1503b98 P 15558bb7548e42789ec21ac818f1b4b8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0312 20:10:04.608791 693 raft_consensus.cc:2700] T 041f375b4ec14b1c9cec5fe2a1503b98 P 15558bb7548e42789ec21ac818f1b4b8: Snoozing failure detection for 1.909s (starting election) I0312 20:10:04.609005 693 raft_consensus.cc:458] T 041f375b4ec14b1c9cec5fe2a1503b98 P 15558bb7548e42789ec21ac818f1b4b8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90c2a1b6af6d40fa8b9148fb40ab6f90" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 43100 } } peers { permanent_uuid: "15558bb7548e42789ec21ac818f1b4b8" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 33977 } } peers { permanent_uuid: "2777b13979e24d25b4f50048a0a1ecb4" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 59907 } } I0312 20:10:04.610149 693 leader_election.cc:231] T 041f375b4ec14b1c9cec5fe2a1503b98 P 15558bb7548e42789ec21ac818f1b4b8 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 90c2a1b6af6d40fa8b9148fb40ab6f90 I0312 20:10:04.610615 693 leader_election.cc:231] T 041f375b4ec14b1c9cec5fe2a1503b98 P 15558bb7548e42789ec21ac818f1b4b8 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 2777b13979e24d25b4f50048a0a1ecb4 I0312 20:10:04.622818 374 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "041f375b4ec14b1c9cec5fe2a1503b98" candidate_uuid: "15558bb7548e42789ec21ac818f1b4b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90c2a1b6af6d40fa8b9148fb40ab6f90" is_pre_election: true I0312 20:10:04.623227 374 raft_consensus.cc:2700] T 041f375b4ec14b1c9cec5fe2a1503b98 P 90c2a1b6af6d40fa8b9148fb40ab6f90: Snoozing failure detection for 1.664s (vote granted) I0312 20:10:04.623430 374 raft_consensus.cc:2215] T 041f375b4ec14b1c9cec5fe2a1503b98 P 90c2a1b6af6d40fa8b9148fb40ab6f90 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 15558bb7548e42789ec21ac818f1b4b8 in term 0. I0312 20:10:04.624250 557 leader_election.cc:363] T 041f375b4ec14b1c9cec5fe2a1503b98 P 15558bb7548e42789ec21ac818f1b4b8 [CANDIDATE]: Term 1 pre-election: Vote granted by peer 90c2a1b6af6d40fa8b9148fb40ab6f90 I0312 20:10:04.624402 557 leader_election.cc:258] T 041f375b4ec14b1c9cec5fe2a1503b98 P 15558bb7548e42789ec21ac818f1b4b8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. I0312 20:10:04.624763 693 raft_consensus.cc:2700] T 041f375b4ec14b1c9cec5fe2a1503b98 P 15558bb7548e42789ec21ac818f1b4b8: Snoozing failure detection for 1.852s (election complete) I0312 20:10:04.624958 693 raft_consensus.cc:2507] T 041f375b4ec14b1c9cec5fe2a1503b98 P 15558bb7548e42789ec21ac818f1b4b8 [term 0 FOLLOWER]: Leader pre-election won for term 1 I0312 20:10:04.625160 693 raft_consensus.cc:436] T 041f375b4ec14b1c9cec5fe2a1503b98 P 15558bb7548e42789ec21ac818f1b4b8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0312 20:10:04.625339 693 raft_consensus.cc:2700] T 041f375b4ec14b1c9cec5fe2a1503b98 P 15558bb7548e42789ec21ac818f1b4b8: Snoozing failure detection for 1.841s (starting election) I0312 20:10:04.625499 693 raft_consensus.cc:2752] T 041f375b4ec14b1c9cec5fe2a1503b98 P 15558bb7548e42789ec21ac818f1b4b8 [term 0 FOLLOWER]: Advancing to term 1 I0312 20:10:04.628749 499 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "041f375b4ec14b1c9cec5fe2a1503b98" candidate_uuid: "15558bb7548e42789ec21ac818f1b4b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2777b13979e24d25b4f50048a0a1ecb4" is_pre_election: true I0312 20:10:04.630378 499 raft_consensus.cc:2700] T 041f375b4ec14b1c9cec5fe2a1503b98 P 2777b13979e24d25b4f50048a0a1ecb4: Snoozing failure detection for 1.548s (vote granted) I0312 20:10:04.630628 499 raft_consensus.cc:2215] T 041f375b4ec14b1c9cec5fe2a1503b98 P 2777b13979e24d25b4f50048a0a1ecb4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 15558bb7548e42789ec21ac818f1b4b8 in term 0. I0312 20:10:04.631209 693 raft_consensus.cc:458] T 041f375b4ec14b1c9cec5fe2a1503b98 P 15558bb7548e42789ec21ac818f1b4b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90c2a1b6af6d40fa8b9148fb40ab6f90" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 43100 } } peers { permanent_uuid: "15558bb7548e42789ec21ac818f1b4b8" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 33977 } } peers { permanent_uuid: "2777b13979e24d25b4f50048a0a1ecb4" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 59907 } } I0312 20:10:04.631764 559 leader_election.cc:363] T 041f375b4ec14b1c9cec5fe2a1503b98 P 15558bb7548e42789ec21ac818f1b4b8 [CANDIDATE]: Term 1 pre-election: Vote granted by peer 2777b13979e24d25b4f50048a0a1ecb4 I0312 20:10:04.631956 693 leader_election.cc:231] T 041f375b4ec14b1c9cec5fe2a1503b98 P 15558bb7548e42789ec21ac818f1b4b8 [CANDIDATE]: Term 1 election: Requesting vote from peer 90c2a1b6af6d40fa8b9148fb40ab6f90 I0312 20:10:04.632272 693 leader_election.cc:231] T 041f375b4ec14b1c9cec5fe2a1503b98 P 15558bb7548e42789ec21ac818f1b4b8 [CANDIDATE]: Term 1 election: Requesting vote from peer 2777b13979e24d25b4f50048a0a1ecb4 I0312 20:10:04.632984 374 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "041f375b4ec14b1c9cec5fe2a1503b98" candidate_uuid: "15558bb7548e42789ec21ac818f1b4b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90c2a1b6af6d40fa8b9148fb40ab6f90" I0312 20:10:04.633337 374 raft_consensus.cc:2752] T 041f375b4ec14b1c9cec5fe2a1503b98 P 90c2a1b6af6d40fa8b9148fb40ab6f90 [term 0 FOLLOWER]: Advancing to term 1 I0312 20:10:04.633538 374 raft_consensus.cc:2700] T 041f375b4ec14b1c9cec5fe2a1503b98 P 90c2a1b6af6d40fa8b9148fb40ab6f90: Snoozing failure detection for 2.018s (vote granted) I0312 20:10:04.633560 499 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "041f375b4ec14b1c9cec5fe2a1503b98" candidate_uuid: "15558bb7548e42789ec21ac818f1b4b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2777b13979e24d25b4f50048a0a1ecb4" I0312 20:10:04.633954 499 raft_consensus.cc:2752] T 041f375b4ec14b1c9cec5fe2a1503b98 P 2777b13979e24d25b4f50048a0a1ecb4 [term 0 FOLLOWER]: Advancing to term 1 I0312 20:10:04.634186 499 raft_consensus.cc:2700] T 041f375b4ec14b1c9cec5fe2a1503b98 P 2777b13979e24d25b4f50048a0a1ecb4: Snoozing failure detection for 1.537s (vote granted) I0312 20:10:04.637239 374 raft_consensus.cc:2215] T 041f375b4ec14b1c9cec5fe2a1503b98 P 90c2a1b6af6d40fa8b9148fb40ab6f90 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 15558bb7548e42789ec21ac818f1b4b8 in term 1. I0312 20:10:04.638947 557 leader_election.cc:363] T 041f375b4ec14b1c9cec5fe2a1503b98 P 15558bb7548e42789ec21ac818f1b4b8 [CANDIDATE]: Term 1 election: Vote granted by peer 90c2a1b6af6d40fa8b9148fb40ab6f90 I0312 20:10:04.639117 557 leader_election.cc:258] T 041f375b4ec14b1c9cec5fe2a1503b98 P 15558bb7548e42789ec21ac818f1b4b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0312 20:10:04.639430 693 raft_consensus.cc:2700] T 041f375b4ec14b1c9cec5fe2a1503b98 P 15558bb7548e42789ec21ac818f1b4b8: Snoozing failure detection for 1.506s (election complete) I0312 20:10:04.639605 693 raft_consensus.cc:2507] T 041f375b4ec14b1c9cec5fe2a1503b98 P 15558bb7548e42789ec21ac818f1b4b8 [term 1 FOLLOWER]: Leader election won for term 1 I0312 20:10:04.639883 499 raft_consensus.cc:2215] T 041f375b4ec14b1c9cec5fe2a1503b98 P 2777b13979e24d25b4f50048a0a1ecb4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 15558bb7548e42789ec21ac818f1b4b8 in term 1. I0312 20:10:04.640683 559 leader_election.cc:363] T 041f375b4ec14b1c9cec5fe2a1503b98 P 15558bb7548e42789ec21ac818f1b4b8 [CANDIDATE]: Term 1 election: Vote granted by peer 2777b13979e24d25b4f50048a0a1ecb4 I0312 20:10:04.642279 693 raft_consensus.cc:572] T 041f375b4ec14b1c9cec5fe2a1503b98 P 15558bb7548e42789ec21ac818f1b4b8 [term 1 LEADER]: Becoming Leader. State: Replica: 15558bb7548e42789ec21ac818f1b4b8, State: Running, Role: LEADER I0312 20:10:04.642725 693 consensus_queue.cc:196] T 041f375b4ec14b1c9cec5fe2a1503b98 P 15558bb7548e42789ec21ac818f1b4b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90c2a1b6af6d40fa8b9148fb40ab6f90" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 43100 } } peers { permanent_uuid: "15558bb7548e42789ec21ac818f1b4b8" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 33977 } } peers { permanent_uuid: "2777b13979e24d25b4f50048a0a1ecb4" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 59907 } } I0312 20:10:04.648891 252 catalog_manager.cc:3592] T 041f375b4ec14b1c9cec5fe2a1503b98 P 15558bb7548e42789ec21ac818f1b4b8 reported cstate change: term changed from 0 to 1, leader changed from to 15558bb7548e42789ec21ac818f1b4b8 (127.0.53.3). New cstate: current_term: 1 leader_uuid: "15558bb7548e42789ec21ac818f1b4b8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90c2a1b6af6d40fa8b9148fb40ab6f90" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 43100 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "15558bb7548e42789ec21ac818f1b4b8" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 33977 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2777b13979e24d25b4f50048a0a1ecb4" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 59907 } health_report { overall_health: UNKNOWN } } } W0312 20:10:04.649924 418 tablet.cc:1679] T 041f375b4ec14b1c9cec5fe2a1503b98 P 90c2a1b6af6d40fa8b9148fb40ab6f90: Can't schedule compaction. Clean time has not been advanced past its initial value. I0312 20:10:04.671703 212 external_mini_cluster.cc:498] 3 TS(s) registered with all masters I0312 20:10:04.673851 212 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 90c2a1b6af6d40fa8b9148fb40ab6f90 to finish bootstrapping I0312 20:10:04.682778 212 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 2777b13979e24d25b4f50048a0a1ecb4 to finish bootstrapping I0312 20:10:04.691844 212 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 15558bb7548e42789ec21ac818f1b4b8 to finish bootstrapping I0312 20:10:04.730600 499 raft_consensus.cc:1058] T 041f375b4ec14b1c9cec5fe2a1503b98 P 2777b13979e24d25b4f50048a0a1ecb4 [term 1 FOLLOWER]: Refusing update from remote peer 15558bb7548e42789ec21ac818f1b4b8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0312 20:10:04.731600 374 raft_consensus.cc:1058] T 041f375b4ec14b1c9cec5fe2a1503b98 P 90c2a1b6af6d40fa8b9148fb40ab6f90 [term 1 FOLLOWER]: Refusing update from remote peer 15558bb7548e42789ec21ac818f1b4b8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0312 20:10:04.731853 697 consensus_queue.cc:1017] T 041f375b4ec14b1c9cec5fe2a1503b98 P 15558bb7548e42789ec21ac818f1b4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2777b13979e24d25b4f50048a0a1ecb4" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 59907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0312 20:10:04.732806 693 consensus_queue.cc:1017] T 041f375b4ec14b1c9cec5fe2a1503b98 P 15558bb7548e42789ec21ac818f1b4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90c2a1b6af6d40fa8b9148fb40ab6f90" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 43100 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s Connected to the Master Fetched info from all 3 Tablet Servers Table TestTable is HEALTHY (1 tablet(s) checked) Table Summary Name | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable -----------+---------+---------------+---------+------------+------------------+------------- TestTable | HEALTHY | 1 | 1 | 0 | 0 | 0 The metadata for 1 table(s) is HEALTHY Checksum finished in 0s: 0/3 replicas remaining (0B from disk, 150 rows summed) ----------------------- TestTable ----------------------- T 041f375b4ec14b1c9cec5fe2a1503b98 P 2777b13979e24d25b4f50048a0a1ecb4 (127.0.53.2:59907): Checksum: 108787617684 T 041f375b4ec14b1c9cec5fe2a1503b98 P 15558bb7548e42789ec21ac818f1b4b8 (127.0.53.3:33977): Checksum: 108787617684 T 041f375b4ec14b1c9cec5fe2a1503b98 P 90c2a1b6af6d40fa8b9148fb40ab6f90 (127.0.53.1:43100): Checksum: 108787617684 I0312 20:10:04.952978 212 log_verifier.cc:121] Checking tablet 041f375b4ec14b1c9cec5fe2a1503b98 I0312 20:10:05.077141 212 log_verifier.cc:172] Verified matching terms for 6 ops in tablet 041f375b4ec14b1c9cec5fe2a1503b98 I0312 20:10:05.299046 212 external_mini_cluster.cc:997] Killing /tmp/dist-test-task3bwPeE/build/tsan/bin/kudu-tserver with pid 546 W0312 20:10:05.327541 235 connection.cc:513] client connection to 127.0.53.3:33977 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0312 20:10:05.328618 237 connection.cc:513] server connection from 127.0.53.3:43785 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0312 20:10:05.339395 306 connection.cc:513] server connection from 127.0.53.3:53446 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0312 20:10:05.340399 430 connection.cc:513] server connection from 127.0.53.3:52486 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) I0312 20:10:05.343243 499 tablet_service.cc:1105] Received Run Leader Election RPC: tablet_id: "041f375b4ec14b1c9cec5fe2a1503b98" dest_uuid: "2777b13979e24d25b4f50048a0a1ecb4" from {username='slave'} at 127.0.0.1:51461 I0312 20:10:05.343641 499 raft_consensus.cc:436] T 041f375b4ec14b1c9cec5fe2a1503b98 P 2777b13979e24d25b4f50048a0a1ecb4 [term 1 FOLLOWER]: Starting forced leader election (received explicit request) I0312 20:10:05.343843 499 raft_consensus.cc:2700] T 041f375b4ec14b1c9cec5fe2a1503b98 P 2777b13979e24d25b4f50048a0a1ecb4: Snoozing failure detection for 1.691s (starting election) I0312 20:10:05.344040 499 raft_consensus.cc:2752] T 041f375b4ec14b1c9cec5fe2a1503b98 P 2777b13979e24d25b4f50048a0a1ecb4 [term 1 FOLLOWER]: Advancing to term 2 I0312 20:10:05.348438 499 raft_consensus.cc:458] T 041f375b4ec14b1c9cec5fe2a1503b98 P 2777b13979e24d25b4f50048a0a1ecb4 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90c2a1b6af6d40fa8b9148fb40ab6f90" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 43100 } } peers { permanent_uuid: "15558bb7548e42789ec21ac818f1b4b8" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 33977 } } peers { permanent_uuid: "2777b13979e24d25b4f50048a0a1ecb4" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 59907 } } I0312 20:10:05.349933 499 leader_election.cc:231] T 041f375b4ec14b1c9cec5fe2a1503b98 P 2777b13979e24d25b4f50048a0a1ecb4 [CANDIDATE]: Term 2 election: Requesting vote from peer 90c2a1b6af6d40fa8b9148fb40ab6f90 I0312 20:10:05.350469 499 leader_election.cc:231] T 041f375b4ec14b1c9cec5fe2a1503b98 P 2777b13979e24d25b4f50048a0a1ecb4 [CANDIDATE]: Term 2 election: Requesting vote from peer 15558bb7548e42789ec21ac818f1b4b8 W0312 20:10:05.329463 223 connection.cc:513] client connection to 127.0.53.3:33977 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0312 20:10:05.401542 432 leader_election.cc:287] T 041f375b4ec14b1c9cec5fe2a1503b98 P 2777b13979e24d25b4f50048a0a1ecb4 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 15558bb7548e42789ec21ac818f1b4b8: Network error: Client connection negotiation failed: client connection to 127.0.53.3:33977: connect: Connection refused (error 111) I0312 20:10:05.402734 374 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "041f375b4ec14b1c9cec5fe2a1503b98" candidate_uuid: "2777b13979e24d25b4f50048a0a1ecb4" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "90c2a1b6af6d40fa8b9148fb40ab6f90" I0312 20:10:05.403174 374 raft_consensus.cc:2752] T 041f375b4ec14b1c9cec5fe2a1503b98 P 90c2a1b6af6d40fa8b9148fb40ab6f90 [term 1 FOLLOWER]: Advancing to term 2 I0312 20:10:05.403406 374 raft_consensus.cc:2700] T 041f375b4ec14b1c9cec5fe2a1503b98 P 90c2a1b6af6d40fa8b9148fb40ab6f90: Snoozing failure detection for 1.661s (vote granted) I0312 20:10:05.407501 374 raft_consensus.cc:2215] T 041f375b4ec14b1c9cec5fe2a1503b98 P 90c2a1b6af6d40fa8b9148fb40ab6f90 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2777b13979e24d25b4f50048a0a1ecb4 in term 2. I0312 20:10:05.408628 432 leader_election.cc:363] T 041f375b4ec14b1c9cec5fe2a1503b98 P 2777b13979e24d25b4f50048a0a1ecb4 [CANDIDATE]: Term 2 election: Vote granted by peer 90c2a1b6af6d40fa8b9148fb40ab6f90 I0312 20:10:05.408849 432 leader_election.cc:258] T 041f375b4ec14b1c9cec5fe2a1503b98 P 2777b13979e24d25b4f50048a0a1ecb4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. I0312 20:10:05.417708 742 raft_consensus.cc:2700] T 041f375b4ec14b1c9cec5fe2a1503b98 P 2777b13979e24d25b4f50048a0a1ecb4: Snoozing failure detection for 2.218s (election complete) I0312 20:10:05.417994 742 raft_consensus.cc:2507] T 041f375b4ec14b1c9cec5fe2a1503b98 P 2777b13979e24d25b4f50048a0a1ecb4 [term 2 FOLLOWER]: Leader election won for term 2 I0312 20:10:05.423002 742 raft_consensus.cc:572] T 041f375b4ec14b1c9cec5fe2a1503b98 P 2777b13979e24d25b4f50048a0a1ecb4 [term 2 LEADER]: Becoming Leader. State: Replica: 2777b13979e24d25b4f50048a0a1ecb4, State: Running, Role: LEADER I0312 20:10:05.423516 742 consensus_queue.cc:196] T 041f375b4ec14b1c9cec5fe2a1503b98 P 2777b13979e24d25b4f50048a0a1ecb4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90c2a1b6af6d40fa8b9148fb40ab6f90" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 43100 } } peers { permanent_uuid: "15558bb7548e42789ec21ac818f1b4b8" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 33977 } } peers { permanent_uuid: "2777b13979e24d25b4f50048a0a1ecb4" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 59907 } } I0312 20:10:05.428375 252 catalog_manager.cc:3592] T 041f375b4ec14b1c9cec5fe2a1503b98 P 2777b13979e24d25b4f50048a0a1ecb4 reported cstate change: term changed from 1 to 2, leader changed from 15558bb7548e42789ec21ac818f1b4b8 (127.0.53.3) to 2777b13979e24d25b4f50048a0a1ecb4 (127.0.53.2). New cstate: current_term: 2 leader_uuid: "2777b13979e24d25b4f50048a0a1ecb4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90c2a1b6af6d40fa8b9148fb40ab6f90" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 43100 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "15558bb7548e42789ec21ac818f1b4b8" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 33977 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2777b13979e24d25b4f50048a0a1ecb4" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 59907 } health_report { overall_health: HEALTHY } } } W0312 20:10:05.341357 674 connection.cc:513] client connection to 127.0.53.3:33977 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0312 20:10:05.341687 220 connection.cc:513] client connection to 127.0.53.3:33977 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0312 20:10:05.483992 674 meta_cache.cc:218] Tablet 041f375b4ec14b1c9cec5fe2a1503b98: Replica 15558bb7548e42789ec21ac818f1b4b8 (127.0.53.3:33977) has failed: Network error: Client connection negotiation failed: client connection to 127.0.53.3:33977: connect: Connection refused (error 111) W0312 20:10:05.548449 432 consensus_peers.cc:436] T 041f375b4ec14b1c9cec5fe2a1503b98 P 2777b13979e24d25b4f50048a0a1ecb4 -> Peer 15558bb7548e42789ec21ac818f1b4b8 (127.0.53.3:33977): Couldn't send request to peer 15558bb7548e42789ec21ac818f1b4b8 for tablet 041f375b4ec14b1c9cec5fe2a1503b98. Status: Network error: Client connection negotiation failed: client connection to 127.0.53.3:33977: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 1 times. I0312 20:10:05.550302 374 raft_consensus.cc:1058] T 041f375b4ec14b1c9cec5fe2a1503b98 P 90c2a1b6af6d40fa8b9148fb40ab6f90 [term 2 FOLLOWER]: Refusing update from remote peer 2777b13979e24d25b4f50048a0a1ecb4: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 8. (index mismatch) I0312 20:10:05.551718 742 consensus_queue.cc:1017] T 041f375b4ec14b1c9cec5fe2a1503b98 P 2777b13979e24d25b4f50048a0a1ecb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90c2a1b6af6d40fa8b9148fb40ab6f90" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 43100 }, Status: LMP_MISMATCH, Last received: 1.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s I0312 20:10:05.660923 212 external_mini_cluster.cc:805] Running /tmp/dist-test-task3bwPeE/build/tsan/bin/kudu-tserver /tmp/dist-test-task3bwPeE/build/tsan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.RunLeaderElection.1520885401735786-212/raft_consensus-itest-cluster/ts-2/wal --fs_data_dirs=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.RunLeaderElection.1520885401735786-212/raft_consensus-itest-cluster/ts-2/data --block_manager=log --rpc_bind_addresses=127.0.53.3:33977 --local_ip_for_outbound_sockets=127.0.53.3 --webserver_port=39202 --webserver_interface=127.0.53.3 --tserver_master_addrs=127.0.53.62:54194 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.RunLeaderElection.1520885401735786-212/raft_consensus-itest-cluster/ts-2/logs --server_dump_info_path=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.RunLeaderElection.1520885401735786-212/raft_consensus-itest-cluster/ts-2/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --log_cache_size_limit_mb=10 --consensus_rpc_timeout_ms=1000 with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W0312 20:10:05.878998 756 flags.cc:412] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0312 20:10:05.879482 756 flags.cc:412] Enabled unsafe flag: --never_fsync=true W0312 20:10:05.902765 756 flags.cc:412] Enabled experimental flag: --ipki_server_key_size=1024 W0312 20:10:05.903214 756 flags.cc:412] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.53.3 I0312 20:10:05.949389 756 tablet_server_main.cc:72] Tablet server non-default flags: --consensus_rpc_timeout_ms=1000 --log_cache_size_limit_mb=10 --fs_data_dirs=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.RunLeaderElection.1520885401735786-212/raft_consensus-itest-cluster/ts-2/data --fs_wal_dir=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.RunLeaderElection.1520885401735786-212/raft_consensus-itest-cluster/ts-2/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.0.53.3:33977 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.RunLeaderElection.1520885401735786-212/raft_consensus-itest-cluster/ts-2/data/info.pb --webserver_interface=127.0.53.3 --webserver_port=39202 --tserver_master_addrs=127.0.53.62:54194 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.0.53.3 --log_dir=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.RunLeaderElection.1520885401735786-212/raft_consensus-itest-cluster/ts-2/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.7.0-SNAPSHOT revision 13e10c63c46de30ac85f34b543cdd6fb44881f61 build type FASTDEBUG built by None at 12 Mar 2018 20:06:40 UTC on jenkins-slave-vn0g build id 12388 TSAN enabled I0312 20:10:05.985438 756 tablet_server_main.cc:79] Initializing tablet server... I0312 20:10:05.999120 756 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 1069776us I0312 20:10:05.999719 756 fs_manager.cc:260] Metadata directory not provided I0312 20:10:05.999899 756 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0312 20:10:06.017515 756 fs_manager.cc:406] Time spent opening directory manager: real 0.014s user 0.004s sys 0.000s I0312 20:10:06.017848 756 env_posix.cc:1620] Not raising this process' open files per process limit of 65536; it is already as high as it can go I0312 20:10:06.018229 756 file_cache.cc:470] Constructed file cache lbm with capacity 26214 I0312 20:10:06.037940 756 fs_manager.cc:418] Time spent opening block manager: real 0.011s user 0.000s sys 0.000s I0312 20:10:06.038095 756 fs_manager.cc:429] Opened local filesystem: /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.RunLeaderElection.1520885401735786-212/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.RunLeaderElection.1520885401735786-212/raft_consensus-itest-cluster/ts-2/wal uuid: "15558bb7548e42789ec21ac818f1b4b8" format_stamp: "Formatted at 2018-03-12 20:10:03 on dist-test-slave-dist-test-slave-f27p" I0312 20:10:06.039794 756 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.RunLeaderElection.1520885401735786-212/raft_consensus-itest-cluster/ts-2/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) W0312 20:10:06.120198 432 consensus_peers.cc:436] T 041f375b4ec14b1c9cec5fe2a1503b98 P 2777b13979e24d25b4f50048a0a1ecb4 -> Peer 15558bb7548e42789ec21ac818f1b4b8 (127.0.53.3:33977): Couldn't send request to peer 15558bb7548e42789ec21ac818f1b4b8 for tablet 041f375b4ec14b1c9cec5fe2a1503b98. Status: Network error: Client connection negotiation failed: client connection to 127.0.53.3:33977: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 2 times. I0312 20:10:06.550180 756 env_posix.cc:1620] Not raising this process' running threads per effective uid limit of 60284; it is already as high as it can go I0312 20:10:06.552062 756 ts_tablet_manager.cc:337] Loading tablet metadata (0/1 complete) I0312 20:10:06.552275 756 ts_tablet_manager.cc:995] T 041f375b4ec14b1c9cec5fe2a1503b98 P 15558bb7548e42789ec21ac818f1b4b8: Loading tablet metadata I0312 20:10:06.558619 756 ts_tablet_manager.cc:349] Loaded tablet metadata (1 live tablets) I0312 20:10:06.563237 756 ts_tablet_manager.cc:1163] T 041f375b4ec14b1c9cec5fe2a1503b98 P 15558bb7548e42789ec21ac818f1b4b8: Registered tablet (data state: TABLET_DATA_READY) I0312 20:10:06.577705 774 ts_tablet_manager.cc:1018] T 041f375b4ec14b1c9cec5fe2a1503b98 P 15558bb7548e42789ec21ac818f1b4b8: Bootstrapping tablet I0312 20:10:06.579066 774 tablet_bootstrap.cc:436] T 041f375b4ec14b1c9cec5fe2a1503b98 P 15558bb7548e42789ec21ac818f1b4b8: Bootstrap starting. I0312 20:10:06.612620 774 log.cc:520] T 041f375b4ec14b1c9cec5fe2a1503b98 P 15558bb7548e42789ec21ac818f1b4b8: Log is configured to *not* fsync() on all Append() calls W0312 20:10:06.633725 432 consensus_peers.cc:436] T 041f375b4ec14b1c9cec5fe2a1503b98 P 2777b13979e24d25b4f50048a0a1ecb4 -> Peer 15558bb7548e42789ec21ac818f1b4b8 (127.0.53.3:33977): Couldn't send request to peer 15558bb7548e42789ec21ac818f1b4b8 for tablet 041f375b4ec14b1c9cec5fe2a1503b98. Status: Network error: Client connection negotiation failed: client connection to 127.0.53.3:33977: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 3 times. I0312 20:10:06.589632 756 tablet_server_main.cc:84] Starting tablet server... I0312 20:10:06.693248 774 tablet_bootstrap.cc:436] T 041f375b4ec14b1c9cec5fe2a1503b98 P 15558bb7548e42789ec21ac818f1b4b8: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=50 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I0312 20:10:06.695382 774 tablet_bootstrap.cc:436] T 041f375b4ec14b1c9cec5fe2a1503b98 P 15558bb7548e42789ec21ac818f1b4b8: Bootstrap complete. I0312 20:10:06.696756 774 ts_tablet_manager.cc:1035] T 041f375b4ec14b1c9cec5fe2a1503b98 P 15558bb7548e42789ec21ac818f1b4b8: Time spent bootstrapping tablet: real 0.118s user 0.048s sys 0.016s I0312 20:10:06.725759 774 raft_consensus.cc:304] T 041f375b4ec14b1c9cec5fe2a1503b98 P 15558bb7548e42789ec21ac818f1b4b8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90c2a1b6af6d40fa8b9148fb40ab6f90" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 43100 } } peers { permanent_uuid: "15558bb7548e42789ec21ac818f1b4b8" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 33977 } } peers { permanent_uuid: "2777b13979e24d25b4f50048a0a1ecb4" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 59907 } } I0312 20:10:06.733827 774 raft_consensus.cc:605] T 041f375b4ec14b1c9cec5fe2a1503b98 P 15558bb7548e42789ec21ac818f1b4b8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15558bb7548e42789ec21ac818f1b4b8, State: Initialized, Role: FOLLOWER I0312 20:10:06.734643 774 consensus_queue.cc:219] T 041f375b4ec14b1c9cec5fe2a1503b98 P 15558bb7548e42789ec21ac818f1b4b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90c2a1b6af6d40fa8b9148fb40ab6f90" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 43100 } } peers { permanent_uuid: "15558bb7548e42789ec21ac818f1b4b8" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 33977 } } peers { permanent_uuid: "2777b13979e24d25b4f50048a0a1ecb4" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 59907 } } I0312 20:10:06.737726 774 ts_tablet_manager.cc:1063] T 041f375b4ec14b1c9cec5fe2a1503b98 P 15558bb7548e42789ec21ac818f1b4b8: Time spent starting tablet: real 0.041s user 0.032s sys 0.000s I0312 20:10:06.904320 756 rpc_server.cc:201] RPC server started. Bound to: 127.0.53.3:33977 I0312 20:10:06.905295 756 webserver.cc:173] Starting webserver on 127.0.53.3:39202 I0312 20:10:06.905453 756 webserver.cc:184] Document root disabled I0312 20:10:06.906838 756 webserver.cc:311] Webserver started. Bound to: http://127.0.53.3:39202/ I0312 20:10:06.907567 756 server_base.cc:607] Dumped server information to /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.RunLeaderElection.1520885401735786-212/raft_consensus-itest-cluster/ts-2/data/info.pb I0312 20:10:06.910789 756 tablet_server_main.cc:87] Tablet server successfully started. I0312 20:10:06.913872 883 process_memory.cc:182] Process hard memory limit is 11.777631 GB I0312 20:10:06.914005 883 process_memory.cc:184] Process soft memory limit is 9.422105 GB I0312 20:10:06.913916 212 external_mini_cluster.cc:867] Started /tmp/dist-test-task3bwPeE/build/tsan/bin/kudu-tserver as pid 756 I0312 20:10:06.914095 883 process_memory.cc:187] Process memory pressure threshold is 7.066578 GB I0312 20:10:06.928454 882 heartbeater.cc:346] Connected to a master server at 127.0.53.62:54194 I0312 20:10:06.928696 882 heartbeater.cc:426] Registering TS with master... I0312 20:10:06.929247 882 heartbeater.cc:475] Master 127.0.53.62:54194 requested a full tablet report, sending... I0312 20:10:06.931313 252 ts_manager.cc:87] Re-registered known tserver with Master: 15558bb7548e42789ec21ac818f1b4b8 (127.0.53.3:33977) I0312 20:10:06.933315 252 master_service.cc:243] Signed X509 certificate for tserver {username='slave'} at 127.0.53.3:60731 Connected to the Master Fetched info from all 3 Tablet Servers Tablet 041f375b4ec14b1c9cec5fe2a1503b98 of table 'TestTable' is conflicted: Tablet 041f375b4ec14b1c9cec5fe2a1503b98 of table 'TestTable' replicas' active configs disagree with the master's 15558bb7548e42789ec21ac818f1b4b8 (127.0.53.3:33977): RUNNING 2777b13979e24d25b4f50048a0a1ecb4 (127.0.53.2:59907): RUNNING [LEADER] 90c2a1b6af6d40fa8b9148fb40ab6f90 (127.0.53.1:43100): RUNNING All the peers reported by the master and tablet servers are: A = 15558bb7548e42789ec21ac818f1b4b8 B = 2777b13979e24d25b4f50048a0a1ecb4 C = 90c2a1b6af6d40fa8b9148fb40ab6f90 The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+--------------+--------------+--------------+------------ master | A B* C | | | Yes A | A B C | 1 | -1 | Yes B | A B* C | 2 | -1 | Yes C | A B* C | 2 | -1 | Yes Table TestTable has 1 tablet(s) with mismatched consensus Table Summary Name | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable -----------+-------------+---------------+---------+------------+------------------+------------- TestTable | UNAVAILABLE | 1 | 0 | 0 | 0 | 1 I0312 20:10:07.030138 212 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Corruption: 1 out of 1 table(s) are not healthy I0312 20:10:07.116642 839 raft_consensus.cc:2752] T 041f375b4ec14b1c9cec5fe2a1503b98 P 15558bb7548e42789ec21ac818f1b4b8 [term 1 FOLLOWER]: Advancing to term 2 I0312 20:10:07.120609 882 heartbeater.cc:467] Master 127.0.53.62:54194 was elected leader, sending a full tablet report... Connected to the Master Fetched info from all 3 Tablet Servers Table TestTable is HEALTHY (1 tablet(s) checked) Table Summary Name | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable -----------+---------+---------------+---------+------------+------------------+------------- TestTable | HEALTHY | 1 | 1 | 0 | 0 | 0 The metadata for 1 table(s) is HEALTHY Checksum finished in 0s: 0/3 replicas remaining (0B from disk, 300 rows summed) ----------------------- TestTable ----------------------- T 041f375b4ec14b1c9cec5fe2a1503b98 P 2777b13979e24d25b4f50048a0a1ecb4 (127.0.53.2:59907): Checksum: 214634478652 T 041f375b4ec14b1c9cec5fe2a1503b98 P 15558bb7548e42789ec21ac818f1b4b8 (127.0.53.3:33977): Checksum: 214634478652 T 041f375b4ec14b1c9cec5fe2a1503b98 P 90c2a1b6af6d40fa8b9148fb40ab6f90 (127.0.53.1:43100): Checksum: 214634478652 I0312 20:10:07.287250 212 log_verifier.cc:121] Checking tablet 041f375b4ec14b1c9cec5fe2a1503b98 I0312 20:10:07.362407 212 log_verifier.cc:172] Verified matching terms for 12 ops in tablet 041f375b4ec14b1c9cec5fe2a1503b98 ================== WARNING: ThreadSanitizer: data race (pid=212) Write of size 8 at 0x7b2c00001768 by main thread: #0 operator delete(void*) /home/jenkins-slave/workspace/kudu-master/2/thirdparty/src/llvm-4.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_new_delete.cc:73 (raft_consensus_election-itest+0x4c6761) #1 std::__1::default_delete::operator()(kudu::ThreadPoolToken*) const /home/jenkins-slave/workspace/kudu-master/2/thirdparty/installed/tsan/include/c++/v1/memory:2397:13 (libkudu_util.so+0x1cd293) #2 std::__1::unique_ptr >::reset(kudu::ThreadPoolToken*) /home/jenkins-slave/workspace/kudu-master/2/thirdparty/installed/tsan/include/c++/v1/memory:2603 (libkudu_util.so+0x1cd293) #3 std::__1::unique_ptr >::~unique_ptr() /home/jenkins-slave/workspace/kudu-master/2/thirdparty/installed/tsan/include/c++/v1/memory:2571 (libkudu_util.so+0x1cd293) #4 kudu::ThreadPool::~ThreadPool() /home/jenkins-slave/workspace/kudu-master/2/src/kudu/util/threadpool.cc:342 (libkudu_util.so+0x1cd293) #5 kudu::DefaultDeleter::operator()(kudu::ThreadPool*) const /home/jenkins-slave/workspace/kudu-master/2/src/kudu/gutil/gscoped_ptr.h:145:5 (libmaster.so+0xce6de) #6 kudu::internal::gscoped_ptr_impl >::~gscoped_ptr_impl() /home/jenkins-slave/workspace/kudu-master/2/src/kudu/gutil/gscoped_ptr.h:228:7 (libmaster.so+0xce6a9) #7 gscoped_ptr >::~gscoped_ptr() /home/jenkins-slave/workspace/kudu-master/2/src/kudu/gutil/gscoped_ptr.h:318:7 (libmaster.so+0xbce39) #8 kudu::DnsResolver::~DnsResolver() /home/jenkins-slave/workspace/kudu-master/2/src/kudu/util/net/dns_resolver.cc:47:1 (libkudu_util.so+0x19097a) #9 kudu::DefaultDeleter::operator()(kudu::DnsResolver*) const /home/jenkins-slave/workspace/kudu-master/2/src/kudu/gutil/gscoped_ptr.h:145:5 (libkudu_client.so+0xd337e) #10 kudu::internal::gscoped_ptr_impl >::reset(kudu::DnsResolver*) /home/jenkins-slave/workspace/kudu-master/2/src/kudu/gutil/gscoped_ptr.h:254:7 (libkudu_client.so+0xd3334) #11 gscoped_ptr >::reset(kudu::DnsResolver*) /home/jenkins-slave/workspace/kudu-master/2/src/kudu/gutil/gscoped_ptr.h:375:46 (libkudu_client.so+0xc8130) #12 kudu::client::KuduClient::Data::~Data() /home/jenkins-slave/workspace/kudu-master/2/src/kudu/client/client-internal.cc:356:17 (libkudu_client.so+0xd8f5d) #13 kudu::client::KuduClient::~KuduClient() /home/jenkins-slave/workspace/kudu-master/2/src/kudu/client/client.cc:359:3 (libkudu_client.so+0xbde0c) #14 std::__1::default_delete::operator()(kudu::client::KuduClient*) const /home/jenkins-slave/workspace/kudu-master/2/thirdparty/installed/tsan/include/c++/v1/memory:2397:13 (libkudu_client.so+0xd2c4b) #15 std::__1::__shared_ptr_pointer, std::__1::allocator >::__on_zero_shared() /home/jenkins-slave/workspace/kudu-master/2/thirdparty/installed/tsan/include/c++/v1/memory:3795 (libkudu_client.so+0xd2c4b) #16 __release_shared /home/jenkins-slave/workspace/kudu-master/2/thirdparty/src/llvm-4.0.0.src/projects/libcxx/src/memory.cpp:67:9 (libc++.so.1+0xc095d) #17 std::__1::__shared_weak_count::__release_shared() /home/jenkins-slave/workspace/kudu-master/2/thirdparty/src/llvm-4.0.0.src/projects/libcxx/src/memory.cpp:92 (libc++.so.1+0xc095d) #18 std::__1::shared_ptr::~shared_ptr() /home/jenkins-slave/workspace/kudu-master/2/thirdparty/installed/tsan/include/c++/v1/memory:4626:19 (raft_consensus_election-itest+0x4d5088) #19 kudu::tserver::TabletServerIntegrationTestBase::~TabletServerIntegrationTestBase() /home/jenkins-slave/workspace/kudu-master/2/src/kudu/integration-tests/ts_itest-base.h:46:7 (raft_consensus_election-itest+0x4d4fcf) #20 kudu::tserver::RaftConsensusITestBase::~RaftConsensusITestBase() /home/jenkins-slave/workspace/kudu-master/2/src/kudu/integration-tests/raft_consensus-itest-base.h:41:7 (raft_consensus_election-itest+0x4d439f) #21 kudu::tserver::RaftConsensusElectionITest_RunLeaderElection_Test::~RaftConsensusElectionITest_RunLeaderElection_Test() /home/jenkins-slave/workspace/kudu-master/2/src/kudu/integration-tests/raft_consensus_election-itest.cc:143:1 (raft_consensus_election-itest+0x4d43c9) #22 testing::Test::DeleteSelf_() /home/jenkins-slave/workspace/kudu-master/2/thirdparty/src/googletest-release-1.8.0/googletest/include/gtest/gtest.h:453:24 (libgmock.so+0x53187) #23 void testing::internal::HandleSehExceptionsInMethodIfSupported(testing::Test*, void (testing::Test::*)(), char const*) /home/jenkins-slave/workspace/kudu-master/2/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2402:10 (libgmock.so+0x52b39) #24 void testing::internal::HandleExceptionsInMethodIfSupported(testing::Test*, void (testing::Test::*)(), char const*) /home/jenkins-slave/workspace/kudu-master/2/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2438 (libgmock.so+0x52b39) #25 testing::TestInfo::Run() /home/jenkins-slave/workspace/kudu-master/2/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2661:3 (libgmock.so+0x340a0) #26 testing::TestCase::Run() /home/jenkins-slave/workspace/kudu-master/2/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2774:28 (libgmock.so+0x34da6) #27 testing::internal::UnitTestImpl::RunAllTests() /home/jenkins-slave/workspace/kudu-master/2/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:4649:43 (libgmock.so+0x40896) #28 bool testing::internal::HandleSehExceptionsInMethodIfSupported(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/jenkins-slave/workspace/kudu-master/2/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2402:10 (libgmock.so+0x53a19) #29 bool testing::internal::HandleExceptionsInMethodIfSupported(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/jenkins-slave/workspace/kudu-master/2/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2438 (libgmock.so+0x53a19) #30 testing::UnitTest::Run() /home/jenkins-slave/workspace/kudu-master/2/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:4257:10 (libgmock.so+0x402e1) #31 RUN_ALL_TESTS() /home/jenkins-slave/workspace/kudu-master/2/thirdparty/installed/tsan/include/gtest/gtest.h:2233:46 (libkudu_test_main.so+0x33db) #32 main /home/jenkins-slave/workspace/kudu-master/2/src/kudu/util/test_main.cc:106:13 (libkudu_test_main.so+0x2b96) Previous read of size 8 at 0x7b2c00001768 by thread T11: #0 scoped_refptr::operator kudu::Histogram* scoped_refptr::*() const /home/jenkins-slave/workspace/kudu-master/2/src/kudu/gutil/ref_counted.h:280:38 (libkrpc.so+0xa60f9) #1 kudu::ThreadPool::DoSubmit(std::__1::shared_ptr, kudu::ThreadPoolToken*) /home/jenkins-slave/workspace/kudu-master/2/src/kudu/util/threadpool.cc:551:7 (libkudu_util.so+0x1cc04b) #2 kudu::ThreadPool::Submit(std::__1::shared_ptr) /home/jenkins-slave/workspace/kudu-master/2/src/kudu/util/threadpool.cc:458:10 (libkudu_util.so+0x1cdcbf) #3 kudu::ThreadPool::SubmitFunc(boost::function) /home/jenkins-slave/workspace/kudu-master/2/src/kudu/util/threadpool.cc:454:10 (libkudu_util.so+0x1cdd59) #4 kudu::DnsResolver::ResolveAddresses(kudu::HostPort const&, std::__1::vector >*, kudu::Callback const&) /home/jenkins-slave/workspace/kudu-master/2/src/kudu/util/net/dns_resolver.cc:59:21 (libkudu_util.so+0x190a2b) #5 kudu::client::internal::RemoteTabletServer::InitProxy(kudu::client::KuduClient*, kudu::Callback const&) /home/jenkins-slave/workspace/kudu-master/2/src/kudu/client/meta_cache.cc:144:33 (libkudu_client.so+0xfc103) #6 kudu::client::internal::MetaCacheServerPicker::PickLeader(kudu::Callback const&, kudu::MonoTime const&) /home/jenkins-slave/workspace/kudu-master/2/src/kudu/client/meta_cache.cc:452:11 (libkudu_client.so+0xfe3bf) #7 kudu::rpc::RetriableRpc::SendRpc() /home/jenkins-slave/workspace/kudu-master/2/src/kudu/rpc/retriable_rpc.h:144:19 (libkudu_client.so+0xb4908) #8 kudu::rpc::RpcRetrier::DelayedRetryCb(kudu::rpc::Rpc*, kudu::Status const&) /home/jenkins-slave/workspace/kudu-master/2/src/kudu/rpc/rpc.cc:94:10 (libkrpc.so+0xd70cc) #9 boost::_mfi::mf2::operator()(kudu::rpc::RpcRetrier*, kudu::rpc::Rpc*, kudu::Status const&) const /home/jenkins-slave/workspace/kudu-master/2/thirdparty/installed/tsan/include/boost/bind/mem_fn_template.hpp:280:29 (libkrpc.so+0xd7ab7) #10 void boost::_bi::list3, boost::_bi::value, boost::arg<1> >::operator(), boost::_bi::rrlist1 >(boost::_bi::type, boost::_mfi::mf2&, boost::_bi::rrlist1&, int) /home/jenkins-slave/workspace/kudu-master/2/thirdparty/installed/tsan/include/boost/bind/bind.hpp:398:9 (libkrpc.so+0xd79f2) #11 void boost::_bi::bind_t, boost::_bi::list3, boost::_bi::value, boost::arg<1> > >::operator()(kudu::Status const&) /home/jenkins-slave/workspace/kudu-master/2/thirdparty/installed/tsan/include/boost/bind/bind.hpp:1234:16 (libkrpc.so+0xd7918) #12 boost::detail::function::void_function_obj_invoker1, boost::_bi::list3, boost::_bi::value, boost::arg<1> > >, void, kudu::Status const&>::invoke(boost::detail::function::function_buffer&, kudu::Status const&) /home/jenkins-slave/workspace/kudu-master/2/thirdparty/installed/tsan/include/boost/function/function_template.hpp:159:11 (libkrpc.so+0xd76b3) #13 boost::function1::operator()(kudu::Status const&) const /home/jenkins-slave/workspace/kudu-master/2/thirdparty/installed/tsan/include/boost/function/function_template.hpp:770:14 (libkrpc.so+0xc5534) #14 kudu::rpc::DelayedTask::TimerHandler(ev::timer&, int) /home/jenkins-slave/workspace/kudu-master/2/src/kudu/rpc/reactor.cc:705:5 (libkrpc.so+0xc2dd6) #15 void ev::base::method_thunk(ev_loop*, ev_timer*, int) /home/jenkins-slave/workspace/kudu-master/2/thirdparty/installed/tsan/include/ev++.h:479:7 (libkrpc.so+0xcad6a) #16 ev_invoke_pending /home/jenkins-slave/workspace/kudu-master/2/thirdparty/src/libev-4.20/ev.c:3155:11 (libev.so.4+0x9800) #17 kudu::rpc::ReactorThread::InvokePendingCb(ev_loop*) /home/jenkins-slave/workspace/kudu-master/2/src/kudu/rpc/reactor.cc:176:3 (libkrpc.so+0xbef55) #18 ev_run /home/jenkins-slave/workspace/kudu-master/2/thirdparty/src/libev-4.20/ev.c:3555:7 (libev.so.4+0xa90d) #19 ev::loop_ref::run(int) /home/jenkins-slave/workspace/kudu-master/2/thirdparty/installed/tsan/include/ev++.h:211:7 (libkrpc.so+0xc4c38) #20 kudu::rpc::ReactorThread::RunThread() /home/jenkins-slave/workspace/kudu-master/2/src/kudu/rpc/reactor.cc:471:9 (libkrpc.so+0xbf09a) #21 boost::_mfi::mf0::operator()(kudu::rpc::ReactorThread*) const /home/jenkins-slave/workspace/kudu-master/2/thirdparty/installed/tsan/include/boost/bind/mem_fn_template.hpp:49:29 (libkrpc.so+0xc8ee6) #22 void boost::_bi::list1 >::operator(), boost::_bi::list0>(boost::_bi::type, boost::_mfi::mf0&, boost::_bi::list0&, int) /home/jenkins-slave/workspace/kudu-master/2/thirdparty/installed/tsan/include/boost/bind/bind.hpp:259:9 (libkrpc.so+0xc8e3a) #23 boost::_bi::bind_t, boost::_bi::list1 > >::operator()() /home/jenkins-slave/workspace/kudu-master/2/thirdparty/installed/tsan/include/boost/bind/bind.hpp:1222:16 (libkrpc.so+0xc8dc3) #24 boost::detail::function::void_function_obj_invoker0, boost::_bi::list1 > >, void>::invoke(boost::detail::function::function_buffer&) /home/jenkins-slave/workspace/kudu-master/2/thirdparty/installed/tsan/include/boost/function/function_template.hpp:159:11 (libkrpc.so+0xc8bc9) #25 boost::function0::operator()() const /home/jenkins-slave/workspace/kudu-master/2/thirdparty/installed/tsan/include/boost/function/function_template.hpp:770:14 (libkrpc.so+0xb8b91) #26 kudu::Thread::SuperviseThread(void*) /home/jenkins-slave/workspace/kudu-master/2/src/kudu/util/thread.cc:608:3 (libkudu_util.so+0x1c5579) As if synchronized via sleep: #0 nanosleep /home/jenkins-slave/workspace/kudu-master/2/thirdparty/src/llvm-4.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:357 (raft_consensus_election-itest+0x461be7) #1 base::SleepForNanoseconds(long) /home/jenkins-slave/workspace/kudu-master/2/src/kudu/gutil/sysinfo.cc:89:10 (libgutil.so+0x6458a) #2 kudu::SleepFor(kudu::MonoDelta const&) /home/jenkins-slave/workspace/kudu-master/2/src/kudu/util/monotime.cc:262:3 (libkudu_util.so+0x18f296) #3 kudu::ClusterVerifier::CheckCluster() /home/jenkins-slave/workspace/kudu-master/2/src/kudu/integration-tests/cluster_verifier.cc:86:5 (libitest_util.so+0x58197) #4 kudu::tserver::TabletServerIntegrationTestBase::AssertAllReplicasAgree(int) /home/jenkins-slave/workspace/kudu-master/2/src/kudu/integration-tests/ts_itest-base.cc:553:3 (libitest_util.so+0x75eb4) #5 kudu::tserver::RaftConsensusElectionITest_RunLeaderElection_Test::TestBody() /home/jenkins-slave/workspace/kudu-master/2/src/kudu/integration-tests/raft_consensus_election-itest.cc:182:3 (raft_consensus_election-itest+0x4c995a) #6 void testing::internal::HandleSehExceptionsInMethodIfSupported(testing::Test*, void (testing::Test::*)(), char const*) /home/jenkins-slave/workspace/kudu-master/2/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2402:10 (libgmock.so+0x52b39) #7 void testing::internal::HandleExceptionsInMethodIfSupported(testing::Test*, void (testing::Test::*)(), char const*) /home/jenkins-slave/workspace/kudu-master/2/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2438 (libgmock.so+0x52b39) #8 testing::Test::Run() /home/jenkins-slave/workspace/kudu-master/2/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2474:5 (libgmock.so+0x32b97) #9 testing::TestInfo::Run() /home/jenkins-slave/workspace/kudu-master/2/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2656:11 (libgmock.so+0x34036) #10 testing::TestCase::Run() /home/jenkins-slave/workspace/kudu-master/2/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2774:28 (libgmock.so+0x34da6) #11 testing::internal::UnitTestImpl::RunAllTests() /home/jenkins-slave/workspace/kudu-master/2/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:4649:43 (libgmock.so+0x40896) #12 bool testing::internal::HandleSehExceptionsInMethodIfSupported(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/jenkins-slave/workspace/kudu-master/2/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2402:10 (libgmock.so+0x53a19) #13 bool testing::internal::HandleExceptionsInMethodIfSupported(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/jenkins-slave/workspace/kudu-master/2/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2438 (libgmock.so+0x53a19) #14 testing::UnitTest::Run() /home/jenkins-slave/workspace/kudu-master/2/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:4257:10 (libgmock.so+0x402e1) #15 RUN_ALL_TESTS() /home/jenkins-slave/workspace/kudu-master/2/thirdparty/installed/tsan/include/gtest/gtest.h:2233:46 (libkudu_test_main.so+0x33db) #16 main /home/jenkins-slave/workspace/kudu-master/2/src/kudu/util/test_main.cc:106:13 (libkudu_test_main.so+0x2b96) Thread T11 'rpc reactor-674' (tid=674, running) created by main thread at: #0 pthread_create /home/jenkins-slave/workspace/kudu-master/2/thirdparty/src/llvm-4.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:897 (raft_consensus_election-itest+0x4662d3) #1 kudu::Thread::StartThread(std::__1::basic_string, std::__1::allocator > const&, std::__1::basic_string, std::__1::allocator > const&, boost::function const&, unsigned long, scoped_refptr*) /home/jenkins-slave/workspace/kudu-master/2/src/kudu/util/thread.cc:561:15 (libkudu_util.so+0x1c4f84) #2 kudu::Status kudu::Thread::Create(std::__1::basic_string, std::__1::allocator > const&, std::__1::basic_string, std::__1::allocator > const&, void (kudu::rpc::ReactorThread::* const&)(), kudu::rpc::ReactorThread* const&, scoped_refptr*) /home/jenkins-slave/workspace/kudu-master/2/src/kudu/util/thread.h:166:12 (libkrpc.so+0xc4345) #3 kudu::rpc::ReactorThread::Init() /home/jenkins-slave/workspace/kudu-master/2/src/kudu/rpc/reactor.cc:168:10 (libkrpc.so+0xbecad) #4 kudu::rpc::Reactor::Init() /home/jenkins-slave/workspace/kudu-master/2/src/kudu/rpc/reactor.cc:722:18 (libkrpc.so+0xc3271) #5 kudu::rpc::Messenger::Init() /home/jenkins-slave/workspace/kudu-master/2/src/kudu/rpc/messenger.cc:437:5 (libkrpc.so+0xa8d32) #6 kudu::rpc::MessengerBuilder::Build(std::__1::shared_ptr*) /home/jenkins-slave/workspace/kudu-master/2/src/kudu/rpc/messenger.cc:200:3 (libkrpc.so+0xa86d1) #7 kudu::client::KuduClientBuilder::Build(std::__1::shared_ptr*) /home/jenkins-slave/workspace/kudu-master/2/src/kudu/client/client.cc:320:3 (libkudu_client.so+0xbd33f) #8 kudu::tserver::TabletServerIntegrationTestBase::CreateClient(std::__1::shared_ptr*) /home/jenkins-slave/workspace/kudu-master/2/src/kudu/integration-tests/ts_itest-base.cc:519:3 (libitest_util.so+0x75162) #9 kudu::tserver::TabletServerIntegrationTestBase::BuildAndStart(std::__1::vector, std::__1::allocator >, std::__1::allocator, std::__1::allocator > > > const&, std::__1::vector, std::__1::allocator >, std::__1::allocator, std::__1::allocator > > > const&) /home/jenkins-slave/workspace/kudu-master/2/src/kudu/integration-tests/ts_itest-base.cc:544:3 (libitest_util.so+0x759a2) #10 kudu::tserver::RaftConsensusElectionITest_RunLeaderElection_Test::TestBody() /home/jenkins-slave/workspace/kudu-master/2/src/kudu/integration-tests/raft_consensus_election-itest.cc:147:3 (raft_consensus_election-itest+0x4c94f9) #11 void testing::internal::HandleSehExceptionsInMethodIfSupported(testing::Test*, void (testing::Test::*)(), char const*) /home/jenkins-slave/workspace/kudu-master/2/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2402:10 (libgmock.so+0x52b39) #12 void testing::internal::HandleExceptionsInMethodIfSupported(testing::Test*, void (testing::Test::*)(), char const*) /home/jenkins-slave/workspace/kudu-master/2/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2438 (libgmock.so+0x52b39) #13 testing::Test::Run() /home/jenkins-slave/workspace/kudu-master/2/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2474:5 (libgmock.so+0x32b97) #14 testing::TestInfo::Run() /home/jenkins-slave/workspace/kudu-master/2/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2656:11 (libgmock.so+0x34036) #15 testing::TestCase::Run() /home/jenkins-slave/workspace/kudu-master/2/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2774:28 (libgmock.so+0x34da6) #16 testing::internal::UnitTestImpl::RunAllTests() /home/jenkins-slave/workspace/kudu-master/2/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:4649:43 (libgmock.so+0x40896) #17 bool testing::internal::HandleSehExceptionsInMethodIfSupported(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/jenkins-slave/workspace/kudu-master/2/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2402:10 (libgmock.so+0x53a19) #18 bool testing::internal::HandleExceptionsInMethodIfSupported(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/jenkins-slave/workspace/kudu-master/2/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2438 (libgmock.so+0x53a19) #19 testing::UnitTest::Run() /home/jenkins-slave/workspace/kudu-master/2/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:4257:10 (libgmock.so+0x402e1) #20 RUN_ALL_TESTS() /home/jenkins-slave/workspace/kudu-master/2/thirdparty/installed/tsan/include/gtest/gtest.h:2233:46 (libkudu_test_main.so+0x33db) #21 main /home/jenkins-slave/workspace/kudu-master/2/src/kudu/util/test_main.cc:106:13 (libkudu_test_main.so+0x2b96) SUMMARY: ThreadSanitizer: data race /home/jenkins-slave/workspace/kudu-master/2/thirdparty/src/llvm-4.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_new_delete.cc:73 in operator delete(void*) ================== ================== WARNING: ThreadSanitizer: data race (pid=212) Write of size 8 at 0x7b50000015a0 by main thread: #0 operator delete(void*) /home/jenkins-slave/workspace/kudu-master/2/thirdparty/src/llvm-4.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_new_delete.cc:73 (raft_consensus_election-itest+0x4c6761) #1 kudu::DefaultDeleter::operator()(kudu::ThreadPool*) const /home/jenkins-slave/workspace/kudu-master/2/src/kudu/gutil/gscoped_ptr.h:145:5 (libmaster.so+0xce6e6) #2 kudu::internal::gscoped_ptr_impl >::~gscoped_ptr_impl() /home/jenkins-slave/workspace/kudu-master/2/src/kudu/gutil/gscoped_ptr.h:228:7 (libmaster.so+0xce6a9) #3 gscoped_ptr >::~gscoped_ptr() /home/jenkins-slave/workspace/kudu-master/2/src/kudu/gutil/gscoped_ptr.h:318:7 (libmaster.so+0xbce39) #4 kudu::DnsResolver::~DnsResolver() /home/jenkins-slave/workspace/kudu-master/2/src/kudu/util/net/dns_resolver.cc:47:1 (libkudu_util.so+0x19097a) #5 kudu::DefaultDeleter::operator()(kudu::DnsResolver*) const /home/jenkins-slave/workspace/kudu-master/2/src/kudu/gutil/gscoped_ptr.h:145:5 (libkudu_client.so+0xd337e) #6 kudu::internal::gscoped_ptr_impl >::reset(kudu::DnsResolver*) /home/jenkins-slave/workspace/kudu-master/2/src/kudu/gutil/gscoped_ptr.h:254:7 (libkudu_client.so+0xd3334) #7 gscoped_ptr >::reset(kudu::DnsResolver*) /home/jenkins-slave/workspace/kudu-master/2/src/kudu/gutil/gscoped_ptr.h:375:46 (libkudu_client.so+0xc8130) #8 kudu::client::KuduClient::Data::~Data() /home/jenkins-slave/workspace/kudu-master/2/src/kudu/client/client-internal.cc:356:17 (libkudu_client.so+0xd8f5d) #9 kudu::client::KuduClient::~KuduClient() /home/jenkins-slave/workspace/kudu-master/2/src/kudu/client/client.cc:359:3 (libkudu_client.so+0xbde0c) #10 std::__1::default_delete::operator()(kudu::client::KuduClient*) const /home/jenkins-slave/workspace/kudu-master/2/thirdparty/installed/tsan/include/c++/v1/memory:2397:13 (libkudu_client.so+0xd2c4b) #11 std::__1::__shared_ptr_pointer, std::__1::allocator >::__on_zero_shared() /home/jenkins-slave/workspace/kudu-master/2/thirdparty/installed/tsan/include/c++/v1/memory:3795 (libkudu_client.so+0xd2c4b) #12 __release_shared /home/jenkins-slave/workspace/kudu-master/2/thirdparty/src/llvm-4.0.0.src/projects/libcxx/src/memory.cpp:67:9 (libc++.so.1+0xc095d) #13 std::__1::__shared_weak_count::__release_shared() /home/jenkins-slave/workspace/kudu-master/2/thirdparty/src/llvm-4.0.0.src/projects/libcxx/src/memory.cpp:92 (libc++.so.1+0xc095d) #14 std::__1::shared_ptr::~shared_ptr() /home/jenkins-slave/workspace/kudu-master/2/thirdparty/installed/tsan/include/c++/v1/memory:4626:19 (raft_consensus_election-itest+0x4d5088) #15 kudu::tserver::TabletServerIntegrationTestBase::~TabletServerIntegrationTestBase() /home/jenkins-slave/workspace/kudu-master/2/src/kudu/integration-tests/ts_itest-base.h:46:7 (raft_consensus_election-itest+0x4d4fcf) #16 kudu::tserver::RaftConsensusITestBase::~RaftConsensusITestBase() /home/jenkins-slave/workspace/kudu-master/2/src/kudu/integration-tests/raft_consensus-itest-base.h:41:7 (raft_consensus_election-itest+0x4d439f) #17 kudu::tserver::RaftConsensusElectionITest_RunLeaderElection_Test::~RaftConsensusElectionITest_RunLeaderElection_Test() /home/jenkins-slave/workspace/kudu-master/2/src/kudu/integration-tests/raft_consensus_election-itest.cc:143:1 (raft_consensus_election-itest+0x4d43c9) #18 testing::Test::DeleteSelf_() /home/jenkins-slave/workspace/kudu-master/2/thirdparty/src/googletest-release-1.8.0/googletest/include/gtest/gtest.h:453:24 (libgmock.so+0x53187) #19 void testing::internal::HandleSehExceptionsInMethodIfSupported(testing::Test*, void (testing::Test::*)(), char const*) /home/jenkins-slave/workspace/kudu-master/2/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2402:10 (libgmock.so+0x52b39) #20 void testing::internal::HandleExceptionsInMethodIfSupported(testing::Test*, void (testing::Test::*)(), char const*) /home/jenkins-slave/workspace/kudu-master/2/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2438 (libgmock.so+0x52b39) #21 testing::TestInfo::Run() /home/jenkins-slave/workspace/kudu-master/2/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2661:3 (libgmock.so+0x340a0) #22 testing::TestCase::Run() /home/jenkins-slave/workspace/kudu-master/2/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2774:28 (libgmock.so+0x34da6) #23 testing::internal::UnitTestImpl::RunAllTests() /home/jenkins-slave/workspace/kudu-master/2/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:4649:43 (libgmock.so+0x40896) #24 bool testing::internal::HandleSehExceptionsInMethodIfSupported(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/jenkins-slave/workspace/kudu-master/2/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2402:10 (libgmock.so+0x53a19) #25 bool testing::internal::HandleExceptionsInMethodIfSupported(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/jenkins-slave/workspace/kudu-master/2/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2438 (libgmock.so+0x53a19) #26 testing::UnitTest::Run() /home/jenkins-slave/workspace/kudu-master/2/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:4257:10 (libgmock.so+0x402e1) #27 RUN_ALL_TESTS() /home/jenkins-slave/workspace/kudu-master/2/thirdparty/installed/tsan/include/gtest/gtest.h:2233:46 (libkudu_test_main.so+0x33db) #28 main /home/jenkins-slave/workspace/kudu-master/2/src/kudu/util/test_main.cc:106:13 (libkudu_test_main.so+0x2b96) Previous read of size 8 at 0x7b50000015a0 by thread T11: #0 scoped_refptr::operator kudu::Histogram* scoped_refptr::*() const /home/jenkins-slave/workspace/kudu-master/2/src/kudu/gutil/ref_counted.h:280:38 (libkrpc.so+0xa60f9) #1 kudu::ThreadPool::DoSubmit(std::__1::shared_ptr, kudu::ThreadPoolToken*) /home/jenkins-slave/workspace/kudu-master/2/src/kudu/util/threadpool.cc:548:7 (libkudu_util.so+0x1cc01c) #2 kudu::ThreadPool::Submit(std::__1::shared_ptr) /home/jenkins-slave/workspace/kudu-master/2/src/kudu/util/threadpool.cc:458:10 (libkudu_util.so+0x1cdcbf) #3 kudu::ThreadPool::SubmitFunc(boost::function) /home/jenkins-slave/workspace/kudu-master/2/src/kudu/util/threadpool.cc:454:10 (libkudu_util.so+0x1cdd59) #4 kudu::DnsResolver::ResolveAddresses(kudu::HostPort const&, std::__1::vector >*, kudu::Callback const&) /home/jenkins-slave/workspace/kudu-master/2/src/kudu/util/net/dns_resolver.cc:59:21 (libkudu_util.so+0x190a2b) #5 kudu::client::internal::RemoteTabletServer::InitProxy(kudu::client::KuduClient*, kudu::Callback const&) /home/jenkins-slave/workspace/kudu-master/2/src/kudu/client/meta_cache.cc:144:33 (libkudu_client.so+0xfc103) #6 kudu::client::internal::MetaCacheServerPicker::PickLeader(kudu::Callback const&, kudu::MonoTime const&) /home/jenkins-slave/workspace/kudu-master/2/src/kudu/client/meta_cache.cc:452:11 (libkudu_client.so+0xfe3bf) #7 kudu::rpc::RetriableRpc::SendRpc() /home/jenkins-slave/workspace/kudu-master/2/src/kudu/rpc/retriable_rpc.h:144:19 (libkudu_client.so+0xb4908) #8 kudu::rpc::RpcRetrier::DelayedRetryCb(kudu::rpc::Rpc*, kudu::Status const&) /home/jenkins-slave/workspace/kudu-master/2/src/kudu/rpc/rpc.cc:94:10 (libkrpc.so+0xd70cc) #9 boost::_mfi::mf2::operator()(kudu::rpc::RpcRetrier*, kudu::rpc::Rpc*, kudu::Status const&) const /home/jenkins-slave/workspace/kudu-master/2/thirdparty/installed/tsan/include/boost/bind/mem_fn_template.hpp:280:29 (libkrpc.so+0xd7ab7) #10 void boost::_bi::list3, boost::_bi::value, boost::arg<1> >::operator(), boost::_bi::rrlist1 >(boost::_bi::type, boost::_mfi::mf2&, boost::_bi::rrlist1&, int) /home/jenkins-slave/workspace/kudu-master/2/thirdparty/installed/tsan/include/boost/bind/bind.hpp:398:9 (libkrpc.so+0xd79f2) #11 void boost::_bi::bind_t, boost::_bi::list3, boost::_bi::value, boost::arg<1> > >::operator()(kudu::Status const&) /home/jenkins-slave/workspace/kudu-master/2/thirdparty/installed/tsan/include/boost/bind/bind.hpp:1234:16 (libkrpc.so+0xd7918) #12 boost::detail::function::void_function_obj_invoker1, boost::_bi::list3, boost::_bi::value, boost::arg<1> > >, void, kudu::Status const&>::invoke(boost::detail::function::function_buffer&, kudu::Status const&) /home/jenkins-slave/workspace/kudu-master/2/thirdparty/installed/tsan/include/boost/function/function_template.hpp:159:11 (libkrpc.so+0xd76b3) #13 boost::function1::operator()(kudu::Status const&) const /home/jenkins-slave/workspace/kudu-master/2/thirdparty/installed/tsan/include/boost/function/function_template.hpp:770:14 (libkrpc.so+0xc5534) #14 kudu::rpc::DelayedTask::TimerHandler(ev::timer&, int) /home/jenkins-slave/workspace/kudu-master/2/src/kudu/rpc/reactor.cc:705:5 (libkrpc.so+0xc2dd6) #15 void ev::base::method_thunk(ev_loop*, ev_timer*, int) /home/jenkins-slave/workspace/kudu-master/2/thirdparty/installed/tsan/include/ev++.h:479:7 (libkrpc.so+0xcad6a) #16 ev_invoke_pending /home/jenkins-slave/workspace/kudu-master/2/thirdparty/src/libev-4.20/ev.c:3155:11 (libev.so.4+0x9800) #17 kudu::rpc::ReactorThread::InvokePendingCb(ev_loop*) /home/jenkins-slave/workspace/kudu-master/2/src/kudu/rpc/reactor.cc:176:3 (libkrpc.so+0xbef55) #18 ev_run /home/jenkins-slave/workspace/kudu-master/2/thirdparty/src/libev-4.20/ev.c:3555:7 (libev.so.4+0xa90d) #19 ev::loop_ref::run(int) /home/jenkins-slave/workspace/kudu-master/2/thirdparty/installed/tsan/include/ev++.h:211:7 (libkrpc.so+0xc4c38) #20 kudu::rpc::ReactorThread::RunThread() /home/jenkins-slave/workspace/kudu-master/2/src/kudu/rpc/reactor.cc:471:9 (libkrpc.so+0xbf09a) #21 boost::_mfi::mf0::operator()(kudu::rpc::ReactorThread*) const /home/jenkins-slave/workspace/kudu-master/2/thirdparty/installed/tsan/include/boost/bind/mem_fn_template.hpp:49:29 (libkrpc.so+0xc8ee6) #22 void boost::_bi::list1 >::operator(), boost::_bi::list0>(boost::_bi::type, boost::_mfi::mf0&, boost::_bi::list0&, int) /home/jenkins-slave/workspace/kudu-master/2/thirdparty/installed/tsan/include/boost/bind/bind.hpp:259:9 (libkrpc.so+0xc8e3a) #23 boost::_bi::bind_t, boost::_bi::list1 > >::operator()() /home/jenkins-slave/workspace/kudu-master/2/thirdparty/installed/tsan/include/boost/bind/bind.hpp:1222:16 (libkrpc.so+0xc8dc3) #24 boost::detail::function::void_function_obj_invoker0, boost::_bi::list1 > >, void>::invoke(boost::detail::function::function_buffer&) /home/jenkins-slave/workspace/kudu-master/2/thirdparty/installed/tsan/include/boost/function/function_template.hpp:159:11 (libkrpc.so+0xc8bc9) #25 boost::function0::operator()() const /home/jenkins-slave/workspace/kudu-master/2/thirdparty/installed/tsan/include/boost/function/function_template.hpp:770:14 (libkrpc.so+0xb8b91) #26 kudu::Thread::SuperviseThread(void*) /home/jenkins-slave/workspace/kudu-master/2/src/kudu/util/thread.cc:608:3 (libkudu_util.so+0x1c5579) As if synchronized via sleep: #0 nanosleep /home/jenkins-slave/workspace/kudu-master/2/thirdparty/src/llvm-4.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:357 (raft_consensus_election-itest+0x461be7) #1 base::SleepForNanoseconds(long) /home/jenkins-slave/workspace/kudu-master/2/src/kudu/gutil/sysinfo.cc:89:10 (libgutil.so+0x6458a) #2 kudu::SleepFor(kudu::MonoDelta const&) /home/jenkins-slave/workspace/kudu-master/2/src/kudu/util/monotime.cc:262:3 (libkudu_util.so+0x18f296) #3 kudu::ClusterVerifier::CheckCluster() /home/jenkins-slave/workspace/kudu-master/2/src/kudu/integration-tests/cluster_verifier.cc:86:5 (libitest_util.so+0x58197) #4 kudu::tserver::TabletServerIntegrationTestBase::AssertAllReplicasAgree(int) /home/jenkins-slave/workspace/kudu-master/2/src/kudu/integration-tests/ts_itest-base.cc:553:3 (libitest_util.so+0x75eb4) #5 kudu::tserver::RaftConsensusElectionITest_RunLeaderElection_Test::TestBody() /home/jenkins-slave/workspace/kudu-master/2/src/kudu/integration-tests/raft_consensus_election-itest.cc:182:3 (raft_consensus_election-itest+0x4c995a) #6 void testing::internal::HandleSehExceptionsInMethodIfSupported(testing::Test*, void (testing::Test::*)(), char const*) /home/jenkins-slave/workspace/kudu-master/2/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2402:10 (libgmock.so+0x52b39) #7 void testing::internal::HandleExceptionsInMethodIfSupported(testing::Test*, void (testing::Test::*)(), char const*) /home/jenkins-slave/workspace/kudu-master/2/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2438 (libgmock.so+0x52b39) #8 testing::Test::Run() /home/jenkins-slave/workspace/kudu-master/2/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2474:5 (libgmock.so+0x32b97) #9 testing::TestInfo::Run() /home/jenkins-slave/workspace/kudu-master/2/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2656:11 (libgmock.so+0x34036) #10 testing::TestCase::Run() /home/jenkins-slave/workspace/kudu-master/2/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2774:28 (libgmock.so+0x34da6) #11 testing::internal::UnitTestImpl::RunAllTests() /home/jenkins-slave/workspace/kudu-master/2/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:4649:43 (libgmock.so+0x40896) #12 bool testing::internal::HandleSehExceptionsInMethodIfSupported(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/jenkins-slave/workspace/kudu-master/2/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2402:10 (libgmock.so+0x53a19) #13 bool testing::internal::HandleExceptionsInMethodIfSupported(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/jenkins-slave/workspace/kudu-master/2/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2438 (libgmock.so+0x53a19) #14 testing::UnitTest::Run() /home/jenkins-slave/workspace/kudu-master/2/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:4257:10 (libgmock.so+0x402e1) #15 RUN_ALL_TESTS() /home/jenkins-slave/workspace/kudu-master/2/thirdparty/installed/tsan/include/gtest/gtest.h:2233:46 (libkudu_test_main.so+0x33db) #16 main /home/jenkins-slave/workspace/kudu-master/2/src/kudu/util/test_main.cc:106:13 (libkudu_test_main.so+0x2b96) Thread T11 'rpc reactor-674' (tid=674, running) created by main thread at: #0 pthread_create /home/jenkins-slave/workspace/kudu-master/2/thirdparty/src/llvm-4.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:897 (raft_consensus_election-itest+0x4662d3) #1 kudu::Thread::StartThread(std::__1::basic_string, std::__1::allocator > const&, std::__1::basic_string, std::__1::allocator > const&, boost::function const&, unsigned long, scoped_refptr*) /home/jenkins-slave/workspace/kudu-master/2/src/kudu/util/thread.cc:561:15 (libkudu_util.so+0x1c4f84) #2 kudu::Status kudu::Thread::Create(std::__1::basic_string, std::__1::allocator > const&, std::__1::basic_string, std::__1::allocator > const&, void (kudu::rpc::ReactorThread::* const&)(), kudu::rpc::ReactorThread* const&, scoped_refptr*) /home/jenkins-slave/workspace/kudu-master/2/src/kudu/util/thread.h:166:12 (libkrpc.so+0xc4345) #3 kudu::rpc::ReactorThread::Init() /home/jenkins-slave/workspace/kudu-master/2/src/kudu/rpc/reactor.cc:168:10 (libkrpc.so+0xbecad) #4 kudu::rpc::Reactor::Init() /home/jenkins-slave/workspace/kudu-master/2/src/kudu/rpc/reactor.cc:722:18 (libkrpc.so+0xc3271) #5 kudu::rpc::Messenger::Init() /home/jenkins-slave/workspace/kudu-master/2/src/kudu/rpc/messenger.cc:437:5 (libkrpc.so+0xa8d32) #6 kudu::rpc::MessengerBuilder::Build(std::__1::shared_ptr*) /home/jenkins-slave/workspace/kudu-master/2/src/kudu/rpc/messenger.cc:200:3 (libkrpc.so+0xa86d1) #7 kudu::client::KuduClientBuilder::Build(std::__1::shared_ptr*) /home/jenkins-slave/workspace/kudu-master/2/src/kudu/client/client.cc:320:3 (libkudu_client.so+0xbd33f) #8 kudu::tserver::TabletServerIntegrationTestBase::CreateClient(std::__1::shared_ptr*) /home/jenkins-slave/workspace/kudu-master/2/src/kudu/integration-tests/ts_itest-base.cc:519:3 (libitest_util.so+0x75162) #9 kudu::tserver::TabletServerIntegrationTestBase::BuildAndStart(std::__1::vector, std::__1::allocator >, std::__1::allocator, std::__1::allocator > > > const&, std::__1::vector, std::__1::allocator >, std::__1::allocator, std::__1::allocator > > > const&) /home/jenkins-slave/workspace/kudu-master/2/src/kudu/integration-tests/ts_itest-base.cc:544:3 (libitest_util.so+0x759a2) #10 kudu::tserver::RaftConsensusElectionITest_RunLeaderElection_Test::TestBody() /home/jenkins-slave/workspace/kudu-master/2/src/kudu/integration-tests/raft_consensus_election-itest.cc:147:3 (raft_consensus_election-itest+0x4c94f9) #11 void testing::internal::HandleSehExceptionsInMethodIfSupported(testing::Test*, void (testing::Test::*)(), char const*) /home/jenkins-slave/workspace/kudu-master/2/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2402:10 (libgmock.so+0x52b39) #12 void testing::internal::HandleExceptionsInMethodIfSupported(testing::Test*, void (testing::Test::*)(), char const*) /home/jenkins-slave/workspace/kudu-master/2/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2438 (libgmock.so+0x52b39) #13 testing::Test::Run() /home/jenkins-slave/workspace/kudu-master/2/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2474:5 (libgmock.so+0x32b97) #14 testing::TestInfo::Run() /home/jenkins-slave/workspace/kudu-master/2/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2656:11 (libgmock.so+0x34036) #15 testing::TestCase::Run() /home/jenkins-slave/workspace/kudu-master/2/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2774:28 (libgmock.so+0x34da6) #16 testing::internal::UnitTestImpl::RunAllTests() /home/jenkins-slave/workspace/kudu-master/2/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:4649:43 (libgmock.so+0x40896) #17 bool testing::internal::HandleSehExceptionsInMethodIfSupported(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/jenkins-slave/workspace/kudu-master/2/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2402:10 (libgmock.so+0x53a19) #18 bool testing::internal::HandleExceptionsInMethodIfSupported(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/jenkins-slave/workspace/kudu-master/2/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2438 (libgmock.so+0x53a19) #19 testing::UnitTest::Run() /home/jenkins-slave/workspace/kudu-master/2/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:4257:10 (libgmock.so+0x402e1) #20 RUN_ALL_TESTS() /home/jenkins-slave/workspace/kudu-master/2/thirdparty/installed/tsan/include/gtest/gtest.h:2233:46 (libkudu_test_main.so+0x33db) #21 main /home/jenkins-slave/workspace/kudu-master/2/src/kudu/util/test_main.cc:106:13 (libkudu_test_main.so+0x2b96) SUMMARY: ThreadSanitizer: data race /home/jenkins-slave/workspace/kudu-master/2/thirdparty/src/llvm-4.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_new_delete.cc:73 in operator delete(void*) ================== I0312 20:10:07.792055 212 external_mini_cluster.cc:997] Killing /tmp/dist-test-task3bwPeE/build/tsan/bin/kudu-tserver with pid 286 W0312 20:10:07.819196 235 connection.cc:513] client connection to 127.0.53.1:43100 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0312 20:10:07.820149 233 connection.cc:513] server connection from 127.0.53.1:39405 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0312 20:10:07.820891 223 connection.cc:513] client connection to 127.0.53.1:43100 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0312 20:10:07.821876 220 connection.cc:513] client connection to 127.0.53.1:43100 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0312 20:10:07.822763 432 connection.cc:513] client connection to 127.0.53.1:43100 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) I0312 20:10:07.832164 212 external_mini_cluster.cc:997] Killing /tmp/dist-test-task3bwPeE/build/tsan/bin/kudu-tserver with pid 421 W0312 20:10:07.856185 237 connection.cc:513] client connection to 127.0.53.2:59907 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0312 20:10:07.856997 237 connection.cc:513] server connection from 127.0.53.2:42574 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0312 20:10:07.857614 223 connection.cc:513] client connection to 127.0.53.2:59907 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0312 20:10:07.858417 222 connection.cc:513] client connection to 127.0.53.2:59907 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0312 20:10:07.859266 765 connection.cc:513] server connection from 127.0.53.2:54915 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) I0312 20:10:07.860297 212 external_mini_cluster.cc:997] Killing /tmp/dist-test-task3bwPeE/build/tsan/bin/kudu-tserver with pid 756 W0312 20:10:07.883934 235 connection.cc:513] server connection from 127.0.53.3:60731 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) I0312 20:10:07.884085 212 external_mini_cluster.cc:997] Killing /tmp/dist-test-task3bwPeE/build/tsan/bin/kudu-master with pid 224 W0312 20:10:07.912341 223 connection.cc:513] client connection to 127.0.53.62:54194 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) [ OK ] RaftConsensusElectionITest.RunLeaderElection (6115 ms) [ RUN ] RaftConsensusElectionITest.ChurnyElections I0312 20:10:07.924394 212 test_util.cc:192] Using random seed: -1166294895 I0312 20:10:07.927569 212 ts_itest-base.cc:113] Starting cluster with: I0312 20:10:07.927675 212 ts_itest-base.cc:114] -------------- I0312 20:10:07.927764 212 ts_itest-base.cc:115] 3 tablet servers I0312 20:10:07.927848 212 ts_itest-base.cc:116] 3 replicas per TS I0312 20:10:07.927939 212 ts_itest-base.cc:117] -------------- I0312 20:10:07.930058 212 external_mini_cluster.cc:805] Running /tmp/dist-test-task3bwPeE/build/tsan/bin/kudu-master /tmp/dist-test-task3bwPeE/build/tsan/bin/kudu-master --fs_wal_dir=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ChurnyElections.1520885401735786-212/raft_consensus-itest-cluster/master-0/wal --fs_data_dirs=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ChurnyElections.1520885401735786-212/raft_consensus-itest-cluster/master-0/data --block_manager=log --webserver_interface=localhost --ipki_ca_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.0.53.62:0 --webserver_interface=127.0.53.62 --webserver_port=0 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ChurnyElections.1520885401735786-212/raft_consensus-itest-cluster/master-0/logs --server_dump_info_path=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ChurnyElections.1520885401735786-212/raft_consensus-itest-cluster/master-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W0312 20:10:08.048758 946 flags.cc:412] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0312 20:10:08.049096 946 flags.cc:412] Enabled unsafe flag: --never_fsync=true W0312 20:10:08.064543 946 flags.cc:412] Enabled experimental flag: --ipki_ca_key_size=1024 W0312 20:10:08.064669 946 flags.cc:412] Enabled experimental flag: --ipki_server_key_size=1024 W0312 20:10:08.064764 946 flags.cc:412] Enabled experimental flag: --tsk_num_rsa_bits=512 I0312 20:10:08.085244 946 master_main.cc:66] Master server non-default flags: --fs_data_dirs=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ChurnyElections.1520885401735786-212/raft_consensus-itest-cluster/master-0/data --fs_wal_dir=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ChurnyElections.1520885401735786-212/raft_consensus-itest-cluster/master-0/wal --ipki_ca_key_size=1024 --ipki_server_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.0.53.62:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ChurnyElections.1520885401735786-212/raft_consensus-itest-cluster/master-0/data/info.pb --webserver_interface=127.0.53.62 --webserver_port=0 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --log_dir=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ChurnyElections.1520885401735786-212/raft_consensus-itest-cluster/master-0/logs --logbuflevel=-1 --logtostderr=true Master server version: kudu 1.7.0-SNAPSHOT revision 13e10c63c46de30ac85f34b543cdd6fb44881f61 build type FASTDEBUG built by None at 12 Mar 2018 20:06:40 UTC on jenkins-slave-vn0g build id 12388 TSAN enabled I0312 20:10:08.094406 946 master_main.cc:73] Initializing master server... I0312 20:10:08.095806 946 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 1071276us I0312 20:10:08.096544 946 fs_manager.cc:260] Metadata directory not provided I0312 20:10:08.096665 946 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0312 20:10:08.096868 946 server_base.cc:434] Could not load existing FS layout: Not found: /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ChurnyElections.1520885401735786-212/raft_consensus-itest-cluster/master-0/data/instance: No such file or directory (error 2) I0312 20:10:08.096976 946 server_base.cc:435] Attempting to create new FS layout instead I0312 20:10:08.104320 946 fs_manager.cc:596] Generated new instance metadata in path /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ChurnyElections.1520885401735786-212/raft_consensus-itest-cluster/master-0/data/instance: uuid: "5014d42b8fa546c5931e69b52774d517" format_stamp: "Formatted at 2018-03-12 20:10:08 on dist-test-slave-dist-test-slave-f27p" I0312 20:10:08.105016 946 fs_manager.cc:596] Generated new instance metadata in path /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ChurnyElections.1520885401735786-212/raft_consensus-itest-cluster/master-0/wal/instance: uuid: "5014d42b8fa546c5931e69b52774d517" format_stamp: "Formatted at 2018-03-12 20:10:08 on dist-test-slave-dist-test-slave-f27p" I0312 20:10:08.113283 946 fs_manager.cc:496] Time spent creating directory manager: real 0.008s user 0.008s sys 0.000s I0312 20:10:08.114428 946 env_posix.cc:1620] Not raising this process' open files per process limit of 65536; it is already as high as it can go I0312 20:10:08.114696 946 file_cache.cc:470] Constructed file cache lbm with capacity 26214 I0312 20:10:08.121106 946 fs_manager.cc:418] Time spent opening block manager: real 0.002s user 0.000s sys 0.000s I0312 20:10:08.121217 946 fs_manager.cc:429] Opened local filesystem: /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ChurnyElections.1520885401735786-212/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ChurnyElections.1520885401735786-212/raft_consensus-itest-cluster/master-0/wal uuid: "5014d42b8fa546c5931e69b52774d517" format_stamp: "Formatted at 2018-03-12 20:10:08 on dist-test-slave-dist-test-slave-f27p" I0312 20:10:08.121414 946 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ChurnyElections.1520885401735786-212/raft_consensus-itest-cluster/master-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0312 20:10:08.335878 946 env_posix.cc:1620] Not raising this process' running threads per effective uid limit of 60284; it is already as high as it can go I0312 20:10:08.336488 946 master_main.cc:76] Starting Master server... I0312 20:10:08.337733 963 process_memory.cc:182] Process hard memory limit is 11.777631 GB I0312 20:10:08.337874 963 process_memory.cc:184] Process soft memory limit is 9.422105 GB I0312 20:10:08.337972 963 process_memory.cc:187] Process memory pressure threshold is 7.066578 GB I0312 20:10:08.375977 946 rpc_server.cc:201] RPC server started. Bound to: 127.0.53.62:57865 I0312 20:10:08.401511 946 webserver.cc:173] Starting webserver on 127.0.53.62:0 I0312 20:10:08.401630 946 webserver.cc:184] Document root disabled I0312 20:10:08.402866 946 webserver.cc:311] Webserver started. Bound to: http://127.0.53.62:52934/ I0312 20:10:08.403636 946 server_base.cc:607] Dumped server information to /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ChurnyElections.1520885401735786-212/raft_consensus-itest-cluster/master-0/data/info.pb I0312 20:10:08.406288 1007 data_dirs.cc:931] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0312 20:10:08.412765 212 external_mini_cluster.cc:867] Started /tmp/dist-test-task3bwPeE/build/tsan/bin/kudu-master as pid 946 I0312 20:10:08.414134 212 external_mini_cluster.cc:805] Running /tmp/dist-test-task3bwPeE/build/tsan/bin/kudu-tserver /tmp/dist-test-task3bwPeE/build/tsan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ChurnyElections.1520885401735786-212/raft_consensus-itest-cluster/ts-0/wal --fs_data_dirs=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ChurnyElections.1520885401735786-212/raft_consensus-itest-cluster/ts-0/data --block_manager=log --rpc_bind_addresses=127.0.53.1:0 --local_ip_for_outbound_sockets=127.0.53.1 --webserver_interface=127.0.53.1 --webserver_port=0 --tserver_master_addrs=127.0.53.62:57865 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ChurnyElections.1520885401735786-212/raft_consensus-itest-cluster/ts-0/logs --server_dump_info_path=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ChurnyElections.1520885401735786-212/raft_consensus-itest-cluster/ts-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_heartbeat_interval_ms=5 with env {} I0312 20:10:08.434052 1007 tablet_bootstrap.cc:436] T 00000000000000000000000000000000 P 5014d42b8fa546c5931e69b52774d517: Bootstrap starting. I0312 20:10:08.436396 1007 tablet_bootstrap.cc:581] T 00000000000000000000000000000000 P 5014d42b8fa546c5931e69b52774d517: No blocks or log segments found. Creating new log. I0312 20:10:08.437609 1007 log.cc:520] T 00000000000000000000000000000000 P 5014d42b8fa546c5931e69b52774d517: Log is configured to *not* fsync() on all Append() calls I0312 20:10:08.446336 1007 tablet_bootstrap.cc:436] T 00000000000000000000000000000000 P 5014d42b8fa546c5931e69b52774d517: No bootstrap required, opened a new log I0312 20:10:08.464174 1007 raft_consensus.cc:304] T 00000000000000000000000000000000 P 5014d42b8fa546c5931e69b52774d517 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5014d42b8fa546c5931e69b52774d517" member_type: VOTER } I0312 20:10:08.464584 1007 raft_consensus.cc:330] T 00000000000000000000000000000000 P 5014d42b8fa546c5931e69b52774d517 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0312 20:10:08.464771 1007 raft_consensus.cc:605] T 00000000000000000000000000000000 P 5014d42b8fa546c5931e69b52774d517 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5014d42b8fa546c5931e69b52774d517, State: Initialized, Role: FOLLOWER I0312 20:10:08.465384 1007 consensus_queue.cc:219] T 00000000000000000000000000000000 P 5014d42b8fa546c5931e69b52774d517 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5014d42b8fa546c5931e69b52774d517" member_type: VOTER } I0312 20:10:08.465855 1007 raft_consensus.cc:344] T 00000000000000000000000000000000 P 5014d42b8fa546c5931e69b52774d517 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0312 20:10:08.466061 1007 raft_consensus.cc:436] T 00000000000000000000000000000000 P 5014d42b8fa546c5931e69b52774d517 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0312 20:10:08.466264 1007 raft_consensus.cc:2700] T 00000000000000000000000000000000 P 5014d42b8fa546c5931e69b52774d517: Snoozing failure detection for 1.575s (starting election) I0312 20:10:08.466464 1007 raft_consensus.cc:2752] T 00000000000000000000000000000000 P 5014d42b8fa546c5931e69b52774d517 [term 0 FOLLOWER]: Advancing to term 1 I0312 20:10:08.470224 1007 raft_consensus.cc:458] T 00000000000000000000000000000000 P 5014d42b8fa546c5931e69b52774d517 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5014d42b8fa546c5931e69b52774d517" member_type: VOTER } I0312 20:10:08.470777 1007 leader_election.cc:258] T 00000000000000000000000000000000 P 5014d42b8fa546c5931e69b52774d517 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0312 20:10:08.472399 1011 raft_consensus.cc:2700] T 00000000000000000000000000000000 P 5014d42b8fa546c5931e69b52774d517: Snoozing failure detection for 1.601s (election complete) I0312 20:10:08.472600 1011 raft_consensus.cc:2507] T 00000000000000000000000000000000 P 5014d42b8fa546c5931e69b52774d517 [term 1 FOLLOWER]: Leader election won for term 1 I0312 20:10:08.493391 1011 raft_consensus.cc:572] T 00000000000000000000000000000000 P 5014d42b8fa546c5931e69b52774d517 [term 1 LEADER]: Becoming Leader. State: Replica: 5014d42b8fa546c5931e69b52774d517, State: Running, Role: LEADER I0312 20:10:08.493705 1011 consensus_queue.cc:196] T 00000000000000000000000000000000 P 5014d42b8fa546c5931e69b52774d517 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5014d42b8fa546c5931e69b52774d517" member_type: VOTER } I0312 20:10:08.506758 1013 sys_catalog.cc:335] T 00000000000000000000000000000000 P 5014d42b8fa546c5931e69b52774d517 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5014d42b8fa546c5931e69b52774d517. Latest consensus state: current_term: 1 leader_uuid: "5014d42b8fa546c5931e69b52774d517" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5014d42b8fa546c5931e69b52774d517" member_type: VOTER } } I0312 20:10:08.506774 1007 sys_catalog.cc:335] T 00000000000000000000000000000000 P 5014d42b8fa546c5931e69b52774d517 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 1 leader_uuid: "5014d42b8fa546c5931e69b52774d517" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5014d42b8fa546c5931e69b52774d517" member_type: VOTER } } I0312 20:10:08.507165 1013 sys_catalog.cc:338] T 00000000000000000000000000000000 P 5014d42b8fa546c5931e69b52774d517 [sys.catalog]: This master's current role is: LEADER I0312 20:10:08.507241 1007 sys_catalog.cc:338] T 00000000000000000000000000000000 P 5014d42b8fa546c5931e69b52774d517 [sys.catalog]: This master's current role is: LEADER I0312 20:10:08.510854 1012 sys_catalog.cc:335] T 00000000000000000000000000000000 P 5014d42b8fa546c5931e69b52774d517 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5014d42b8fa546c5931e69b52774d517" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5014d42b8fa546c5931e69b52774d517" member_type: VOTER } } I0312 20:10:08.511842 1012 sys_catalog.cc:338] T 00000000000000000000000000000000 P 5014d42b8fa546c5931e69b52774d517 [sys.catalog]: This master's current role is: LEADER I0312 20:10:08.512792 1007 sys_catalog.cc:422] T 00000000000000000000000000000000 P 5014d42b8fa546c5931e69b52774d517 [sys.catalog]: configured and running, proceeding with master startup. I0312 20:10:08.512991 1015 catalog_manager.cc:972] Loading table and tablet metadata into memory... I0312 20:10:08.514672 946 master_main.cc:79] Master server successfully started. I0312 20:10:08.527835 1015 catalog_manager.cc:983] Initializing Kudu internal certificate authority... WARNING: Logging before InitGoogleLogging() is written to STDERR W0312 20:10:08.553570 1008 flags.cc:412] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0312 20:10:08.553905 1008 flags.cc:412] Enabled unsafe flag: --never_fsync=true W0312 20:10:08.568147 1008 flags.cc:412] Enabled experimental flag: --ipki_server_key_size=1024 W0312 20:10:08.568459 1008 flags.cc:412] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.53.1 I0312 20:10:08.587927 1008 tablet_server_main.cc:72] Tablet server non-default flags: --raft_heartbeat_interval_ms=5 --fs_data_dirs=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ChurnyElections.1520885401735786-212/raft_consensus-itest-cluster/ts-0/data --fs_wal_dir=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ChurnyElections.1520885401735786-212/raft_consensus-itest-cluster/ts-0/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.0.53.1:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ChurnyElections.1520885401735786-212/raft_consensus-itest-cluster/ts-0/data/info.pb --webserver_interface=127.0.53.1 --webserver_port=0 --tserver_master_addrs=127.0.53.62:57865 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.0.53.1 --log_dir=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ChurnyElections.1520885401735786-212/raft_consensus-itest-cluster/ts-0/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.7.0-SNAPSHOT revision 13e10c63c46de30ac85f34b543cdd6fb44881f61 build type FASTDEBUG built by None at 12 Mar 2018 20:06:40 UTC on jenkins-slave-vn0g build id 12388 TSAN enabled W0312 20:10:08.588316 963 tablet.cc:1679] T 00000000000000000000000000000000 P 5014d42b8fa546c5931e69b52774d517: Can't schedule compaction. Clean time has not been advanced past its initial value. I0312 20:10:08.597983 1008 tablet_server_main.cc:79] Initializing tablet server... I0312 20:10:08.599864 1008 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 1071276us I0312 20:10:08.600507 1008 fs_manager.cc:260] Metadata directory not provided I0312 20:10:08.600706 1008 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0312 20:10:08.600983 1008 server_base.cc:434] Could not load existing FS layout: Not found: /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ChurnyElections.1520885401735786-212/raft_consensus-itest-cluster/ts-0/data/instance: No such file or directory (error 2) I0312 20:10:08.601130 1008 server_base.cc:435] Attempting to create new FS layout instead I0312 20:10:08.602727 1015 catalog_manager.cc:870] Generated new certificate authority record I0312 20:10:08.605027 1015 catalog_manager.cc:992] Loading token signing keys... I0312 20:10:08.608664 1008 fs_manager.cc:596] Generated new instance metadata in path /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ChurnyElections.1520885401735786-212/raft_consensus-itest-cluster/ts-0/data/instance: uuid: "ded9b7f8677940fb91e1a8c8bceceaed" format_stamp: "Formatted at 2018-03-12 20:10:08 on dist-test-slave-dist-test-slave-f27p" I0312 20:10:08.609388 1008 fs_manager.cc:596] Generated new instance metadata in path /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ChurnyElections.1520885401735786-212/raft_consensus-itest-cluster/ts-0/wal/instance: uuid: "ded9b7f8677940fb91e1a8c8bceceaed" format_stamp: "Formatted at 2018-03-12 20:10:08 on dist-test-slave-dist-test-slave-f27p" I0312 20:10:08.620164 1015 catalog_manager.cc:3868] T 00000000000000000000000000000000 P 5014d42b8fa546c5931e69b52774d517: Generated new TSK 0 I0312 20:10:08.620563 1008 fs_manager.cc:496] Time spent creating directory manager: real 0.011s user 0.008s sys 0.000s I0312 20:10:08.622068 1008 env_posix.cc:1620] Not raising this process' open files per process limit of 65536; it is already as high as it can go I0312 20:10:08.622364 1008 file_cache.cc:470] Constructed file cache lbm with capacity 26214 I0312 20:10:08.629261 1008 fs_manager.cc:418] Time spent opening block manager: real 0.002s user 0.004s sys 0.000s I0312 20:10:08.629376 1008 fs_manager.cc:429] Opened local filesystem: /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ChurnyElections.1520885401735786-212/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ChurnyElections.1520885401735786-212/raft_consensus-itest-cluster/ts-0/wal uuid: "ded9b7f8677940fb91e1a8c8bceceaed" format_stamp: "Formatted at 2018-03-12 20:10:08 on dist-test-slave-dist-test-slave-f27p" I0312 20:10:08.629586 1008 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ChurnyElections.1520885401735786-212/raft_consensus-itest-cluster/ts-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0312 20:10:08.861945 1008 env_posix.cc:1620] Not raising this process' running threads per effective uid limit of 60284; it is already as high as it can go I0312 20:10:08.863255 1008 ts_tablet_manager.cc:349] Loaded tablet metadata (0 live tablets) I0312 20:10:08.864420 1008 tablet_server_main.cc:84] Starting tablet server... I0312 20:10:08.968510 1008 rpc_server.cc:201] RPC server started. Bound to: 127.0.53.1:35037 I0312 20:10:08.969439 1008 webserver.cc:173] Starting webserver on 127.0.53.1:0 I0312 20:10:08.969549 1008 webserver.cc:184] Document root disabled I0312 20:10:08.970927 1008 webserver.cc:311] Webserver started. Bound to: http://127.0.53.1:54888/ I0312 20:10:08.971657 1008 server_base.cc:607] Dumped server information to /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ChurnyElections.1520885401735786-212/raft_consensus-itest-cluster/ts-0/data/info.pb I0312 20:10:08.974437 1008 tablet_server_main.cc:87] Tablet server successfully started. I0312 20:10:08.980695 212 external_mini_cluster.cc:867] Started /tmp/dist-test-task3bwPeE/build/tsan/bin/kudu-tserver as pid 1008 I0312 20:10:08.982013 212 external_mini_cluster.cc:805] Running /tmp/dist-test-task3bwPeE/build/tsan/bin/kudu-tserver /tmp/dist-test-task3bwPeE/build/tsan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ChurnyElections.1520885401735786-212/raft_consensus-itest-cluster/ts-1/wal --fs_data_dirs=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ChurnyElections.1520885401735786-212/raft_consensus-itest-cluster/ts-1/data --block_manager=log --rpc_bind_addresses=127.0.53.2:0 --local_ip_for_outbound_sockets=127.0.53.2 --webserver_interface=127.0.53.2 --webserver_port=0 --tserver_master_addrs=127.0.53.62:57865 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ChurnyElections.1520885401735786-212/raft_consensus-itest-cluster/ts-1/logs --server_dump_info_path=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ChurnyElections.1520885401735786-212/raft_consensus-itest-cluster/ts-1/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_heartbeat_interval_ms=5 with env {} I0312 20:10:08.975136 1140 process_memory.cc:182] Process hard memory limit is 11.777631 GB I0312 20:10:09.002722 1140 process_memory.cc:184] Process soft memory limit is 9.422105 GB I0312 20:10:09.002833 1140 process_memory.cc:187] Process memory pressure threshold is 7.066578 GB I0312 20:10:09.023085 1139 heartbeater.cc:346] Connected to a master server at 127.0.53.62:57865 I0312 20:10:09.023324 1139 heartbeater.cc:426] Registering TS with master... I0312 20:10:09.023754 1139 heartbeater.cc:475] Master 127.0.53.62:57865 requested a full tablet report, sending... I0312 20:10:09.025331 974 ts_manager.cc:81] Registered new tserver with Master: ded9b7f8677940fb91e1a8c8bceceaed (127.0.53.1:35037) I0312 20:10:09.026993 974 master_service.cc:243] Signed X509 certificate for tserver {username='slave'} at 127.0.53.1:39786 WARNING: Logging before InitGoogleLogging() is written to STDERR W0312 20:10:09.165341 1143 flags.cc:412] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0312 20:10:09.165794 1143 flags.cc:412] Enabled unsafe flag: --never_fsync=true W0312 20:10:09.189762 1143 flags.cc:412] Enabled experimental flag: --ipki_server_key_size=1024 W0312 20:10:09.190228 1143 flags.cc:412] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.53.2 I0312 20:10:09.220538 1143 tablet_server_main.cc:72] Tablet server non-default flags: --raft_heartbeat_interval_ms=5 --fs_data_dirs=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ChurnyElections.1520885401735786-212/raft_consensus-itest-cluster/ts-1/data --fs_wal_dir=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ChurnyElections.1520885401735786-212/raft_consensus-itest-cluster/ts-1/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.0.53.2:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ChurnyElections.1520885401735786-212/raft_consensus-itest-cluster/ts-1/data/info.pb --webserver_interface=127.0.53.2 --webserver_port=0 --tserver_master_addrs=127.0.53.62:57865 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.0.53.2 --log_dir=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ChurnyElections.1520885401735786-212/raft_consensus-itest-cluster/ts-1/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.7.0-SNAPSHOT revision 13e10c63c46de30ac85f34b543cdd6fb44881f61 build type FASTDEBUG built by None at 12 Mar 2018 20:06:40 UTC on jenkins-slave-vn0g build id 12388 TSAN enabled I0312 20:10:09.235347 1143 tablet_server_main.cc:79] Initializing tablet server... I0312 20:10:09.237568 1143 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 1071776us I0312 20:10:09.238131 1143 fs_manager.cc:260] Metadata directory not provided I0312 20:10:09.238281 1143 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0312 20:10:09.238528 1143 server_base.cc:434] Could not load existing FS layout: Not found: /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ChurnyElections.1520885401735786-212/raft_consensus-itest-cluster/ts-1/data/instance: No such file or directory (error 2) I0312 20:10:09.238672 1143 server_base.cc:435] Attempting to create new FS layout instead I0312 20:10:09.250270 1143 fs_manager.cc:596] Generated new instance metadata in path /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ChurnyElections.1520885401735786-212/raft_consensus-itest-cluster/ts-1/data/instance: uuid: "6885de38238442d6a9e5f1eb9fb569ff" format_stamp: "Formatted at 2018-03-12 20:10:09 on dist-test-slave-dist-test-slave-f27p" I0312 20:10:09.251212 1143 fs_manager.cc:596] Generated new instance metadata in path /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ChurnyElections.1520885401735786-212/raft_consensus-itest-cluster/ts-1/wal/instance: uuid: "6885de38238442d6a9e5f1eb9fb569ff" format_stamp: "Formatted at 2018-03-12 20:10:09 on dist-test-slave-dist-test-slave-f27p" I0312 20:10:09.263942 1143 fs_manager.cc:496] Time spent creating directory manager: real 0.012s user 0.016s sys 0.000s I0312 20:10:09.265678 1143 env_posix.cc:1620] Not raising this process' open files per process limit of 65536; it is already as high as it can go I0312 20:10:09.266077 1143 file_cache.cc:470] Constructed file cache lbm with capacity 26214 I0312 20:10:09.276409 1143 fs_manager.cc:418] Time spent opening block manager: real 0.002s user 0.004s sys 0.000s I0312 20:10:09.276563 1143 fs_manager.cc:429] Opened local filesystem: /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ChurnyElections.1520885401735786-212/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ChurnyElections.1520885401735786-212/raft_consensus-itest-cluster/ts-1/wal uuid: "6885de38238442d6a9e5f1eb9fb569ff" format_stamp: "Formatted at 2018-03-12 20:10:09 on dist-test-slave-dist-test-slave-f27p" I0312 20:10:09.276839 1143 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ChurnyElections.1520885401735786-212/raft_consensus-itest-cluster/ts-1/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0312 20:10:09.489271 1143 env_posix.cc:1620] Not raising this process' running threads per effective uid limit of 60284; it is already as high as it can go I0312 20:10:09.491071 1143 ts_tablet_manager.cc:349] Loaded tablet metadata (0 live tablets) I0312 20:10:09.492328 1143 tablet_server_main.cc:84] Starting tablet server... I0312 20:10:09.618582 1143 rpc_server.cc:201] RPC server started. Bound to: 127.0.53.2:51697 I0312 20:10:09.619483 1143 webserver.cc:173] Starting webserver on 127.0.53.2:0 I0312 20:10:09.619593 1143 webserver.cc:184] Document root disabled I0312 20:10:09.621052 1143 webserver.cc:311] Webserver started. Bound to: http://127.0.53.2:50281/ I0312 20:10:09.621814 1143 server_base.cc:607] Dumped server information to /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ChurnyElections.1520885401735786-212/raft_consensus-itest-cluster/ts-1/data/info.pb I0312 20:10:09.627095 1265 process_memory.cc:182] Process hard memory limit is 11.777631 GB I0312 20:10:09.627336 1265 process_memory.cc:184] Process soft memory limit is 9.422105 GB I0312 20:10:09.627493 1265 process_memory.cc:187] Process memory pressure threshold is 7.066578 GB I0312 20:10:09.628718 212 external_mini_cluster.cc:867] Started /tmp/dist-test-task3bwPeE/build/tsan/bin/kudu-tserver as pid 1143 I0312 20:10:09.628906 1143 tablet_server_main.cc:87] Tablet server successfully started. I0312 20:10:09.629847 212 external_mini_cluster.cc:805] Running /tmp/dist-test-task3bwPeE/build/tsan/bin/kudu-tserver /tmp/dist-test-task3bwPeE/build/tsan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ChurnyElections.1520885401735786-212/raft_consensus-itest-cluster/ts-2/wal --fs_data_dirs=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ChurnyElections.1520885401735786-212/raft_consensus-itest-cluster/ts-2/data --block_manager=log --rpc_bind_addresses=127.0.53.3:0 --local_ip_for_outbound_sockets=127.0.53.3 --webserver_interface=127.0.53.3 --webserver_port=0 --tserver_master_addrs=127.0.53.62:57865 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ChurnyElections.1520885401735786-212/raft_consensus-itest-cluster/ts-2/logs --server_dump_info_path=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ChurnyElections.1520885401735786-212/raft_consensus-itest-cluster/ts-2/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_heartbeat_interval_ms=5 with env {} I0312 20:10:09.654326 1264 heartbeater.cc:346] Connected to a master server at 127.0.53.62:57865 I0312 20:10:09.654603 1264 heartbeater.cc:426] Registering TS with master... I0312 20:10:09.655151 1264 heartbeater.cc:475] Master 127.0.53.62:57865 requested a full tablet report, sending... I0312 20:10:09.656884 974 ts_manager.cc:81] Registered new tserver with Master: 6885de38238442d6a9e5f1eb9fb569ff (127.0.53.2:51697) I0312 20:10:09.658572 974 master_service.cc:243] Signed X509 certificate for tserver {username='slave'} at 127.0.53.2:43315 WARNING: Logging before InitGoogleLogging() is written to STDERR W0312 20:10:09.746156 1268 flags.cc:412] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0312 20:10:09.746474 1268 flags.cc:412] Enabled unsafe flag: --never_fsync=true W0312 20:10:09.761008 1268 flags.cc:412] Enabled experimental flag: --ipki_server_key_size=1024 W0312 20:10:09.761327 1268 flags.cc:412] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.53.3 I0312 20:10:09.780993 1268 tablet_server_main.cc:72] Tablet server non-default flags: --raft_heartbeat_interval_ms=5 --fs_data_dirs=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ChurnyElections.1520885401735786-212/raft_consensus-itest-cluster/ts-2/data --fs_wal_dir=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ChurnyElections.1520885401735786-212/raft_consensus-itest-cluster/ts-2/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.0.53.3:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ChurnyElections.1520885401735786-212/raft_consensus-itest-cluster/ts-2/data/info.pb --webserver_interface=127.0.53.3 --webserver_port=0 --tserver_master_addrs=127.0.53.62:57865 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.0.53.3 --log_dir=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ChurnyElections.1520885401735786-212/raft_consensus-itest-cluster/ts-2/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.7.0-SNAPSHOT revision 13e10c63c46de30ac85f34b543cdd6fb44881f61 build type FASTDEBUG built by None at 12 Mar 2018 20:06:40 UTC on jenkins-slave-vn0g build id 12388 TSAN enabled I0312 20:10:09.791275 1268 tablet_server_main.cc:79] Initializing tablet server... I0312 20:10:09.793092 1268 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 1071776us I0312 20:10:09.793530 1268 fs_manager.cc:260] Metadata directory not provided I0312 20:10:09.793645 1268 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0312 20:10:09.793855 1268 server_base.cc:434] Could not load existing FS layout: Not found: /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ChurnyElections.1520885401735786-212/raft_consensus-itest-cluster/ts-2/data/instance: No such file or directory (error 2) I0312 20:10:09.794023 1268 server_base.cc:435] Attempting to create new FS layout instead I0312 20:10:09.801606 1268 fs_manager.cc:596] Generated new instance metadata in path /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ChurnyElections.1520885401735786-212/raft_consensus-itest-cluster/ts-2/data/instance: uuid: "f78a2351568c4a268ee1112681e22bec" format_stamp: "Formatted at 2018-03-12 20:10:09 on dist-test-slave-dist-test-slave-f27p" I0312 20:10:09.802345 1268 fs_manager.cc:596] Generated new instance metadata in path /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ChurnyElections.1520885401735786-212/raft_consensus-itest-cluster/ts-2/wal/instance: uuid: "f78a2351568c4a268ee1112681e22bec" format_stamp: "Formatted at 2018-03-12 20:10:09 on dist-test-slave-dist-test-slave-f27p" I0312 20:10:09.810914 1268 fs_manager.cc:496] Time spent creating directory manager: real 0.008s user 0.008s sys 0.000s I0312 20:10:09.812067 1268 env_posix.cc:1620] Not raising this process' open files per process limit of 65536; it is already as high as it can go I0312 20:10:09.812352 1268 file_cache.cc:470] Constructed file cache lbm with capacity 26214 I0312 20:10:09.818891 1268 fs_manager.cc:418] Time spent opening block manager: real 0.002s user 0.000s sys 0.000s I0312 20:10:09.819025 1268 fs_manager.cc:429] Opened local filesystem: /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ChurnyElections.1520885401735786-212/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ChurnyElections.1520885401735786-212/raft_consensus-itest-cluster/ts-2/wal uuid: "f78a2351568c4a268ee1112681e22bec" format_stamp: "Formatted at 2018-03-12 20:10:09 on dist-test-slave-dist-test-slave-f27p" I0312 20:10:09.819254 1268 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ChurnyElections.1520885401735786-212/raft_consensus-itest-cluster/ts-2/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0312 20:10:10.025599 1268 env_posix.cc:1620] Not raising this process' running threads per effective uid limit of 60284; it is already as high as it can go I0312 20:10:10.026917 1268 ts_tablet_manager.cc:349] Loaded tablet metadata (0 live tablets) I0312 20:10:10.027899 1268 tablet_server_main.cc:84] Starting tablet server... I0312 20:10:10.029467 1139 heartbeater.cc:467] Master 127.0.53.62:57865 was elected leader, sending a full tablet report... I0312 20:10:10.115052 1268 rpc_server.cc:201] RPC server started. Bound to: 127.0.53.3:43274 I0312 20:10:10.115916 1268 webserver.cc:173] Starting webserver on 127.0.53.3:0 I0312 20:10:10.116014 1268 webserver.cc:184] Document root disabled I0312 20:10:10.117259 1268 webserver.cc:311] Webserver started. Bound to: http://127.0.53.3:40044/ I0312 20:10:10.117981 1268 server_base.cc:607] Dumped server information to /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ChurnyElections.1520885401735786-212/raft_consensus-itest-cluster/ts-2/data/info.pb I0312 20:10:10.121217 1268 tablet_server_main.cc:87] Tablet server successfully started. I0312 20:10:10.123288 212 external_mini_cluster.cc:867] Started /tmp/dist-test-task3bwPeE/build/tsan/bin/kudu-tserver as pid 1268 I0312 20:10:10.121218 1390 process_memory.cc:182] Process hard memory limit is 11.777631 GB I0312 20:10:10.147342 1390 process_memory.cc:184] Process soft memory limit is 9.422105 GB I0312 20:10:10.147469 1390 process_memory.cc:187] Process memory pressure threshold is 7.066578 GB I0312 20:10:10.167109 1389 heartbeater.cc:346] Connected to a master server at 127.0.53.62:57865 I0312 20:10:10.167346 1389 heartbeater.cc:426] Registering TS with master... I0312 20:10:10.167856 1389 heartbeater.cc:475] Master 127.0.53.62:57865 requested a full tablet report, sending... I0312 20:10:10.169276 974 ts_manager.cc:81] Registered new tserver with Master: f78a2351568c4a268ee1112681e22bec (127.0.53.3:43274) I0312 20:10:10.170454 974 master_service.cc:243] Signed X509 certificate for tserver {username='slave'} at 127.0.53.3:40529 I0312 20:10:10.171386 212 external_mini_cluster.cc:498] 3 TS(s) registered with all masters I0312 20:10:10.174388 212 test_util.cc:192] Using random seed: -1164044899 I0312 20:10:10.190675 974 catalog_manager.cc:1294] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40812: name: "test-workload" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 3 split_rows_range_bounds { } partition_schema { range_schema { columns { name: "key" } } } W0312 20:10:10.199229 974 catalog_manager.cc:1450] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a replica failure: 4 tablet servers are needed, 3 are alive. Consider bringing up additional tablet server(s) or running both the masters and all tablet servers with --raft_prepare_replacement_before_eviction=false flag (not recommended). I0312 20:10:10.239231 1326 tablet_service.cc:755] Processing CreateTablet for tablet 92318a1b744e4baa9c7d3f3b9758739b (table=test-workload [id=f4dc283773f24c1e91504cc03e44eb02]), partition=RANGE (key) PARTITION UNBOUNDED I0312 20:10:10.239964 1326 data_dirs.cc:931] Could only allocate 1 dirs of requested 3 for tablet 92318a1b744e4baa9c7d3f3b9758739b. 1 dirs total, 0 dirs full, 0 dirs failed I0312 20:10:10.241320 1201 tablet_service.cc:755] Processing CreateTablet for tablet 92318a1b744e4baa9c7d3f3b9758739b (table=test-workload [id=f4dc283773f24c1e91504cc03e44eb02]), partition=RANGE (key) PARTITION UNBOUNDED I0312 20:10:10.242182 1201 data_dirs.cc:931] Could only allocate 1 dirs of requested 3 for tablet 92318a1b744e4baa9c7d3f3b9758739b. 1 dirs total, 0 dirs full, 0 dirs failed I0312 20:10:10.242641 1076 tablet_service.cc:755] Processing CreateTablet for tablet 92318a1b744e4baa9c7d3f3b9758739b (table=test-workload [id=f4dc283773f24c1e91504cc03e44eb02]), partition=RANGE (key) PARTITION UNBOUNDED I0312 20:10:10.243468 1076 data_dirs.cc:931] Could only allocate 1 dirs of requested 3 for tablet 92318a1b744e4baa9c7d3f3b9758739b. 1 dirs total, 0 dirs full, 0 dirs failed I0312 20:10:10.252171 1326 ts_tablet_manager.cc:1163] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec: Registered tablet (data state: TABLET_DATA_READY) I0312 20:10:10.258479 1201 ts_tablet_manager.cc:1163] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff: Registered tablet (data state: TABLET_DATA_READY) I0312 20:10:10.259363 1076 ts_tablet_manager.cc:1163] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed: Registered tablet (data state: TABLET_DATA_READY) I0312 20:10:10.260340 1410 ts_tablet_manager.cc:1018] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff: Bootstrapping tablet I0312 20:10:10.260361 1409 ts_tablet_manager.cc:1018] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec: Bootstrapping tablet I0312 20:10:10.260769 1410 tablet_bootstrap.cc:436] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff: Bootstrap starting. I0312 20:10:10.261492 1409 tablet_bootstrap.cc:436] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec: Bootstrap starting. I0312 20:10:10.263631 1410 tablet_bootstrap.cc:581] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff: No blocks or log segments found. Creating new log. I0312 20:10:10.264467 1409 tablet_bootstrap.cc:581] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec: No blocks or log segments found. Creating new log. I0312 20:10:10.264827 1410 log.cc:520] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff: Log is configured to *not* fsync() on all Append() calls I0312 20:10:10.266017 1409 log.cc:520] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec: Log is configured to *not* fsync() on all Append() calls I0312 20:10:10.266054 1411 ts_tablet_manager.cc:1018] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed: Bootstrapping tablet I0312 20:10:10.266582 1411 tablet_bootstrap.cc:436] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed: Bootstrap starting. I0312 20:10:10.269573 1411 tablet_bootstrap.cc:581] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed: No blocks or log segments found. Creating new log. I0312 20:10:10.269726 1410 tablet_bootstrap.cc:436] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff: No bootstrap required, opened a new log I0312 20:10:10.271153 1411 log.cc:520] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed: Log is configured to *not* fsync() on all Append() calls I0312 20:10:10.270117 1410 ts_tablet_manager.cc:1035] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s I0312 20:10:10.273778 1409 tablet_bootstrap.cc:436] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec: No bootstrap required, opened a new log I0312 20:10:10.274062 1409 ts_tablet_manager.cc:1035] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.008s I0312 20:10:10.274369 1411 tablet_bootstrap.cc:436] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed: No bootstrap required, opened a new log I0312 20:10:10.274727 1411 ts_tablet_manager.cc:1035] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.004s I0312 20:10:10.283926 1409 raft_consensus.cc:304] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:10.284461 1409 raft_consensus.cc:330] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0312 20:10:10.284637 1409 raft_consensus.cc:605] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f78a2351568c4a268ee1112681e22bec, State: Initialized, Role: FOLLOWER I0312 20:10:10.289273 1410 raft_consensus.cc:304] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:10.290016 1410 raft_consensus.cc:330] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0312 20:10:10.290190 1410 raft_consensus.cc:605] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6885de38238442d6a9e5f1eb9fb569ff, State: Initialized, Role: FOLLOWER I0312 20:10:10.289988 1411 raft_consensus.cc:304] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:10.290709 1411 raft_consensus.cc:330] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0312 20:10:10.290892 1411 raft_consensus.cc:605] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ded9b7f8677940fb91e1a8c8bceceaed, State: Initialized, Role: FOLLOWER I0312 20:10:10.290832 1410 consensus_queue.cc:219] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:10.291517 1411 consensus_queue.cc:219] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:10.294430 1264 heartbeater.cc:467] Master 127.0.53.62:57865 was elected leader, sending a full tablet report... I0312 20:10:10.295339 1418 raft_consensus.cc:436] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0312 20:10:10.295650 1418 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed: Snoozing failure detection for 0.020s (starting election) I0312 20:10:10.295564 1409 consensus_queue.cc:219] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:10.295902 1418 raft_consensus.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:10.298000 1418 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer f78a2351568c4a268ee1112681e22bec I0312 20:10:10.298064 1409 ts_tablet_manager.cc:1063] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec: Time spent starting tablet: real 0.024s user 0.008s sys 0.016s I0312 20:10:10.298478 1389 heartbeater.cc:467] Master 127.0.53.62:57865 was elected leader, sending a full tablet report... I0312 20:10:10.301380 1411 ts_tablet_manager.cc:1063] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed: Time spent starting tablet: real 0.026s user 0.016s sys 0.004s I0312 20:10:10.301940 1415 raft_consensus.cc:436] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0312 20:10:10.302208 1415 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec: Snoozing failure detection for 0.015s (starting election) I0312 20:10:10.302906 1410 ts_tablet_manager.cc:1063] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff: Time spent starting tablet: real 0.031s user 0.020s sys 0.000s I0312 20:10:10.302587 1415 raft_consensus.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:10.303856 1415 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 6885de38238442d6a9e5f1eb9fb569ff I0312 20:10:10.305660 1415 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer ded9b7f8677940fb91e1a8c8bceceaed I0312 20:10:10.305555 1418 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 6885de38238442d6a9e5f1eb9fb569ff I0312 20:10:10.318528 1415 raft_consensus.cc:436] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0312 20:10:10.318735 1415 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec: Snoozing failure detection for 0.021s (starting election) I0312 20:10:10.318948 1415 raft_consensus.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } W0312 20:10:10.319396 1027 leader_election.cc:287] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer f78a2351568c4a268ee1112681e22bec: Timed out: connection negotiation to 127.0.53.3:43274 for RPC RequestConsensusVote timed out after 0.020s (ON_OUTBOUND_QUEUE) I0312 20:10:10.319896 1415 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 6885de38238442d6a9e5f1eb9fb569ff W0312 20:10:10.320765 1279 leader_election.cc:287] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 6885de38238442d6a9e5f1eb9fb569ff: Timed out: connection negotiation to 127.0.53.2:51697 for RPC RequestConsensusVote timed out after 0.015s (ON_OUTBOUND_QUEUE) W0312 20:10:10.321930 1278 leader_election.cc:287] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer ded9b7f8677940fb91e1a8c8bceceaed: Timed out: connection negotiation to 127.0.53.1:35037 for RPC RequestConsensusVote timed out after 0.015s (ON_OUTBOUND_QUEUE) I0312 20:10:10.322201 1278 leader_election.cc:258] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. I0312 20:10:10.322939 1420 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec: Snoozing failure detection for 0.015s (election complete) I0312 20:10:10.323196 1420 raft_consensus.cc:2455] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority I0312 20:10:10.320854 1415 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer ded9b7f8677940fb91e1a8c8bceceaed W0312 20:10:10.334477 1029 leader_election.cc:287] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 6885de38238442d6a9e5f1eb9fb569ff: Timed out: connection negotiation to 127.0.53.2:51697 for RPC RequestConsensusVote timed out after 0.020s (ON_OUTBOUND_QUEUE) I0312 20:10:10.334695 1029 leader_election.cc:258] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. I0312 20:10:10.335621 1417 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed: Snoozing failure detection for 0.016s (election complete) I0312 20:10:10.335876 1417 raft_consensus.cc:2455] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority I0312 20:10:10.336127 1417 raft_consensus.cc:436] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0312 20:10:10.336297 1417 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed: Snoozing failure detection for 0.032s (starting election) I0312 20:10:10.336521 1417 raft_consensus.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:10.338141 1417 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer f78a2351568c4a268ee1112681e22bec I0312 20:10:10.338562 1417 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 6885de38238442d6a9e5f1eb9fb569ff I0312 20:10:10.339083 1415 raft_consensus.cc:436] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0312 20:10:10.339287 1415 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec: Snoozing failure detection for 0.024s (starting election) I0312 20:10:10.339507 1415 raft_consensus.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:10.340415 1415 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 6885de38238442d6a9e5f1eb9fb569ff I0312 20:10:10.340762 1415 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer ded9b7f8677940fb91e1a8c8bceceaed W0312 20:10:10.342725 1279 leader_election.cc:287] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 6885de38238442d6a9e5f1eb9fb569ff: Timed out: connection negotiation to 127.0.53.2:51697 for RPC RequestConsensusVote timed out after 0.021s (ON_OUTBOUND_QUEUE) I0312 20:10:10.345151 1096 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "f78a2351568c4a268ee1112681e22bec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" is_pre_election: true I0312 20:10:10.345718 1096 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed: Snoozing failure detection for 0.033s (vote granted) I0312 20:10:10.345933 1096 raft_consensus.cc:2215] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f78a2351568c4a268ee1112681e22bec in term 0. W0312 20:10:10.346770 1278 leader_election.cc:287] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer ded9b7f8677940fb91e1a8c8bceceaed: Timed out: RequestConsensusVote RPC to 127.0.53.1:35037 timed out after 0.021s (SENT) I0312 20:10:10.346971 1278 leader_election.cc:258] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. I0312 20:10:10.347479 1415 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec: Snoozing failure detection for 0.030s (election complete) I0312 20:10:10.347671 1415 raft_consensus.cc:2455] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority I0312 20:10:10.348335 1095 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "f78a2351568c4a268ee1112681e22bec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" is_pre_election: true I0312 20:10:10.348804 1095 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed: Snoozing failure detection for 0.029s (vote granted) I0312 20:10:10.349028 1095 raft_consensus.cc:2215] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f78a2351568c4a268ee1112681e22bec in term 0. I0312 20:10:10.350495 1278 leader_election.cc:363] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 1 pre-election: Vote granted by peer ded9b7f8677940fb91e1a8c8bceceaed I0312 20:10:10.350719 1278 leader_election.cc:258] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. I0312 20:10:10.351229 1415 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec: Snoozing failure detection for 0.042s (election complete) I0312 20:10:10.351438 1415 raft_consensus.cc:2507] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 0 FOLLOWER]: Leader pre-election won for term 1 I0312 20:10:10.351616 1415 raft_consensus.cc:436] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0312 20:10:10.351788 1415 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec: Snoozing failure detection for 0.035s (starting election) I0312 20:10:10.351954 1415 raft_consensus.cc:2752] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 0 FOLLOWER]: Advancing to term 1 I0312 20:10:10.356226 1415 raft_consensus.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:10.361717 1415 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 1 election: Requesting vote from peer 6885de38238442d6a9e5f1eb9fb569ff I0312 20:10:10.362107 1415 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 1 election: Requesting vote from peer ded9b7f8677940fb91e1a8c8bceceaed I0312 20:10:10.363395 1095 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "f78a2351568c4a268ee1112681e22bec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" I0312 20:10:10.363765 1095 raft_consensus.cc:2752] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 0 FOLLOWER]: Advancing to term 1 I0312 20:10:10.363996 1095 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed: Snoozing failure detection for 0.024s (vote granted) W0312 20:10:10.366566 1279 leader_election.cc:287] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 6885de38238442d6a9e5f1eb9fb569ff: Timed out: connection negotiation to 127.0.53.2:51697 for RPC RequestConsensusVote timed out after 0.024s (ON_OUTBOUND_QUEUE) I0312 20:10:10.368098 1095 raft_consensus.cc:2215] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f78a2351568c4a268ee1112681e22bec in term 1. I0312 20:10:10.369087 1278 leader_election.cc:363] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 1 election: Vote granted by peer ded9b7f8677940fb91e1a8c8bceceaed I0312 20:10:10.369339 1278 leader_election.cc:258] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0312 20:10:10.369742 1420 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec: Snoozing failure detection for 0.033s (election complete) I0312 20:10:10.369946 1420 raft_consensus.cc:2507] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 1 FOLLOWER]: Leader election won for term 1 I0312 20:10:10.370203 1420 raft_consensus.cc:572] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 1 LEADER]: Becoming Leader. State: Replica: f78a2351568c4a268ee1112681e22bec, State: Running, Role: LEADER I0312 20:10:10.370651 1420 consensus_queue.cc:196] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } W0312 20:10:10.375790 1029 leader_election.cc:287] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 6885de38238442d6a9e5f1eb9fb569ff: Timed out: connection negotiation to 127.0.53.2:51697 for RPC RequestConsensusVote timed out after 0.032s (ON_OUTBOUND_QUEUE) W0312 20:10:10.376302 1027 leader_election.cc:287] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer f78a2351568c4a268ee1112681e22bec: Timed out: connection negotiation to 127.0.53.3:43274 for RPC RequestConsensusVote timed out after 0.032s (ON_OUTBOUND_QUEUE) I0312 20:10:10.376507 1027 leader_election.cc:258] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. I0312 20:10:10.377056 1418 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed: Snoozing failure detection for 0.015s (election complete) I0312 20:10:10.377274 1418 raft_consensus.cc:2455] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority I0312 20:10:10.389128 974 catalog_manager.cc:3592] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed reported cstate change: term changed from 0 to 1, leader changed from to f78a2351568c4a268ee1112681e22bec (127.0.53.3). New cstate: current_term: 1 leader_uuid: "f78a2351568c4a268ee1112681e22bec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } } W0312 20:10:10.397836 1390 tablet.cc:1679] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec: Can't schedule compaction. Clean time has not been advanced past its initial value. W0312 20:10:10.398427 1279 leader_election.cc:287] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 6885de38238442d6a9e5f1eb9fb569ff: Timed out: connection negotiation to 127.0.53.2:51697 for RPC RequestConsensusVote timed out after 0.035s (ON_OUTBOUND_QUEUE) I0312 20:10:10.302145 1416 raft_consensus.cc:436] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0312 20:10:10.501971 1416 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff: Snoozing failure detection for 0.021s (starting election) I0312 20:10:10.502282 1416 raft_consensus.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:10.503708 1416 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer f78a2351568c4a268ee1112681e22bec I0312 20:10:10.504245 1416 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer ded9b7f8677940fb91e1a8c8bceceaed I0312 20:10:10.538552 1418 raft_consensus.cc:436] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f78a2351568c4a268ee1112681e22bec) I0312 20:10:10.538789 1418 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed: Snoozing failure detection for 0.016s (starting election) I0312 20:10:10.539001 1418 raft_consensus.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:10.540212 1418 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer f78a2351568c4a268ee1112681e22bec I0312 20:10:10.540562 1418 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 6885de38238442d6a9e5f1eb9fb569ff I0312 20:10:10.542551 1346 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f78a2351568c4a268ee1112681e22bec" is_pre_election: true I0312 20:10:10.543212 1346 raft_consensus.cc:2176] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate ded9b7f8677940fb91e1a8c8bceceaed for term 2 because replica is either leader or believes a valid leader to be alive. I0312 20:10:10.544606 1027 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 2 pre-election: Vote denied by peer f78a2351568c4a268ee1112681e22bec. Message: Invalid argument: T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate ded9b7f8677940fb91e1a8c8bceceaed for term 2 because replica is either leader or believes a valid leader to be alive. W0312 20:10:10.558240 1029 leader_election.cc:287] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6885de38238442d6a9e5f1eb9fb569ff: Timed out: RequestConsensusVote RPC to 127.0.53.2:51697 timed out after 0.016s (SENT) I0312 20:10:10.558442 1029 leader_election.cc:258] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. I0312 20:10:10.559012 1418 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed: Snoozing failure detection for 0.019s (election complete) I0312 20:10:10.559204 1418 raft_consensus.cc:2455] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority I0312 20:10:10.605348 1418 raft_consensus.cc:436] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f78a2351568c4a268ee1112681e22bec) I0312 20:10:10.605592 1418 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed: Snoozing failure detection for 0.028s (starting election) I0312 20:10:10.605808 1418 raft_consensus.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:10.606766 1418 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer f78a2351568c4a268ee1112681e22bec I0312 20:10:10.607134 1418 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 6885de38238442d6a9e5f1eb9fb569ff I0312 20:10:10.610549 1346 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "f78a2351568c4a268ee1112681e22bec" is_pre_election: true I0312 20:10:10.611280 1346 raft_consensus.cc:2176] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate ded9b7f8677940fb91e1a8c8bceceaed for term 2 because replica is either leader or believes a valid leader to be alive. I0312 20:10:10.612937 1027 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 2 pre-election: Vote denied by peer f78a2351568c4a268ee1112681e22bec. Message: Invalid argument: T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate ded9b7f8677940fb91e1a8c8bceceaed for term 2 because replica is either leader or believes a valid leader to be alive. I0312 20:10:10.634255 1418 raft_consensus.cc:436] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f78a2351568c4a268ee1112681e22bec) I0312 20:10:10.634486 1418 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed: Snoozing failure detection for 0.033s (starting election) I0312 20:10:10.634699 1418 raft_consensus.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:10.635632 1418 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer f78a2351568c4a268ee1112681e22bec I0312 20:10:10.635973 1418 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 6885de38238442d6a9e5f1eb9fb569ff W0312 20:10:10.637387 1029 leader_election.cc:287] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6885de38238442d6a9e5f1eb9fb569ff: Timed out: RequestConsensusVote RPC to 127.0.53.2:51697 timed out after 0.028s (SENT) I0312 20:10:10.637595 1029 leader_election.cc:258] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. I0312 20:10:10.639041 1418 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed: Snoozing failure detection for 0.018s (election complete) I0312 20:10:10.639241 1418 raft_consensus.cc:2455] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority W0312 20:10:10.378140 1265 tablet.cc:1679] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff: Can't schedule compaction. Clean time has not been advanced past its initial value. I0312 20:10:10.640507 1346 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "f78a2351568c4a268ee1112681e22bec" is_pre_election: true I0312 20:10:10.641014 1346 raft_consensus.cc:2176] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate ded9b7f8677940fb91e1a8c8bceceaed for term 2 because replica is either leader or believes a valid leader to be alive. I0312 20:10:10.642218 1027 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 2 pre-election: Vote denied by peer f78a2351568c4a268ee1112681e22bec. Message: Invalid argument: T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate ded9b7f8677940fb91e1a8c8bceceaed for term 2 because replica is either leader or believes a valid leader to be alive. W0312 20:10:10.647094 1153 leader_election.cc:287] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer ded9b7f8677940fb91e1a8c8bceceaed: Timed out: connection negotiation to 127.0.53.1:35037 for RPC RequestConsensusVote timed out after 0.021s (ON_OUTBOUND_QUEUE) I0312 20:10:10.647459 1419 raft_consensus.cc:436] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0312 20:10:10.648895 1419 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff: Snoozing failure detection for 0.015s (starting election) I0312 20:10:10.649159 1419 raft_consensus.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:10.650465 1419 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer f78a2351568c4a268ee1112681e22bec I0312 20:10:10.650646 1221 raft_consensus.cc:2752] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 0 FOLLOWER]: Advancing to term 1 I0312 20:10:10.650960 1419 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer ded9b7f8677940fb91e1a8c8bceceaed W0312 20:10:10.651943 1152 leader_election.cc:287] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer f78a2351568c4a268ee1112681e22bec: Timed out: connection negotiation to 127.0.53.3:43274 for RPC RequestConsensusVote timed out after 0.021s (ON_OUTBOUND_QUEUE) I0312 20:10:10.652317 1152 leader_election.cc:258] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. I0312 20:10:10.654345 1220 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6885de38238442d6a9e5f1eb9fb569ff" is_pre_election: true I0312 20:10:10.655738 1219 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "6885de38238442d6a9e5f1eb9fb569ff" is_pre_election: true I0312 20:10:10.657407 1419 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff: Snoozing failure detection for 0.017s (election complete) I0312 20:10:10.657572 1419 raft_consensus.cc:2455] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority I0312 20:10:10.657891 1220 raft_consensus.cc:2190] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ded9b7f8677940fb91e1a8c8bceceaed for term 2 because replica is already servicing an update from a current leader or another vote. I0312 20:10:10.658869 1219 raft_consensus.cc:2190] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ded9b7f8677940fb91e1a8c8bceceaed for term 2 because replica is already servicing an update from a current leader or another vote. I0312 20:10:10.664074 1220 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "6885de38238442d6a9e5f1eb9fb569ff" is_pre_election: true I0312 20:10:10.664594 1220 raft_consensus.cc:2176] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ded9b7f8677940fb91e1a8c8bceceaed for term 2 because replica is either leader or believes a valid leader to be alive. I0312 20:10:10.666080 1029 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 2 pre-election: Vote denied by peer 6885de38238442d6a9e5f1eb9fb569ff. Message: Invalid argument: T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ded9b7f8677940fb91e1a8c8bceceaed for term 2 because replica is either leader or believes a valid leader to be alive. I0312 20:10:10.666302 1029 leader_election.cc:258] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. I0312 20:10:10.666894 1418 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed: Snoozing failure detection for 0.036s (election complete) I0312 20:10:10.667090 1418 raft_consensus.cc:2455] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority W0312 20:10:10.670104 1152 leader_election.cc:287] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer f78a2351568c4a268ee1112681e22bec: Timed out: connection negotiation to 127.0.53.3:43274 for RPC RequestConsensusVote timed out after 0.015s (ON_OUTBOUND_QUEUE) W0312 20:10:10.671489 1153 leader_election.cc:287] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer ded9b7f8677940fb91e1a8c8bceceaed: Timed out: connection negotiation to 127.0.53.1:35037 for RPC RequestConsensusVote timed out after 0.015s (ON_OUTBOUND_QUEUE) I0312 20:10:10.671738 1153 leader_election.cc:258] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. I0312 20:10:10.672152 1419 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff: Snoozing failure detection for 0.016s (election complete) I0312 20:10:10.672358 1419 raft_consensus.cc:2455] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority I0312 20:10:10.703161 1418 raft_consensus.cc:436] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f78a2351568c4a268ee1112681e22bec) I0312 20:10:10.703394 1418 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed: Snoozing failure detection for 0.054s (starting election) I0312 20:10:10.703603 1418 raft_consensus.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:10.704601 1418 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer f78a2351568c4a268ee1112681e22bec I0312 20:10:10.704964 1418 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 6885de38238442d6a9e5f1eb9fb569ff I0312 20:10:10.709170 1219 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" candidate_term: 2 candidate_status { last_received { term: 1 index: 20 } } ignore_live_leader: false dest_uuid: "6885de38238442d6a9e5f1eb9fb569ff" is_pre_election: true I0312 20:10:10.712442 1346 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" candidate_term: 2 candidate_status { last_received { term: 1 index: 20 } } ignore_live_leader: false dest_uuid: "f78a2351568c4a268ee1112681e22bec" is_pre_election: true I0312 20:10:10.713001 1346 raft_consensus.cc:2176] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate ded9b7f8677940fb91e1a8c8bceceaed for term 2 because replica is either leader or believes a valid leader to be alive. I0312 20:10:10.714542 1027 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 2 pre-election: Vote denied by peer f78a2351568c4a268ee1112681e22bec. Message: Invalid argument: T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate ded9b7f8677940fb91e1a8c8bceceaed for term 2 because replica is either leader or believes a valid leader to be alive. I0312 20:10:10.719377 1219 raft_consensus.cc:2190] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ded9b7f8677940fb91e1a8c8bceceaed for term 2 because replica is already servicing an update from a current leader or another vote. I0312 20:10:10.720398 1419 raft_consensus.cc:436] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f78a2351568c4a268ee1112681e22bec) I0312 20:10:10.720631 1419 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff: Snoozing failure detection for 0.019s (starting election) I0312 20:10:10.720866 1419 raft_consensus.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:10.722043 1419 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer f78a2351568c4a268ee1112681e22bec I0312 20:10:10.722390 1419 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer ded9b7f8677940fb91e1a8c8bceceaed I0312 20:10:10.724745 1346 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "6885de38238442d6a9e5f1eb9fb569ff" candidate_term: 2 candidate_status { last_received { term: 1 index: 17 } } ignore_live_leader: false dest_uuid: "f78a2351568c4a268ee1112681e22bec" is_pre_election: true I0312 20:10:10.726974 1346 raft_consensus.cc:2176] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate 6885de38238442d6a9e5f1eb9fb569ff for term 2 because replica is either leader or believes a valid leader to be alive. I0312 20:10:10.727103 1029 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 2 pre-election: Vote denied by peer 6885de38238442d6a9e5f1eb9fb569ff. Message: Service unavailable: T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ded9b7f8677940fb91e1a8c8bceceaed for term 2 because replica is already servicing an update from a current leader or another vote. I0312 20:10:10.727320 1029 leader_election.cc:258] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. I0312 20:10:10.728224 1418 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed: Snoozing failure detection for 0.023s (election complete) I0312 20:10:10.728430 1418 raft_consensus.cc:2455] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority I0312 20:10:10.728673 1152 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 2 pre-election: Vote denied by peer f78a2351568c4a268ee1112681e22bec. Message: Invalid argument: T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate 6885de38238442d6a9e5f1eb9fb569ff for term 2 because replica is either leader or believes a valid leader to be alive. I0312 20:10:10.728870 1096 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "6885de38238442d6a9e5f1eb9fb569ff" candidate_term: 2 candidate_status { last_received { term: 1 index: 17 } } ignore_live_leader: false dest_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" is_pre_election: true I0312 20:10:10.729463 1096 raft_consensus.cc:2176] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6885de38238442d6a9e5f1eb9fb569ff for term 2 because replica is either leader or believes a valid leader to be alive. I0312 20:10:10.730433 1153 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 2 pre-election: Vote denied by peer ded9b7f8677940fb91e1a8c8bceceaed. Message: Invalid argument: T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6885de38238442d6a9e5f1eb9fb569ff for term 2 because replica is either leader or believes a valid leader to be alive. I0312 20:10:10.730657 1153 leader_election.cc:258] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. I0312 20:10:10.731310 1419 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff: Snoozing failure detection for 0.017s (election complete) I0312 20:10:10.731513 1419 raft_consensus.cc:2455] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority I0312 20:10:10.762192 1419 raft_consensus.cc:436] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f78a2351568c4a268ee1112681e22bec) I0312 20:10:10.762439 1419 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff: Snoozing failure detection for 0.042s (starting election) I0312 20:10:10.762665 1419 raft_consensus.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:10.763790 1419 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer f78a2351568c4a268ee1112681e22bec I0312 20:10:10.764147 1419 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer ded9b7f8677940fb91e1a8c8bceceaed I0312 20:10:10.765903 1095 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "6885de38238442d6a9e5f1eb9fb569ff" candidate_term: 2 candidate_status { last_received { term: 1 index: 24 } } ignore_live_leader: false dest_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" is_pre_election: true I0312 20:10:10.766491 1095 raft_consensus.cc:2176] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6885de38238442d6a9e5f1eb9fb569ff for term 2 because replica is either leader or believes a valid leader to be alive. I0312 20:10:10.767382 1153 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 2 pre-election: Vote denied by peer ded9b7f8677940fb91e1a8c8bceceaed. Message: Invalid argument: T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6885de38238442d6a9e5f1eb9fb569ff for term 2 because replica is either leader or believes a valid leader to be alive. I0312 20:10:10.767668 1346 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "6885de38238442d6a9e5f1eb9fb569ff" candidate_term: 2 candidate_status { last_received { term: 1 index: 24 } } ignore_live_leader: false dest_uuid: "f78a2351568c4a268ee1112681e22bec" is_pre_election: true I0312 20:10:10.770217 1346 raft_consensus.cc:2176] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate 6885de38238442d6a9e5f1eb9fb569ff for term 2 because replica is either leader or believes a valid leader to be alive. I0312 20:10:10.771697 1152 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 2 pre-election: Vote denied by peer f78a2351568c4a268ee1112681e22bec. Message: Invalid argument: T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate 6885de38238442d6a9e5f1eb9fb569ff for term 2 because replica is either leader or believes a valid leader to be alive. I0312 20:10:10.771924 1152 leader_election.cc:258] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. I0312 20:10:10.775650 1418 raft_consensus.cc:436] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f78a2351568c4a268ee1112681e22bec) I0312 20:10:10.775895 1418 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed: Snoozing failure detection for 0.057s (starting election) I0312 20:10:10.776115 1418 raft_consensus.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:10.777243 1418 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer f78a2351568c4a268ee1112681e22bec I0312 20:10:10.777596 1418 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 6885de38238442d6a9e5f1eb9fb569ff I0312 20:10:10.778340 1419 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff: Snoozing failure detection for 0.041s (election complete) I0312 20:10:10.778543 1419 raft_consensus.cc:2455] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority I0312 20:10:10.778857 1219 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" candidate_term: 2 candidate_status { last_received { term: 1 index: 27 } } ignore_live_leader: false dest_uuid: "6885de38238442d6a9e5f1eb9fb569ff" is_pre_election: true I0312 20:10:10.779404 1219 raft_consensus.cc:2176] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ded9b7f8677940fb91e1a8c8bceceaed for term 2 because replica is either leader or believes a valid leader to be alive. I0312 20:10:10.779345 1346 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" candidate_term: 2 candidate_status { last_received { term: 1 index: 27 } } ignore_live_leader: false dest_uuid: "f78a2351568c4a268ee1112681e22bec" is_pre_election: true I0312 20:10:10.779930 1346 raft_consensus.cc:2176] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate ded9b7f8677940fb91e1a8c8bceceaed for term 2 because replica is either leader or believes a valid leader to be alive. I0312 20:10:10.780710 1029 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 2 pre-election: Vote denied by peer 6885de38238442d6a9e5f1eb9fb569ff. Message: Invalid argument: T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ded9b7f8677940fb91e1a8c8bceceaed for term 2 because replica is either leader or believes a valid leader to be alive. I0312 20:10:10.781095 1027 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 2 pre-election: Vote denied by peer f78a2351568c4a268ee1112681e22bec. Message: Invalid argument: T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate ded9b7f8677940fb91e1a8c8bceceaed for term 2 because replica is either leader or believes a valid leader to be alive. I0312 20:10:10.781314 1027 leader_election.cc:258] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. I0312 20:10:10.781726 1418 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed: Snoozing failure detection for 0.104s (election complete) I0312 20:10:10.781947 1418 raft_consensus.cc:2455] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority I0312 20:10:10.800339 1418 raft_consensus.cc:436] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f78a2351568c4a268ee1112681e22bec) I0312 20:10:10.800623 1418 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed: Snoozing failure detection for 0.123s (starting election) I0312 20:10:10.800894 1418 raft_consensus.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:10.801977 1418 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer f78a2351568c4a268ee1112681e22bec I0312 20:10:10.802352 1418 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 6885de38238442d6a9e5f1eb9fb569ff I0312 20:10:10.803423 1346 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" candidate_term: 2 candidate_status { last_received { term: 1 index: 29 } } ignore_live_leader: false dest_uuid: "f78a2351568c4a268ee1112681e22bec" is_pre_election: true I0312 20:10:10.806571 1221 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" candidate_term: 2 candidate_status { last_received { term: 1 index: 29 } } ignore_live_leader: false dest_uuid: "6885de38238442d6a9e5f1eb9fb569ff" is_pre_election: true I0312 20:10:10.807090 1221 raft_consensus.cc:2190] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ded9b7f8677940fb91e1a8c8bceceaed for term 2 because replica is already servicing an update from a current leader or another vote. I0312 20:10:10.807994 1029 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 2 pre-election: Vote denied by peer 6885de38238442d6a9e5f1eb9fb569ff. Message: Service unavailable: T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ded9b7f8677940fb91e1a8c8bceceaed for term 2 because replica is already servicing an update from a current leader or another vote. I0312 20:10:10.809048 1346 raft_consensus.cc:2190] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate ded9b7f8677940fb91e1a8c8bceceaed for term 2 because replica is already servicing an update from a current leader or another vote. I0312 20:10:10.810262 1027 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 2 pre-election: Vote denied by peer f78a2351568c4a268ee1112681e22bec. Message: Service unavailable: T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate ded9b7f8677940fb91e1a8c8bceceaed for term 2 because replica is already servicing an update from a current leader or another vote. I0312 20:10:10.810516 1027 leader_election.cc:258] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. I0312 20:10:10.811157 1418 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed: Snoozing failure detection for 0.021s (election complete) I0312 20:10:10.811388 1418 raft_consensus.cc:2455] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority I0312 20:10:10.874657 1419 raft_consensus.cc:436] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f78a2351568c4a268ee1112681e22bec) I0312 20:10:10.874933 1419 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff: Snoozing failure detection for 0.102s (starting election) I0312 20:10:10.875180 1419 raft_consensus.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:10.876256 1419 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer f78a2351568c4a268ee1112681e22bec I0312 20:10:10.876619 1419 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer ded9b7f8677940fb91e1a8c8bceceaed I0312 20:10:10.877888 1418 raft_consensus.cc:436] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f78a2351568c4a268ee1112681e22bec) I0312 20:10:10.878116 1418 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed: Snoozing failure detection for 0.235s (starting election) I0312 20:10:10.878231 1096 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "6885de38238442d6a9e5f1eb9fb569ff" candidate_term: 2 candidate_status { last_received { term: 1 index: 34 } } ignore_live_leader: false dest_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" is_pre_election: true I0312 20:10:10.878346 1418 raft_consensus.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:10.879442 1418 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer f78a2351568c4a268ee1112681e22bec I0312 20:10:10.879798 1418 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 6885de38238442d6a9e5f1eb9fb569ff I0312 20:10:10.880255 1346 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "6885de38238442d6a9e5f1eb9fb569ff" candidate_term: 2 candidate_status { last_received { term: 1 index: 34 } } ignore_live_leader: false dest_uuid: "f78a2351568c4a268ee1112681e22bec" is_pre_election: true I0312 20:10:10.880872 1346 raft_consensus.cc:2176] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate 6885de38238442d6a9e5f1eb9fb569ff for term 2 because replica is either leader or believes a valid leader to be alive. I0312 20:10:10.881245 1096 raft_consensus.cc:2190] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6885de38238442d6a9e5f1eb9fb569ff for term 2 because replica is already servicing an update from a current leader or another vote. I0312 20:10:10.882699 1346 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" candidate_term: 2 candidate_status { last_received { term: 1 index: 34 } } ignore_live_leader: false dest_uuid: "f78a2351568c4a268ee1112681e22bec" is_pre_election: true I0312 20:10:10.882864 1152 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 2 pre-election: Vote denied by peer f78a2351568c4a268ee1112681e22bec. Message: Invalid argument: T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate 6885de38238442d6a9e5f1eb9fb569ff for term 2 because replica is either leader or believes a valid leader to be alive. I0312 20:10:10.883261 1346 raft_consensus.cc:2176] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate ded9b7f8677940fb91e1a8c8bceceaed for term 2 because replica is either leader or believes a valid leader to be alive. I0312 20:10:10.883544 1153 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 2 pre-election: Vote denied by peer ded9b7f8677940fb91e1a8c8bceceaed. Message: Service unavailable: T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6885de38238442d6a9e5f1eb9fb569ff for term 2 because replica is already servicing an update from a current leader or another vote. I0312 20:10:10.883821 1153 leader_election.cc:258] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. I0312 20:10:10.884234 1027 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 2 pre-election: Vote denied by peer f78a2351568c4a268ee1112681e22bec. Message: Invalid argument: T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate ded9b7f8677940fb91e1a8c8bceceaed for term 2 because replica is either leader or believes a valid leader to be alive. I0312 20:10:10.884244 1419 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff: Snoozing failure detection for 0.084s (election complete) I0312 20:10:10.884467 1419 raft_consensus.cc:2455] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority I0312 20:10:10.885124 1220 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" candidate_term: 2 candidate_status { last_received { term: 1 index: 34 } } ignore_live_leader: false dest_uuid: "6885de38238442d6a9e5f1eb9fb569ff" is_pre_election: true I0312 20:10:10.885722 1220 raft_consensus.cc:2176] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ded9b7f8677940fb91e1a8c8bceceaed for term 2 because replica is either leader or believes a valid leader to be alive. I0312 20:10:10.886607 1029 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 2 pre-election: Vote denied by peer 6885de38238442d6a9e5f1eb9fb569ff. Message: Invalid argument: T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ded9b7f8677940fb91e1a8c8bceceaed for term 2 because replica is either leader or believes a valid leader to be alive. I0312 20:10:10.886831 1029 leader_election.cc:258] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. I0312 20:10:10.887486 1418 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed: Snoozing failure detection for 0.022s (election complete) I0312 20:10:10.887711 1418 raft_consensus.cc:2455] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority I0312 20:10:10.904234 1418 raft_consensus.cc:436] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f78a2351568c4a268ee1112681e22bec) I0312 20:10:10.904453 1418 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed: Snoozing failure detection for 0.075s (starting election) I0312 20:10:10.904669 1418 raft_consensus.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:10.907104 1418 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer f78a2351568c4a268ee1112681e22bec I0312 20:10:10.907481 1418 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 6885de38238442d6a9e5f1eb9fb569ff I0312 20:10:10.909162 1219 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" candidate_term: 2 candidate_status { last_received { term: 1 index: 36 } } ignore_live_leader: false dest_uuid: "6885de38238442d6a9e5f1eb9fb569ff" is_pre_election: true I0312 20:10:10.909641 1219 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff: Snoozing failure detection for 0.168s (vote granted) I0312 20:10:10.909858 1219 raft_consensus.cc:2215] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ded9b7f8677940fb91e1a8c8bceceaed in term 1. I0312 20:10:10.911931 1029 leader_election.cc:363] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 2 pre-election: Vote granted by peer 6885de38238442d6a9e5f1eb9fb569ff I0312 20:10:10.912124 1029 leader_election.cc:258] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. I0312 20:10:10.912638 1418 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed: Snoozing failure detection for 0.120s (election complete) I0312 20:10:10.912843 1418 raft_consensus.cc:2507] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 1 FOLLOWER]: Leader pre-election won for term 2 I0312 20:10:10.913022 1418 raft_consensus.cc:436] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 1 FOLLOWER]: Starting leader election (detected failure of leader f78a2351568c4a268ee1112681e22bec) I0312 20:10:10.913224 1418 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed: Snoozing failure detection for 0.229s (starting election) I0312 20:10:10.913405 1418 raft_consensus.cc:2752] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 1 FOLLOWER]: Advancing to term 2 I0312 20:10:10.914554 1346 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" candidate_term: 2 candidate_status { last_received { term: 1 index: 36 } } ignore_live_leader: false dest_uuid: "f78a2351568c4a268ee1112681e22bec" is_pre_election: true I0312 20:10:10.915081 1346 raft_consensus.cc:2176] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate ded9b7f8677940fb91e1a8c8bceceaed for term 2 because replica is either leader or believes a valid leader to be alive. I0312 20:10:10.915974 1027 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 2 pre-election: Vote denied by peer f78a2351568c4a268ee1112681e22bec. Message: Invalid argument: T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate ded9b7f8677940fb91e1a8c8bceceaed for term 2 because replica is either leader or believes a valid leader to be alive. I0312 20:10:10.917960 1418 raft_consensus.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:10.918802 1418 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 2 election: Requesting vote from peer f78a2351568c4a268ee1112681e22bec I0312 20:10:10.918946 1096 raft_consensus.cc:1026] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 2 FOLLOWER]: Rejecting Update request from peer f78a2351568c4a268ee1112681e22bec for earlier term 1. Current term is 2. Ops: [] I0312 20:10:10.920203 1418 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 2 election: Requesting vote from peer 6885de38238442d6a9e5f1eb9fb569ff I0312 20:10:10.920140 1345 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" candidate_term: 2 candidate_status { last_received { term: 1 index: 36 } } ignore_live_leader: false dest_uuid: "f78a2351568c4a268ee1112681e22bec" I0312 20:10:10.920658 1345 raft_consensus.cc:2176] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 1 LEADER]: Leader election vote request: Denying vote to candidate ded9b7f8677940fb91e1a8c8bceceaed for term 2 because replica is either leader or believes a valid leader to be alive. I0312 20:10:10.921476 1220 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" candidate_term: 2 candidate_status { last_received { term: 1 index: 36 } } ignore_live_leader: false dest_uuid: "6885de38238442d6a9e5f1eb9fb569ff" I0312 20:10:10.921880 1027 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 2 election: Vote denied by peer f78a2351568c4a268ee1112681e22bec. Message: Invalid argument: T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 1 LEADER]: Leader election vote request: Denying vote to candidate ded9b7f8677940fb91e1a8c8bceceaed for term 2 because replica is either leader or believes a valid leader to be alive. I0312 20:10:10.922017 1220 raft_consensus.cc:2190] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate ded9b7f8677940fb91e1a8c8bceceaed for term 2 because replica is already servicing an update from a current leader or another vote. I0312 20:10:10.922770 1029 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 2 election: Vote denied by peer 6885de38238442d6a9e5f1eb9fb569ff. Message: Service unavailable: T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate ded9b7f8677940fb91e1a8c8bceceaed for term 2 because replica is already servicing an update from a current leader or another vote. I0312 20:10:10.922987 1029 leader_election.cc:258] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. I0312 20:10:10.923399 1417 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed: Snoozing failure detection for 0.186s (election complete) I0312 20:10:10.923440 1415 consensus_queue.cc:1028] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 }, Status: INVALID_TERM, Last received: 1.36, Next index: 37, Last known committed idx: 34, Time since last communication: 0.000s I0312 20:10:10.923619 1417 raft_consensus.cc:2455] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority I0312 20:10:10.923853 1420 raft_consensus.cc:2747] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 1 LEADER]: Stepping down as leader of term 1 I0312 20:10:10.924046 1420 raft_consensus.cc:605] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f78a2351568c4a268ee1112681e22bec, State: Running, Role: LEADER I0312 20:10:10.924671 1420 consensus_queue.cc:219] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 38, Committed index: 38, Last appended: 1.38, Last appended by leader: 38, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:10.925343 1420 consensus_peers.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec -> Peer ded9b7f8677940fb91e1a8c8bceceaed (127.0.53.1:35037): Closing peer: ded9b7f8677940fb91e1a8c8bceceaed I0312 20:10:10.925354 1415 consensus_peers.cc:230] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec -> Peer 6885de38238442d6a9e5f1eb9fb569ff (127.0.53.2:51697): Could not obtain request from queue for peer: 6885de38238442d6a9e5f1eb9fb569ff. Status: Not found: Peer not tracked or queue not in leader mode. I0312 20:10:10.925710 1420 consensus_peers.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec -> Peer 6885de38238442d6a9e5f1eb9fb569ff (127.0.53.2:51697): Closing peer: 6885de38238442d6a9e5f1eb9fb569ff I0312 20:10:10.926160 1420 raft_consensus.cc:2752] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 1 FOLLOWER]: Advancing to term 2 I0312 20:10:10.932929 1419 raft_consensus.cc:436] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f78a2351568c4a268ee1112681e22bec) I0312 20:10:10.933168 1419 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff: Snoozing failure detection for 0.074s (starting election) I0312 20:10:10.933395 1419 raft_consensus.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:10.934401 1419 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer f78a2351568c4a268ee1112681e22bec I0312 20:10:10.934765 1419 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer ded9b7f8677940fb91e1a8c8bceceaed I0312 20:10:10.936563 1346 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "6885de38238442d6a9e5f1eb9fb569ff" candidate_term: 2 candidate_status { last_received { term: 1 index: 38 } } ignore_live_leader: false dest_uuid: "f78a2351568c4a268ee1112681e22bec" is_pre_election: true I0312 20:10:10.937021 1346 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec: Snoozing failure detection for 0.021s (vote granted) I0312 20:10:10.937255 1346 raft_consensus.cc:2215] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6885de38238442d6a9e5f1eb9fb569ff in term 2. I0312 20:10:10.937798 1096 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "6885de38238442d6a9e5f1eb9fb569ff" candidate_term: 2 candidate_status { last_received { term: 1 index: 38 } } ignore_live_leader: false dest_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" is_pre_election: true I0312 20:10:10.938406 1096 raft_consensus.cc:2145] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6885de38238442d6a9e5f1eb9fb569ff in current term 2: Already voted for candidate ded9b7f8677940fb91e1a8c8bceceaed in this term. I0312 20:10:10.939071 1152 leader_election.cc:363] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 2 pre-election: Vote granted by peer f78a2351568c4a268ee1112681e22bec I0312 20:10:10.939296 1152 leader_election.cc:258] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. I0312 20:10:10.939793 1419 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff: Snoozing failure detection for 0.115s (election complete) I0312 20:10:10.939872 1153 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 2 pre-election: Vote denied by peer ded9b7f8677940fb91e1a8c8bceceaed. Message: Invalid argument: T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6885de38238442d6a9e5f1eb9fb569ff in current term 2: Already voted for candidate ded9b7f8677940fb91e1a8c8bceceaed in this term. I0312 20:10:10.940006 1419 raft_consensus.cc:2507] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 1 FOLLOWER]: Leader pre-election won for term 2 I0312 20:10:10.940178 1419 raft_consensus.cc:436] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 1 FOLLOWER]: Starting leader election (detected failure of leader f78a2351568c4a268ee1112681e22bec) I0312 20:10:10.940356 1419 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff: Snoozing failure detection for 0.040s (starting election) I0312 20:10:10.940526 1419 raft_consensus.cc:2752] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 1 FOLLOWER]: Advancing to term 2 I0312 20:10:10.944659 1419 raft_consensus.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:10.945669 1419 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 2 election: Requesting vote from peer f78a2351568c4a268ee1112681e22bec I0312 20:10:10.946413 1419 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 2 election: Requesting vote from peer ded9b7f8677940fb91e1a8c8bceceaed I0312 20:10:10.946853 1346 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "6885de38238442d6a9e5f1eb9fb569ff" candidate_term: 2 candidate_status { last_received { term: 1 index: 38 } } ignore_live_leader: false dest_uuid: "f78a2351568c4a268ee1112681e22bec" I0312 20:10:10.947257 1346 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec: Snoozing failure detection for 0.020s (vote granted) I0312 20:10:10.948487 1096 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "6885de38238442d6a9e5f1eb9fb569ff" candidate_term: 2 candidate_status { last_received { term: 1 index: 38 } } ignore_live_leader: false dest_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" I0312 20:10:10.948979 1096 raft_consensus.cc:2145] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 6885de38238442d6a9e5f1eb9fb569ff in current term 2: Already voted for candidate ded9b7f8677940fb91e1a8c8bceceaed in this term. I0312 20:10:10.949955 1153 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 2 election: Vote denied by peer ded9b7f8677940fb91e1a8c8bceceaed. Message: Invalid argument: T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 6885de38238442d6a9e5f1eb9fb569ff in current term 2: Already voted for candidate ded9b7f8677940fb91e1a8c8bceceaed in this term. I0312 20:10:10.951421 1346 raft_consensus.cc:2215] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6885de38238442d6a9e5f1eb9fb569ff in term 2. I0312 20:10:10.952232 1152 leader_election.cc:363] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 2 election: Vote granted by peer f78a2351568c4a268ee1112681e22bec I0312 20:10:10.952445 1152 leader_election.cc:258] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. I0312 20:10:10.953336 1419 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff: Snoozing failure detection for 0.076s (election complete) I0312 20:10:10.953582 1419 raft_consensus.cc:2507] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 2 FOLLOWER]: Leader election won for term 2 I0312 20:10:10.954329 1419 raft_consensus.cc:572] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 2 LEADER]: Becoming Leader. State: Replica: 6885de38238442d6a9e5f1eb9fb569ff, State: Running, Role: LEADER I0312 20:10:10.954826 1419 consensus_queue.cc:196] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 36, Committed index: 36, Last appended: 1.38, Last appended by leader: 38, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:10.959960 974 catalog_manager.cc:3592] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff reported cstate change: term changed from 1 to 2, leader changed from f78a2351568c4a268ee1112681e22bec (127.0.53.3) to 6885de38238442d6a9e5f1eb9fb569ff (127.0.53.2). New cstate: current_term: 2 leader_uuid: "6885de38238442d6a9e5f1eb9fb569ff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } health_report { overall_health: UNKNOWN } } } I0312 20:10:10.964036 1096 raft_consensus.cc:1058] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 2 FOLLOWER]: Refusing update from remote peer 6885de38238442d6a9e5f1eb9fb569ff: Log matching property violated. Preceding OpId in replica: term: 1 index: 36. Preceding OpId from leader: term: 2 index: 39. (index mismatch) I0312 20:10:10.965634 1416 consensus_queue.cc:1017] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 }, Status: LMP_MISMATCH, Last received: 1.36, Next index: 37, Last known committed idx: 34, Time since last communication: 0.001s I0312 20:10:10.982362 1346 raft_consensus.cc:1058] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 2 FOLLOWER]: Refusing update from remote peer 6885de38238442d6a9e5f1eb9fb569ff: Log matching property violated. Preceding OpId in replica: term: 1 index: 38. Preceding OpId from leader: term: 2 index: 39. (index mismatch) I0312 20:10:10.984680 1418 raft_consensus.cc:436] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6885de38238442d6a9e5f1eb9fb569ff) I0312 20:10:10.984917 1418 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed: Snoozing failure detection for 0.021s (starting election) I0312 20:10:10.985157 1418 raft_consensus.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:10.986160 1418 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 3 pre-election: Requesting pre-vote from peer f78a2351568c4a268ee1112681e22bec I0312 20:10:10.986510 1418 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 3 pre-election: Requesting pre-vote from peer 6885de38238442d6a9e5f1eb9fb569ff I0312 20:10:10.987884 1415 raft_consensus.cc:436] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0312 20:10:10.987944 1346 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" candidate_term: 3 candidate_status { last_received { term: 2 index: 39 } } ignore_live_leader: false dest_uuid: "f78a2351568c4a268ee1112681e22bec" is_pre_election: true I0312 20:10:10.988116 1415 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec: Snoozing failure detection for 0.020s (starting election) I0312 20:10:10.988368 1415 raft_consensus.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:10.989473 1415 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 3 pre-election: Requesting pre-vote from peer 6885de38238442d6a9e5f1eb9fb569ff I0312 20:10:10.989504 1346 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec: Snoozing failure detection for 0.021s (vote granted) I0312 20:10:10.989724 1346 raft_consensus.cc:2215] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ded9b7f8677940fb91e1a8c8bceceaed in term 2. I0312 20:10:10.989892 1415 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 3 pre-election: Requesting pre-vote from peer ded9b7f8677940fb91e1a8c8bceceaed I0312 20:10:10.990310 1415 raft_consensus.cc:436] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0312 20:10:10.990499 1027 leader_election.cc:363] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 3 pre-election: Vote granted by peer f78a2351568c4a268ee1112681e22bec I0312 20:10:10.990504 1415 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec: Snoozing failure detection for 0.019s (starting election) I0312 20:10:10.990651 1027 leader_election.cc:258] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. I0312 20:10:10.990972 1418 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed: Snoozing failure detection for 0.019s (election complete) I0312 20:10:10.991125 1418 raft_consensus.cc:2507] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 2 FOLLOWER]: Leader pre-election won for term 3 I0312 20:10:10.990723 1415 raft_consensus.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:10.991257 1418 raft_consensus.cc:436] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 2 FOLLOWER]: Starting leader election (detected failure of leader 6885de38238442d6a9e5f1eb9fb569ff) I0312 20:10:10.991375 1418 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed: Snoozing failure detection for 0.021s (starting election) I0312 20:10:10.991489 1418 raft_consensus.cc:2752] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 2 FOLLOWER]: Advancing to term 3 I0312 20:10:10.992740 1415 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 3 pre-election: Requesting pre-vote from peer 6885de38238442d6a9e5f1eb9fb569ff I0312 20:10:10.992619 1096 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "f78a2351568c4a268ee1112681e22bec" candidate_term: 3 candidate_status { last_received { term: 1 index: 38 } } ignore_live_leader: false dest_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" is_pre_election: true I0312 20:10:10.993980 1415 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 3 pre-election: Requesting pre-vote from peer ded9b7f8677940fb91e1a8c8bceceaed I0312 20:10:10.994580 1418 raft_consensus.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:10.995223 1095 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "f78a2351568c4a268ee1112681e22bec" candidate_term: 3 candidate_status { last_received { term: 1 index: 38 } } ignore_live_leader: false dest_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" is_pre_election: true I0312 20:10:10.995584 1418 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 3 election: Requesting vote from peer f78a2351568c4a268ee1112681e22bec I0312 20:10:10.995743 1095 raft_consensus.cc:2190] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 3 because replica is already servicing an update from a current leader or another vote. I0312 20:10:10.996713 1278 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 3 pre-election: Vote denied by peer ded9b7f8677940fb91e1a8c8bceceaed. Message: Service unavailable: T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 3 because replica is already servicing an update from a current leader or another vote. I0312 20:10:10.996706 1346 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" candidate_term: 3 candidate_status { last_received { term: 2 index: 39 } } ignore_live_leader: false dest_uuid: "f78a2351568c4a268ee1112681e22bec" I0312 20:10:10.997128 1346 raft_consensus.cc:2752] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 2 FOLLOWER]: Advancing to term 3 I0312 20:10:10.997270 1096 raft_consensus.cc:2145] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec in current term 3: Already voted for candidate ded9b7f8677940fb91e1a8c8bceceaed in this term. I0312 20:10:10.997349 1346 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec: Snoozing failure detection for 0.020s (vote granted) I0312 20:10:10.998353 1278 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 3 pre-election: Vote denied by peer ded9b7f8677940fb91e1a8c8bceceaed. Message: Invalid argument: T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec in current term 3: Already voted for candidate ded9b7f8677940fb91e1a8c8bceceaed in this term. I0312 20:10:10.998649 1418 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 3 election: Requesting vote from peer 6885de38238442d6a9e5f1eb9fb569ff I0312 20:10:11.000777 1346 raft_consensus.cc:2215] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ded9b7f8677940fb91e1a8c8bceceaed in term 3. I0312 20:10:11.001579 1027 leader_election.cc:363] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 3 election: Vote granted by peer f78a2351568c4a268ee1112681e22bec I0312 20:10:11.001787 1027 leader_election.cc:258] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. I0312 20:10:11.002116 1418 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed: Snoozing failure detection for 0.020s (election complete) I0312 20:10:11.002010 1219 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "f78a2351568c4a268ee1112681e22bec" candidate_term: 3 candidate_status { last_received { term: 1 index: 38 } } ignore_live_leader: false dest_uuid: "6885de38238442d6a9e5f1eb9fb569ff" is_pre_election: true I0312 20:10:11.002341 1418 raft_consensus.cc:2507] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 3 FOLLOWER]: Leader election won for term 3 I0312 20:10:11.002549 1418 raft_consensus.cc:572] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 3 LEADER]: Becoming Leader. State: Replica: ded9b7f8677940fb91e1a8c8bceceaed, State: Running, Role: LEADER I0312 20:10:11.002984 1418 consensus_queue.cc:196] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 36, Committed index: 36, Last appended: 2.39, Last appended by leader: 39, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:11.003177 1416 consensus_queue.cc:1017] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 }, Status: LMP_MISMATCH, Last received: 1.38, Next index: 39, Last known committed idx: 38, Time since last communication: 0.000s I0312 20:10:11.004246 1220 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "f78a2351568c4a268ee1112681e22bec" candidate_term: 3 candidate_status { last_received { term: 1 index: 38 } } ignore_live_leader: false dest_uuid: "6885de38238442d6a9e5f1eb9fb569ff" is_pre_election: true I0312 20:10:11.005223 1096 raft_consensus.cc:1026] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 3 LEADER]: Rejecting Update request from peer 6885de38238442d6a9e5f1eb9fb569ff for earlier term 2. Current term is 3. Ops: [] I0312 20:10:11.007078 974 catalog_manager.cc:3592] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed reported cstate change: term changed from 2 to 3, leader changed from 6885de38238442d6a9e5f1eb9fb569ff (127.0.53.2) to ded9b7f8677940fb91e1a8c8bceceaed (127.0.53.1). New cstate: current_term: 3 leader_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } health_report { overall_health: HEALTHY } } } I0312 20:10:11.008380 1416 consensus_queue.cc:1004] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [LEADER]: Peer ded9b7f8677940fb91e1a8c8bceceaed log is divergent from this leader: its last log entry 3.40 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 36 I0312 20:10:11.008610 1416 consensus_queue.cc:1028] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 }, Status: INVALID_TERM, Last received: 2.39, Next index: 37, Last known committed idx: 36, Time since last communication: 0.001s W0312 20:10:11.012187 1279 leader_election.cc:287] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6885de38238442d6a9e5f1eb9fb569ff: Timed out: RequestConsensusVote RPC to 127.0.53.2:51697 timed out after 0.020s (SENT) I0312 20:10:11.012418 1279 leader_election.cc:258] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. I0312 20:10:11.013093 1420 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec: Snoozing failure detection for 0.021s (election complete) I0312 20:10:11.013304 1420 raft_consensus.cc:2455] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority W0312 20:10:11.014147 1279 leader_election.cc:287] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6885de38238442d6a9e5f1eb9fb569ff: Timed out: RequestConsensusVote RPC to 127.0.53.2:51697 timed out after 0.019s (SENT) I0312 20:10:11.014364 1279 leader_election.cc:258] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. W0312 20:10:11.014734 1029 leader_election.cc:287] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6885de38238442d6a9e5f1eb9fb569ff: Timed out: RequestConsensusVote RPC to 127.0.53.2:51697 timed out after 0.021s (SENT) I0312 20:10:11.015050 1420 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec: Snoozing failure detection for 0.018s (election complete) I0312 20:10:11.015377 1420 raft_consensus.cc:2455] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority I0312 20:10:11.009141 1221 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" candidate_term: 3 candidate_status { last_received { term: 2 index: 39 } } ignore_live_leader: false dest_uuid: "6885de38238442d6a9e5f1eb9fb569ff" is_pre_election: true I0312 20:10:11.009990 1218 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" candidate_term: 3 candidate_status { last_received { term: 2 index: 39 } } ignore_live_leader: false dest_uuid: "6885de38238442d6a9e5f1eb9fb569ff" I0312 20:10:11.016733 1219 raft_consensus.cc:2176] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 2 LEADER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 3 because replica is either leader or believes a valid leader to be alive. I0312 20:10:11.016819 1345 raft_consensus.cc:1026] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 3 FOLLOWER]: Rejecting Update request from peer 6885de38238442d6a9e5f1eb9fb569ff for earlier term 2. Current term is 3. Ops: [] W0312 20:10:11.017223 1219 rpcz_store.cc:251] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.53.3:38196 (request call id 23) took 15ms (client timeout 19). I0312 20:10:11.018046 1346 raft_consensus.cc:1058] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 3 FOLLOWER]: Refusing update from remote peer ded9b7f8677940fb91e1a8c8bceceaed: Log matching property violated. Preceding OpId in replica: term: 1 index: 38. Preceding OpId from leader: term: 3 index: 40. (index mismatch) W0312 20:10:11.018137 1219 rpcz_store.cc:255] Trace: 0312 20:10:11.001761 (+ 0us) service_pool.cc:163] Inserting onto call queue 0312 20:10:11.001868 (+ 107us) service_pool.cc:222] Handling call 0312 20:10:11.016540 (+ 14672us) spinlock_profiling.cc:261] Waited 7.8 ms on lock 0x7b5000070b18. stack: 00007efc5efb2c74 00007efc5efb2b70 00007efc5daa2314 00007efc66aef294 00007efc66aef229 00007efc65658ec0 00007efc65699627 00007efc66b2d56b 00007efc614e0b24 00007efc614e0ab1 00007efc614e09c4 00007efc60b3cc29 00007efc60b3c6ec 00007efc60b3dbcd 00007efc60b3f946 00007efc60b3f89a 0312 20:10:11.017039 (+ 499us) spinlock_profiling.cc:261] Waited 6.12 ms on lock 0x7b480003009c. stack: 00007efc5efb2c74 00007efc5efb2b70 00007efc5daa2314 00007efc66aef294 00007efc66aef229 00007efc65699af3 00007efc66b2d56b 00007efc614e0b24 00007efc614e0ab1 00007efc614e09c4 00007efc60b3cc29 00007efc60b3c6ec 00007efc60b3dbcd 00007efc60b3f946 00007efc60b3f89a 00007efc60b3f823 0312 20:10:11.017167 (+ 128us) inbound_call.cc:157] Queueing success response Metrics: {"spinlock_wait_cycles":32015360} I0312 20:10:11.019994 1416 raft_consensus.cc:2747] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 2 LEADER]: Stepping down as leader of term 2 I0312 20:10:11.020242 1416 raft_consensus.cc:605] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 6885de38238442d6a9e5f1eb9fb569ff, State: Running, Role: LEADER I0312 20:10:11.020501 1452 consensus_queue.cc:1028] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 }, Status: INVALID_TERM, Last received: 1.38, Next index: 39, Last known committed idx: 38, Time since last communication: 0.000s W0312 20:10:11.020797 1029 leader_election.cc:287] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 6885de38238442d6a9e5f1eb9fb569ff: Timed out: RequestConsensusVote RPC to 127.0.53.2:51697 timed out after 0.021s (SENT) I0312 20:10:11.021688 1418 consensus_queue.cc:1017] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [LEADER]: Connected to new peer: Peer: permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 }, Status: LMP_MISMATCH, Last received: 1.38, Next index: 39, Last known committed idx: 38, Time since last communication: 0.000s I0312 20:10:11.024070 1096 raft_consensus.cc:1003] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 3 LEADER]: Deduplicated request from leader. Original: 1.36->[1.37-2.39] Dedup: 2.39->[] I0312 20:10:11.025223 1096 raft_consensus.cc:1026] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 3 LEADER]: Rejecting Update request from peer 6885de38238442d6a9e5f1eb9fb569ff for earlier term 2. Current term is 3. Ops: [1.37-2.39] I0312 20:10:11.026316 1416 consensus_queue.cc:219] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 39, Committed index: 39, Last appended: 2.39, Last appended by leader: 39, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:11.029731 1416 consensus_peers.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff -> Peer ded9b7f8677940fb91e1a8c8bceceaed (127.0.53.1:35037): Closing peer: ded9b7f8677940fb91e1a8c8bceceaed I0312 20:10:11.030185 1452 consensus_queue.cc:1004] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [NON_LEADER]: Peer ded9b7f8677940fb91e1a8c8bceceaed log is divergent from this leader: its last log entry 3.40 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 36 I0312 20:10:11.030560 1452 consensus_queue.cc:1028] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 }, Status: INVALID_TERM, Last received: 2.39, Next index: 37, Last known committed idx: 36, Time since last communication: 0.001s I0312 20:10:11.031213 1416 consensus_peers.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff -> Peer f78a2351568c4a268ee1112681e22bec (127.0.53.3:43274): Closing peer: f78a2351568c4a268ee1112681e22bec I0312 20:10:11.031766 1416 raft_consensus.cc:2752] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 2 FOLLOWER]: Advancing to term 3 I0312 20:10:11.037967 1345 raft_consensus.cc:1003] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 3 FOLLOWER]: Deduplicated request from leader. Original: 1.38->[2.39-2.39] Dedup: 2.39->[] I0312 20:10:11.038278 1345 raft_consensus.cc:1026] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 3 FOLLOWER]: Rejecting Update request from peer 6885de38238442d6a9e5f1eb9fb569ff for earlier term 2. Current term is 3. Ops: [2.39-2.39] I0312 20:10:11.041276 1220 raft_consensus.cc:2190] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 3 because replica is already servicing an update from a current leader or another vote. I0312 20:10:11.041607 1218 raft_consensus.cc:2190] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate ded9b7f8677940fb91e1a8c8bceceaed for term 3 because replica is already servicing an update from a current leader or another vote. W0312 20:10:11.042078 1218 rpcz_store.cc:251] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.53.1:43660 (request call id 13) took 32ms (client timeout 21). W0312 20:10:11.042716 1218 rpcz_store.cc:255] Trace: 0312 20:10:11.009646 (+ 0us) service_pool.cc:163] Inserting onto call queue 0312 20:10:11.009739 (+ 93us) service_pool.cc:222] Handling call 0312 20:10:11.041953 (+ 32214us) spinlock_profiling.cc:261] Waited 25.5 ms on lock 0x7b480003009c. stack: 00007efc5efb2c74 00007efc5efb2b70 00007efc5daa2314 00007efc66aef294 00007efc66aef229 00007efc65699606 00007efc66b2d56b 00007efc614e0b24 00007efc614e0ab1 00007efc614e09c4 00007efc60b3cc29 00007efc60b3c6ec 00007efc60b3dbcd 00007efc60b3f946 00007efc60b3f89a 00007efc60b3f823 0312 20:10:11.042032 (+ 79us) inbound_call.cc:157] Queueing success response Metrics: {"spinlock_wait_cycles":58642304} W0312 20:10:11.043486 1220 rpcz_store.cc:251] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.53.3:38196 (request call id 22) took 42ms (client timeout 20). W0312 20:10:11.044441 1220 rpcz_store.cc:255] Trace: 0312 20:10:11.001417 (+ 0us) service_pool.cc:163] Inserting onto call queue 0312 20:10:11.004104 (+ 2687us) service_pool.cc:222] Handling call 0312 20:10:11.043355 (+ 39251us) spinlock_profiling.cc:261] Waited 36.5 ms on lock 0x7b480003009c. stack: 00007efc5efb2c74 00007efc5efb2b70 00007efc5daa2314 00007efc66aef294 00007efc66aef229 00007efc65699606 00007efc66b2d56b 00007efc614e0b24 00007efc614e0ab1 00007efc614e09c4 00007efc60b3cc29 00007efc60b3c6ec 00007efc60b3dbcd 00007efc60b3f946 00007efc60b3f89a 00007efc60b3f823 0312 20:10:11.043440 (+ 85us) inbound_call.cc:157] Queueing success response Metrics: {"spinlock_wait_cycles":83925888} I0312 20:10:11.043804 1454 raft_consensus.cc:436] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0312 20:10:11.045383 1454 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff: Snoozing failure detection for 0.017s (starting election) I0312 20:10:11.045660 1454 raft_consensus.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:11.047027 1221 raft_consensus.cc:2190] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ded9b7f8677940fb91e1a8c8bceceaed for term 3 because replica is already servicing an update from a current leader or another vote. W0312 20:10:11.047478 1221 rpcz_store.cc:251] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.53.1:43660 (request call id 12) took 39ms (client timeout 21). I0312 20:10:11.047747 1420 raft_consensus.cc:436] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ded9b7f8677940fb91e1a8c8bceceaed) I0312 20:10:11.047977 1420 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec: Snoozing failure detection for 0.015s (starting election) W0312 20:10:11.048116 1221 rpcz_store.cc:255] Trace: 0312 20:10:11.008885 (+ 0us) service_pool.cc:163] Inserting onto call queue 0312 20:10:11.008979 (+ 94us) service_pool.cc:222] Handling call 0312 20:10:11.047335 (+ 38356us) spinlock_profiling.cc:261] Waited 31.1 ms on lock 0x7b480003009c. stack: 00007efc5efb2c74 00007efc5efb2b70 00007efc5daa2314 00007efc66aef294 00007efc66aef229 00007efc65699606 00007efc66b2d56b 00007efc614e0b24 00007efc614e0ab1 00007efc614e09c4 00007efc60b3cc29 00007efc60b3c6ec 00007efc60b3dbcd 00007efc60b3f946 00007efc60b3f89a 00007efc60b3f823 0312 20:10:11.047428 (+ 93us) inbound_call.cc:157] Queueing success response Metrics: {"spinlock_wait_cycles":71503616} I0312 20:10:11.048193 1420 raft_consensus.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:11.048802 1219 raft_consensus.cc:1058] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 3 FOLLOWER]: Refusing update from remote peer ded9b7f8677940fb91e1a8c8bceceaed: Log matching property violated. Preceding OpId in replica: term: 2 index: 39. Preceding OpId from leader: term: 3 index: 40. (index mismatch) I0312 20:10:11.049206 1420 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 4 pre-election: Requesting pre-vote from peer 6885de38238442d6a9e5f1eb9fb569ff I0312 20:10:11.049543 1420 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 4 pre-election: Requesting pre-vote from peer ded9b7f8677940fb91e1a8c8bceceaed I0312 20:10:11.049753 1454 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 4 pre-election: Requesting pre-vote from peer f78a2351568c4a268ee1112681e22bec I0312 20:10:11.050101 1454 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 4 pre-election: Requesting pre-vote from peer ded9b7f8677940fb91e1a8c8bceceaed I0312 20:10:11.051340 1096 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "f78a2351568c4a268ee1112681e22bec" candidate_term: 4 candidate_status { last_received { term: 3 index: 40 } } ignore_live_leader: false dest_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" is_pre_election: true I0312 20:10:11.051479 1095 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "6885de38238442d6a9e5f1eb9fb569ff" candidate_term: 4 candidate_status { last_received { term: 2 index: 39 } } ignore_live_leader: false dest_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" is_pre_election: true I0312 20:10:11.051888 1096 raft_consensus.cc:2176] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 3 LEADER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 4 because replica is either leader or believes a valid leader to be alive. I0312 20:10:11.052166 1095 raft_consensus.cc:2190] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 3 LEADER]: Leader pre-election vote request: Denying vote to candidate 6885de38238442d6a9e5f1eb9fb569ff for term 4 because replica is already servicing an update from a current leader or another vote. I0312 20:10:11.052544 1219 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "f78a2351568c4a268ee1112681e22bec" candidate_term: 4 candidate_status { last_received { term: 3 index: 40 } } ignore_live_leader: false dest_uuid: "6885de38238442d6a9e5f1eb9fb569ff" is_pre_election: true I0312 20:10:11.052969 1219 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff: Snoozing failure detection for 0.019s (vote granted) I0312 20:10:11.053153 1153 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 4 pre-election: Vote denied by peer ded9b7f8677940fb91e1a8c8bceceaed. Message: Service unavailable: T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 3 LEADER]: Leader pre-election vote request: Denying vote to candidate 6885de38238442d6a9e5f1eb9fb569ff for term 4 because replica is already servicing an update from a current leader or another vote. I0312 20:10:11.053267 1219 raft_consensus.cc:2215] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f78a2351568c4a268ee1112681e22bec in term 3. I0312 20:10:11.054051 1279 leader_election.cc:363] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 4 pre-election: Vote granted by peer 6885de38238442d6a9e5f1eb9fb569ff I0312 20:10:11.054261 1279 leader_election.cc:258] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. I0312 20:10:11.054538 1278 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 4 pre-election: Vote denied by peer ded9b7f8677940fb91e1a8c8bceceaed. Message: Invalid argument: T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 3 LEADER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 4 because replica is either leader or believes a valid leader to be alive. I0312 20:10:11.054761 1420 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec: Snoozing failure detection for 0.018s (election complete) I0312 20:10:11.054975 1420 raft_consensus.cc:2507] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 3 FOLLOWER]: Leader pre-election won for term 4 I0312 20:10:11.055147 1420 raft_consensus.cc:436] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 3 FOLLOWER]: Starting leader election (detected failure of leader ded9b7f8677940fb91e1a8c8bceceaed) I0312 20:10:11.055327 1420 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec: Snoozing failure detection for 0.021s (starting election) I0312 20:10:11.055492 1420 raft_consensus.cc:2752] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 3 FOLLOWER]: Advancing to term 4 I0312 20:10:11.059980 1420 raft_consensus.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:11.061203 1420 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 4 election: Requesting vote from peer 6885de38238442d6a9e5f1eb9fb569ff I0312 20:10:11.062394 1219 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "f78a2351568c4a268ee1112681e22bec" candidate_term: 4 candidate_status { last_received { term: 3 index: 40 } } ignore_live_leader: false dest_uuid: "6885de38238442d6a9e5f1eb9fb569ff" I0312 20:10:11.062891 1219 raft_consensus.cc:2752] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 3 FOLLOWER]: Advancing to term 4 I0312 20:10:11.063132 1219 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff: Snoozing failure detection for 0.016s (vote granted) I0312 20:10:11.067420 1219 raft_consensus.cc:2215] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f78a2351568c4a268ee1112681e22bec in term 4. I0312 20:10:11.067677 1420 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 4 election: Requesting vote from peer ded9b7f8677940fb91e1a8c8bceceaed I0312 20:10:11.069372 1096 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "f78a2351568c4a268ee1112681e22bec" candidate_term: 4 candidate_status { last_received { term: 3 index: 40 } } ignore_live_leader: false dest_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" I0312 20:10:11.070001 1096 raft_consensus.cc:2176] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 3 LEADER]: Leader election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 4 because replica is either leader or believes a valid leader to be alive. I0312 20:10:11.070940 1278 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 4 election: Vote denied by peer ded9b7f8677940fb91e1a8c8bceceaed. Message: Invalid argument: T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 3 LEADER]: Leader election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 4 because replica is either leader or believes a valid leader to be alive. I0312 20:10:11.073129 1279 leader_election.cc:363] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 4 election: Vote granted by peer 6885de38238442d6a9e5f1eb9fb569ff I0312 20:10:11.073343 1279 leader_election.cc:258] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. I0312 20:10:11.073768 1418 consensus_queue.cc:1017] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [LEADER]: Connected to new peer: Peer: permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 }, Status: LMP_MISMATCH, Last received: 2.39, Next index: 40, Last known committed idx: 39, Time since last communication: 0.000s I0312 20:10:11.074234 1420 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec: Snoozing failure detection for 0.021s (election complete) I0312 20:10:11.074499 1420 raft_consensus.cc:2507] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 4 FOLLOWER]: Leader election won for term 4 I0312 20:10:11.075417 1420 raft_consensus.cc:572] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 4 LEADER]: Becoming Leader. State: Replica: f78a2351568c4a268ee1112681e22bec, State: Running, Role: LEADER I0312 20:10:11.075938 1420 consensus_queue.cc:196] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 36, Committed index: 36, Last appended: 3.40, Last appended by leader: 40, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } W0312 20:10:11.072347 1152 leader_election.cc:287] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer f78a2351568c4a268ee1112681e22bec: Timed out: RequestConsensusVote RPC to 127.0.53.3:43274 timed out after 0.017s (ON_OUTBOUND_QUEUE) I0312 20:10:11.077215 1152 leader_election.cc:258] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. I0312 20:10:11.078222 1416 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff: Snoozing failure detection for 0.018s (election complete) I0312 20:10:11.078464 1416 raft_consensus.cc:2455] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority I0312 20:10:11.079244 1219 raft_consensus.cc:1026] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 4 FOLLOWER]: Rejecting Update request from peer ded9b7f8677940fb91e1a8c8bceceaed for earlier term 3. Current term is 4. Ops: [] I0312 20:10:11.081470 1417 consensus_queue.cc:1028] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 }, Status: INVALID_TERM, Last received: 2.39, Next index: 40, Last known committed idx: 39, Time since last communication: 0.000s I0312 20:10:11.082185 1417 raft_consensus.cc:2747] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 3 LEADER]: Stepping down as leader of term 3 I0312 20:10:11.082403 1417 raft_consensus.cc:605] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 3 LEADER]: Becoming Follower/Learner. State: Replica: ded9b7f8677940fb91e1a8c8bceceaed, State: Running, Role: LEADER I0312 20:10:11.086024 1346 raft_consensus.cc:1026] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 4 LEADER]: Rejecting Update request from peer ded9b7f8677940fb91e1a8c8bceceaed for earlier term 3. Current term is 4. Ops: [] I0312 20:10:11.092130 1219 raft_consensus.cc:1058] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 4 FOLLOWER]: Refusing update from remote peer f78a2351568c4a268ee1112681e22bec: Log matching property violated. Preceding OpId in replica: term: 2 index: 39. Preceding OpId from leader: term: 4 index: 41. (index mismatch) I0312 20:10:11.092340 974 catalog_manager.cc:3592] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec reported cstate change: term changed from 3 to 4, leader changed from ded9b7f8677940fb91e1a8c8bceceaed (127.0.53.1) to f78a2351568c4a268ee1112681e22bec (127.0.53.3). New cstate: current_term: 4 leader_uuid: "f78a2351568c4a268ee1112681e22bec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } health_report { overall_health: UNKNOWN } } } I0312 20:10:11.093384 1415 consensus_queue.cc:1017] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [LEADER]: Connected to new peer: Peer: permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 }, Status: LMP_MISMATCH, Last received: 2.39, Next index: 40, Last known committed idx: 39, Time since last communication: 0.000s I0312 20:10:11.083379 1417 consensus_queue.cc:219] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 40, Committed index: 40, Last appended: 3.40, Last appended by leader: 40, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:11.108678 1455 consensus_queue.cc:1004] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [NON_LEADER]: Peer f78a2351568c4a268ee1112681e22bec log is divergent from this leader: its last log entry 4.41 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 36 I0312 20:10:11.108866 1455 consensus_queue.cc:1028] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 }, Status: INVALID_TERM, Last received: 3.40, Next index: 37, Last known committed idx: 36, Time since last communication: 0.001s I0312 20:10:11.110244 1418 consensus_peers.cc:230] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed -> Peer 6885de38238442d6a9e5f1eb9fb569ff (127.0.53.2:51697): Could not obtain request from queue for peer: 6885de38238442d6a9e5f1eb9fb569ff. Status: Not found: Peer not tracked or queue not in leader mode. I0312 20:10:11.114493 1458 consensus_peers.cc:230] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed -> Peer 6885de38238442d6a9e5f1eb9fb569ff (127.0.53.2:51697): Could not obtain request from queue for peer: 6885de38238442d6a9e5f1eb9fb569ff. Status: Not found: Peer not tracked or queue not in leader mode. I0312 20:10:11.114572 1418 consensus_peers.cc:230] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed -> Peer f78a2351568c4a268ee1112681e22bec (127.0.53.3:43274): Could not obtain request from queue for peer: f78a2351568c4a268ee1112681e22bec. Status: Not found: Peer not tracked or queue not in leader mode. I0312 20:10:11.119117 1418 consensus_peers.cc:230] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed -> Peer f78a2351568c4a268ee1112681e22bec (127.0.53.3:43274): Could not obtain request from queue for peer: f78a2351568c4a268ee1112681e22bec. Status: Not found: Peer not tracked or queue not in leader mode. I0312 20:10:11.119143 1417 consensus_peers.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed -> Peer 6885de38238442d6a9e5f1eb9fb569ff (127.0.53.2:51697): Closing peer: 6885de38238442d6a9e5f1eb9fb569ff I0312 20:10:11.119484 1417 consensus_peers.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed -> Peer f78a2351568c4a268ee1112681e22bec (127.0.53.3:43274): Closing peer: f78a2351568c4a268ee1112681e22bec I0312 20:10:11.120676 1417 raft_consensus.cc:2752] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 3 FOLLOWER]: Advancing to term 4 I0312 20:10:11.162133 1455 raft_consensus.cc:436] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0312 20:10:11.162359 1455 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed: Snoozing failure detection for 0.017s (starting election) I0312 20:10:11.162595 1455 raft_consensus.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:11.163689 1455 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 5 pre-election: Requesting pre-vote from peer f78a2351568c4a268ee1112681e22bec I0312 20:10:11.164063 1455 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 5 pre-election: Requesting pre-vote from peer 6885de38238442d6a9e5f1eb9fb569ff I0312 20:10:11.164896 1096 raft_consensus.cc:1058] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 4 FOLLOWER]: Refusing update from remote peer f78a2351568c4a268ee1112681e22bec: Log matching property violated. Preceding OpId in replica: term: 3 index: 40. Preceding OpId from leader: term: 4 index: 41. (index mismatch) I0312 20:10:11.166431 1424 consensus_queue.cc:1017] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [LEADER]: Connected to new peer: Peer: permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 }, Status: LMP_MISMATCH, Last received: 3.40, Next index: 41, Last known committed idx: 40, Time since last communication: 0.000s I0312 20:10:11.167855 1219 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" candidate_term: 5 candidate_status { last_received { term: 3 index: 40 } } ignore_live_leader: false dest_uuid: "6885de38238442d6a9e5f1eb9fb569ff" is_pre_election: true I0312 20:10:11.167999 1346 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" candidate_term: 5 candidate_status { last_received { term: 3 index: 40 } } ignore_live_leader: false dest_uuid: "f78a2351568c4a268ee1112681e22bec" is_pre_election: true I0312 20:10:11.168560 1346 raft_consensus.cc:2176] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 4 LEADER]: Leader pre-election vote request: Denying vote to candidate ded9b7f8677940fb91e1a8c8bceceaed for term 5 because replica is either leader or believes a valid leader to be alive. I0312 20:10:11.171864 1219 raft_consensus.cc:2190] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ded9b7f8677940fb91e1a8c8bceceaed for term 5 because replica is already servicing an update from a current leader or another vote. I0312 20:10:11.172991 1029 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 5 pre-election: Vote denied by peer 6885de38238442d6a9e5f1eb9fb569ff. Message: Service unavailable: T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ded9b7f8677940fb91e1a8c8bceceaed for term 5 because replica is already servicing an update from a current leader or another vote. I0312 20:10:11.185432 1027 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 5 pre-election: Vote denied by peer f78a2351568c4a268ee1112681e22bec. Message: Invalid argument: T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 4 LEADER]: Leader pre-election vote request: Denying vote to candidate ded9b7f8677940fb91e1a8c8bceceaed for term 5 because replica is either leader or believes a valid leader to be alive. I0312 20:10:11.185775 1027 leader_election.cc:258] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. I0312 20:10:11.186308 1456 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed: Snoozing failure detection for 0.020s (election complete) I0312 20:10:11.186568 1456 raft_consensus.cc:2455] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority I0312 20:10:11.186815 1455 raft_consensus.cc:436] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 4 FOLLOWER]: Starting pre-election (detected failure of leader f78a2351568c4a268ee1112681e22bec) I0312 20:10:11.186991 1455 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed: Snoozing failure detection for 0.026s (starting election) I0312 20:10:11.187166 1455 raft_consensus.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:11.187860 1455 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 5 pre-election: Requesting pre-vote from peer f78a2351568c4a268ee1112681e22bec I0312 20:10:11.188148 1455 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 5 pre-election: Requesting pre-vote from peer 6885de38238442d6a9e5f1eb9fb569ff I0312 20:10:11.188976 1346 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" candidate_term: 5 candidate_status { last_received { term: 4 index: 47 } } ignore_live_leader: false dest_uuid: "f78a2351568c4a268ee1112681e22bec" is_pre_election: true I0312 20:10:11.189507 1346 raft_consensus.cc:2176] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 4 LEADER]: Leader pre-election vote request: Denying vote to candidate ded9b7f8677940fb91e1a8c8bceceaed for term 5 because replica is either leader or believes a valid leader to be alive. I0312 20:10:11.190603 1027 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 5 pre-election: Vote denied by peer f78a2351568c4a268ee1112681e22bec. Message: Invalid argument: T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 4 LEADER]: Leader pre-election vote request: Denying vote to candidate ded9b7f8677940fb91e1a8c8bceceaed for term 5 because replica is either leader or believes a valid leader to be alive. I0312 20:10:11.192955 1221 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" candidate_term: 5 candidate_status { last_received { term: 4 index: 47 } } ignore_live_leader: false dest_uuid: "6885de38238442d6a9e5f1eb9fb569ff" is_pre_election: true I0312 20:10:11.193862 1221 raft_consensus.cc:2162] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ded9b7f8677940fb91e1a8c8bceceaed for term 5 because replica has last-logged OpId of term: 4 index: 48, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 47. I0312 20:10:11.195279 1029 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 5 pre-election: Vote denied by peer 6885de38238442d6a9e5f1eb9fb569ff. Message: Invalid argument: T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ded9b7f8677940fb91e1a8c8bceceaed for term 5 because replica has last-logged OpId of term: 4 index: 48, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 47. I0312 20:10:11.196563 1029 leader_election.cc:258] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. I0312 20:10:11.199843 1455 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed: Snoozing failure detection for 0.024s (election complete) I0312 20:10:11.200026 1455 raft_consensus.cc:2455] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority I0312 20:10:11.206717 1454 raft_consensus.cc:436] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 4 FOLLOWER]: Starting pre-election (detected failure of leader f78a2351568c4a268ee1112681e22bec) I0312 20:10:11.206944 1454 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff: Snoozing failure detection for 0.016s (starting election) I0312 20:10:11.207219 1454 raft_consensus.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:11.208299 1454 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 5 pre-election: Requesting pre-vote from peer f78a2351568c4a268ee1112681e22bec I0312 20:10:11.208652 1454 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 5 pre-election: Requesting pre-vote from peer ded9b7f8677940fb91e1a8c8bceceaed I0312 20:10:11.210271 1095 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "6885de38238442d6a9e5f1eb9fb569ff" candidate_term: 5 candidate_status { last_received { term: 4 index: 49 } } ignore_live_leader: false dest_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" is_pre_election: true I0312 20:10:11.211918 1346 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "6885de38238442d6a9e5f1eb9fb569ff" candidate_term: 5 candidate_status { last_received { term: 4 index: 49 } } ignore_live_leader: false dest_uuid: "f78a2351568c4a268ee1112681e22bec" is_pre_election: true I0312 20:10:11.213452 1095 raft_consensus.cc:2190] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6885de38238442d6a9e5f1eb9fb569ff for term 5 because replica is already servicing an update from a current leader or another vote. I0312 20:10:11.213536 1346 raft_consensus.cc:2176] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 4 LEADER]: Leader pre-election vote request: Denying vote to candidate 6885de38238442d6a9e5f1eb9fb569ff for term 5 because replica is either leader or believes a valid leader to be alive. I0312 20:10:11.214751 1153 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 5 pre-election: Vote denied by peer ded9b7f8677940fb91e1a8c8bceceaed. Message: Service unavailable: T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6885de38238442d6a9e5f1eb9fb569ff for term 5 because replica is already servicing an update from a current leader or another vote. I0312 20:10:11.216270 1152 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 5 pre-election: Vote denied by peer f78a2351568c4a268ee1112681e22bec. Message: Invalid argument: T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 4 LEADER]: Leader pre-election vote request: Denying vote to candidate 6885de38238442d6a9e5f1eb9fb569ff for term 5 because replica is either leader or believes a valid leader to be alive. I0312 20:10:11.216526 1152 leader_election.cc:258] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. I0312 20:10:11.219957 1454 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff: Snoozing failure detection for 0.021s (election complete) I0312 20:10:11.220171 1454 raft_consensus.cc:2455] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority I0312 20:10:11.241901 1454 raft_consensus.cc:436] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 4 FOLLOWER]: Starting pre-election (detected failure of leader f78a2351568c4a268ee1112681e22bec) I0312 20:10:11.242108 1454 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff: Snoozing failure detection for 0.023s (starting election) I0312 20:10:11.242342 1454 raft_consensus.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:11.243309 1454 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 5 pre-election: Requesting pre-vote from peer f78a2351568c4a268ee1112681e22bec I0312 20:10:11.243640 1454 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 5 pre-election: Requesting pre-vote from peer ded9b7f8677940fb91e1a8c8bceceaed I0312 20:10:11.245941 1095 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "6885de38238442d6a9e5f1eb9fb569ff" candidate_term: 5 candidate_status { last_received { term: 4 index: 52 } } ignore_live_leader: false dest_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" is_pre_election: true I0312 20:10:11.246430 1095 raft_consensus.cc:2176] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6885de38238442d6a9e5f1eb9fb569ff for term 5 because replica is either leader or believes a valid leader to be alive. I0312 20:10:11.246970 1346 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "6885de38238442d6a9e5f1eb9fb569ff" candidate_term: 5 candidate_status { last_received { term: 4 index: 52 } } ignore_live_leader: false dest_uuid: "f78a2351568c4a268ee1112681e22bec" is_pre_election: true I0312 20:10:11.247318 1153 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 5 pre-election: Vote denied by peer ded9b7f8677940fb91e1a8c8bceceaed. Message: Invalid argument: T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6885de38238442d6a9e5f1eb9fb569ff for term 5 because replica is either leader or believes a valid leader to be alive. I0312 20:10:11.247459 1346 raft_consensus.cc:2176] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 4 LEADER]: Leader pre-election vote request: Denying vote to candidate 6885de38238442d6a9e5f1eb9fb569ff for term 5 because replica is either leader or believes a valid leader to be alive. I0312 20:10:11.248219 1152 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 5 pre-election: Vote denied by peer f78a2351568c4a268ee1112681e22bec. Message: Invalid argument: T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 4 LEADER]: Leader pre-election vote request: Denying vote to candidate 6885de38238442d6a9e5f1eb9fb569ff for term 5 because replica is either leader or believes a valid leader to be alive. I0312 20:10:11.248433 1152 leader_election.cc:258] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. I0312 20:10:11.249001 1454 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff: Snoozing failure detection for 0.022s (election complete) I0312 20:10:11.249222 1454 raft_consensus.cc:2455] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority I0312 20:10:11.271963 1454 raft_consensus.cc:436] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 4 FOLLOWER]: Starting pre-election (detected failure of leader f78a2351568c4a268ee1112681e22bec) I0312 20:10:11.272176 1454 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff: Snoozing failure detection for 0.047s (starting election) I0312 20:10:11.272367 1454 raft_consensus.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:11.273131 1454 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 5 pre-election: Requesting pre-vote from peer f78a2351568c4a268ee1112681e22bec I0312 20:10:11.273398 1454 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 5 pre-election: Requesting pre-vote from peer ded9b7f8677940fb91e1a8c8bceceaed I0312 20:10:11.274466 1095 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "6885de38238442d6a9e5f1eb9fb569ff" candidate_term: 5 candidate_status { last_received { term: 4 index: 54 } } ignore_live_leader: false dest_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" is_pre_election: true I0312 20:10:11.276002 1346 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "6885de38238442d6a9e5f1eb9fb569ff" candidate_term: 5 candidate_status { last_received { term: 4 index: 54 } } ignore_live_leader: false dest_uuid: "f78a2351568c4a268ee1112681e22bec" is_pre_election: true I0312 20:10:11.276515 1346 raft_consensus.cc:2176] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 4 LEADER]: Leader pre-election vote request: Denying vote to candidate 6885de38238442d6a9e5f1eb9fb569ff for term 5 because replica is either leader or believes a valid leader to be alive. I0312 20:10:11.281553 1152 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 5 pre-election: Vote denied by peer f78a2351568c4a268ee1112681e22bec. Message: Invalid argument: T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 4 LEADER]: Leader pre-election vote request: Denying vote to candidate 6885de38238442d6a9e5f1eb9fb569ff for term 5 because replica is either leader or believes a valid leader to be alive. I0312 20:10:11.284078 1095 raft_consensus.cc:2190] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6885de38238442d6a9e5f1eb9fb569ff for term 5 because replica is already servicing an update from a current leader or another vote. I0312 20:10:11.285351 1153 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 5 pre-election: Vote denied by peer ded9b7f8677940fb91e1a8c8bceceaed. Message: Service unavailable: T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6885de38238442d6a9e5f1eb9fb569ff for term 5 because replica is already servicing an update from a current leader or another vote. I0312 20:10:11.285571 1153 leader_election.cc:258] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. I0312 20:10:11.286522 1454 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff: Snoozing failure detection for 0.030s (election complete) I0312 20:10:11.286731 1454 raft_consensus.cc:2455] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority I0312 20:10:11.289366 1455 raft_consensus.cc:436] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 4 FOLLOWER]: Starting pre-election (detected failure of leader f78a2351568c4a268ee1112681e22bec) I0312 20:10:11.289618 1455 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed: Snoozing failure detection for 0.034s (starting election) I0312 20:10:11.289839 1455 raft_consensus.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:11.290830 1455 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 5 pre-election: Requesting pre-vote from peer f78a2351568c4a268ee1112681e22bec I0312 20:10:11.291180 1455 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 5 pre-election: Requesting pre-vote from peer 6885de38238442d6a9e5f1eb9fb569ff I0312 20:10:11.292738 1219 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" candidate_term: 5 candidate_status { last_received { term: 4 index: 57 } } ignore_live_leader: false dest_uuid: "6885de38238442d6a9e5f1eb9fb569ff" is_pre_election: true I0312 20:10:11.293236 1219 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff: Snoozing failure detection for 0.060s (vote granted) I0312 20:10:11.293439 1219 raft_consensus.cc:2215] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ded9b7f8677940fb91e1a8c8bceceaed in term 4. I0312 20:10:11.294406 1029 leader_election.cc:363] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 5 pre-election: Vote granted by peer 6885de38238442d6a9e5f1eb9fb569ff I0312 20:10:11.294602 1029 leader_election.cc:258] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. I0312 20:10:11.295357 1455 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed: Snoozing failure detection for 0.026s (election complete) I0312 20:10:11.295568 1455 raft_consensus.cc:2507] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 4 FOLLOWER]: Leader pre-election won for term 5 I0312 20:10:11.295754 1455 raft_consensus.cc:436] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 4 FOLLOWER]: Starting leader election (detected failure of leader f78a2351568c4a268ee1112681e22bec) I0312 20:10:11.295943 1455 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed: Snoozing failure detection for 0.036s (starting election) I0312 20:10:11.296113 1455 raft_consensus.cc:2752] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 4 FOLLOWER]: Advancing to term 5 I0312 20:10:11.298167 1346 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" candidate_term: 5 candidate_status { last_received { term: 4 index: 57 } } ignore_live_leader: false dest_uuid: "f78a2351568c4a268ee1112681e22bec" is_pre_election: true I0312 20:10:11.298681 1346 raft_consensus.cc:2176] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 4 LEADER]: Leader pre-election vote request: Denying vote to candidate ded9b7f8677940fb91e1a8c8bceceaed for term 5 because replica is either leader or believes a valid leader to be alive. I0312 20:10:11.299633 1027 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 5 pre-election: Vote denied by peer f78a2351568c4a268ee1112681e22bec. Message: Invalid argument: T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 4 LEADER]: Leader pre-election vote request: Denying vote to candidate ded9b7f8677940fb91e1a8c8bceceaed for term 5 because replica is either leader or believes a valid leader to be alive. I0312 20:10:11.300956 1455 raft_consensus.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:11.302075 1455 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 5 election: Requesting vote from peer f78a2351568c4a268ee1112681e22bec I0312 20:10:11.302763 1095 raft_consensus.cc:1026] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 5 FOLLOWER]: Rejecting Update request from peer f78a2351568c4a268ee1112681e22bec for earlier term 4. Current term is 5. Ops: [] I0312 20:10:11.305554 1415 consensus_queue.cc:1028] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 }, Status: INVALID_TERM, Last received: 4.57, Next index: 58, Last known committed idx: 56, Time since last communication: 0.000s I0312 20:10:11.306077 1424 raft_consensus.cc:2747] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 4 LEADER]: Stepping down as leader of term 4 I0312 20:10:11.306279 1424 raft_consensus.cc:605] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 4 LEADER]: Becoming Follower/Learner. State: Replica: f78a2351568c4a268ee1112681e22bec, State: Running, Role: LEADER I0312 20:10:11.307093 1455 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 5 election: Requesting vote from peer 6885de38238442d6a9e5f1eb9fb569ff I0312 20:10:11.306695 1424 consensus_queue.cc:219] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 57, Committed index: 57, Last appended: 4.58, Last appended by leader: 58, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:11.307327 1424 consensus_peers.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec -> Peer ded9b7f8677940fb91e1a8c8bceceaed (127.0.53.1:35037): Closing peer: ded9b7f8677940fb91e1a8c8bceceaed I0312 20:10:11.307513 1424 consensus_peers.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec -> Peer 6885de38238442d6a9e5f1eb9fb569ff (127.0.53.2:51697): Closing peer: 6885de38238442d6a9e5f1eb9fb569ff I0312 20:10:11.307804 1424 raft_consensus.cc:2752] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 4 FOLLOWER]: Advancing to term 5 I0312 20:10:11.308310 1219 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" candidate_term: 5 candidate_status { last_received { term: 4 index: 57 } } ignore_live_leader: false dest_uuid: "6885de38238442d6a9e5f1eb9fb569ff" I0312 20:10:11.308704 1219 raft_consensus.cc:2752] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 4 FOLLOWER]: Advancing to term 5 I0312 20:10:11.308919 1219 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff: Snoozing failure detection for 0.068s (vote granted) I0312 20:10:11.309772 1346 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" candidate_term: 5 candidate_status { last_received { term: 4 index: 57 } } ignore_live_leader: false dest_uuid: "f78a2351568c4a268ee1112681e22bec" I0312 20:10:11.314241 1219 raft_consensus.cc:2215] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ded9b7f8677940fb91e1a8c8bceceaed in term 5. I0312 20:10:11.314435 1346 raft_consensus.cc:2162] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate ded9b7f8677940fb91e1a8c8bceceaed for term 5 because replica has last-logged OpId of term: 4 index: 58, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 57. I0312 20:10:11.315156 1029 leader_election.cc:363] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 5 election: Vote granted by peer 6885de38238442d6a9e5f1eb9fb569ff I0312 20:10:11.315356 1029 leader_election.cc:258] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. I0312 20:10:11.315552 1027 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 5 election: Vote denied by peer f78a2351568c4a268ee1112681e22bec. Message: Invalid argument: T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate ded9b7f8677940fb91e1a8c8bceceaed for term 5 because replica has last-logged OpId of term: 4 index: 58, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 57. I0312 20:10:11.315806 1456 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed: Snoozing failure detection for 0.030s (election complete) I0312 20:10:11.316062 1456 raft_consensus.cc:2507] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 5 FOLLOWER]: Leader election won for term 5 I0312 20:10:11.316320 1456 raft_consensus.cc:572] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 5 LEADER]: Becoming Leader. State: Replica: ded9b7f8677940fb91e1a8c8bceceaed, State: Running, Role: LEADER I0312 20:10:11.316715 1456 consensus_queue.cc:196] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 56, Committed index: 56, Last appended: 4.57, Last appended by leader: 57, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:11.320825 974 catalog_manager.cc:3592] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed reported cstate change: term changed from 4 to 5, leader changed from f78a2351568c4a268ee1112681e22bec (127.0.53.3) to ded9b7f8677940fb91e1a8c8bceceaed (127.0.53.1). New cstate: current_term: 5 leader_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } health_report { overall_health: HEALTHY } } } I0312 20:10:11.322407 1221 raft_consensus.cc:1026] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 5 FOLLOWER]: Rejecting Update request from peer f78a2351568c4a268ee1112681e22bec for earlier term 4. Current term is 5. Ops: [] I0312 20:10:11.331964 1221 raft_consensus.cc:1058] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 5 FOLLOWER]: Refusing update from remote peer ded9b7f8677940fb91e1a8c8bceceaed: Log matching property violated. Preceding OpId in replica: term: 4 index: 57. Preceding OpId from leader: term: 5 index: 58. (index mismatch) I0312 20:10:11.332180 1346 raft_consensus.cc:1058] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 5 FOLLOWER]: Refusing update from remote peer ded9b7f8677940fb91e1a8c8bceceaed: Log matching property violated. Preceding OpId in replica: term: 4 index: 58. Preceding OpId from leader: term: 5 index: 58. (term mismatch) I0312 20:10:11.333544 1346 pending_rounds.cc:76] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec: Aborting all transactions after (but not including) 57 I0312 20:10:11.333725 1346 pending_rounds.cc:97] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec: Aborting uncommitted WRITE_OP operation due to leader change: 4.58 I0312 20:10:11.334302 1417 consensus_queue.cc:1017] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [LEADER]: Connected to new peer: Peer: permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 }, Status: LMP_MISMATCH, Last received: 4.57, Next index: 58, Last known committed idx: 56, Time since last communication: 0.000s I0312 20:10:11.341286 1415 raft_consensus.cc:436] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0312 20:10:11.341506 1415 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec: Snoozing failure detection for 0.018s (starting election) I0312 20:10:11.341738 1415 raft_consensus.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:11.342754 1415 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 6 pre-election: Requesting pre-vote from peer 6885de38238442d6a9e5f1eb9fb569ff I0312 20:10:11.343124 1415 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 6 pre-election: Requesting pre-vote from peer ded9b7f8677940fb91e1a8c8bceceaed I0312 20:10:11.343542 1415 raft_consensus.cc:436] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0312 20:10:11.343744 1415 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec: Snoozing failure detection for 0.018s (starting election) I0312 20:10:11.343966 1415 raft_consensus.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:11.348249 1415 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 6 pre-election: Requesting pre-vote from peer 6885de38238442d6a9e5f1eb9fb569ff I0312 20:10:11.349611 1220 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "f78a2351568c4a268ee1112681e22bec" candidate_term: 6 candidate_status { last_received { term: 4 index: 57 } } ignore_live_leader: false dest_uuid: "6885de38238442d6a9e5f1eb9fb569ff" is_pre_election: true I0312 20:10:11.350194 1415 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 6 pre-election: Requesting pre-vote from peer ded9b7f8677940fb91e1a8c8bceceaed I0312 20:10:11.350925 1417 consensus_queue.cc:1017] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [LEADER]: Connected to new peer: Peer: permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 }, Status: LMP_MISMATCH, Last received: 4.57, Next index: 58, Last known committed idx: 57, Time since last communication: 0.000s I0312 20:10:11.353147 1454 raft_consensus.cc:436] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ded9b7f8677940fb91e1a8c8bceceaed) I0312 20:10:11.353384 1454 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff: Snoozing failure detection for 0.016s (starting election) I0312 20:10:11.353613 1454 raft_consensus.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:11.354614 1454 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 6 pre-election: Requesting pre-vote from peer f78a2351568c4a268ee1112681e22bec I0312 20:10:11.354745 1220 raft_consensus.cc:2190] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 6 because replica is already servicing an update from a current leader or another vote. I0312 20:10:11.354977 1454 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 6 pre-election: Requesting pre-vote from peer ded9b7f8677940fb91e1a8c8bceceaed I0312 20:10:11.356158 1279 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 6 pre-election: Vote denied by peer 6885de38238442d6a9e5f1eb9fb569ff. Message: Service unavailable: T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 6 because replica is already servicing an update from a current leader or another vote. I0312 20:10:11.356734 1095 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "f78a2351568c4a268ee1112681e22bec" candidate_term: 6 candidate_status { last_received { term: 4 index: 57 } } ignore_live_leader: false dest_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" is_pre_election: true I0312 20:10:11.359067 1095 raft_consensus.cc:2176] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 5 LEADER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 6 because replica is either leader or believes a valid leader to be alive. I0312 20:10:11.359606 1221 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "f78a2351568c4a268ee1112681e22bec" candidate_term: 6 candidate_status { last_received { term: 4 index: 57 } } ignore_live_leader: false dest_uuid: "6885de38238442d6a9e5f1eb9fb569ff" is_pre_election: true I0312 20:10:11.360188 1221 raft_consensus.cc:2162] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 6 because replica has last-logged OpId of term: 5 index: 58, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 57. W0312 20:10:11.360438 1221 rpcz_store.cc:251] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.53.3:38196 (request call id 48) took 14ms (client timeout 18). W0312 20:10:11.360930 1221 rpcz_store.cc:255] Trace: 0312 20:10:11.345945 (+ 0us) service_pool.cc:163] Inserting onto call queue 0312 20:10:11.359460 (+ 13515us) service_pool.cc:222] Handling call 0312 20:10:11.360397 (+ 937us) inbound_call.cc:157] Queueing success response Metrics: {} I0312 20:10:11.362051 1279 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 6 pre-election: Vote denied by peer 6885de38238442d6a9e5f1eb9fb569ff. Message: Invalid argument: T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 6 because replica has last-logged OpId of term: 5 index: 58, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 57. W0312 20:10:11.363970 1278 leader_election.cc:287] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer ded9b7f8677940fb91e1a8c8bceceaed: Timed out: RequestConsensusVote RPC to 127.0.53.1:35037 timed out after 0.018s (SENT) I0312 20:10:11.364192 1278 leader_election.cc:258] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. I0312 20:10:11.365090 1415 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec: Snoozing failure detection for 0.017s (election complete) I0312 20:10:11.365334 1415 raft_consensus.cc:2455] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority W0312 20:10:11.369197 1278 leader_election.cc:287] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer ded9b7f8677940fb91e1a8c8bceceaed: Timed out: RequestConsensusVote RPC to 127.0.53.1:35037 timed out after 0.018s (SENT) I0312 20:10:11.369428 1278 leader_election.cc:258] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. I0312 20:10:11.370030 1415 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec: Snoozing failure detection for 0.027s (election complete) I0312 20:10:11.370256 1415 raft_consensus.cc:2455] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority I0312 20:10:11.371453 1346 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "6885de38238442d6a9e5f1eb9fb569ff" candidate_term: 6 candidate_status { last_received { term: 5 index: 58 } } ignore_live_leader: false dest_uuid: "f78a2351568c4a268ee1112681e22bec" is_pre_election: true I0312 20:10:11.371882 1346 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec: Snoozing failure detection for 0.030s (vote granted) I0312 20:10:11.372076 1346 raft_consensus.cc:2215] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6885de38238442d6a9e5f1eb9fb569ff in term 5. W0312 20:10:11.372162 1153 leader_election.cc:287] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer ded9b7f8677940fb91e1a8c8bceceaed: Timed out: RequestConsensusVote RPC to 127.0.53.1:35037 timed out after 0.016s (SENT) I0312 20:10:11.373857 1095 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "f78a2351568c4a268ee1112681e22bec" candidate_term: 6 candidate_status { last_received { term: 4 index: 57 } } ignore_live_leader: false dest_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" is_pre_election: true I0312 20:10:11.374338 1095 raft_consensus.cc:2176] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 5 LEADER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 6 because replica is either leader or believes a valid leader to be alive. I0312 20:10:11.374303 1096 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "6885de38238442d6a9e5f1eb9fb569ff" candidate_term: 6 candidate_status { last_received { term: 5 index: 58 } } ignore_live_leader: false dest_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" is_pre_election: true I0312 20:10:11.374855 1096 raft_consensus.cc:2176] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 5 LEADER]: Leader pre-election vote request: Denying vote to candidate 6885de38238442d6a9e5f1eb9fb569ff for term 6 because replica is either leader or believes a valid leader to be alive. W0312 20:10:11.375172 1096 rpcz_store.cc:251] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.53.2:38849 (request call id 16) took 17ms (client timeout 16). W0312 20:10:11.375327 1152 leader_election.cc:287] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer f78a2351568c4a268ee1112681e22bec: Timed out: RequestConsensusVote RPC to 127.0.53.3:43274 timed out after 0.016s (SENT) I0312 20:10:11.375540 1152 leader_election.cc:258] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. I0312 20:10:11.376503 1454 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff: Snoozing failure detection for 0.020s (election complete) I0312 20:10:11.376709 1454 raft_consensus.cc:2455] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority W0312 20:10:11.385476 1096 rpcz_store.cc:255] Trace: 0312 20:10:11.357546 (+ 0us) service_pool.cc:163] Inserting onto call queue 0312 20:10:11.374164 (+ 16618us) service_pool.cc:222] Handling call 0312 20:10:11.375122 (+ 958us) inbound_call.cc:157] Queueing success response Metrics: {} I0312 20:10:11.412729 1424 raft_consensus.cc:436] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ded9b7f8677940fb91e1a8c8bceceaed) I0312 20:10:11.413000 1424 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec: Snoozing failure detection for 0.018s (starting election) I0312 20:10:11.413270 1424 raft_consensus.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:11.414327 1424 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 6 pre-election: Requesting pre-vote from peer 6885de38238442d6a9e5f1eb9fb569ff I0312 20:10:11.415590 1424 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 6 pre-election: Requesting pre-vote from peer ded9b7f8677940fb91e1a8c8bceceaed I0312 20:10:11.415592 1221 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "f78a2351568c4a268ee1112681e22bec" candidate_term: 6 candidate_status { last_received { term: 5 index: 61 } } ignore_live_leader: false dest_uuid: "6885de38238442d6a9e5f1eb9fb569ff" is_pre_election: true I0312 20:10:11.416141 1221 raft_consensus.cc:2176] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 6 because replica is either leader or believes a valid leader to be alive. I0312 20:10:11.417309 1279 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 6 pre-election: Vote denied by peer 6885de38238442d6a9e5f1eb9fb569ff. Message: Invalid argument: T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 6 because replica is either leader or believes a valid leader to be alive. I0312 20:10:11.425845 1096 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "f78a2351568c4a268ee1112681e22bec" candidate_term: 6 candidate_status { last_received { term: 5 index: 61 } } ignore_live_leader: false dest_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" is_pre_election: true I0312 20:10:11.426373 1096 raft_consensus.cc:2176] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 5 LEADER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 6 because replica is either leader or believes a valid leader to be alive. I0312 20:10:11.431526 1278 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 6 pre-election: Vote denied by peer ded9b7f8677940fb91e1a8c8bceceaed. Message: Invalid argument: T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 5 LEADER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 6 because replica is either leader or believes a valid leader to be alive. I0312 20:10:11.431784 1278 leader_election.cc:258] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. I0312 20:10:11.432987 1424 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec: Snoozing failure detection for 0.022s (election complete) I0312 20:10:11.433257 1424 raft_consensus.cc:2455] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority I0312 20:10:11.435783 1454 raft_consensus.cc:436] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ded9b7f8677940fb91e1a8c8bceceaed) I0312 20:10:11.436014 1454 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff: Snoozing failure detection for 0.020s (starting election) I0312 20:10:11.436233 1454 raft_consensus.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:11.437330 1454 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 6 pre-election: Requesting pre-vote from peer f78a2351568c4a268ee1112681e22bec I0312 20:10:11.437672 1454 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 6 pre-election: Requesting pre-vote from peer ded9b7f8677940fb91e1a8c8bceceaed I0312 20:10:11.438948 1096 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "6885de38238442d6a9e5f1eb9fb569ff" candidate_term: 6 candidate_status { last_received { term: 5 index: 61 } } ignore_live_leader: false dest_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" is_pre_election: true I0312 20:10:11.439445 1096 raft_consensus.cc:2176] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 5 LEADER]: Leader pre-election vote request: Denying vote to candidate 6885de38238442d6a9e5f1eb9fb569ff for term 6 because replica is either leader or believes a valid leader to be alive. I0312 20:10:11.439723 1345 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "6885de38238442d6a9e5f1eb9fb569ff" candidate_term: 6 candidate_status { last_received { term: 5 index: 61 } } ignore_live_leader: false dest_uuid: "f78a2351568c4a268ee1112681e22bec" is_pre_election: true I0312 20:10:11.440419 1153 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 6 pre-election: Vote denied by peer ded9b7f8677940fb91e1a8c8bceceaed. Message: Invalid argument: T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 5 LEADER]: Leader pre-election vote request: Denying vote to candidate 6885de38238442d6a9e5f1eb9fb569ff for term 6 because replica is either leader or believes a valid leader to be alive. I0312 20:10:11.441813 1345 raft_consensus.cc:2190] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6885de38238442d6a9e5f1eb9fb569ff for term 6 because replica is already servicing an update from a current leader or another vote. I0312 20:10:11.442960 1152 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 6 pre-election: Vote denied by peer f78a2351568c4a268ee1112681e22bec. Message: Service unavailable: T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6885de38238442d6a9e5f1eb9fb569ff for term 6 because replica is already servicing an update from a current leader or another vote. I0312 20:10:11.443223 1152 leader_election.cc:258] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. I0312 20:10:11.443831 1454 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff: Snoozing failure detection for 0.015s (election complete) I0312 20:10:11.444031 1454 raft_consensus.cc:2455] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority I0312 20:10:11.466666 1424 raft_consensus.cc:436] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ded9b7f8677940fb91e1a8c8bceceaed) I0312 20:10:11.466898 1424 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec: Snoozing failure detection for 0.030s (starting election) I0312 20:10:11.467118 1424 raft_consensus.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:11.468082 1424 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 6 pre-election: Requesting pre-vote from peer 6885de38238442d6a9e5f1eb9fb569ff I0312 20:10:11.469395 1221 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "f78a2351568c4a268ee1112681e22bec" candidate_term: 6 candidate_status { last_received { term: 5 index: 63 } } ignore_live_leader: false dest_uuid: "6885de38238442d6a9e5f1eb9fb569ff" is_pre_election: true I0312 20:10:11.470011 1221 raft_consensus.cc:2176] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 6 because replica is either leader or believes a valid leader to be alive. I0312 20:10:11.470980 1279 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 6 pre-election: Vote denied by peer 6885de38238442d6a9e5f1eb9fb569ff. Message: Invalid argument: T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 6 because replica is either leader or believes a valid leader to be alive. I0312 20:10:11.471894 1424 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 6 pre-election: Requesting pre-vote from peer ded9b7f8677940fb91e1a8c8bceceaed I0312 20:10:11.487053 1096 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "f78a2351568c4a268ee1112681e22bec" candidate_term: 6 candidate_status { last_received { term: 5 index: 63 } } ignore_live_leader: false dest_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" is_pre_election: true I0312 20:10:11.487578 1096 raft_consensus.cc:2176] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 5 LEADER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 6 because replica is either leader or believes a valid leader to be alive. I0312 20:10:11.496681 1454 raft_consensus.cc:436] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ded9b7f8677940fb91e1a8c8bceceaed) I0312 20:10:11.496918 1454 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff: Snoozing failure detection for 0.030s (starting election) I0312 20:10:11.497153 1454 raft_consensus.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:11.498225 1454 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 6 pre-election: Requesting pre-vote from peer f78a2351568c4a268ee1112681e22bec I0312 20:10:11.498580 1454 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 6 pre-election: Requesting pre-vote from peer ded9b7f8677940fb91e1a8c8bceceaed I0312 20:10:11.501621 1096 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "6885de38238442d6a9e5f1eb9fb569ff" candidate_term: 6 candidate_status { last_received { term: 5 index: 64 } } ignore_live_leader: false dest_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" is_pre_election: true I0312 20:10:11.501760 1424 raft_consensus.cc:436] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ded9b7f8677940fb91e1a8c8bceceaed) I0312 20:10:11.501983 1424 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec: Snoozing failure detection for 0.024s (starting election) I0312 20:10:11.502122 1096 raft_consensus.cc:2176] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 5 LEADER]: Leader pre-election vote request: Denying vote to candidate 6885de38238442d6a9e5f1eb9fb569ff for term 6 because replica is either leader or believes a valid leader to be alive. I0312 20:10:11.502212 1424 raft_consensus.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } W0312 20:10:11.502887 1278 leader_election.cc:287] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer ded9b7f8677940fb91e1a8c8bceceaed: Timed out: RequestConsensusVote RPC to 127.0.53.1:35037 timed out after 0.030s (SENT) I0312 20:10:11.503121 1278 leader_election.cc:258] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. I0312 20:10:11.503160 1153 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 6 pre-election: Vote denied by peer ded9b7f8677940fb91e1a8c8bceceaed. Message: Invalid argument: T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 5 LEADER]: Leader pre-election vote request: Denying vote to candidate 6885de38238442d6a9e5f1eb9fb569ff for term 6 because replica is either leader or believes a valid leader to be alive. I0312 20:10:11.504758 1415 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec: Snoozing failure detection for 0.028s (election complete) I0312 20:10:11.504987 1415 raft_consensus.cc:2455] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority I0312 20:10:11.505470 1424 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 6 pre-election: Requesting pre-vote from peer 6885de38238442d6a9e5f1eb9fb569ff I0312 20:10:11.506850 1220 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "f78a2351568c4a268ee1112681e22bec" candidate_term: 6 candidate_status { last_received { term: 5 index: 64 } } ignore_live_leader: false dest_uuid: "6885de38238442d6a9e5f1eb9fb569ff" is_pre_election: true I0312 20:10:11.507385 1220 raft_consensus.cc:2190] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 6 because replica is already servicing an update from a current leader or another vote. I0312 20:10:11.508445 1279 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 6 pre-election: Vote denied by peer 6885de38238442d6a9e5f1eb9fb569ff. Message: Service unavailable: T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 6 because replica is already servicing an update from a current leader or another vote. I0312 20:10:11.508817 1424 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 6 pre-election: Requesting pre-vote from peer ded9b7f8677940fb91e1a8c8bceceaed I0312 20:10:11.514425 1096 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "f78a2351568c4a268ee1112681e22bec" candidate_term: 6 candidate_status { last_received { term: 5 index: 64 } } ignore_live_leader: false dest_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" is_pre_election: true I0312 20:10:11.515807 1096 raft_consensus.cc:2176] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 5 LEADER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 6 because replica is either leader or believes a valid leader to be alive. I0312 20:10:11.516587 1346 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "6885de38238442d6a9e5f1eb9fb569ff" candidate_term: 6 candidate_status { last_received { term: 5 index: 64 } } ignore_live_leader: false dest_uuid: "f78a2351568c4a268ee1112681e22bec" is_pre_election: true I0312 20:10:11.517092 1346 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec: Snoozing failure detection for 0.017s (vote granted) I0312 20:10:11.517321 1346 raft_consensus.cc:2215] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6885de38238442d6a9e5f1eb9fb569ff in term 5. I0312 20:10:11.526530 1152 leader_election.cc:363] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 6 pre-election: Vote granted by peer f78a2351568c4a268ee1112681e22bec I0312 20:10:11.526759 1152 leader_election.cc:258] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. I0312 20:10:11.527199 1454 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff: Snoozing failure detection for 0.045s (election complete) I0312 20:10:11.527405 1454 raft_consensus.cc:2507] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 5 FOLLOWER]: Leader pre-election won for term 6 I0312 20:10:11.527596 1454 raft_consensus.cc:436] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 5 FOLLOWER]: Starting leader election (detected failure of leader ded9b7f8677940fb91e1a8c8bceceaed) I0312 20:10:11.527791 1454 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff: Snoozing failure detection for 0.043s (starting election) I0312 20:10:11.527958 1454 raft_consensus.cc:2752] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 5 FOLLOWER]: Advancing to term 6 I0312 20:10:11.532274 1278 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 6 pre-election: Vote denied by peer ded9b7f8677940fb91e1a8c8bceceaed. Message: Invalid argument: T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 5 LEADER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 6 because replica is either leader or believes a valid leader to be alive. I0312 20:10:11.532495 1278 leader_election.cc:258] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. I0312 20:10:11.533223 1424 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec: Snoozing failure detection for 0.025s (election complete) I0312 20:10:11.533424 1424 raft_consensus.cc:2455] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority I0312 20:10:11.536695 1454 raft_consensus.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:11.546185 1454 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 6 election: Requesting vote from peer f78a2351568c4a268ee1112681e22bec I0312 20:10:11.546551 1454 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 6 election: Requesting vote from peer ded9b7f8677940fb91e1a8c8bceceaed I0312 20:10:11.547715 1096 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "6885de38238442d6a9e5f1eb9fb569ff" candidate_term: 6 candidate_status { last_received { term: 5 index: 66 } } ignore_live_leader: false dest_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" I0312 20:10:11.548200 1096 raft_consensus.cc:2176] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 5 LEADER]: Leader election vote request: Denying vote to candidate 6885de38238442d6a9e5f1eb9fb569ff for term 6 because replica is either leader or believes a valid leader to be alive. I0312 20:10:11.549160 1153 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 6 election: Vote denied by peer ded9b7f8677940fb91e1a8c8bceceaed. Message: Invalid argument: T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 5 LEADER]: Leader election vote request: Denying vote to candidate 6885de38238442d6a9e5f1eb9fb569ff for term 6 because replica is either leader or believes a valid leader to be alive. I0312 20:10:11.549738 1220 raft_consensus.cc:1026] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 6 FOLLOWER]: Rejecting Update request from peer ded9b7f8677940fb91e1a8c8bceceaed for earlier term 5. Current term is 6. Ops: [] I0312 20:10:11.551748 1455 consensus_queue.cc:1028] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 }, Status: INVALID_TERM, Last received: 5.66, Next index: 67, Last known committed idx: 65, Time since last communication: 0.000s I0312 20:10:11.552444 1417 raft_consensus.cc:2747] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 5 LEADER]: Stepping down as leader of term 5 I0312 20:10:11.552633 1417 raft_consensus.cc:605] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 5 LEADER]: Becoming Follower/Learner. State: Replica: ded9b7f8677940fb91e1a8c8bceceaed, State: Running, Role: LEADER I0312 20:10:11.553040 1417 consensus_queue.cc:219] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 66, Committed index: 66, Last appended: 5.68, Last appended by leader: 68, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:11.553655 1417 consensus_peers.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed -> Peer 6885de38238442d6a9e5f1eb9fb569ff (127.0.53.2:51697): Closing peer: 6885de38238442d6a9e5f1eb9fb569ff I0312 20:10:11.553833 1417 consensus_peers.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed -> Peer f78a2351568c4a268ee1112681e22bec (127.0.53.3:43274): Closing peer: f78a2351568c4a268ee1112681e22bec I0312 20:10:11.554011 1417 raft_consensus.cc:2752] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 5 FOLLOWER]: Advancing to term 6 I0312 20:10:11.561380 1424 raft_consensus.cc:436] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ded9b7f8677940fb91e1a8c8bceceaed) I0312 20:10:11.561635 1424 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec: Snoozing failure detection for 0.039s (starting election) I0312 20:10:11.561857 1424 raft_consensus.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:11.562857 1424 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 6 pre-election: Requesting pre-vote from peer 6885de38238442d6a9e5f1eb9fb569ff I0312 20:10:11.563220 1424 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 6 pre-election: Requesting pre-vote from peer ded9b7f8677940fb91e1a8c8bceceaed I0312 20:10:11.564941 1220 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "f78a2351568c4a268ee1112681e22bec" candidate_term: 6 candidate_status { last_received { term: 5 index: 68 } } ignore_live_leader: false dest_uuid: "6885de38238442d6a9e5f1eb9fb569ff" is_pre_election: true I0312 20:10:11.565521 1220 raft_consensus.cc:2145] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec in current term 6: Already voted for candidate 6885de38238442d6a9e5f1eb9fb569ff in this term. I0312 20:10:11.565850 1346 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "6885de38238442d6a9e5f1eb9fb569ff" candidate_term: 6 candidate_status { last_received { term: 5 index: 66 } } ignore_live_leader: false dest_uuid: "f78a2351568c4a268ee1112681e22bec" I0312 20:10:11.566310 1346 raft_consensus.cc:2752] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 5 FOLLOWER]: Advancing to term 6 I0312 20:10:11.566515 1279 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 6 pre-election: Vote denied by peer 6885de38238442d6a9e5f1eb9fb569ff. Message: Invalid argument: T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec in current term 6: Already voted for candidate 6885de38238442d6a9e5f1eb9fb569ff in this term. I0312 20:10:11.567416 1096 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "f78a2351568c4a268ee1112681e22bec" candidate_term: 6 candidate_status { last_received { term: 5 index: 68 } } ignore_live_leader: false dest_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" is_pre_election: true I0312 20:10:11.569082 1096 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed: Snoozing failure detection for 0.020s (vote granted) I0312 20:10:11.569284 1096 raft_consensus.cc:2215] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f78a2351568c4a268ee1112681e22bec in term 6. I0312 20:10:11.570202 1278 leader_election.cc:363] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 6 pre-election: Vote granted by peer ded9b7f8677940fb91e1a8c8bceceaed I0312 20:10:11.570399 1278 leader_election.cc:258] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. I0312 20:10:11.571451 1346 raft_consensus.cc:2162] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 6885de38238442d6a9e5f1eb9fb569ff for term 6 because replica has last-logged OpId of term: 5 index: 68, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 66. I0312 20:10:11.571678 1424 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec: Snoozing failure detection for 0.027s (election complete) I0312 20:10:11.571869 1424 raft_consensus.cc:2470] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 6 FOLLOWER]: Leader pre-election decision vote started in defunct term 5: won I0312 20:10:11.572350 1454 raft_consensus.cc:436] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0312 20:10:11.572616 1152 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 6 election: Vote denied by peer f78a2351568c4a268ee1112681e22bec. Message: Invalid argument: T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 6885de38238442d6a9e5f1eb9fb569ff for term 6 because replica has last-logged OpId of term: 5 index: 68, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 66. I0312 20:10:11.572763 1454 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff: Snoozing failure detection for 0.042s (starting election) I0312 20:10:11.572897 1152 leader_election.cc:258] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. I0312 20:10:11.572998 1454 raft_consensus.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:11.574225 1416 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff: Snoozing failure detection for 0.023s (election complete) I0312 20:10:11.574443 1416 raft_consensus.cc:2455] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority I0312 20:10:11.586181 1454 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 7 pre-election: Requesting pre-vote from peer f78a2351568c4a268ee1112681e22bec I0312 20:10:11.587466 1346 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "6885de38238442d6a9e5f1eb9fb569ff" candidate_term: 7 candidate_status { last_received { term: 5 index: 66 } } ignore_live_leader: false dest_uuid: "f78a2351568c4a268ee1112681e22bec" is_pre_election: true I0312 20:10:11.588080 1346 raft_consensus.cc:2162] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6885de38238442d6a9e5f1eb9fb569ff for term 7 because replica has last-logged OpId of term: 5 index: 68, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 66. I0312 20:10:11.588959 1152 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 7 pre-election: Vote denied by peer f78a2351568c4a268ee1112681e22bec. Message: Invalid argument: T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6885de38238442d6a9e5f1eb9fb569ff for term 7 because replica has last-logged OpId of term: 5 index: 68, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 66. I0312 20:10:11.589396 1454 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 7 pre-election: Requesting pre-vote from peer ded9b7f8677940fb91e1a8c8bceceaed I0312 20:10:11.591140 1417 raft_consensus.cc:436] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0312 20:10:11.591356 1417 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed: Snoozing failure detection for 0.015s (starting election) I0312 20:10:11.591728 1096 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "6885de38238442d6a9e5f1eb9fb569ff" candidate_term: 7 candidate_status { last_received { term: 5 index: 66 } } ignore_live_leader: false dest_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" is_pre_election: true I0312 20:10:11.591570 1417 raft_consensus.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:11.592543 1417 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 7 pre-election: Requesting pre-vote from peer f78a2351568c4a268ee1112681e22bec I0312 20:10:11.592892 1417 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 7 pre-election: Requesting pre-vote from peer 6885de38238442d6a9e5f1eb9fb569ff I0312 20:10:11.594403 1220 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" candidate_term: 7 candidate_status { last_received { term: 5 index: 68 } } ignore_live_leader: false dest_uuid: "6885de38238442d6a9e5f1eb9fb569ff" is_pre_election: true I0312 20:10:11.594827 1220 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff: Snoozing failure detection for 0.072s (vote granted) I0312 20:10:11.595021 1220 raft_consensus.cc:2215] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ded9b7f8677940fb91e1a8c8bceceaed in term 6. I0312 20:10:11.596246 1029 leader_election.cc:363] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 7 pre-election: Vote granted by peer 6885de38238442d6a9e5f1eb9fb569ff I0312 20:10:11.596439 1029 leader_election.cc:258] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. I0312 20:10:11.597180 1096 raft_consensus.cc:2162] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6885de38238442d6a9e5f1eb9fb569ff for term 7 because replica has last-logged OpId of term: 5 index: 68, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 66. I0312 20:10:11.599308 1153 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 7 pre-election: Vote denied by peer ded9b7f8677940fb91e1a8c8bceceaed. Message: Invalid argument: T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6885de38238442d6a9e5f1eb9fb569ff for term 7 because replica has last-logged OpId of term: 5 index: 68, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 66. I0312 20:10:11.599527 1153 leader_election.cc:258] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. I0312 20:10:11.600127 1454 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff: Snoozing failure detection for 0.042s (election complete) I0312 20:10:11.600322 1454 raft_consensus.cc:2455] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority I0312 20:10:11.600878 1424 raft_consensus.cc:436] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0312 20:10:11.601124 1424 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec: Snoozing failure detection for 0.057s (starting election) I0312 20:10:11.601354 1424 raft_consensus.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:11.602371 1424 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 7 pre-election: Requesting pre-vote from peer 6885de38238442d6a9e5f1eb9fb569ff I0312 20:10:11.602730 1424 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 7 pre-election: Requesting pre-vote from peer ded9b7f8677940fb91e1a8c8bceceaed I0312 20:10:11.604382 1220 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "f78a2351568c4a268ee1112681e22bec" candidate_term: 7 candidate_status { last_received { term: 5 index: 68 } } ignore_live_leader: false dest_uuid: "6885de38238442d6a9e5f1eb9fb569ff" is_pre_election: true I0312 20:10:11.604804 1220 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff: Snoozing failure detection for 0.053s (vote granted) I0312 20:10:11.605007 1220 raft_consensus.cc:2215] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f78a2351568c4a268ee1112681e22bec in term 6. I0312 20:10:11.606076 1417 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed: Snoozing failure detection for 0.018s (election complete) I0312 20:10:11.606297 1417 raft_consensus.cc:2507] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 6 FOLLOWER]: Leader pre-election won for term 7 I0312 20:10:11.606475 1417 raft_consensus.cc:436] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0312 20:10:11.606647 1417 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed: Snoozing failure detection for 0.016s (starting election) I0312 20:10:11.606817 1417 raft_consensus.cc:2752] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 6 FOLLOWER]: Advancing to term 7 I0312 20:10:11.609810 1279 leader_election.cc:363] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 7 pre-election: Vote granted by peer 6885de38238442d6a9e5f1eb9fb569ff I0312 20:10:11.610023 1279 leader_election.cc:258] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. I0312 20:10:11.610509 1424 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec: Snoozing failure detection for 0.036s (election complete) I0312 20:10:11.610730 1424 raft_consensus.cc:2507] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 6 FOLLOWER]: Leader pre-election won for term 7 I0312 20:10:11.610918 1424 raft_consensus.cc:436] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0312 20:10:11.611099 1424 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec: Snoozing failure detection for 0.046s (starting election) I0312 20:10:11.611273 1424 raft_consensus.cc:2752] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 6 FOLLOWER]: Advancing to term 7 I0312 20:10:11.614310 1346 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" candidate_term: 7 candidate_status { last_received { term: 5 index: 68 } } ignore_live_leader: false dest_uuid: "f78a2351568c4a268ee1112681e22bec" is_pre_election: true I0312 20:10:11.614636 1096 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "f78a2351568c4a268ee1112681e22bec" candidate_term: 7 candidate_status { last_received { term: 5 index: 68 } } ignore_live_leader: false dest_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" is_pre_election: true I0312 20:10:11.621053 1424 raft_consensus.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:11.622082 1424 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 7 election: Requesting vote from peer 6885de38238442d6a9e5f1eb9fb569ff I0312 20:10:11.622241 1346 raft_consensus.cc:2145] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ded9b7f8677940fb91e1a8c8bceceaed in current term 7: Already voted for candidate f78a2351568c4a268ee1112681e22bec in this term. I0312 20:10:11.623365 1027 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 7 pre-election: Vote denied by peer f78a2351568c4a268ee1112681e22bec. Message: Invalid argument: T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ded9b7f8677940fb91e1a8c8bceceaed in current term 7: Already voted for candidate f78a2351568c4a268ee1112681e22bec in this term. I0312 20:10:11.623281 1220 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "f78a2351568c4a268ee1112681e22bec" candidate_term: 7 candidate_status { last_received { term: 5 index: 68 } } ignore_live_leader: false dest_uuid: "6885de38238442d6a9e5f1eb9fb569ff" I0312 20:10:11.623668 1220 raft_consensus.cc:2752] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 6 FOLLOWER]: Advancing to term 7 I0312 20:10:11.623873 1220 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff: Snoozing failure detection for 0.053s (vote granted) I0312 20:10:11.630087 1220 raft_consensus.cc:2215] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f78a2351568c4a268ee1112681e22bec in term 7. I0312 20:10:11.630352 1424 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 7 election: Requesting vote from peer ded9b7f8677940fb91e1a8c8bceceaed I0312 20:10:11.631767 1095 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "f78a2351568c4a268ee1112681e22bec" candidate_term: 7 candidate_status { last_received { term: 5 index: 68 } } ignore_live_leader: false dest_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" I0312 20:10:11.618934 1417 raft_consensus.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:11.633095 1455 raft_consensus.cc:436] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0312 20:10:11.633303 1455 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed: Snoozing failure detection for 0.017s (starting election) I0312 20:10:11.633519 1455 raft_consensus.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:11.634605 1455 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 8 pre-election: Requesting pre-vote from peer f78a2351568c4a268ee1112681e22bec I0312 20:10:11.634946 1455 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 8 pre-election: Requesting pre-vote from peer 6885de38238442d6a9e5f1eb9fb569ff I0312 20:10:11.636281 1346 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" candidate_term: 8 candidate_status { last_received { term: 5 index: 68 } } ignore_live_leader: false dest_uuid: "f78a2351568c4a268ee1112681e22bec" is_pre_election: true I0312 20:10:11.636695 1346 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec: Snoozing failure detection for 0.069s (vote granted) I0312 20:10:11.636911 1346 raft_consensus.cc:2215] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ded9b7f8677940fb91e1a8c8bceceaed in term 7. I0312 20:10:11.639112 1027 leader_election.cc:363] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 8 pre-election: Vote granted by peer f78a2351568c4a268ee1112681e22bec I0312 20:10:11.639320 1027 leader_election.cc:258] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. I0312 20:10:11.640538 1455 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed: Snoozing failure detection for 0.022s (election complete) I0312 20:10:11.640753 1455 raft_consensus.cc:2507] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 7 FOLLOWER]: Leader pre-election won for term 8 I0312 20:10:11.640928 1455 raft_consensus.cc:436] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0312 20:10:11.641124 1455 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed: Snoozing failure detection for 0.016s (starting election) I0312 20:10:11.641300 1455 raft_consensus.cc:2752] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 7 FOLLOWER]: Advancing to term 8 I0312 20:10:11.646003 1455 raft_consensus.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:11.648154 1095 raft_consensus.cc:2190] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 7 because replica is already servicing an update from a current leader or another vote. W0312 20:10:11.649405 1278 leader_election.cc:347] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 7 election: Vote denied by peer ded9b7f8677940fb91e1a8c8bceceaed with higher term. Message: Service unavailable: T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 7 because replica is already servicing an update from a current leader or another vote. I0312 20:10:11.649590 1278 leader_election.cc:350] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 7 election: Cancelling election due to peer responding with higher term I0312 20:10:11.650087 1415 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec: Snoozing failure detection for 0.030s (election complete) I0312 20:10:11.650254 1096 raft_consensus.cc:2120] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for earlier term 7. Current term is 8. I0312 20:10:11.650293 1415 raft_consensus.cc:2752] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 7 FOLLOWER]: Advancing to term 8 I0312 20:10:11.651202 1455 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 8 election: Requesting vote from peer f78a2351568c4a268ee1112681e22bec W0312 20:10:11.653729 1278 leader_election.cc:347] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 7 pre-election: Vote denied by peer ded9b7f8677940fb91e1a8c8bceceaed with higher term. Message: Invalid argument: T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for earlier term 7. Current term is 8. I0312 20:10:11.656590 1415 raft_consensus.cc:2455] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 8 FOLLOWER]: Leader election lost for term 7. Reason: Vote denied by peer ded9b7f8677940fb91e1a8c8bceceaed with higher term. Message: Service unavailable: T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 7 because replica is already servicing an update from a current leader or another vote. W0312 20:10:11.657579 1029 leader_election.cc:287] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 6885de38238442d6a9e5f1eb9fb569ff: Timed out: RequestConsensusVote RPC to 127.0.53.2:51697 timed out after 0.017s (SENT) I0312 20:10:11.657621 1456 raft_consensus.cc:436] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0312 20:10:11.657860 1456 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed: Snoozing failure detection for 0.017s (starting election) I0312 20:10:11.658071 1456 raft_consensus.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:11.658993 1456 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 9 pre-election: Requesting pre-vote from peer f78a2351568c4a268ee1112681e22bec I0312 20:10:11.659312 1456 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 9 pre-election: Requesting pre-vote from peer 6885de38238442d6a9e5f1eb9fb569ff I0312 20:10:11.662691 1346 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" candidate_term: 8 candidate_status { last_received { term: 5 index: 68 } } ignore_live_leader: false dest_uuid: "f78a2351568c4a268ee1112681e22bec" I0312 20:10:11.663130 1346 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec: Snoozing failure detection for 0.054s (vote granted) I0312 20:10:11.663358 1345 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" candidate_term: 9 candidate_status { last_received { term: 5 index: 68 } } ignore_live_leader: false dest_uuid: "f78a2351568c4a268ee1112681e22bec" is_pre_election: true I0312 20:10:11.640662 1221 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" candidate_term: 8 candidate_status { last_received { term: 5 index: 68 } } ignore_live_leader: false dest_uuid: "6885de38238442d6a9e5f1eb9fb569ff" is_pre_election: true I0312 20:10:11.660585 1219 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" candidate_term: 9 candidate_status { last_received { term: 5 index: 68 } } ignore_live_leader: false dest_uuid: "6885de38238442d6a9e5f1eb9fb569ff" is_pre_election: true I0312 20:10:11.665551 1221 raft_consensus.cc:2190] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ded9b7f8677940fb91e1a8c8bceceaed for term 8 because replica is already servicing an update from a current leader or another vote. W0312 20:10:11.665832 1221 rpcz_store.cc:251] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.53.1:43660 (request call id 38) took 25ms (client timeout 17). W0312 20:10:11.666381 1221 rpcz_store.cc:255] Trace: 0312 20:10:11.640406 (+ 0us) service_pool.cc:163] Inserting onto call queue 0312 20:10:11.640517 (+ 111us) service_pool.cc:222] Handling call 0312 20:10:11.665785 (+ 25268us) inbound_call.cc:157] Queueing success response Metrics: {"spinlock_wait_cycles":515968} I0312 20:10:11.666918 1346 raft_consensus.cc:2215] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ded9b7f8677940fb91e1a8c8bceceaed in term 8. I0312 20:10:11.667228 1219 raft_consensus.cc:2190] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ded9b7f8677940fb91e1a8c8bceceaed for term 9 because replica is already servicing an update from a current leader or another vote. I0312 20:10:11.667299 1345 raft_consensus.cc:2190] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ded9b7f8677940fb91e1a8c8bceceaed for term 9 because replica is already servicing an update from a current leader or another vote. I0312 20:10:11.668612 1029 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 9 pre-election: Vote denied by peer 6885de38238442d6a9e5f1eb9fb569ff. Message: Service unavailable: T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ded9b7f8677940fb91e1a8c8bceceaed for term 9 because replica is already servicing an update from a current leader or another vote. W0312 20:10:11.669088 1279 leader_election.cc:287] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer 6885de38238442d6a9e5f1eb9fb569ff: Timed out: RequestConsensusVote RPC to 127.0.53.2:51697 timed out after 0.046s (SENT) I0312 20:10:11.669121 1417 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 7 election: Requesting vote from peer f78a2351568c4a268ee1112681e22bec I0312 20:10:11.669490 1417 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 7 election: Requesting vote from peer 6885de38238442d6a9e5f1eb9fb569ff I0312 20:10:11.669735 1027 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 9 pre-election: Vote denied by peer f78a2351568c4a268ee1112681e22bec. Message: Service unavailable: T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ded9b7f8677940fb91e1a8c8bceceaed for term 9 because replica is already servicing an update from a current leader or another vote. I0312 20:10:11.670011 1027 leader_election.cc:258] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. I0312 20:10:11.670423 1456 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed: Snoozing failure detection for 0.017s (election complete) I0312 20:10:11.671319 1456 raft_consensus.cc:2455] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority I0312 20:10:11.671581 1219 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" candidate_term: 7 candidate_status { last_received { term: 5 index: 68 } } ignore_live_leader: false dest_uuid: "6885de38238442d6a9e5f1eb9fb569ff" I0312 20:10:11.672075 1219 raft_consensus.cc:2145] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate ded9b7f8677940fb91e1a8c8bceceaed in current term 7: Already voted for candidate f78a2351568c4a268ee1112681e22bec in this term. I0312 20:10:11.672256 1027 leader_election.cc:363] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 8 election: Vote granted by peer f78a2351568c4a268ee1112681e22bec I0312 20:10:11.672466 1027 leader_election.cc:258] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. I0312 20:10:11.672888 1456 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed: Snoozing failure detection for 0.029s (election complete) I0312 20:10:11.672935 1029 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 7 election: Vote denied by peer 6885de38238442d6a9e5f1eb9fb569ff. Message: Invalid argument: T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate ded9b7f8677940fb91e1a8c8bceceaed in current term 7: Already voted for candidate f78a2351568c4a268ee1112681e22bec in this term. I0312 20:10:11.673126 1456 raft_consensus.cc:2507] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 8 FOLLOWER]: Leader election won for term 8 I0312 20:10:11.673251 1455 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 8 election: Requesting vote from peer 6885de38238442d6a9e5f1eb9fb569ff W0312 20:10:11.668952 1220 rpcz_store.cc:251] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.53.3:38196 (request call id 55) took 45ms (client timeout 46). W0312 20:10:11.673945 1220 rpcz_store.cc:255] Trace: 0312 20:10:11.623051 (+ 0us) service_pool.cc:163] Inserting onto call queue 0312 20:10:11.623144 (+ 93us) service_pool.cc:222] Handling call 0312 20:10:11.668905 (+ 45761us) inbound_call.cc:157] Queueing success response Metrics: {} I0312 20:10:11.674038 1456 raft_consensus.cc:572] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 8 LEADER]: Becoming Leader. State: Replica: ded9b7f8677940fb91e1a8c8bceceaed, State: Running, Role: LEADER I0312 20:10:11.674535 1456 consensus_queue.cc:196] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 66, Committed index: 66, Last appended: 5.68, Last appended by leader: 68, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:11.675046 1219 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" candidate_term: 8 candidate_status { last_received { term: 5 index: 68 } } ignore_live_leader: false dest_uuid: "6885de38238442d6a9e5f1eb9fb569ff" I0312 20:10:11.675611 1219 raft_consensus.cc:2752] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 7 FOLLOWER]: Advancing to term 8 I0312 20:10:11.675823 1219 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff: Snoozing failure detection for 0.088s (vote granted) I0312 20:10:11.679092 973 catalog_manager.cc:3592] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed reported cstate change: term changed from 5 to 8. New cstate: current_term: 8 leader_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } health_report { overall_health: HEALTHY } } } I0312 20:10:11.679774 1219 raft_consensus.cc:2215] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ded9b7f8677940fb91e1a8c8bceceaed in term 8. I0312 20:10:11.683485 1029 leader_election.cc:363] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 8 election: Vote granted by peer 6885de38238442d6a9e5f1eb9fb569ff I0312 20:10:11.684906 1346 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" candidate_term: 7 candidate_status { last_received { term: 5 index: 68 } } ignore_live_leader: false dest_uuid: "f78a2351568c4a268ee1112681e22bec" I0312 20:10:11.685457 1346 raft_consensus.cc:2120] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate ded9b7f8677940fb91e1a8c8bceceaed for earlier term 7. Current term is 8. W0312 20:10:11.686782 1027 leader_election.cc:347] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 7 election: Vote denied by peer f78a2351568c4a268ee1112681e22bec with higher term. Message: Invalid argument: T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate ded9b7f8677940fb91e1a8c8bceceaed for earlier term 7. Current term is 8. I0312 20:10:11.687036 1027 leader_election.cc:350] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 7 election: Cancelling election due to peer responding with higher term I0312 20:10:11.688606 1346 raft_consensus.cc:1058] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 8 FOLLOWER]: Refusing update from remote peer ded9b7f8677940fb91e1a8c8bceceaed: Log matching property violated. Preceding OpId in replica: term: 5 index: 68. Preceding OpId from leader: term: 8 index: 69. (index mismatch) I0312 20:10:11.689458 1417 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed: Snoozing failure detection for 0.016s (election complete) I0312 20:10:11.689880 1417 raft_consensus.cc:2455] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 8 LEADER]: Leader election lost for term 7. Reason: Vote denied by peer f78a2351568c4a268ee1112681e22bec with higher term. Message: Invalid argument: T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate ded9b7f8677940fb91e1a8c8bceceaed for earlier term 7. Current term is 8. I0312 20:10:11.691427 1456 consensus_queue.cc:1017] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [LEADER]: Connected to new peer: Peer: permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 }, Status: LMP_MISMATCH, Last received: 5.68, Next index: 69, Last known committed idx: 66, Time since last communication: 0.000s I0312 20:10:11.692103 1220 raft_consensus.cc:1058] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 8 FOLLOWER]: Refusing update from remote peer ded9b7f8677940fb91e1a8c8bceceaed: Log matching property violated. Preceding OpId in replica: term: 5 index: 66. Preceding OpId from leader: term: 8 index: 69. (index mismatch) I0312 20:10:11.694200 1456 consensus_queue.cc:1017] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [LEADER]: Connected to new peer: Peer: permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 }, Status: LMP_MISMATCH, Last received: 5.66, Next index: 67, Last known committed idx: 65, Time since last communication: 0.000s I0312 20:10:11.716059 1424 raft_consensus.cc:436] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 8 FOLLOWER]: Starting pre-election (detected failure of leader ded9b7f8677940fb91e1a8c8bceceaed) I0312 20:10:11.716317 1424 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec: Snoozing failure detection for 0.016s (starting election) I0312 20:10:11.716563 1424 raft_consensus.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:11.717602 1424 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 9 pre-election: Requesting pre-vote from peer 6885de38238442d6a9e5f1eb9fb569ff I0312 20:10:11.718974 1219 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "f78a2351568c4a268ee1112681e22bec" candidate_term: 9 candidate_status { last_received { term: 8 index: 69 } } ignore_live_leader: false dest_uuid: "6885de38238442d6a9e5f1eb9fb569ff" is_pre_election: true I0312 20:10:11.719491 1424 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 9 pre-election: Requesting pre-vote from peer ded9b7f8677940fb91e1a8c8bceceaed I0312 20:10:11.730684 1454 raft_consensus.cc:436] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 8 FOLLOWER]: Starting pre-election (detected failure of leader ded9b7f8677940fb91e1a8c8bceceaed) I0312 20:10:11.730906 1454 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff: Snoozing failure detection for 0.017s (starting election) I0312 20:10:11.731117 1454 raft_consensus.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:11.732062 1454 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 9 pre-election: Requesting pre-vote from peer f78a2351568c4a268ee1112681e22bec I0312 20:10:11.732391 1454 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 9 pre-election: Requesting pre-vote from peer ded9b7f8677940fb91e1a8c8bceceaed I0312 20:10:11.733819 1096 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "6885de38238442d6a9e5f1eb9fb569ff" candidate_term: 9 candidate_status { last_received { term: 8 index: 69 } } ignore_live_leader: false dest_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" is_pre_election: true I0312 20:10:11.734341 1096 raft_consensus.cc:2176] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 8 LEADER]: Leader pre-election vote request: Denying vote to candidate 6885de38238442d6a9e5f1eb9fb569ff for term 9 because replica is either leader or believes a valid leader to be alive. W0312 20:10:11.734261 1279 leader_election.cc:287] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 6885de38238442d6a9e5f1eb9fb569ff: Timed out: RequestConsensusVote RPC to 127.0.53.2:51697 timed out after 0.016s (SENT) I0312 20:10:11.735043 1096 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "f78a2351568c4a268ee1112681e22bec" candidate_term: 9 candidate_status { last_received { term: 8 index: 69 } } ignore_live_leader: false dest_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" is_pre_election: true I0312 20:10:11.735352 1153 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 9 pre-election: Vote denied by peer ded9b7f8677940fb91e1a8c8bceceaed. Message: Invalid argument: T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 8 LEADER]: Leader pre-election vote request: Denying vote to candidate 6885de38238442d6a9e5f1eb9fb569ff for term 9 because replica is either leader or believes a valid leader to be alive. I0312 20:10:11.735556 1096 raft_consensus.cc:2176] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 8 LEADER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 9 because replica is either leader or believes a valid leader to be alive. W0312 20:10:11.736127 1278 leader_election.cc:287] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer ded9b7f8677940fb91e1a8c8bceceaed: Timed out: RequestConsensusVote RPC to 127.0.53.1:35037 timed out after 0.016s (SENT) I0312 20:10:11.736340 1278 leader_election.cc:258] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. I0312 20:10:11.736979 1424 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec: Snoozing failure detection for 0.015s (election complete) I0312 20:10:11.737232 1424 raft_consensus.cc:2455] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority I0312 20:10:11.740042 1346 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "6885de38238442d6a9e5f1eb9fb569ff" candidate_term: 9 candidate_status { last_received { term: 8 index: 69 } } ignore_live_leader: false dest_uuid: "f78a2351568c4a268ee1112681e22bec" is_pre_election: true I0312 20:10:11.741168 1219 raft_consensus.cc:2190] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 9 because replica is already servicing an update from a current leader or another vote. I0312 20:10:11.742508 1346 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec: Snoozing failure detection for 0.021s (vote granted) W0312 20:10:11.742072 1219 rpcz_store.cc:251] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.53.3:38196 (request call id 56) took 23ms (client timeout 16). W0312 20:10:11.743827 1219 rpcz_store.cc:255] Trace: 0312 20:10:11.718750 (+ 0us) service_pool.cc:163] Inserting onto call queue 0312 20:10:11.718843 (+ 93us) service_pool.cc:222] Handling call 0312 20:10:11.741946 (+ 23103us) spinlock_profiling.cc:261] Waited 21.7 ms on lock 0x7b480003009c. stack: 00007efc5efb2c74 00007efc5efb2b70 00007efc5daa2314 00007efc66aef294 00007efc66aef229 00007efc65699606 00007efc66b2d56b 00007efc614e0b24 00007efc614e0ab1 00007efc614e09c4 00007efc60b3cc29 00007efc60b3c6ec 00007efc60b3dbcd 00007efc60b3f946 00007efc60b3f89a 00007efc60b3f823 0312 20:10:11.742031 (+ 85us) inbound_call.cc:157] Queueing success response Metrics: {"spinlock_wait_cycles":49819392} I0312 20:10:11.743206 1346 raft_consensus.cc:2215] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6885de38238442d6a9e5f1eb9fb569ff in term 8. I0312 20:10:11.745245 1152 leader_election.cc:363] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 9 pre-election: Vote granted by peer f78a2351568c4a268ee1112681e22bec I0312 20:10:11.745960 1152 leader_election.cc:258] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. I0312 20:10:11.746611 1454 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff: Snoozing failure detection for 0.021s (election complete) I0312 20:10:11.746834 1454 raft_consensus.cc:2507] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 8 FOLLOWER]: Leader pre-election won for term 9 I0312 20:10:11.747023 1454 raft_consensus.cc:436] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 8 FOLLOWER]: Starting leader election (detected failure of leader ded9b7f8677940fb91e1a8c8bceceaed) I0312 20:10:11.747234 1454 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff: Snoozing failure detection for 0.017s (starting election) I0312 20:10:11.747448 1454 raft_consensus.cc:2752] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 8 FOLLOWER]: Advancing to term 9 I0312 20:10:11.753274 1454 raft_consensus.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:11.755188 1454 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 9 election: Requesting vote from peer f78a2351568c4a268ee1112681e22bec I0312 20:10:11.758021 1219 raft_consensus.cc:1026] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 9 FOLLOWER]: Rejecting Update request from peer ded9b7f8677940fb91e1a8c8bceceaed for earlier term 8. Current term is 9. Ops: [] I0312 20:10:11.763556 1456 consensus_queue.cc:1028] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 }, Status: INVALID_TERM, Last received: 8.69, Next index: 70, Last known committed idx: 69, Time since last communication: 0.000s I0312 20:10:11.764176 1417 raft_consensus.cc:2747] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 8 LEADER]: Stepping down as leader of term 8 I0312 20:10:11.764392 1417 raft_consensus.cc:605] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 8 LEADER]: Becoming Follower/Learner. State: Replica: ded9b7f8677940fb91e1a8c8bceceaed, State: Running, Role: LEADER I0312 20:10:11.764894 1417 consensus_queue.cc:219] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 70, Committed index: 70, Last appended: 8.72, Last appended by leader: 72, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:11.765538 1417 consensus_peers.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed -> Peer 6885de38238442d6a9e5f1eb9fb569ff (127.0.53.2:51697): Closing peer: 6885de38238442d6a9e5f1eb9fb569ff I0312 20:10:11.765727 1417 consensus_peers.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed -> Peer f78a2351568c4a268ee1112681e22bec (127.0.53.3:43274): Closing peer: f78a2351568c4a268ee1112681e22bec I0312 20:10:11.765741 1424 raft_consensus.cc:436] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 8 FOLLOWER]: Starting pre-election (detected failure of leader ded9b7f8677940fb91e1a8c8bceceaed) I0312 20:10:11.765913 1417 raft_consensus.cc:2752] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 8 FOLLOWER]: Advancing to term 9 I0312 20:10:11.765972 1424 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec: Snoozing failure detection for 0.024s (starting election) I0312 20:10:11.766191 1424 raft_consensus.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:11.767124 1424 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 9 pre-election: Requesting pre-vote from peer 6885de38238442d6a9e5f1eb9fb569ff I0312 20:10:11.767196 1416 raft_consensus.cc:436] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0312 20:10:11.767391 1416 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff: Snoozing failure detection for 0.020s (starting election) I0312 20:10:11.767484 1424 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 9 pre-election: Requesting pre-vote from peer ded9b7f8677940fb91e1a8c8bceceaed I0312 20:10:11.767611 1416 raft_consensus.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:11.768555 1416 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 10 pre-election: Requesting pre-vote from peer f78a2351568c4a268ee1112681e22bec I0312 20:10:11.768893 1416 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 10 pre-election: Requesting pre-vote from peer ded9b7f8677940fb91e1a8c8bceceaed I0312 20:10:11.770184 1096 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "6885de38238442d6a9e5f1eb9fb569ff" candidate_term: 10 candidate_status { last_received { term: 8 index: 69 } } ignore_live_leader: false dest_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" is_pre_election: true I0312 20:10:11.770793 1096 raft_consensus.cc:2162] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6885de38238442d6a9e5f1eb9fb569ff for term 10 because replica has last-logged OpId of term: 8 index: 72, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 69. I0312 20:10:11.770777 1454 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 9 election: Requesting vote from peer ded9b7f8677940fb91e1a8c8bceceaed I0312 20:10:11.771317 1095 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "f78a2351568c4a268ee1112681e22bec" candidate_term: 9 candidate_status { last_received { term: 8 index: 70 } } ignore_live_leader: false dest_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" is_pre_election: true I0312 20:10:11.771906 1095 raft_consensus.cc:2162] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 9 because replica has last-logged OpId of term: 8 index: 72, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 70. I0312 20:10:11.773330 1219 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "f78a2351568c4a268ee1112681e22bec" candidate_term: 9 candidate_status { last_received { term: 8 index: 70 } } ignore_live_leader: false dest_uuid: "6885de38238442d6a9e5f1eb9fb569ff" is_pre_election: true I0312 20:10:11.773725 1346 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "6885de38238442d6a9e5f1eb9fb569ff" candidate_term: 9 candidate_status { last_received { term: 8 index: 69 } } ignore_live_leader: false dest_uuid: "f78a2351568c4a268ee1112681e22bec" I0312 20:10:11.774029 1219 raft_consensus.cc:2145] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec in current term 9: Already voted for candidate 6885de38238442d6a9e5f1eb9fb569ff in this term. I0312 20:10:11.773939 1095 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "6885de38238442d6a9e5f1eb9fb569ff" candidate_term: 9 candidate_status { last_received { term: 8 index: 69 } } ignore_live_leader: false dest_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" I0312 20:10:11.774307 1346 raft_consensus.cc:2176] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 6885de38238442d6a9e5f1eb9fb569ff for term 9 because replica is either leader or believes a valid leader to be alive. I0312 20:10:11.774258 1153 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 10 pre-election: Vote denied by peer ded9b7f8677940fb91e1a8c8bceceaed. Message: Invalid argument: T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6885de38238442d6a9e5f1eb9fb569ff for term 10 because replica has last-logged OpId of term: 8 index: 72, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 69. I0312 20:10:11.774477 1095 raft_consensus.cc:2162] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 6885de38238442d6a9e5f1eb9fb569ff for term 9 because replica has last-logged OpId of term: 8 index: 72, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 69. I0312 20:10:11.775094 1279 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 9 pre-election: Vote denied by peer 6885de38238442d6a9e5f1eb9fb569ff. Message: Invalid argument: T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec in current term 9: Already voted for candidate 6885de38238442d6a9e5f1eb9fb569ff in this term. I0312 20:10:11.775279 1153 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 9 election: Vote denied by peer ded9b7f8677940fb91e1a8c8bceceaed. Message: Invalid argument: T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 6885de38238442d6a9e5f1eb9fb569ff for term 9 because replica has last-logged OpId of term: 8 index: 72, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 69. I0312 20:10:11.775825 1152 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 9 election: Vote denied by peer f78a2351568c4a268ee1112681e22bec. Message: Invalid argument: T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 6885de38238442d6a9e5f1eb9fb569ff for term 9 because replica is either leader or believes a valid leader to be alive. I0312 20:10:11.776067 1152 leader_election.cc:258] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. I0312 20:10:11.776521 1454 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff: Snoozing failure detection for 0.017s (election complete) I0312 20:10:11.776751 1454 raft_consensus.cc:2455] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority I0312 20:10:11.777274 1346 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "6885de38238442d6a9e5f1eb9fb569ff" candidate_term: 10 candidate_status { last_received { term: 8 index: 69 } } ignore_live_leader: false dest_uuid: "f78a2351568c4a268ee1112681e22bec" is_pre_election: true I0312 20:10:11.777621 1456 raft_consensus.cc:436] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0312 20:10:11.777863 1456 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed: Snoozing failure detection for 0.024s (starting election) I0312 20:10:11.777925 1346 raft_consensus.cc:2176] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6885de38238442d6a9e5f1eb9fb569ff for term 10 because replica is either leader or believes a valid leader to be alive. I0312 20:10:11.778108 1456 raft_consensus.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:11.778887 1152 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 10 pre-election: Vote denied by peer f78a2351568c4a268ee1112681e22bec. Message: Invalid argument: T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6885de38238442d6a9e5f1eb9fb569ff for term 10 because replica is either leader or believes a valid leader to be alive. I0312 20:10:11.779120 1456 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 10 pre-election: Requesting pre-vote from peer f78a2351568c4a268ee1112681e22bec I0312 20:10:11.779253 1152 leader_election.cc:258] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. I0312 20:10:11.779484 1456 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 10 pre-election: Requesting pre-vote from peer 6885de38238442d6a9e5f1eb9fb569ff I0312 20:10:11.780195 1454 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff: Snoozing failure detection for 0.023s (election complete) I0312 20:10:11.781164 1454 raft_consensus.cc:2455] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority I0312 20:10:11.780740 1219 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" candidate_term: 10 candidate_status { last_received { term: 8 index: 72 } } ignore_live_leader: false dest_uuid: "6885de38238442d6a9e5f1eb9fb569ff" is_pre_election: true I0312 20:10:11.781446 1278 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 9 pre-election: Vote denied by peer ded9b7f8677940fb91e1a8c8bceceaed. Message: Invalid argument: T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 9 because replica has last-logged OpId of term: 8 index: 72, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 70. I0312 20:10:11.781576 1219 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff: Snoozing failure detection for 0.037s (vote granted) I0312 20:10:11.781667 1278 leader_election.cc:258] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. I0312 20:10:11.781775 1219 raft_consensus.cc:2215] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ded9b7f8677940fb91e1a8c8bceceaed in term 9. I0312 20:10:11.782243 1424 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec: Snoozing failure detection for 0.032s (election complete) I0312 20:10:11.782450 1424 raft_consensus.cc:2752] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 8 FOLLOWER]: Advancing to term 9 I0312 20:10:11.786527 1424 raft_consensus.cc:2455] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority I0312 20:10:11.787303 1029 leader_election.cc:363] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 10 pre-election: Vote granted by peer 6885de38238442d6a9e5f1eb9fb569ff I0312 20:10:11.787547 1029 leader_election.cc:258] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. I0312 20:10:11.788069 1456 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed: Snoozing failure detection for 0.017s (election complete) I0312 20:10:11.788317 1456 raft_consensus.cc:2507] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 9 FOLLOWER]: Leader pre-election won for term 10 I0312 20:10:11.788542 1456 raft_consensus.cc:436] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0312 20:10:11.789918 1346 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" candidate_term: 10 candidate_status { last_received { term: 8 index: 72 } } ignore_live_leader: false dest_uuid: "f78a2351568c4a268ee1112681e22bec" is_pre_election: true I0312 20:10:11.790359 1346 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec: Snoozing failure detection for 0.021s (vote granted) I0312 20:10:11.790575 1346 raft_consensus.cc:2215] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ded9b7f8677940fb91e1a8c8bceceaed in term 9. I0312 20:10:11.791354 1456 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed: Snoozing failure detection for 0.021s (starting election) I0312 20:10:11.791592 1456 raft_consensus.cc:2752] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 9 FOLLOWER]: Advancing to term 10 I0312 20:10:11.793020 1027 leader_election.cc:363] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 10 pre-election: Vote granted by peer f78a2351568c4a268ee1112681e22bec I0312 20:10:11.796913 1456 raft_consensus.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:11.797968 1456 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 10 election: Requesting vote from peer f78a2351568c4a268ee1112681e22bec I0312 20:10:11.799310 1346 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" candidate_term: 10 candidate_status { last_received { term: 8 index: 72 } } ignore_live_leader: false dest_uuid: "f78a2351568c4a268ee1112681e22bec" I0312 20:10:11.799711 1346 raft_consensus.cc:2752] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 9 FOLLOWER]: Advancing to term 10 I0312 20:10:11.799927 1346 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec: Snoozing failure detection for 0.033s (vote granted) I0312 20:10:11.803889 1346 raft_consensus.cc:2215] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ded9b7f8677940fb91e1a8c8bceceaed in term 10. I0312 20:10:11.805044 1027 leader_election.cc:363] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 10 election: Vote granted by peer f78a2351568c4a268ee1112681e22bec I0312 20:10:11.805290 1027 leader_election.cc:258] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. I0312 20:10:11.805856 1417 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed: Snoozing failure detection for 0.018s (election complete) I0312 20:10:11.806071 1417 raft_consensus.cc:2507] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 10 FOLLOWER]: Leader election won for term 10 I0312 20:10:11.806339 1417 raft_consensus.cc:572] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 10 LEADER]: Becoming Leader. State: Replica: ded9b7f8677940fb91e1a8c8bceceaed, State: Running, Role: LEADER I0312 20:10:11.807227 1417 consensus_queue.cc:196] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 70, Committed index: 70, Last appended: 8.72, Last appended by leader: 72, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:11.810703 1456 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 10 election: Requesting vote from peer 6885de38238442d6a9e5f1eb9fb569ff I0312 20:10:11.812172 1219 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" candidate_term: 10 candidate_status { last_received { term: 8 index: 72 } } ignore_live_leader: false dest_uuid: "6885de38238442d6a9e5f1eb9fb569ff" I0312 20:10:11.811650 974 catalog_manager.cc:3592] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed reported cstate change: term changed from 8 to 10. New cstate: current_term: 10 leader_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } health_report { overall_health: HEALTHY } } } I0312 20:10:11.812585 1219 raft_consensus.cc:2752] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 9 FOLLOWER]: Advancing to term 10 I0312 20:10:11.812809 1219 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff: Snoozing failure detection for 0.019s (vote granted) I0312 20:10:11.816895 1219 raft_consensus.cc:2215] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ded9b7f8677940fb91e1a8c8bceceaed in term 10. I0312 20:10:11.819504 1346 raft_consensus.cc:1058] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 10 FOLLOWER]: Refusing update from remote peer ded9b7f8677940fb91e1a8c8bceceaed: Log matching property violated. Preceding OpId in replica: term: 8 index: 71. Preceding OpId from leader: term: 10 index: 73. (index mismatch) I0312 20:10:11.819936 1220 raft_consensus.cc:1058] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 10 FOLLOWER]: Refusing update from remote peer ded9b7f8677940fb91e1a8c8bceceaed: Log matching property violated. Preceding OpId in replica: term: 8 index: 69. Preceding OpId from leader: term: 10 index: 73. (index mismatch) I0312 20:10:11.820662 1417 consensus_queue.cc:1017] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [LEADER]: Connected to new peer: Peer: permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 }, Status: LMP_MISMATCH, Last received: 8.71, Next index: 72, Last known committed idx: 70, Time since last communication: 0.000s I0312 20:10:11.821413 1455 consensus_queue.cc:1017] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [LEADER]: Connected to new peer: Peer: permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 }, Status: LMP_MISMATCH, Last received: 8.69, Next index: 70, Last known committed idx: 69, Time since last communication: 0.000s I0312 20:10:11.822978 1029 leader_election.cc:363] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 10 election: Vote granted by peer 6885de38238442d6a9e5f1eb9fb569ff I0312 20:10:11.839524 1424 raft_consensus.cc:436] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 10 FOLLOWER]: Starting pre-election (detected failure of leader ded9b7f8677940fb91e1a8c8bceceaed) I0312 20:10:11.839754 1424 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec: Snoozing failure detection for 0.015s (starting election) I0312 20:10:11.840015 1454 raft_consensus.cc:436] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 10 FOLLOWER]: Starting pre-election (detected failure of leader ded9b7f8677940fb91e1a8c8bceceaed) I0312 20:10:11.840234 1454 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff: Snoozing failure detection for 0.016s (starting election) I0312 20:10:11.839984 1424 raft_consensus.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:11.840456 1454 raft_consensus.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:11.840950 1424 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 11 pre-election: Requesting pre-vote from peer 6885de38238442d6a9e5f1eb9fb569ff I0312 20:10:11.841415 1454 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 11 pre-election: Requesting pre-vote from peer f78a2351568c4a268ee1112681e22bec I0312 20:10:11.841420 1424 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 11 pre-election: Requesting pre-vote from peer ded9b7f8677940fb91e1a8c8bceceaed I0312 20:10:11.841768 1454 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 11 pre-election: Requesting pre-vote from peer ded9b7f8677940fb91e1a8c8bceceaed I0312 20:10:11.842960 1221 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "f78a2351568c4a268ee1112681e22bec" candidate_term: 11 candidate_status { last_received { term: 10 index: 73 } } ignore_live_leader: false dest_uuid: "6885de38238442d6a9e5f1eb9fb569ff" is_pre_election: true I0312 20:10:11.843099 1095 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "6885de38238442d6a9e5f1eb9fb569ff" candidate_term: 11 candidate_status { last_received { term: 10 index: 73 } } ignore_live_leader: false dest_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" is_pre_election: true I0312 20:10:11.843447 1221 raft_consensus.cc:2190] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 11 because replica is already servicing an update from a current leader or another vote. I0312 20:10:11.843644 1095 raft_consensus.cc:2176] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 10 LEADER]: Leader pre-election vote request: Denying vote to candidate 6885de38238442d6a9e5f1eb9fb569ff for term 11 because replica is either leader or believes a valid leader to be alive. I0312 20:10:11.844646 1279 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 11 pre-election: Vote denied by peer 6885de38238442d6a9e5f1eb9fb569ff. Message: Service unavailable: T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 11 because replica is already servicing an update from a current leader or another vote. I0312 20:10:11.844753 1153 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 11 pre-election: Vote denied by peer ded9b7f8677940fb91e1a8c8bceceaed. Message: Invalid argument: T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 10 LEADER]: Leader pre-election vote request: Denying vote to candidate 6885de38238442d6a9e5f1eb9fb569ff for term 11 because replica is either leader or believes a valid leader to be alive. I0312 20:10:11.845825 1345 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "6885de38238442d6a9e5f1eb9fb569ff" candidate_term: 11 candidate_status { last_received { term: 10 index: 73 } } ignore_live_leader: false dest_uuid: "f78a2351568c4a268ee1112681e22bec" is_pre_election: true I0312 20:10:11.846267 1345 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec: Snoozing failure detection for 0.015s (vote granted) I0312 20:10:11.847342 1345 raft_consensus.cc:2215] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6885de38238442d6a9e5f1eb9fb569ff in term 10. I0312 20:10:11.852174 1152 leader_election.cc:363] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 11 pre-election: Vote granted by peer f78a2351568c4a268ee1112681e22bec I0312 20:10:11.852437 1152 leader_election.cc:258] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. I0312 20:10:11.853054 1454 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff: Snoozing failure detection for 0.016s (election complete) I0312 20:10:11.852891 1095 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "f78a2351568c4a268ee1112681e22bec" candidate_term: 11 candidate_status { last_received { term: 10 index: 73 } } ignore_live_leader: false dest_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" is_pre_election: true I0312 20:10:11.853276 1454 raft_consensus.cc:2507] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 10 FOLLOWER]: Leader pre-election won for term 11 I0312 20:10:11.853386 1095 raft_consensus.cc:2176] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 10 LEADER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 11 because replica is either leader or believes a valid leader to be alive. I0312 20:10:11.853452 1454 raft_consensus.cc:436] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 10 FOLLOWER]: Starting leader election (detected failure of leader ded9b7f8677940fb91e1a8c8bceceaed) I0312 20:10:11.853629 1454 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff: Snoozing failure detection for 0.017s (starting election) I0312 20:10:11.853794 1454 raft_consensus.cc:2752] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 10 FOLLOWER]: Advancing to term 11 I0312 20:10:11.854501 1278 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 11 pre-election: Vote denied by peer ded9b7f8677940fb91e1a8c8bceceaed. Message: Invalid argument: T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 10 LEADER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 11 because replica is either leader or believes a valid leader to be alive. I0312 20:10:11.854717 1278 leader_election.cc:258] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. I0312 20:10:11.855439 1424 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec: Snoozing failure detection for 0.022s (election complete) I0312 20:10:11.855636 1424 raft_consensus.cc:2455] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority I0312 20:10:11.859215 1454 raft_consensus.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:11.860590 1454 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 11 election: Requesting vote from peer f78a2351568c4a268ee1112681e22bec I0312 20:10:11.862694 1219 raft_consensus.cc:1026] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 11 FOLLOWER]: Rejecting Update request from peer ded9b7f8677940fb91e1a8c8bceceaed for earlier term 10. Current term is 11. Ops: [] I0312 20:10:11.863838 1455 consensus_queue.cc:1028] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 }, Status: INVALID_TERM, Last received: 10.73, Next index: 74, Last known committed idx: 70, Time since last communication: 0.000s I0312 20:10:11.864337 1455 raft_consensus.cc:2747] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 10 LEADER]: Stepping down as leader of term 10 I0312 20:10:11.864534 1455 raft_consensus.cc:605] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 10 LEADER]: Becoming Follower/Learner. State: Replica: ded9b7f8677940fb91e1a8c8bceceaed, State: Running, Role: LEADER I0312 20:10:11.864956 1455 consensus_queue.cc:219] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 73, Committed index: 73, Last appended: 10.74, Last appended by leader: 74, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:11.865928 1455 consensus_peers.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed -> Peer 6885de38238442d6a9e5f1eb9fb569ff (127.0.53.2:51697): Closing peer: 6885de38238442d6a9e5f1eb9fb569ff I0312 20:10:11.866621 1346 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "6885de38238442d6a9e5f1eb9fb569ff" candidate_term: 11 candidate_status { last_received { term: 10 index: 73 } } ignore_live_leader: false dest_uuid: "f78a2351568c4a268ee1112681e22bec" I0312 20:10:11.867175 1346 raft_consensus.cc:2752] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 10 FOLLOWER]: Advancing to term 11 I0312 20:10:11.867548 1346 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec: Snoozing failure detection for 0.023s (vote granted) I0312 20:10:11.871621 1346 raft_consensus.cc:2215] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6885de38238442d6a9e5f1eb9fb569ff in term 11. I0312 20:10:11.871989 1416 raft_consensus.cc:436] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0312 20:10:11.872182 1416 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff: Snoozing failure detection for 0.015s (starting election) I0312 20:10:11.872390 1416 raft_consensus.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:11.873348 1416 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 12 pre-election: Requesting pre-vote from peer f78a2351568c4a268ee1112681e22bec I0312 20:10:11.873713 1416 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 12 pre-election: Requesting pre-vote from peer ded9b7f8677940fb91e1a8c8bceceaed I0312 20:10:11.878535 1095 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "6885de38238442d6a9e5f1eb9fb569ff" candidate_term: 12 candidate_status { last_received { term: 10 index: 73 } } ignore_live_leader: false dest_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" is_pre_election: true I0312 20:10:11.879159 1455 consensus_peers.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed -> Peer f78a2351568c4a268ee1112681e22bec (127.0.53.3:43274): Closing peer: f78a2351568c4a268ee1112681e22bec I0312 20:10:11.879489 1455 raft_consensus.cc:2752] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 10 FOLLOWER]: Advancing to term 11 I0312 20:10:11.883604 1418 raft_consensus.cc:436] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0312 20:10:11.883795 1418 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed: Snoozing failure detection for 0.022s (starting election) I0312 20:10:11.883997 1418 raft_consensus.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:11.885049 1418 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 12 pre-election: Requesting pre-vote from peer f78a2351568c4a268ee1112681e22bec I0312 20:10:11.885408 1418 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 12 pre-election: Requesting pre-vote from peer 6885de38238442d6a9e5f1eb9fb569ff I0312 20:10:11.886075 1095 raft_consensus.cc:2162] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6885de38238442d6a9e5f1eb9fb569ff for term 12 because replica has last-logged OpId of term: 10 index: 74, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 73. I0312 20:10:11.891135 1153 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 12 pre-election: Vote denied by peer ded9b7f8677940fb91e1a8c8bceceaed. Message: Invalid argument: T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6885de38238442d6a9e5f1eb9fb569ff for term 12 because replica has last-logged OpId of term: 10 index: 74, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 73. W0312 20:10:11.891834 1152 leader_election.cc:287] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 11 election: RPC error from VoteRequest() call to peer f78a2351568c4a268ee1112681e22bec: Timed out: RequestConsensusVote RPC to 127.0.53.3:43274 timed out after 0.017s (SENT) I0312 20:10:11.896384 1219 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" candidate_term: 12 candidate_status { last_received { term: 10 index: 74 } } ignore_live_leader: false dest_uuid: "6885de38238442d6a9e5f1eb9fb569ff" is_pre_election: true I0312 20:10:11.896847 1219 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff: Snoozing failure detection for 0.018s (vote granted) I0312 20:10:11.897079 1219 raft_consensus.cc:2215] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ded9b7f8677940fb91e1a8c8bceceaed in term 11. I0312 20:10:11.897861 1029 leader_election.cc:363] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 12 pre-election: Vote granted by peer 6885de38238442d6a9e5f1eb9fb569ff I0312 20:10:11.898053 1029 leader_election.cc:258] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. I0312 20:10:11.898541 1417 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed: Snoozing failure detection for 0.015s (election complete) I0312 20:10:11.898901 1417 raft_consensus.cc:2507] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 11 FOLLOWER]: Leader pre-election won for term 12 I0312 20:10:11.899111 1417 raft_consensus.cc:436] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0312 20:10:11.899305 1417 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed: Snoozing failure detection for 0.021s (starting election) I0312 20:10:11.899513 1417 raft_consensus.cc:2752] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 11 FOLLOWER]: Advancing to term 12 I0312 20:10:11.902078 1454 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 11 election: Requesting vote from peer ded9b7f8677940fb91e1a8c8bceceaed I0312 20:10:11.903414 1095 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "6885de38238442d6a9e5f1eb9fb569ff" candidate_term: 11 candidate_status { last_received { term: 10 index: 73 } } ignore_live_leader: false dest_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" I0312 20:10:11.890130 1344 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" candidate_term: 12 candidate_status { last_received { term: 10 index: 74 } } ignore_live_leader: false dest_uuid: "f78a2351568c4a268ee1112681e22bec" is_pre_election: true I0312 20:10:11.895475 1345 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "6885de38238442d6a9e5f1eb9fb569ff" candidate_term: 12 candidate_status { last_received { term: 10 index: 73 } } ignore_live_leader: false dest_uuid: "f78a2351568c4a268ee1112681e22bec" is_pre_election: true I0312 20:10:11.906103 1424 raft_consensus.cc:436] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0312 20:10:11.906322 1424 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec: Snoozing failure detection for 0.027s (starting election) W0312 20:10:11.906210 1346 rpcz_store.cc:251] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.53.2:37068 (request call id 24) took 44ms (client timeout 17). I0312 20:10:11.906580 1424 raft_consensus.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } W0312 20:10:11.907284 1152 leader_election.cc:287] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer f78a2351568c4a268ee1112681e22bec: Timed out: RequestConsensusVote RPC to 127.0.53.3:43274 timed out after 0.015s (SENT) I0312 20:10:11.907500 1152 leader_election.cc:258] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. I0312 20:10:11.908077 1454 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff: Snoozing failure detection for 0.018s (election complete) I0312 20:10:11.908267 1454 raft_consensus.cc:2455] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority I0312 20:10:11.908320 1344 raft_consensus.cc:2190] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ded9b7f8677940fb91e1a8c8bceceaed for term 12 because replica is already servicing an update from a current leader or another vote. I0312 20:10:11.908673 1345 raft_consensus.cc:2190] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6885de38238442d6a9e5f1eb9fb569ff for term 12 because replica is already servicing an update from a current leader or another vote. I0312 20:10:11.908953 1417 raft_consensus.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:11.910033 1095 raft_consensus.cc:2120] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 6885de38238442d6a9e5f1eb9fb569ff for earlier term 11. Current term is 12. W0312 20:10:11.911186 1153 leader_election.cc:347] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 11 election: Vote denied by peer ded9b7f8677940fb91e1a8c8bceceaed with higher term. Message: Invalid argument: T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 6885de38238442d6a9e5f1eb9fb569ff for earlier term 11. Current term is 12. I0312 20:10:11.911381 1153 leader_election.cc:350] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 11 election: Cancelling election due to peer responding with higher term W0312 20:10:11.912051 1027 leader_election.cc:287] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer f78a2351568c4a268ee1112681e22bec: Timed out: RequestConsensusVote RPC to 127.0.53.3:43274 timed out after 0.022s (SENT) I0312 20:10:11.912508 1454 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff: Snoozing failure detection for 0.029s (election complete) I0312 20:10:11.913408 1454 raft_consensus.cc:2752] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 11 FOLLOWER]: Advancing to term 12 W0312 20:10:11.915469 1346 rpcz_store.cc:255] Trace: 0312 20:10:11.861845 (+ 0us) service_pool.cc:163] Inserting onto call queue 0312 20:10:11.866483 (+ 4638us) service_pool.cc:222] Handling call 0312 20:10:11.906148 (+ 39665us) inbound_call.cc:157] Queueing success response Metrics: {} I0312 20:10:11.915565 1424 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 12 pre-election: Requesting pre-vote from peer 6885de38238442d6a9e5f1eb9fb569ff I0312 20:10:11.915971 1424 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 12 pre-election: Requesting pre-vote from peer ded9b7f8677940fb91e1a8c8bceceaed I0312 20:10:11.917145 1417 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 12 election: Requesting vote from peer f78a2351568c4a268ee1112681e22bec I0312 20:10:11.917520 1417 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 12 election: Requesting vote from peer 6885de38238442d6a9e5f1eb9fb569ff I0312 20:10:11.917512 1095 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "f78a2351568c4a268ee1112681e22bec" candidate_term: 12 candidate_status { last_received { term: 10 index: 73 } } ignore_live_leader: false dest_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" is_pre_election: true I0312 20:10:11.918045 1095 raft_consensus.cc:2145] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec in current term 12: Already voted for candidate ded9b7f8677940fb91e1a8c8bceceaed in this term. I0312 20:10:11.918836 1219 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" candidate_term: 12 candidate_status { last_received { term: 10 index: 74 } } ignore_live_leader: false dest_uuid: "6885de38238442d6a9e5f1eb9fb569ff" I0312 20:10:11.919528 1278 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 12 pre-election: Vote denied by peer ded9b7f8677940fb91e1a8c8bceceaed. Message: Invalid argument: T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec in current term 12: Already voted for candidate ded9b7f8677940fb91e1a8c8bceceaed in this term. I0312 20:10:11.920316 1220 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "f78a2351568c4a268ee1112681e22bec" candidate_term: 12 candidate_status { last_received { term: 10 index: 73 } } ignore_live_leader: false dest_uuid: "6885de38238442d6a9e5f1eb9fb569ff" is_pre_election: true I0312 20:10:11.920835 1454 raft_consensus.cc:2455] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 12 FOLLOWER]: Leader election lost for term 11. Reason: Vote denied by peer ded9b7f8677940fb91e1a8c8bceceaed with higher term. Message: Invalid argument: T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 6885de38238442d6a9e5f1eb9fb569ff for earlier term 11. Current term is 12. I0312 20:10:11.921113 1219 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff: Snoozing failure detection for 0.044s (vote granted) W0312 20:10:11.923758 1344 rpcz_store.cc:251] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.53.1:47760 (request call id 45) took 33ms (client timeout 22). W0312 20:10:11.924441 1344 rpcz_store.cc:255] Trace: 0312 20:10:11.889882 (+ 0us) service_pool.cc:163] Inserting onto call queue 0312 20:10:11.889984 (+ 102us) service_pool.cc:222] Handling call 0312 20:10:11.923620 (+ 33636us) spinlock_profiling.cc:261] Waited 2.28 ms on lock 0x7b480003009c. stack: 00007f173be47c74 00007f173be47b70 00007f173a937314 00007f1743984294 00007f1743984229 00007f174252e606 00007f17439c256b 00007f173e375b24 00007f173e375ab1 00007f173e3759c4 00007f173d9d1c29 00007f173d9d16ec 00007f173d9d2bcd 00007f173d9d4946 00007f173d9d489a 00007f173d9d4823 0312 20:10:11.923710 (+ 90us) inbound_call.cc:157] Queueing success response Metrics: {"spinlock_wait_cycles":5252352} I0312 20:10:11.925401 1219 raft_consensus.cc:2215] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ded9b7f8677940fb91e1a8c8bceceaed in term 12. W0312 20:10:11.925614 1345 rpcz_store.cc:251] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.53.2:37068 (request call id 25) took 30ms (client timeout 15). W0312 20:10:11.926280 1345 rpcz_store.cc:255] Trace: 0312 20:10:11.895225 (+ 0us) service_pool.cc:163] Inserting onto call queue 0312 20:10:11.895329 (+ 104us) service_pool.cc:222] Handling call 0312 20:10:11.925472 (+ 30143us) spinlock_profiling.cc:261] Waited 2.61 ms on lock 0x7b480003009c. stack: 00007f173be47c74 00007f173be47b70 00007f173a937314 00007f1743984294 00007f1743984229 00007f174252e606 00007f17439c256b 00007f173e375b24 00007f173e375ab1 00007f173e3759c4 00007f173d9d1c29 00007f173d9d16ec 00007f173d9d2bcd 00007f173d9d4946 00007f173d9d489a 00007f173d9d4823 0312 20:10:11.925567 (+ 95us) inbound_call.cc:157] Queueing success response Metrics: {"spinlock_wait_cycles":6002560} I0312 20:10:11.926380 1029 leader_election.cc:363] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 12 election: Vote granted by peer 6885de38238442d6a9e5f1eb9fb569ff I0312 20:10:11.926568 1029 leader_election.cc:258] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. I0312 20:10:11.927016 1455 raft_consensus.cc:436] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0312 20:10:11.927070 1220 raft_consensus.cc:2190] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 12 because replica is already servicing an update from a current leader or another vote. I0312 20:10:11.927213 1455 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed: Snoozing failure detection for 0.018s (starting election) I0312 20:10:11.928323 1279 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 12 pre-election: Vote denied by peer 6885de38238442d6a9e5f1eb9fb569ff. Message: Service unavailable: T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 12 because replica is already servicing an update from a current leader or another vote. I0312 20:10:11.928550 1279 leader_election.cc:258] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. I0312 20:10:11.928337 1455 raft_consensus.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:11.929026 1415 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec: Snoozing failure detection for 0.020s (election complete) I0312 20:10:11.929266 1415 raft_consensus.cc:2752] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 11 FOLLOWER]: Advancing to term 12 I0312 20:10:11.929770 1455 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 13 pre-election: Requesting pre-vote from peer f78a2351568c4a268ee1112681e22bec I0312 20:10:11.930443 1455 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 13 pre-election: Requesting pre-vote from peer 6885de38238442d6a9e5f1eb9fb569ff I0312 20:10:11.931849 1220 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" candidate_term: 13 candidate_status { last_received { term: 10 index: 74 } } ignore_live_leader: false dest_uuid: "6885de38238442d6a9e5f1eb9fb569ff" is_pre_election: true I0312 20:10:11.932235 1220 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff: Snoozing failure detection for 0.047s (vote granted) I0312 20:10:11.932201 1345 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" candidate_term: 13 candidate_status { last_received { term: 10 index: 74 } } ignore_live_leader: false dest_uuid: "f78a2351568c4a268ee1112681e22bec" is_pre_election: true I0312 20:10:11.932433 1220 raft_consensus.cc:2215] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ded9b7f8677940fb91e1a8c8bceceaed in term 12. I0312 20:10:11.932735 1456 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed: Snoozing failure detection for 0.018s (election complete) I0312 20:10:11.933336 1456 raft_consensus.cc:2507] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 12 FOLLOWER]: Leader election won for term 12 I0312 20:10:11.933652 1456 raft_consensus.cc:572] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 12 LEADER]: Becoming Leader. State: Replica: ded9b7f8677940fb91e1a8c8bceceaed, State: Running, Role: LEADER I0312 20:10:11.936776 1029 leader_election.cc:363] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 13 pre-election: Vote granted by peer 6885de38238442d6a9e5f1eb9fb569ff I0312 20:10:11.937412 1029 leader_election.cc:258] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. I0312 20:10:11.938109 1456 consensus_queue.cc:196] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 73, Committed index: 73, Last appended: 10.74, Last appended by leader: 74, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:11.939117 1415 raft_consensus.cc:2455] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority I0312 20:10:11.939477 1345 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec: Snoozing failure detection for 0.033s (vote granted) I0312 20:10:11.939693 1345 raft_consensus.cc:2215] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ded9b7f8677940fb91e1a8c8bceceaed in term 12. W0312 20:10:11.939605 1027 leader_election.cc:287] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 12 election: RPC error from VoteRequest() call to peer f78a2351568c4a268ee1112681e22bec: Timed out: RequestConsensusVote RPC to 127.0.53.3:43274 timed out after 0.021s (SENT) I0312 20:10:11.940906 1027 leader_election.cc:363] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 13 pre-election: Vote granted by peer f78a2351568c4a268ee1112681e22bec I0312 20:10:11.944988 974 catalog_manager.cc:3592] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed reported cstate change: term changed from 10 to 12. New cstate: current_term: 12 leader_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } health_report { overall_health: HEALTHY } } } I0312 20:10:11.950341 1220 raft_consensus.cc:1058] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 12 FOLLOWER]: Refusing update from remote peer ded9b7f8677940fb91e1a8c8bceceaed: Log matching property violated. Preceding OpId in replica: term: 10 index: 73. Preceding OpId from leader: term: 12 index: 75. (index mismatch) I0312 20:10:11.951772 1418 consensus_queue.cc:1017] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [LEADER]: Connected to new peer: Peer: permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 }, Status: LMP_MISMATCH, Last received: 10.73, Next index: 74, Last known committed idx: 70, Time since last communication: 0.000s I0312 20:10:11.953743 1417 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed: Snoozing failure detection for 0.020s (election complete) I0312 20:10:11.953846 1345 raft_consensus.cc:1058] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 12 FOLLOWER]: Refusing update from remote peer ded9b7f8677940fb91e1a8c8bceceaed: Log matching property violated. Preceding OpId in replica: term: 10 index: 73. Preceding OpId from leader: term: 12 index: 75. (index mismatch) W0312 20:10:11.954134 1346 rpcz_store.cc:251] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.53.1:47760 (request call id 46) took 35ms (client timeout 21). W0312 20:10:11.954964 1346 rpcz_store.cc:255] Trace: 0312 20:10:11.919178 (+ 0us) service_pool.cc:163] Inserting onto call queue 0312 20:10:11.953997 (+ 34819us) service_pool.cc:207] Skipping call since client already timed out 0312 20:10:11.954086 (+ 89us) inbound_call.cc:157] Queueing failure response Metrics: {} I0312 20:10:11.955062 1417 consensus_queue.cc:1017] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [LEADER]: Connected to new peer: Peer: permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 }, Status: LMP_MISMATCH, Last received: 10.73, Next index: 74, Last known committed idx: 73, Time since last communication: 0.000s I0312 20:10:11.971518 1416 raft_consensus.cc:436] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 12 FOLLOWER]: Starting pre-election (detected failure of leader ded9b7f8677940fb91e1a8c8bceceaed) I0312 20:10:11.971750 1416 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff: Snoozing failure detection for 0.017s (starting election) I0312 20:10:11.971956 1416 raft_consensus.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:11.972900 1416 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 13 pre-election: Requesting pre-vote from peer f78a2351568c4a268ee1112681e22bec I0312 20:10:11.973356 1416 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 13 pre-election: Requesting pre-vote from peer ded9b7f8677940fb91e1a8c8bceceaed I0312 20:10:11.974642 1095 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "6885de38238442d6a9e5f1eb9fb569ff" candidate_term: 13 candidate_status { last_received { term: 12 index: 76 } } ignore_live_leader: false dest_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" is_pre_election: true I0312 20:10:11.975149 1095 raft_consensus.cc:2176] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 12 LEADER]: Leader pre-election vote request: Denying vote to candidate 6885de38238442d6a9e5f1eb9fb569ff for term 13 because replica is either leader or believes a valid leader to be alive. I0312 20:10:11.976084 1153 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 13 pre-election: Vote denied by peer ded9b7f8677940fb91e1a8c8bceceaed. Message: Invalid argument: T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 12 LEADER]: Leader pre-election vote request: Denying vote to candidate 6885de38238442d6a9e5f1eb9fb569ff for term 13 because replica is either leader or believes a valid leader to be alive. I0312 20:10:11.978286 1344 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "6885de38238442d6a9e5f1eb9fb569ff" candidate_term: 13 candidate_status { last_received { term: 12 index: 76 } } ignore_live_leader: false dest_uuid: "f78a2351568c4a268ee1112681e22bec" is_pre_election: true I0312 20:10:11.978713 1344 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec: Snoozing failure detection for 0.021s (vote granted) I0312 20:10:11.978904 1344 raft_consensus.cc:2215] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6885de38238442d6a9e5f1eb9fb569ff in term 12. I0312 20:10:11.980208 1152 leader_election.cc:363] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 13 pre-election: Vote granted by peer f78a2351568c4a268ee1112681e22bec I0312 20:10:11.980438 1152 leader_election.cc:258] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. I0312 20:10:11.992930 1416 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff: Snoozing failure detection for 0.019s (election complete) I0312 20:10:11.993178 1416 raft_consensus.cc:2507] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 12 FOLLOWER]: Leader pre-election won for term 13 I0312 20:10:11.995159 1415 raft_consensus.cc:436] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 12 FOLLOWER]: Starting pre-election (detected failure of leader ded9b7f8677940fb91e1a8c8bceceaed) I0312 20:10:11.995493 1415 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec: Snoozing failure detection for 0.017s (starting election) I0312 20:10:11.995755 1415 raft_consensus.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:11.996891 1415 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 13 pre-election: Requesting pre-vote from peer 6885de38238442d6a9e5f1eb9fb569ff I0312 20:10:11.998801 1220 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "f78a2351568c4a268ee1112681e22bec" candidate_term: 13 candidate_status { last_received { term: 12 index: 76 } } ignore_live_leader: false dest_uuid: "6885de38238442d6a9e5f1eb9fb569ff" is_pre_election: true I0312 20:10:11.999364 1220 raft_consensus.cc:2176] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 13 because replica is either leader or believes a valid leader to be alive. I0312 20:10:12.000073 1415 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 13 pre-election: Requesting pre-vote from peer ded9b7f8677940fb91e1a8c8bceceaed I0312 20:10:12.000501 1279 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 13 pre-election: Vote denied by peer 6885de38238442d6a9e5f1eb9fb569ff. Message: Invalid argument: T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 13 because replica is either leader or believes a valid leader to be alive. I0312 20:10:12.001890 1095 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "f78a2351568c4a268ee1112681e22bec" candidate_term: 13 candidate_status { last_received { term: 12 index: 76 } } ignore_live_leader: false dest_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" is_pre_election: true I0312 20:10:12.002398 1095 raft_consensus.cc:2176] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 12 LEADER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 13 because replica is either leader or believes a valid leader to be alive. I0312 20:10:12.004773 1278 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 13 pre-election: Vote denied by peer ded9b7f8677940fb91e1a8c8bceceaed. Message: Invalid argument: T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 12 LEADER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 13 because replica is either leader or believes a valid leader to be alive. I0312 20:10:12.005007 1278 leader_election.cc:258] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. I0312 20:10:12.005595 1415 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec: Snoozing failure detection for 0.021s (election complete) I0312 20:10:12.005796 1415 raft_consensus.cc:2455] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority I0312 20:10:12.012661 1416 raft_consensus.cc:436] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 12 FOLLOWER]: Starting leader election (detected failure of leader ded9b7f8677940fb91e1a8c8bceceaed) I0312 20:10:12.012882 1416 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff: Snoozing failure detection for 0.018s (starting election) I0312 20:10:12.013125 1416 raft_consensus.cc:2752] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 12 FOLLOWER]: Advancing to term 13 I0312 20:10:12.018343 1416 raft_consensus.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:12.019448 1416 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 13 election: Requesting vote from peer f78a2351568c4a268ee1112681e22bec I0312 20:10:12.019570 1220 raft_consensus.cc:1026] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 13 FOLLOWER]: Rejecting Update request from peer ded9b7f8677940fb91e1a8c8bceceaed for earlier term 12. Current term is 13. Ops: [] I0312 20:10:12.019948 1416 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 13 election: Requesting vote from peer ded9b7f8677940fb91e1a8c8bceceaed I0312 20:10:12.021649 1346 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "6885de38238442d6a9e5f1eb9fb569ff" candidate_term: 13 candidate_status { last_received { term: 12 index: 78 } } ignore_live_leader: false dest_uuid: "f78a2351568c4a268ee1112681e22bec" I0312 20:10:12.022141 1346 raft_consensus.cc:2176] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 6885de38238442d6a9e5f1eb9fb569ff for term 13 because replica is either leader or believes a valid leader to be alive. I0312 20:10:12.024170 1095 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "6885de38238442d6a9e5f1eb9fb569ff" candidate_term: 13 candidate_status { last_received { term: 12 index: 78 } } ignore_live_leader: false dest_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" I0312 20:10:12.024657 1095 raft_consensus.cc:2176] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 12 LEADER]: Leader election vote request: Denying vote to candidate 6885de38238442d6a9e5f1eb9fb569ff for term 13 because replica is either leader or believes a valid leader to be alive. I0312 20:10:12.024695 1415 raft_consensus.cc:436] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 12 FOLLOWER]: Starting pre-election (detected failure of leader ded9b7f8677940fb91e1a8c8bceceaed) I0312 20:10:12.024914 1415 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec: Snoozing failure detection for 0.024s (starting election) I0312 20:10:12.025192 1415 raft_consensus.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:12.025749 1153 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 13 election: Vote denied by peer ded9b7f8677940fb91e1a8c8bceceaed. Message: Invalid argument: T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 12 LEADER]: Leader election vote request: Denying vote to candidate 6885de38238442d6a9e5f1eb9fb569ff for term 13 because replica is either leader or believes a valid leader to be alive. I0312 20:10:12.026285 1415 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 13 pre-election: Requesting pre-vote from peer 6885de38238442d6a9e5f1eb9fb569ff I0312 20:10:12.027390 1415 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 13 pre-election: Requesting pre-vote from peer ded9b7f8677940fb91e1a8c8bceceaed I0312 20:10:12.027848 1152 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 13 election: Vote denied by peer f78a2351568c4a268ee1112681e22bec. Message: Invalid argument: T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 6885de38238442d6a9e5f1eb9fb569ff for term 13 because replica is either leader or believes a valid leader to be alive. I0312 20:10:12.028903 1095 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "f78a2351568c4a268ee1112681e22bec" candidate_term: 13 candidate_status { last_received { term: 12 index: 78 } } ignore_live_leader: false dest_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" is_pre_election: true I0312 20:10:12.029399 1095 raft_consensus.cc:2176] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 12 LEADER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 13 because replica is either leader or believes a valid leader to be alive. I0312 20:10:12.030257 1278 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 13 pre-election: Vote denied by peer ded9b7f8677940fb91e1a8c8bceceaed. Message: Invalid argument: T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 12 LEADER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 13 because replica is either leader or believes a valid leader to be alive. I0312 20:10:12.028110 1152 leader_election.cc:258] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. I0312 20:10:12.031522 1220 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "f78a2351568c4a268ee1112681e22bec" candidate_term: 13 candidate_status { last_received { term: 12 index: 78 } } ignore_live_leader: false dest_uuid: "6885de38238442d6a9e5f1eb9fb569ff" is_pre_election: true I0312 20:10:12.032516 1220 raft_consensus.cc:2145] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec in current term 13: Already voted for candidate 6885de38238442d6a9e5f1eb9fb569ff in this term. I0312 20:10:12.034976 1279 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 13 pre-election: Vote denied by peer 6885de38238442d6a9e5f1eb9fb569ff. Message: Invalid argument: T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec in current term 13: Already voted for candidate 6885de38238442d6a9e5f1eb9fb569ff in this term. I0312 20:10:12.035223 1279 leader_election.cc:258] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. I0312 20:10:12.036159 1415 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec: Snoozing failure detection for 0.025s (election complete) I0312 20:10:12.036406 1415 raft_consensus.cc:2752] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 12 FOLLOWER]: Advancing to term 13 I0312 20:10:12.042546 1454 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff: Snoozing failure detection for 0.021s (election complete) I0312 20:10:12.042750 1454 raft_consensus.cc:2455] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority I0312 20:10:12.042982 1454 raft_consensus.cc:436] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0312 20:10:12.043155 1454 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff: Snoozing failure detection for 0.026s (starting election) I0312 20:10:12.043366 1454 raft_consensus.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:12.044308 1454 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 14 pre-election: Requesting pre-vote from peer f78a2351568c4a268ee1112681e22bec I0312 20:10:12.044644 1454 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 14 pre-election: Requesting pre-vote from peer ded9b7f8677940fb91e1a8c8bceceaed I0312 20:10:12.046038 1095 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "6885de38238442d6a9e5f1eb9fb569ff" candidate_term: 14 candidate_status { last_received { term: 12 index: 78 } } ignore_live_leader: false dest_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" is_pre_election: true I0312 20:10:12.046524 1095 raft_consensus.cc:2176] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 12 LEADER]: Leader pre-election vote request: Denying vote to candidate 6885de38238442d6a9e5f1eb9fb569ff for term 14 because replica is either leader or believes a valid leader to be alive. I0312 20:10:12.047468 1153 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 14 pre-election: Vote denied by peer ded9b7f8677940fb91e1a8c8bceceaed. Message: Invalid argument: T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 12 LEADER]: Leader pre-election vote request: Denying vote to candidate 6885de38238442d6a9e5f1eb9fb569ff for term 14 because replica is either leader or believes a valid leader to be alive. I0312 20:10:12.048981 1346 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "6885de38238442d6a9e5f1eb9fb569ff" candidate_term: 14 candidate_status { last_received { term: 12 index: 78 } } ignore_live_leader: false dest_uuid: "f78a2351568c4a268ee1112681e22bec" is_pre_election: true I0312 20:10:12.048385 1455 consensus_queue.cc:1028] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 }, Status: INVALID_TERM, Last received: 12.78, Next index: 79, Last known committed idx: 78, Time since last communication: 0.000s I0312 20:10:12.049751 1455 raft_consensus.cc:2747] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 12 LEADER]: Stepping down as leader of term 12 I0312 20:10:12.049937 1455 raft_consensus.cc:605] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 12 LEADER]: Becoming Follower/Learner. State: Replica: ded9b7f8677940fb91e1a8c8bceceaed, State: Running, Role: LEADER I0312 20:10:12.050333 1455 consensus_queue.cc:219] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 80, Committed index: 80, Last appended: 12.80, Last appended by leader: 80, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:12.051477 1418 consensus_peers.cc:230] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed -> Peer 6885de38238442d6a9e5f1eb9fb569ff (127.0.53.2:51697): Could not obtain request from queue for peer: 6885de38238442d6a9e5f1eb9fb569ff. Status: Not found: Peer not tracked or queue not in leader mode. I0312 20:10:12.051935 1455 consensus_peers.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed -> Peer 6885de38238442d6a9e5f1eb9fb569ff (127.0.53.2:51697): Closing peer: 6885de38238442d6a9e5f1eb9fb569ff I0312 20:10:12.052168 1455 consensus_peers.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed -> Peer f78a2351568c4a268ee1112681e22bec (127.0.53.3:43274): Closing peer: f78a2351568c4a268ee1112681e22bec I0312 20:10:12.052382 1455 raft_consensus.cc:2752] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 12 FOLLOWER]: Advancing to term 13 I0312 20:10:12.059638 1415 raft_consensus.cc:2455] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority I0312 20:10:12.063048 1455 raft_consensus.cc:436] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0312 20:10:12.063170 1346 raft_consensus.cc:2190] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6885de38238442d6a9e5f1eb9fb569ff for term 14 because replica is already servicing an update from a current leader or another vote. I0312 20:10:12.063288 1455 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed: Snoozing failure detection for 0.015s (starting election) I0312 20:10:12.063510 1455 raft_consensus.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:12.064462 1455 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 14 pre-election: Requesting pre-vote from peer f78a2351568c4a268ee1112681e22bec I0312 20:10:12.064590 1152 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 14 pre-election: Vote denied by peer f78a2351568c4a268ee1112681e22bec. Message: Service unavailable: T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6885de38238442d6a9e5f1eb9fb569ff for term 14 because replica is already servicing an update from a current leader or another vote. I0312 20:10:12.064841 1152 leader_election.cc:258] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. I0312 20:10:12.064846 1455 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 14 pre-election: Requesting pre-vote from peer 6885de38238442d6a9e5f1eb9fb569ff I0312 20:10:12.065150 1416 raft_consensus.cc:436] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0312 20:10:12.066155 1416 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff: Snoozing failure detection for 0.017s (starting election) I0312 20:10:12.066380 1416 raft_consensus.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:12.067330 1416 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 14 pre-election: Requesting pre-vote from peer f78a2351568c4a268ee1112681e22bec I0312 20:10:12.067667 1416 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 14 pre-election: Requesting pre-vote from peer ded9b7f8677940fb91e1a8c8bceceaed I0312 20:10:12.068629 1424 raft_consensus.cc:436] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0312 20:10:12.068863 1424 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec: Snoozing failure detection for 0.022s (starting election) I0312 20:10:12.069650 1095 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "6885de38238442d6a9e5f1eb9fb569ff" candidate_term: 14 candidate_status { last_received { term: 12 index: 78 } } ignore_live_leader: false dest_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" is_pre_election: true I0312 20:10:12.070205 1095 raft_consensus.cc:2162] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6885de38238442d6a9e5f1eb9fb569ff for term 14 because replica has last-logged OpId of term: 12 index: 80, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 78. I0312 20:10:12.073151 1424 raft_consensus.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:12.074276 1424 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 14 pre-election: Requesting pre-vote from peer 6885de38238442d6a9e5f1eb9fb569ff I0312 20:10:12.074637 1424 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 14 pre-election: Requesting pre-vote from peer ded9b7f8677940fb91e1a8c8bceceaed I0312 20:10:12.072990 1153 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 14 pre-election: Vote denied by peer ded9b7f8677940fb91e1a8c8bceceaed. Message: Invalid argument: T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6885de38238442d6a9e5f1eb9fb569ff for term 14 because replica has last-logged OpId of term: 12 index: 80, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 78. I0312 20:10:12.076791 1220 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "f78a2351568c4a268ee1112681e22bec" candidate_term: 14 candidate_status { last_received { term: 12 index: 80 } } ignore_live_leader: false dest_uuid: "6885de38238442d6a9e5f1eb9fb569ff" is_pre_election: true I0312 20:10:12.077251 1220 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff: Snoozing failure detection for 0.022s (vote granted) I0312 20:10:12.077445 1220 raft_consensus.cc:2215] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f78a2351568c4a268ee1112681e22bec in term 13. I0312 20:10:12.079599 1346 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "6885de38238442d6a9e5f1eb9fb569ff" candidate_term: 14 candidate_status { last_received { term: 12 index: 78 } } ignore_live_leader: false dest_uuid: "f78a2351568c4a268ee1112681e22bec" is_pre_election: true I0312 20:10:12.080090 1346 raft_consensus.cc:2190] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6885de38238442d6a9e5f1eb9fb569ff for term 14 because replica is already servicing an update from a current leader or another vote. W0312 20:10:12.080812 1029 leader_election.cc:287] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 14 pre-election: RPC error from VoteRequest() call to peer 6885de38238442d6a9e5f1eb9fb569ff: Timed out: RequestConsensusVote RPC to 127.0.53.2:51697 timed out after 0.015s (SENT) I0312 20:10:12.081262 1455 raft_consensus.cc:436] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0312 20:10:12.081466 1455 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed: Snoozing failure detection for 0.020s (starting election) I0312 20:10:12.081836 1220 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" candidate_term: 14 candidate_status { last_received { term: 12 index: 80 } } ignore_live_leader: false dest_uuid: "6885de38238442d6a9e5f1eb9fb569ff" is_pre_election: true I0312 20:10:12.081673 1455 raft_consensus.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:12.082229 1220 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff: Snoozing failure detection for 0.016s (vote granted) I0312 20:10:12.082424 1220 raft_consensus.cc:2215] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ded9b7f8677940fb91e1a8c8bceceaed in term 13. I0312 20:10:12.082597 1455 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 14 pre-election: Requesting pre-vote from peer f78a2351568c4a268ee1112681e22bec I0312 20:10:12.082931 1455 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 14 pre-election: Requesting pre-vote from peer 6885de38238442d6a9e5f1eb9fb569ff I0312 20:10:12.084444 1279 leader_election.cc:363] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 14 pre-election: Vote granted by peer 6885de38238442d6a9e5f1eb9fb569ff I0312 20:10:12.086498 1279 leader_election.cc:258] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. I0312 20:10:12.087882 1152 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 14 pre-election: Vote denied by peer f78a2351568c4a268ee1112681e22bec. Message: Service unavailable: T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6885de38238442d6a9e5f1eb9fb569ff for term 14 because replica is already servicing an update from a current leader or another vote. I0312 20:10:12.088106 1152 leader_election.cc:258] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. I0312 20:10:12.088542 1454 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff: Snoozing failure detection for 0.015s (election complete) I0312 20:10:12.088748 1454 raft_consensus.cc:2455] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority I0312 20:10:12.089203 1424 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec: Snoozing failure detection for 0.015s (election complete) I0312 20:10:12.089448 1424 raft_consensus.cc:2507] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 13 FOLLOWER]: Leader pre-election won for term 14 I0312 20:10:12.089648 1424 raft_consensus.cc:436] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0312 20:10:12.089841 1424 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec: Snoozing failure detection for 0.023s (starting election) I0312 20:10:12.090023 1424 raft_consensus.cc:2752] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 13 FOLLOWER]: Advancing to term 14 I0312 20:10:12.088991 1454 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff: Snoozing failure detection for 0.031s (election complete) I0312 20:10:12.093243 1454 raft_consensus.cc:2455] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority I0312 20:10:12.089030 1220 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" candidate_term: 14 candidate_status { last_received { term: 12 index: 80 } } ignore_live_leader: false dest_uuid: "6885de38238442d6a9e5f1eb9fb569ff" is_pre_election: true I0312 20:10:12.093580 1220 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff: Snoozing failure detection for 0.025s (vote granted) I0312 20:10:12.093773 1220 raft_consensus.cc:2215] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ded9b7f8677940fb91e1a8c8bceceaed in term 13. I0312 20:10:12.094643 1029 leader_election.cc:363] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 14 pre-election: Vote granted by peer 6885de38238442d6a9e5f1eb9fb569ff I0312 20:10:12.094847 1029 leader_election.cc:258] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. I0312 20:10:12.095291 1455 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed: Snoozing failure detection for 0.019s (election complete) I0312 20:10:12.095495 1455 raft_consensus.cc:2507] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 13 FOLLOWER]: Leader pre-election won for term 14 I0312 20:10:12.095667 1455 raft_consensus.cc:436] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0312 20:10:12.095841 1455 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed: Snoozing failure detection for 0.019s (starting election) I0312 20:10:12.096009 1455 raft_consensus.cc:2752] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 13 FOLLOWER]: Advancing to term 14 W0312 20:10:12.097975 1278 leader_election.cc:287] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 14 pre-election: RPC error from VoteRequest() call to peer ded9b7f8677940fb91e1a8c8bceceaed: Timed out: RequestConsensusVote RPC to 127.0.53.1:35037 timed out after 0.022s (SENT) I0312 20:10:12.099160 1424 raft_consensus.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:12.100505 1345 raft_consensus.cc:1026] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 14 FOLLOWER]: Rejecting Update request from peer ded9b7f8677940fb91e1a8c8bceceaed for earlier term 12. Current term is 14. Ops: [] I0312 20:10:12.100811 1455 raft_consensus.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:12.101752 1455 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 14 election: Requesting vote from peer f78a2351568c4a268ee1112681e22bec I0312 20:10:12.102102 1455 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 14 election: Requesting vote from peer 6885de38238442d6a9e5f1eb9fb569ff I0312 20:10:12.101881 1424 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 14 election: Requesting vote from peer 6885de38238442d6a9e5f1eb9fb569ff I0312 20:10:12.103380 1220 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" candidate_term: 14 candidate_status { last_received { term: 12 index: 80 } } ignore_live_leader: false dest_uuid: "6885de38238442d6a9e5f1eb9fb569ff" I0312 20:10:12.103543 1219 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "f78a2351568c4a268ee1112681e22bec" candidate_term: 14 candidate_status { last_received { term: 12 index: 80 } } ignore_live_leader: false dest_uuid: "6885de38238442d6a9e5f1eb9fb569ff" I0312 20:10:12.103760 1220 raft_consensus.cc:2752] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 13 FOLLOWER]: Advancing to term 14 I0312 20:10:12.103973 1220 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff: Snoozing failure detection for 0.059s (vote granted) I0312 20:10:12.105124 1424 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 14 election: Requesting vote from peer ded9b7f8677940fb91e1a8c8bceceaed I0312 20:10:12.106492 1345 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" candidate_term: 14 candidate_status { last_received { term: 12 index: 80 } } ignore_live_leader: false dest_uuid: "f78a2351568c4a268ee1112681e22bec" is_pre_election: true I0312 20:10:12.107055 1345 raft_consensus.cc:2145] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ded9b7f8677940fb91e1a8c8bceceaed in current term 14: Already voted for candidate f78a2351568c4a268ee1112681e22bec in this term. I0312 20:10:12.107952 1220 raft_consensus.cc:2215] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ded9b7f8677940fb91e1a8c8bceceaed in term 14. I0312 20:10:12.108296 1219 raft_consensus.cc:2190] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 14 because replica is already servicing an update from a current leader or another vote. I0312 20:10:12.108459 1345 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" candidate_term: 14 candidate_status { last_received { term: 12 index: 80 } } ignore_live_leader: false dest_uuid: "f78a2351568c4a268ee1112681e22bec" is_pre_election: true I0312 20:10:12.108840 1029 leader_election.cc:363] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 14 election: Vote granted by peer 6885de38238442d6a9e5f1eb9fb569ff I0312 20:10:12.108991 1345 raft_consensus.cc:2145] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ded9b7f8677940fb91e1a8c8bceceaed in current term 14: Already voted for candidate f78a2351568c4a268ee1112681e22bec in this term. I0312 20:10:12.109036 1029 leader_election.cc:258] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. I0312 20:10:12.109397 1279 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 14 election: Vote denied by peer 6885de38238442d6a9e5f1eb9fb569ff. Message: Service unavailable: T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 14 because replica is already servicing an update from a current leader or another vote. I0312 20:10:12.109503 1417 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed: Snoozing failure detection for 0.018s (election complete) I0312 20:10:12.109704 1417 raft_consensus.cc:2507] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 14 FOLLOWER]: Leader election won for term 14 I0312 20:10:12.109989 1417 raft_consensus.cc:572] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 14 LEADER]: Becoming Leader. State: Replica: ded9b7f8677940fb91e1a8c8bceceaed, State: Running, Role: LEADER I0312 20:10:12.110416 1345 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" candidate_term: 14 candidate_status { last_received { term: 12 index: 80 } } ignore_live_leader: false dest_uuid: "f78a2351568c4a268ee1112681e22bec" I0312 20:10:12.110371 1417 consensus_queue.cc:196] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 80, Committed index: 80, Last appended: 12.80, Last appended by leader: 80, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:12.111080 1345 raft_consensus.cc:2145] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate ded9b7f8677940fb91e1a8c8bceceaed in current term 14: Already voted for candidate f78a2351568c4a268ee1112681e22bec in this term. I0312 20:10:12.112263 1095 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "f78a2351568c4a268ee1112681e22bec" candidate_term: 14 candidate_status { last_received { term: 12 index: 80 } } ignore_live_leader: false dest_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" is_pre_election: true I0312 20:10:12.112813 1095 raft_consensus.cc:2176] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 14 LEADER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 14 because replica is either leader or believes a valid leader to be alive. I0312 20:10:12.113492 1424 raft_consensus.cc:436] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0312 20:10:12.113765 1424 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec: Snoozing failure detection for 0.015s (starting election) I0312 20:10:12.114018 1424 raft_consensus.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:12.115010 1424 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 15 pre-election: Requesting pre-vote from peer 6885de38238442d6a9e5f1eb9fb569ff I0312 20:10:12.114887 974 catalog_manager.cc:3592] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed reported cstate change: term changed from 12 to 14. New cstate: current_term: 14 leader_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } health_report { overall_health: HEALTHY } } } I0312 20:10:12.119249 1027 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 14 pre-election: Vote denied by peer f78a2351568c4a268ee1112681e22bec. Message: Invalid argument: T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ded9b7f8677940fb91e1a8c8bceceaed in current term 14: Already voted for candidate f78a2351568c4a268ee1112681e22bec in this term. I0312 20:10:12.119524 1027 leader_election.cc:258] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. I0312 20:10:12.119915 1417 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed: Snoozing failure detection for 0.016s (election complete) I0312 20:10:12.120095 1417 raft_consensus.cc:2455] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 14 LEADER]: Leader pre-election lost for term 14. Reason: could not achieve majority I0312 20:10:12.122205 1219 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "f78a2351568c4a268ee1112681e22bec" candidate_term: 15 candidate_status { last_received { term: 12 index: 80 } } ignore_live_leader: false dest_uuid: "6885de38238442d6a9e5f1eb9fb569ff" is_pre_election: true I0312 20:10:12.122617 1219 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff: Snoozing failure detection for 0.024s (vote granted) I0312 20:10:12.122804 1219 raft_consensus.cc:2215] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f78a2351568c4a268ee1112681e22bec in term 14. I0312 20:10:12.123705 1279 leader_election.cc:363] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 15 pre-election: Vote granted by peer 6885de38238442d6a9e5f1eb9fb569ff I0312 20:10:12.123898 1279 leader_election.cc:258] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. I0312 20:10:12.124390 1415 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec: Snoozing failure detection for 0.040s (election complete) I0312 20:10:12.124622 1415 raft_consensus.cc:2507] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 14 FOLLOWER]: Leader pre-election won for term 15 I0312 20:10:12.124826 1415 raft_consensus.cc:436] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0312 20:10:12.125022 1415 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec: Snoozing failure detection for 0.037s (starting election) I0312 20:10:12.125247 1415 raft_consensus.cc:2752] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 14 FOLLOWER]: Advancing to term 15 I0312 20:10:12.126802 1219 raft_consensus.cc:1058] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 14 FOLLOWER]: Refusing update from remote peer ded9b7f8677940fb91e1a8c8bceceaed: Log matching property violated. Preceding OpId in replica: term: 12 index: 78. Preceding OpId from leader: term: 14 index: 81. (index mismatch) I0312 20:10:12.127918 1417 consensus_queue.cc:1017] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [LEADER]: Connected to new peer: Peer: permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 }, Status: LMP_MISMATCH, Last received: 12.78, Next index: 79, Last known committed idx: 78, Time since last communication: 0.000s I0312 20:10:12.128516 1095 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "f78a2351568c4a268ee1112681e22bec" candidate_term: 14 candidate_status { last_received { term: 12 index: 80 } } ignore_live_leader: false dest_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" W0312 20:10:12.129222 1278 leader_election.cc:287] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 14 election: RPC error from VoteRequest() call to peer ded9b7f8677940fb91e1a8c8bceceaed: Timed out: RequestConsensusVote RPC to 127.0.53.1:35037 timed out after 0.023s (SENT) I0312 20:10:12.129451 1278 leader_election.cc:258] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. I0312 20:10:12.129024 1095 raft_consensus.cc:2176] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 14 LEADER]: Leader election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 14 because replica is either leader or believes a valid leader to be alive. W0312 20:10:12.136029 1027 leader_election.cc:287] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 14 election: RPC error from VoteRequest() call to peer f78a2351568c4a268ee1112681e22bec: Timed out: RequestConsensusVote RPC to 127.0.53.3:43274 timed out after 0.019s (SENT) W0312 20:10:12.136281 1027 leader_election.cc:287] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 14 pre-election: RPC error from VoteRequest() call to peer f78a2351568c4a268ee1112681e22bec: Timed out: RequestConsensusVote RPC to 127.0.53.3:43274 timed out after 0.020s (SENT) I0312 20:10:12.141203 1424 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 15 pre-election: Requesting pre-vote from peer ded9b7f8677940fb91e1a8c8bceceaed I0312 20:10:12.142531 1415 raft_consensus.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:12.146147 1415 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 15 election: Requesting vote from peer 6885de38238442d6a9e5f1eb9fb569ff I0312 20:10:12.147346 1220 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "f78a2351568c4a268ee1112681e22bec" candidate_term: 15 candidate_status { last_received { term: 12 index: 80 } } ignore_live_leader: false dest_uuid: "6885de38238442d6a9e5f1eb9fb569ff" I0312 20:10:12.147737 1424 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec: Snoozing failure detection for 0.016s (election complete) I0312 20:10:12.147900 1424 raft_consensus.cc:2455] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 15 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority I0312 20:10:12.148134 1345 raft_consensus.cc:1026] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 15 FOLLOWER]: Rejecting Update request from peer ded9b7f8677940fb91e1a8c8bceceaed for earlier term 14. Current term is 15. Ops: [] I0312 20:10:12.148836 1415 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 15 election: Requesting vote from peer ded9b7f8677940fb91e1a8c8bceceaed I0312 20:10:12.149618 1417 consensus_queue.cc:1028] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 }, Status: INVALID_TERM, Last received: 12.80, Next index: 81, Last known committed idx: 78, Time since last communication: 0.000s I0312 20:10:12.150228 1418 raft_consensus.cc:2747] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 14 LEADER]: Stepping down as leader of term 14 I0312 20:10:12.150404 1418 raft_consensus.cc:605] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 14 LEADER]: Becoming Follower/Learner. State: Replica: ded9b7f8677940fb91e1a8c8bceceaed, State: Running, Role: LEADER I0312 20:10:12.150796 1418 consensus_queue.cc:219] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 80, Committed index: 80, Last appended: 14.83, Last appended by leader: 83, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:12.151418 1418 consensus_peers.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed -> Peer 6885de38238442d6a9e5f1eb9fb569ff (127.0.53.2:51697): Closing peer: 6885de38238442d6a9e5f1eb9fb569ff I0312 20:10:12.152340 1418 consensus_peers.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed -> Peer f78a2351568c4a268ee1112681e22bec (127.0.53.3:43274): Closing peer: f78a2351568c4a268ee1112681e22bec I0312 20:10:12.152420 1220 raft_consensus.cc:2190] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 15 because replica is already servicing an update from a current leader or another vote. I0312 20:10:12.152544 1418 raft_consensus.cc:2752] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 14 FOLLOWER]: Advancing to term 15 I0312 20:10:12.156661 1279 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 15 election: Vote denied by peer 6885de38238442d6a9e5f1eb9fb569ff. Message: Service unavailable: T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 15 because replica is already servicing an update from a current leader or another vote. I0312 20:10:12.157407 1416 raft_consensus.cc:436] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 14 FOLLOWER]: Starting pre-election (detected failure of leader ded9b7f8677940fb91e1a8c8bceceaed) I0312 20:10:12.157636 1416 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff: Snoozing failure detection for 0.016s (starting election) I0312 20:10:12.157865 1416 raft_consensus.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:12.158819 1416 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 15 pre-election: Requesting pre-vote from peer f78a2351568c4a268ee1112681e22bec I0312 20:10:12.159152 1416 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 15 pre-election: Requesting pre-vote from peer ded9b7f8677940fb91e1a8c8bceceaed I0312 20:10:12.160306 1095 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "6885de38238442d6a9e5f1eb9fb569ff" candidate_term: 15 candidate_status { last_received { term: 14 index: 82 } } ignore_live_leader: false dest_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" is_pre_election: true I0312 20:10:12.160894 1095 raft_consensus.cc:2162] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6885de38238442d6a9e5f1eb9fb569ff for term 15 because replica has last-logged OpId of term: 14 index: 83, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 82. I0312 20:10:12.161939 1153 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 15 pre-election: Vote denied by peer ded9b7f8677940fb91e1a8c8bceceaed. Message: Invalid argument: T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6885de38238442d6a9e5f1eb9fb569ff for term 15 because replica has last-logged OpId of term: 14 index: 83, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 82. I0312 20:10:12.163122 1345 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "6885de38238442d6a9e5f1eb9fb569ff" candidate_term: 15 candidate_status { last_received { term: 14 index: 82 } } ignore_live_leader: false dest_uuid: "f78a2351568c4a268ee1112681e22bec" is_pre_election: true I0312 20:10:12.163661 1345 raft_consensus.cc:2145] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6885de38238442d6a9e5f1eb9fb569ff in current term 15: Already voted for candidate f78a2351568c4a268ee1112681e22bec in this term. I0312 20:10:12.164697 1415 raft_consensus.cc:436] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0312 20:10:12.164942 1415 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec: Snoozing failure detection for 0.022s (starting election) I0312 20:10:12.165210 1415 raft_consensus.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:12.166178 1415 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 16 pre-election: Requesting pre-vote from peer 6885de38238442d6a9e5f1eb9fb569ff I0312 20:10:12.167634 1220 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "f78a2351568c4a268ee1112681e22bec" candidate_term: 16 candidate_status { last_received { term: 12 index: 80 } } ignore_live_leader: false dest_uuid: "6885de38238442d6a9e5f1eb9fb569ff" is_pre_election: true I0312 20:10:12.168287 1220 raft_consensus.cc:2162] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 16 because replica has last-logged OpId of term: 14 index: 82, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 80. I0312 20:10:12.169162 1279 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 16 pre-election: Vote denied by peer 6885de38238442d6a9e5f1eb9fb569ff. Message: Invalid argument: T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 16 because replica has last-logged OpId of term: 14 index: 82, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 80. I0312 20:10:12.169417 1152 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 15 pre-election: Vote denied by peer f78a2351568c4a268ee1112681e22bec. Message: Invalid argument: T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6885de38238442d6a9e5f1eb9fb569ff in current term 15: Already voted for candidate f78a2351568c4a268ee1112681e22bec in this term. I0312 20:10:12.169560 1415 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 16 pre-election: Requesting pre-vote from peer ded9b7f8677940fb91e1a8c8bceceaed I0312 20:10:12.169642 1152 leader_election.cc:258] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. I0312 20:10:12.170042 1416 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff: Snoozing failure detection for 0.021s (election complete) I0312 20:10:12.170073 1455 raft_consensus.cc:436] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0312 20:10:12.170241 1416 raft_consensus.cc:2752] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 14 FOLLOWER]: Advancing to term 15 I0312 20:10:12.170333 1455 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed: Snoozing failure detection for 0.019s (starting election) I0312 20:10:12.171546 1455 raft_consensus.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:12.172606 1455 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 16 pre-election: Requesting pre-vote from peer f78a2351568c4a268ee1112681e22bec I0312 20:10:12.173866 1345 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" candidate_term: 16 candidate_status { last_received { term: 14 index: 83 } } ignore_live_leader: false dest_uuid: "f78a2351568c4a268ee1112681e22bec" is_pre_election: true I0312 20:10:12.174270 1345 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec: Snoozing failure detection for 0.054s (vote granted) I0312 20:10:12.174455 1345 raft_consensus.cc:2215] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ded9b7f8677940fb91e1a8c8bceceaed in term 15. I0312 20:10:12.174885 1416 raft_consensus.cc:2455] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority I0312 20:10:12.172987 1455 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 16 pre-election: Requesting pre-vote from peer 6885de38238442d6a9e5f1eb9fb569ff W0312 20:10:12.175964 1278 outbound_call.cc:286] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.53.1:35037, user_credentials={real_user=slave}} blocked reactor thread for 46782.1us I0312 20:10:12.176512 1220 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" candidate_term: 16 candidate_status { last_received { term: 14 index: 83 } } ignore_live_leader: false dest_uuid: "6885de38238442d6a9e5f1eb9fb569ff" is_pre_election: true I0312 20:10:12.176978 1220 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff: Snoozing failure detection for 0.017s (vote granted) I0312 20:10:12.177232 1220 raft_consensus.cc:2215] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ded9b7f8677940fb91e1a8c8bceceaed in term 15. I0312 20:10:12.178073 1029 leader_election.cc:363] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 16 pre-election: Vote granted by peer 6885de38238442d6a9e5f1eb9fb569ff I0312 20:10:12.178262 1029 leader_election.cc:258] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. I0312 20:10:12.179648 1418 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed: Snoozing failure detection for 0.026s (election complete) I0312 20:10:12.179895 1418 raft_consensus.cc:2507] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 15 FOLLOWER]: Leader pre-election won for term 16 I0312 20:10:12.180102 1418 raft_consensus.cc:436] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0312 20:10:12.180302 1418 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed: Snoozing failure detection for 0.017s (starting election) I0312 20:10:12.180501 1418 raft_consensus.cc:2752] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 15 FOLLOWER]: Advancing to term 16 I0312 20:10:12.183241 1095 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "f78a2351568c4a268ee1112681e22bec" candidate_term: 15 candidate_status { last_received { term: 12 index: 80 } } ignore_live_leader: false dest_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" is_pre_election: true I0312 20:10:12.183885 1027 leader_election.cc:363] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 16 pre-election: Vote granted by peer f78a2351568c4a268ee1112681e22bec I0312 20:10:12.185816 1096 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "f78a2351568c4a268ee1112681e22bec" candidate_term: 15 candidate_status { last_received { term: 12 index: 80 } } ignore_live_leader: false dest_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" I0312 20:10:12.187783 1094 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "f78a2351568c4a268ee1112681e22bec" candidate_term: 16 candidate_status { last_received { term: 12 index: 80 } } ignore_live_leader: false dest_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" is_pre_election: true I0312 20:10:12.189019 1418 raft_consensus.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:12.191375 1095 raft_consensus.cc:2120] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for earlier term 15. Current term is 16. W0312 20:10:12.191627 1278 leader_election.cc:287] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 15 pre-election: RPC error from VoteRequest() call to peer ded9b7f8677940fb91e1a8c8bceceaed: Timed out: RequestConsensusVote RPC to 127.0.53.1:35037 timed out after 0.015s (SENT) I0312 20:10:12.192087 1094 raft_consensus.cc:2190] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 16 because replica is already servicing an update from a current leader or another vote. I0312 20:10:12.192868 1096 raft_consensus.cc:2190] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 15 because replica is already servicing an update from a current leader or another vote. I0312 20:10:12.194380 1278 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 16 pre-election: Vote denied by peer ded9b7f8677940fb91e1a8c8bceceaed. Message: Service unavailable: T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 16 because replica is already servicing an update from a current leader or another vote. I0312 20:10:12.194597 1278 leader_election.cc:258] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. I0312 20:10:12.195283 1464 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec: Snoozing failure detection for 0.031s (election complete) I0312 20:10:12.195528 1464 raft_consensus.cc:2752] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 15 FOLLOWER]: Advancing to term 16 I0312 20:10:12.196844 1416 raft_consensus.cc:436] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0312 20:10:12.197048 1416 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff: Snoozing failure detection for 0.023s (starting election) I0312 20:10:12.197294 1416 raft_consensus.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:12.198205 1416 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 16 pre-election: Requesting pre-vote from peer f78a2351568c4a268ee1112681e22bec I0312 20:10:12.198549 1416 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 16 pre-election: Requesting pre-vote from peer ded9b7f8677940fb91e1a8c8bceceaed I0312 20:10:12.200783 1464 raft_consensus.cc:2455] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority I0312 20:10:12.200867 1096 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "6885de38238442d6a9e5f1eb9fb569ff" candidate_term: 16 candidate_status { last_received { term: 14 index: 82 } } ignore_live_leader: false dest_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" is_pre_election: true I0312 20:10:12.200985 1345 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "6885de38238442d6a9e5f1eb9fb569ff" candidate_term: 16 candidate_status { last_received { term: 14 index: 82 } } ignore_live_leader: false dest_uuid: "f78a2351568c4a268ee1112681e22bec" is_pre_election: true I0312 20:10:12.201403 1096 raft_consensus.cc:2145] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6885de38238442d6a9e5f1eb9fb569ff in current term 16: Already voted for candidate ded9b7f8677940fb91e1a8c8bceceaed in this term. I0312 20:10:12.201483 1345 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec: Snoozing failure detection for 0.049s (vote granted) I0312 20:10:12.201681 1345 raft_consensus.cc:2215] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6885de38238442d6a9e5f1eb9fb569ff in term 16. I0312 20:10:12.202481 1153 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 16 pre-election: Vote denied by peer ded9b7f8677940fb91e1a8c8bceceaed. Message: Invalid argument: T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6885de38238442d6a9e5f1eb9fb569ff in current term 16: Already voted for candidate ded9b7f8677940fb91e1a8c8bceceaed in this term. I0312 20:10:12.202723 1152 leader_election.cc:363] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 16 pre-election: Vote granted by peer f78a2351568c4a268ee1112681e22bec I0312 20:10:12.202852 1455 raft_consensus.cc:436] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0312 20:10:12.202929 1152 leader_election.cc:258] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. I0312 20:10:12.203110 1455 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed: Snoozing failure detection for 0.020s (starting election) I0312 20:10:12.203498 1416 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff: Snoozing failure detection for 0.016s (election complete) I0312 20:10:12.203707 1416 raft_consensus.cc:2507] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 15 FOLLOWER]: Leader pre-election won for term 16 I0312 20:10:12.203420 1455 raft_consensus.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:12.203889 1416 raft_consensus.cc:436] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0312 20:10:12.204064 1416 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff: Snoozing failure detection for 0.024s (starting election) I0312 20:10:12.204238 1416 raft_consensus.cc:2752] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 15 FOLLOWER]: Advancing to term 16 I0312 20:10:12.204530 1455 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 17 pre-election: Requesting pre-vote from peer f78a2351568c4a268ee1112681e22bec I0312 20:10:12.204907 1455 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 17 pre-election: Requesting pre-vote from peer 6885de38238442d6a9e5f1eb9fb569ff W0312 20:10:12.205628 1278 leader_election.cc:347] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 15 election: Vote denied by peer ded9b7f8677940fb91e1a8c8bceceaed with higher term. Message: Service unavailable: T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 15 because replica is already servicing an update from a current leader or another vote. I0312 20:10:12.205818 1278 leader_election.cc:350] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 15 election: Cancelling election due to peer responding with higher term I0312 20:10:12.206416 1464 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec: Snoozing failure detection for 0.061s (election complete) I0312 20:10:12.206616 1464 raft_consensus.cc:2455] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 16 FOLLOWER]: Leader election lost for term 15. Reason: Vote denied by peer ded9b7f8677940fb91e1a8c8bceceaed with higher term. Message: Service unavailable: T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 15 because replica is already servicing an update from a current leader or another vote. I0312 20:10:12.209009 1220 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" candidate_term: 17 candidate_status { last_received { term: 14 index: 83 } } ignore_live_leader: false dest_uuid: "6885de38238442d6a9e5f1eb9fb569ff" is_pre_election: true I0312 20:10:12.209463 1416 raft_consensus.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:12.210453 1220 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff: Snoozing failure detection for 0.026s (vote granted) I0312 20:10:12.210661 1220 raft_consensus.cc:2215] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ded9b7f8677940fb91e1a8c8bceceaed in term 16. I0312 20:10:12.210515 1345 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" candidate_term: 17 candidate_status { last_received { term: 14 index: 83 } } ignore_live_leader: false dest_uuid: "f78a2351568c4a268ee1112681e22bec" is_pre_election: true I0312 20:10:12.210955 1345 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec: Snoozing failure detection for 0.156s (vote granted) I0312 20:10:12.211155 1345 raft_consensus.cc:2215] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ded9b7f8677940fb91e1a8c8bceceaed in term 16. I0312 20:10:12.212625 1029 leader_election.cc:363] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 17 pre-election: Vote granted by peer 6885de38238442d6a9e5f1eb9fb569ff I0312 20:10:12.212831 1029 leader_election.cc:258] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. I0312 20:10:12.213351 1417 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed: Snoozing failure detection for 0.023s (election complete) I0312 20:10:12.213568 1417 raft_consensus.cc:2507] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 16 FOLLOWER]: Leader pre-election won for term 17 I0312 20:10:12.213762 1417 raft_consensus.cc:436] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0312 20:10:12.213937 1417 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed: Snoozing failure detection for 0.032s (starting election) I0312 20:10:12.214110 1417 raft_consensus.cc:2752] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 16 FOLLOWER]: Advancing to term 17 I0312 20:10:12.219297 1417 raft_consensus.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:12.220252 1416 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 16 election: Requesting vote from peer f78a2351568c4a268ee1112681e22bec I0312 20:10:12.220531 1417 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 17 election: Requesting vote from peer f78a2351568c4a268ee1112681e22bec I0312 20:10:12.220927 1417 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 17 election: Requesting vote from peer 6885de38238442d6a9e5f1eb9fb569ff I0312 20:10:12.221402 1345 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "6885de38238442d6a9e5f1eb9fb569ff" candidate_term: 16 candidate_status { last_received { term: 14 index: 82 } } ignore_live_leader: false dest_uuid: "f78a2351568c4a268ee1112681e22bec" I0312 20:10:12.221803 1345 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec: Snoozing failure detection for 0.081s (vote granted) I0312 20:10:12.221534 1027 leader_election.cc:363] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 17 pre-election: Vote granted by peer f78a2351568c4a268ee1112681e22bec I0312 20:10:12.223345 1418 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 16 election: Requesting vote from peer f78a2351568c4a268ee1112681e22bec I0312 20:10:12.223404 1346 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" candidate_term: 17 candidate_status { last_received { term: 14 index: 83 } } ignore_live_leader: false dest_uuid: "f78a2351568c4a268ee1112681e22bec" I0312 20:10:12.223935 1418 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 16 election: Requesting vote from peer 6885de38238442d6a9e5f1eb9fb569ff I0312 20:10:12.225847 1344 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" candidate_term: 16 candidate_status { last_received { term: 14 index: 83 } } ignore_live_leader: false dest_uuid: "f78a2351568c4a268ee1112681e22bec" I0312 20:10:12.226514 1220 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" candidate_term: 17 candidate_status { last_received { term: 14 index: 83 } } ignore_live_leader: false dest_uuid: "6885de38238442d6a9e5f1eb9fb569ff" I0312 20:10:12.226899 1220 raft_consensus.cc:2752] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 16 FOLLOWER]: Advancing to term 17 I0312 20:10:12.227118 1220 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff: Snoozing failure detection for 0.030s (vote granted) I0312 20:10:12.231178 1220 raft_consensus.cc:2215] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ded9b7f8677940fb91e1a8c8bceceaed in term 17. I0312 20:10:12.232293 1029 leader_election.cc:363] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 17 election: Vote granted by peer 6885de38238442d6a9e5f1eb9fb569ff I0312 20:10:12.232482 1029 leader_election.cc:258] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. I0312 20:10:12.233111 1418 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed: Snoozing failure detection for 0.032s (election complete) I0312 20:10:12.233347 1418 raft_consensus.cc:2507] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 17 FOLLOWER]: Leader election won for term 17 I0312 20:10:12.233520 1345 raft_consensus.cc:2215] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6885de38238442d6a9e5f1eb9fb569ff in term 16. I0312 20:10:12.233897 1346 raft_consensus.cc:2190] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate ded9b7f8677940fb91e1a8c8bceceaed for term 17 because replica is already servicing an update from a current leader or another vote. I0312 20:10:12.233690 1418 raft_consensus.cc:572] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 17 LEADER]: Becoming Leader. State: Replica: ded9b7f8677940fb91e1a8c8bceceaed, State: Running, Role: LEADER I0312 20:10:12.234607 1344 raft_consensus.cc:2190] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate ded9b7f8677940fb91e1a8c8bceceaed for term 16 because replica is already servicing an update from a current leader or another vote. I0312 20:10:12.234870 1418 consensus_queue.cc:196] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 80, Committed index: 80, Last appended: 14.83, Last appended by leader: 83, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:12.235893 1027 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 17 election: Vote denied by peer f78a2351568c4a268ee1112681e22bec. Message: Service unavailable: T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate ded9b7f8677940fb91e1a8c8bceceaed for term 17 because replica is already servicing an update from a current leader or another vote. I0312 20:10:12.237632 1027 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 16 election: Vote denied by peer f78a2351568c4a268ee1112681e22bec. Message: Service unavailable: T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate ded9b7f8677940fb91e1a8c8bceceaed for term 16 because replica is already servicing an update from a current leader or another vote. I0312 20:10:12.238670 1152 leader_election.cc:363] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 16 election: Vote granted by peer f78a2351568c4a268ee1112681e22bec I0312 20:10:12.238860 1152 leader_election.cc:258] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. I0312 20:10:12.239271 1454 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff: Snoozing failure detection for 0.023s (election complete) I0312 20:10:12.239477 1454 raft_consensus.cc:2470] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 17 FOLLOWER]: Leader election decision vote started in defunct term 16: won I0312 20:10:12.239902 1220 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" candidate_term: 16 candidate_status { last_received { term: 14 index: 83 } } ignore_live_leader: false dest_uuid: "6885de38238442d6a9e5f1eb9fb569ff" I0312 20:10:12.240402 1220 raft_consensus.cc:2120] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate ded9b7f8677940fb91e1a8c8bceceaed for earlier term 16. Current term is 17. W0312 20:10:12.241252 1029 leader_election.cc:347] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 16 election: Vote denied by peer 6885de38238442d6a9e5f1eb9fb569ff with higher term. Message: Invalid argument: T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate ded9b7f8677940fb91e1a8c8bceceaed for earlier term 16. Current term is 17. I0312 20:10:12.241425 1029 leader_election.cc:350] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 16 election: Cancelling election due to peer responding with higher term I0312 20:10:12.240972 974 catalog_manager.cc:3592] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed reported cstate change: term changed from 14 to 17. New cstate: current_term: 17 leader_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } health_report { overall_health: HEALTHY } } } I0312 20:10:12.242560 1456 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed: Snoozing failure detection for 0.019s (election complete) I0312 20:10:12.242754 1456 raft_consensus.cc:2455] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 17 LEADER]: Leader election lost for term 16. Reason: Vote denied by peer 6885de38238442d6a9e5f1eb9fb569ff with higher term. Message: Invalid argument: T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate ded9b7f8677940fb91e1a8c8bceceaed for earlier term 16. Current term is 17. I0312 20:10:12.244859 1345 raft_consensus.cc:2752] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 16 FOLLOWER]: Advancing to term 17 I0312 20:10:12.246794 1416 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 16 election: Requesting vote from peer ded9b7f8677940fb91e1a8c8bceceaed I0312 20:10:12.248560 1220 raft_consensus.cc:1058] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 17 FOLLOWER]: Refusing update from remote peer ded9b7f8677940fb91e1a8c8bceceaed: Log matching property violated. Preceding OpId in replica: term: 14 index: 82. Preceding OpId from leader: term: 17 index: 84. (index mismatch) I0312 20:10:12.249223 1096 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "6885de38238442d6a9e5f1eb9fb569ff" candidate_term: 16 candidate_status { last_received { term: 14 index: 82 } } ignore_live_leader: false dest_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" I0312 20:10:12.249734 1096 raft_consensus.cc:2176] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 17 LEADER]: Leader election vote request: Denying vote to candidate 6885de38238442d6a9e5f1eb9fb569ff for term 16 because replica is either leader or believes a valid leader to be alive. I0312 20:10:12.249855 1455 consensus_queue.cc:1017] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [LEADER]: Connected to new peer: Peer: permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 }, Status: LMP_MISMATCH, Last received: 14.82, Next index: 83, Last known committed idx: 80, Time since last communication: 0.000s W0312 20:10:12.251060 1153 leader_election.cc:347] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 16 election: Vote denied by peer ded9b7f8677940fb91e1a8c8bceceaed with higher term. Message: Invalid argument: T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 17 LEADER]: Leader election vote request: Denying vote to candidate 6885de38238442d6a9e5f1eb9fb569ff for term 16 because replica is either leader or believes a valid leader to be alive. I0312 20:10:12.260643 1345 raft_consensus.cc:1058] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 17 FOLLOWER]: Refusing update from remote peer ded9b7f8677940fb91e1a8c8bceceaed: Log matching property violated. Preceding OpId in replica: term: 12 index: 80. Preceding OpId from leader: term: 17 index: 84. (index mismatch) I0312 20:10:12.262620 1417 consensus_queue.cc:1017] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [LEADER]: Connected to new peer: Peer: permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 }, Status: LMP_MISMATCH, Last received: 12.80, Next index: 81, Last known committed idx: 78, Time since last communication: 0.000s I0312 20:10:12.269004 1416 raft_consensus.cc:436] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 17 FOLLOWER]: Starting pre-election (detected failure of leader ded9b7f8677940fb91e1a8c8bceceaed) I0312 20:10:12.269250 1416 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff: Snoozing failure detection for 0.021s (starting election) I0312 20:10:12.269469 1416 raft_consensus.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:12.270439 1416 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 18 pre-election: Requesting pre-vote from peer f78a2351568c4a268ee1112681e22bec I0312 20:10:12.270787 1416 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 18 pre-election: Requesting pre-vote from peer ded9b7f8677940fb91e1a8c8bceceaed I0312 20:10:12.272054 1344 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "6885de38238442d6a9e5f1eb9fb569ff" candidate_term: 18 candidate_status { last_received { term: 17 index: 84 } } ignore_live_leader: false dest_uuid: "f78a2351568c4a268ee1112681e22bec" is_pre_election: true I0312 20:10:12.273561 1096 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "6885de38238442d6a9e5f1eb9fb569ff" candidate_term: 18 candidate_status { last_received { term: 17 index: 84 } } ignore_live_leader: false dest_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" is_pre_election: true I0312 20:10:12.273970 1344 raft_consensus.cc:2190] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6885de38238442d6a9e5f1eb9fb569ff for term 18 because replica is already servicing an update from a current leader or another vote. I0312 20:10:12.274051 1096 raft_consensus.cc:2176] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 17 LEADER]: Leader pre-election vote request: Denying vote to candidate 6885de38238442d6a9e5f1eb9fb569ff for term 18 because replica is either leader or believes a valid leader to be alive. I0312 20:10:12.275051 1153 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 18 pre-election: Vote denied by peer ded9b7f8677940fb91e1a8c8bceceaed. Message: Invalid argument: T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 17 LEADER]: Leader pre-election vote request: Denying vote to candidate 6885de38238442d6a9e5f1eb9fb569ff for term 18 because replica is either leader or believes a valid leader to be alive. I0312 20:10:12.275606 1152 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 18 pre-election: Vote denied by peer f78a2351568c4a268ee1112681e22bec. Message: Service unavailable: T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6885de38238442d6a9e5f1eb9fb569ff for term 18 because replica is already servicing an update from a current leader or another vote. I0312 20:10:12.275835 1152 leader_election.cc:258] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. I0312 20:10:12.277212 1416 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff: Snoozing failure detection for 0.016s (election complete) I0312 20:10:12.277412 1416 raft_consensus.cc:2455] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority I0312 20:10:12.286903 1464 raft_consensus.cc:436] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 17 FOLLOWER]: Starting pre-election (detected failure of leader ded9b7f8677940fb91e1a8c8bceceaed) I0312 20:10:12.287127 1464 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec: Snoozing failure detection for 0.021s (starting election) I0312 20:10:12.287341 1464 raft_consensus.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:12.288339 1464 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 18 pre-election: Requesting pre-vote from peer 6885de38238442d6a9e5f1eb9fb569ff I0312 20:10:12.288718 1464 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 18 pre-election: Requesting pre-vote from peer ded9b7f8677940fb91e1a8c8bceceaed I0312 20:10:12.290446 1220 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "f78a2351568c4a268ee1112681e22bec" candidate_term: 18 candidate_status { last_received { term: 17 index: 84 } } ignore_live_leader: false dest_uuid: "6885de38238442d6a9e5f1eb9fb569ff" is_pre_election: true I0312 20:10:12.290839 1220 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff: Snoozing failure detection for 0.029s (vote granted) I0312 20:10:12.291036 1220 raft_consensus.cc:2215] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f78a2351568c4a268ee1112681e22bec in term 17. I0312 20:10:12.291570 1096 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "f78a2351568c4a268ee1112681e22bec" candidate_term: 18 candidate_status { last_received { term: 17 index: 84 } } ignore_live_leader: false dest_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" is_pre_election: true I0312 20:10:12.292034 1096 raft_consensus.cc:2176] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 17 LEADER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 18 because replica is either leader or believes a valid leader to be alive. I0312 20:10:12.293336 1279 leader_election.cc:363] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 18 pre-election: Vote granted by peer 6885de38238442d6a9e5f1eb9fb569ff I0312 20:10:12.293541 1279 leader_election.cc:258] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. I0312 20:10:12.293970 1464 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec: Snoozing failure detection for 0.022s (election complete) I0312 20:10:12.294955 1464 raft_consensus.cc:2507] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 17 FOLLOWER]: Leader pre-election won for term 18 I0312 20:10:12.295138 1464 raft_consensus.cc:436] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 17 FOLLOWER]: Starting leader election (detected failure of leader ded9b7f8677940fb91e1a8c8bceceaed) I0312 20:10:12.295322 1464 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec: Snoozing failure detection for 0.017s (starting election) I0312 20:10:12.295493 1464 raft_consensus.cc:2752] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 17 FOLLOWER]: Advancing to term 18 I0312 20:10:12.300182 1464 raft_consensus.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:12.301303 1464 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 18 election: Requesting vote from peer 6885de38238442d6a9e5f1eb9fb569ff I0312 20:10:12.303031 1220 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "f78a2351568c4a268ee1112681e22bec" candidate_term: 18 candidate_status { last_received { term: 17 index: 84 } } ignore_live_leader: false dest_uuid: "6885de38238442d6a9e5f1eb9fb569ff" I0312 20:10:12.303514 1220 raft_consensus.cc:2176] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 18 because replica is either leader or believes a valid leader to be alive. I0312 20:10:12.304460 1279 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 18 election: Vote denied by peer 6885de38238442d6a9e5f1eb9fb569ff. Message: Invalid argument: T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 18 because replica is either leader or believes a valid leader to be alive. I0312 20:10:12.304798 1464 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 18 election: Requesting vote from peer ded9b7f8677940fb91e1a8c8bceceaed W0312 20:10:12.311151 1278 leader_election.cc:287] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 18 pre-election: RPC error from VoteRequest() call to peer ded9b7f8677940fb91e1a8c8bceceaed: Timed out: RequestConsensusVote RPC to 127.0.53.1:35037 timed out after 0.021s (SENT) I0312 20:10:12.314818 1096 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "f78a2351568c4a268ee1112681e22bec" candidate_term: 18 candidate_status { last_received { term: 17 index: 84 } } ignore_live_leader: false dest_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" I0312 20:10:12.315317 1096 raft_consensus.cc:2176] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 17 LEADER]: Leader election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 18 because replica is either leader or believes a valid leader to be alive. I0312 20:10:12.319638 1278 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 18 election: Vote denied by peer ded9b7f8677940fb91e1a8c8bceceaed. Message: Invalid argument: T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 17 LEADER]: Leader election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 18 because replica is either leader or believes a valid leader to be alive. I0312 20:10:12.319869 1278 leader_election.cc:258] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. I0312 20:10:12.320436 1415 raft_consensus.cc:436] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0312 20:10:12.320631 1415 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec: Snoozing failure detection for 0.018s (starting election) I0312 20:10:12.320984 1415 raft_consensus.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:12.322051 1415 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 19 pre-election: Requesting pre-vote from peer 6885de38238442d6a9e5f1eb9fb569ff I0312 20:10:12.322409 1415 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 19 pre-election: Requesting pre-vote from peer ded9b7f8677940fb91e1a8c8bceceaed I0312 20:10:12.322801 1415 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec: Snoozing failure detection for 0.019s (election complete) I0312 20:10:12.322989 1415 raft_consensus.cc:2455] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority I0312 20:10:12.324647 1220 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "f78a2351568c4a268ee1112681e22bec" candidate_term: 19 candidate_status { last_received { term: 17 index: 84 } } ignore_live_leader: false dest_uuid: "6885de38238442d6a9e5f1eb9fb569ff" is_pre_election: true I0312 20:10:12.325095 1345 raft_consensus.cc:1026] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 18 FOLLOWER]: Rejecting Update request from peer ded9b7f8677940fb91e1a8c8bceceaed for earlier term 17. Current term is 18. Ops: [] I0312 20:10:12.325726 1220 raft_consensus.cc:2176] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 19 because replica is either leader or believes a valid leader to be alive. I0312 20:10:12.329581 1279 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 19 pre-election: Vote denied by peer 6885de38238442d6a9e5f1eb9fb569ff. Message: Invalid argument: T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 19 because replica is either leader or believes a valid leader to be alive. I0312 20:10:12.331130 1456 consensus_queue.cc:1028] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 }, Status: INVALID_TERM, Last received: 17.84, Next index: 85, Last known committed idx: 80, Time since last communication: 0.000s I0312 20:10:12.332188 1456 raft_consensus.cc:2747] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 17 LEADER]: Stepping down as leader of term 17 I0312 20:10:12.332376 1456 raft_consensus.cc:605] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 17 LEADER]: Becoming Follower/Learner. State: Replica: ded9b7f8677940fb91e1a8c8bceceaed, State: Running, Role: LEADER I0312 20:10:12.333406 1096 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "f78a2351568c4a268ee1112681e22bec" candidate_term: 19 candidate_status { last_received { term: 17 index: 84 } } ignore_live_leader: false dest_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" is_pre_election: true I0312 20:10:12.332784 1456 consensus_queue.cc:219] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 84, Committed index: 84, Last appended: 17.86, Last appended by leader: 86, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:12.334564 1456 consensus_peers.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed -> Peer 6885de38238442d6a9e5f1eb9fb569ff (127.0.53.2:51697): Closing peer: 6885de38238442d6a9e5f1eb9fb569ff I0312 20:10:12.334779 1456 consensus_peers.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed -> Peer f78a2351568c4a268ee1112681e22bec (127.0.53.3:43274): Closing peer: f78a2351568c4a268ee1112681e22bec I0312 20:10:12.334995 1456 raft_consensus.cc:2752] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 17 FOLLOWER]: Advancing to term 18 I0312 20:10:12.337636 1416 raft_consensus.cc:436] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 17 FOLLOWER]: Starting pre-election (detected failure of leader ded9b7f8677940fb91e1a8c8bceceaed) I0312 20:10:12.337870 1416 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff: Snoozing failure detection for 0.020s (starting election) I0312 20:10:12.338090 1416 raft_consensus.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:12.339067 1416 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 18 pre-election: Requesting pre-vote from peer f78a2351568c4a268ee1112681e22bec I0312 20:10:12.340636 1416 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 18 pre-election: Requesting pre-vote from peer ded9b7f8677940fb91e1a8c8bceceaed I0312 20:10:12.342106 1094 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "6885de38238442d6a9e5f1eb9fb569ff" candidate_term: 18 candidate_status { last_received { term: 17 index: 84 } } ignore_live_leader: false dest_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" is_pre_election: true W0312 20:10:12.342775 1278 leader_election.cc:287] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 19 pre-election: RPC error from VoteRequest() call to peer ded9b7f8677940fb91e1a8c8bceceaed: Timed out: RequestConsensusVote RPC to 127.0.53.1:35037 timed out after 0.018s (SENT) I0312 20:10:12.342815 1415 raft_consensus.cc:436] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0312 20:10:12.343014 1278 leader_election.cc:258] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. I0312 20:10:12.343057 1415 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec: Snoozing failure detection for 0.029s (starting election) I0312 20:10:12.343322 1415 raft_consensus.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:12.344297 1424 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec: Snoozing failure detection for 0.021s (election complete) I0312 20:10:12.344512 1424 raft_consensus.cc:2455] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority I0312 20:10:12.344794 1415 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 19 pre-election: Requesting pre-vote from peer 6885de38238442d6a9e5f1eb9fb569ff I0312 20:10:12.345170 1415 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 19 pre-election: Requesting pre-vote from peer ded9b7f8677940fb91e1a8c8bceceaed I0312 20:10:12.346388 1219 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "f78a2351568c4a268ee1112681e22bec" candidate_term: 19 candidate_status { last_received { term: 17 index: 84 } } ignore_live_leader: false dest_uuid: "6885de38238442d6a9e5f1eb9fb569ff" is_pre_election: true I0312 20:10:12.347848 1095 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "f78a2351568c4a268ee1112681e22bec" candidate_term: 19 candidate_status { last_received { term: 17 index: 84 } } ignore_live_leader: false dest_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" is_pre_election: true I0312 20:10:12.348695 1345 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "6885de38238442d6a9e5f1eb9fb569ff" candidate_term: 18 candidate_status { last_received { term: 17 index: 84 } } ignore_live_leader: false dest_uuid: "f78a2351568c4a268ee1112681e22bec" is_pre_election: true I0312 20:10:12.349292 1345 raft_consensus.cc:2145] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6885de38238442d6a9e5f1eb9fb569ff in current term 18: Already voted for candidate f78a2351568c4a268ee1112681e22bec in this term. I0312 20:10:12.351109 1152 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 18 pre-election: Vote denied by peer f78a2351568c4a268ee1112681e22bec. Message: Invalid argument: T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6885de38238442d6a9e5f1eb9fb569ff in current term 18: Already voted for candidate f78a2351568c4a268ee1112681e22bec in this term. I0312 20:10:12.353878 1096 raft_consensus.cc:2162] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 19 because replica has last-logged OpId of term: 17 index: 86, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 84. I0312 20:10:12.353955 1219 raft_consensus.cc:2190] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 19 because replica is already servicing an update from a current leader or another vote. W0312 20:10:12.354347 1096 rpcz_store.cc:251] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.53.3:55789 (request call id 97) took 22ms (client timeout 18). I0312 20:10:12.354653 1095 raft_consensus.cc:2190] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 19 because replica is already servicing an update from a current leader or another vote. I0312 20:10:12.355481 1279 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 19 pre-election: Vote denied by peer 6885de38238442d6a9e5f1eb9fb569ff. Message: Service unavailable: T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 19 because replica is already servicing an update from a current leader or another vote. W0312 20:10:12.356015 1096 rpcz_store.cc:255] Trace: 0312 20:10:12.332060 (+ 0us) service_pool.cc:163] Inserting onto call queue 0312 20:10:12.333268 (+ 1208us) service_pool.cc:222] Handling call 0312 20:10:12.354216 (+ 20948us) spinlock_profiling.cc:261] Waited 19.8 ms on lock 0x7b480003c09c. stack: 00007f2232dd2c74 00007f2232dd2b70 00007f22318c2314 00007f223a90f294 00007f223a90f229 00007f22394b9af3 00007f223a94d56b 00007f2235300b24 00007f2235300ab1 00007f22353009c4 00007f223495cc29 00007f223495c6ec 00007f223495dbcd 00007f223495f946 00007f223495f89a 00007f223495f823 0312 20:10:12.354304 (+ 88us) inbound_call.cc:157] Queueing success response Metrics: {"spinlock_wait_cycles":45629312} I0312 20:10:12.356556 1417 raft_consensus.cc:436] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0312 20:10:12.356840 1417 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed: Snoozing failure detection for 0.028s (starting election) I0312 20:10:12.357142 1417 raft_consensus.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:12.357798 1278 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 19 pre-election: Vote denied by peer ded9b7f8677940fb91e1a8c8bceceaed. Message: Service unavailable: T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 19 because replica is already servicing an update from a current leader or another vote. I0312 20:10:12.358026 1278 leader_election.cc:258] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. I0312 20:10:12.358458 1464 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec: Snoozing failure detection for 0.034s (election complete) I0312 20:10:12.358716 1464 raft_consensus.cc:2455] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority I0312 20:10:12.359071 1094 raft_consensus.cc:2190] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6885de38238442d6a9e5f1eb9fb569ff for term 18 because replica is already servicing an update from a current leader or another vote. W0312 20:10:12.359520 1094 rpcz_store.cc:251] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.53.2:38849 (request call id 35) took 17ms (client timeout 20). W0312 20:10:12.360255 1094 rpcz_store.cc:255] Trace: 0312 20:10:12.341857 (+ 0us) service_pool.cc:163] Inserting onto call queue 0312 20:10:12.341962 (+ 105us) service_pool.cc:222] Handling call 0312 20:10:12.359393 (+ 17431us) spinlock_profiling.cc:261] Waited 16.5 ms on lock 0x7b480003c09c. stack: 00007f2232dd2c74 00007f2232dd2b70 00007f22318c2314 00007f223a90f294 00007f223a90f229 00007f22394b9606 00007f223a94d56b 00007f2235300b24 00007f2235300ab1 00007f22353009c4 00007f223495cc29 00007f223495c6ec 00007f223495dbcd 00007f223495f946 00007f223495f89a 00007f223495f823 0312 20:10:12.359476 (+ 83us) inbound_call.cc:157] Queueing success response Metrics: {"spinlock_wait_cycles":37865216} I0312 20:10:12.360584 1416 raft_consensus.cc:436] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 17 FOLLOWER]: Starting pre-election (detected failure of leader ded9b7f8677940fb91e1a8c8bceceaed) I0312 20:10:12.360822 1416 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff: Snoozing failure detection for 0.016s (starting election) I0312 20:10:12.361171 1153 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 18 pre-election: Vote denied by peer ded9b7f8677940fb91e1a8c8bceceaed. Message: Service unavailable: T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6885de38238442d6a9e5f1eb9fb569ff for term 18 because replica is already servicing an update from a current leader or another vote. I0312 20:10:12.361418 1153 leader_election.cc:258] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. I0312 20:10:12.361039 1416 raft_consensus.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:12.362418 1417 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 19 pre-election: Requesting pre-vote from peer f78a2351568c4a268ee1112681e22bec I0312 20:10:12.363499 1454 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff: Snoozing failure detection for 0.024s (election complete) I0312 20:10:12.363724 1454 raft_consensus.cc:2752] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 17 FOLLOWER]: Advancing to term 18 I0312 20:10:12.364199 1345 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" candidate_term: 19 candidate_status { last_received { term: 17 index: 86 } } ignore_live_leader: false dest_uuid: "f78a2351568c4a268ee1112681e22bec" is_pre_election: true I0312 20:10:12.364627 1345 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec: Snoozing failure detection for 0.024s (vote granted) I0312 20:10:12.364831 1345 raft_consensus.cc:2215] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ded9b7f8677940fb91e1a8c8bceceaed in term 18. I0312 20:10:12.365741 1027 leader_election.cc:363] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 19 pre-election: Vote granted by peer f78a2351568c4a268ee1112681e22bec I0312 20:10:12.365928 1027 leader_election.cc:258] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. I0312 20:10:12.366339 1456 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed: Snoozing failure detection for 0.033s (election complete) I0312 20:10:12.366675 1456 raft_consensus.cc:2507] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 18 FOLLOWER]: Leader pre-election won for term 19 I0312 20:10:12.366868 1456 raft_consensus.cc:436] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0312 20:10:12.367034 1456 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed: Snoozing failure detection for 0.029s (starting election) I0312 20:10:12.367194 1456 raft_consensus.cc:2752] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 18 FOLLOWER]: Advancing to term 19 I0312 20:10:12.368033 1454 raft_consensus.cc:2455] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority I0312 20:10:12.368502 1416 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 18 pre-election: Requesting pre-vote from peer f78a2351568c4a268ee1112681e22bec I0312 20:10:12.368937 1416 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 18 pre-election: Requesting pre-vote from peer ded9b7f8677940fb91e1a8c8bceceaed I0312 20:10:12.370379 1094 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "6885de38238442d6a9e5f1eb9fb569ff" candidate_term: 18 candidate_status { last_received { term: 17 index: 85 } } ignore_live_leader: false dest_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" is_pre_election: true I0312 20:10:12.371178 1456 raft_consensus.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:12.373138 1417 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 19 pre-election: Requesting pre-vote from peer 6885de38238442d6a9e5f1eb9fb569ff I0312 20:10:12.374047 1345 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "6885de38238442d6a9e5f1eb9fb569ff" candidate_term: 18 candidate_status { last_received { term: 17 index: 85 } } ignore_live_leader: false dest_uuid: "f78a2351568c4a268ee1112681e22bec" is_pre_election: true I0312 20:10:12.374621 1345 raft_consensus.cc:2145] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6885de38238442d6a9e5f1eb9fb569ff in current term 18: Already voted for candidate f78a2351568c4a268ee1112681e22bec in this term. I0312 20:10:12.376101 1220 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" candidate_term: 19 candidate_status { last_received { term: 17 index: 86 } } ignore_live_leader: false dest_uuid: "6885de38238442d6a9e5f1eb9fb569ff" is_pre_election: true I0312 20:10:12.376535 1220 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff: Snoozing failure detection for 0.025s (vote granted) I0312 20:10:12.376751 1220 raft_consensus.cc:2215] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ded9b7f8677940fb91e1a8c8bceceaed in term 18. I0312 20:10:12.377246 1094 raft_consensus.cc:2120] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6885de38238442d6a9e5f1eb9fb569ff for earlier term 18. Current term is 19. I0312 20:10:12.377431 1152 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 18 pre-election: Vote denied by peer f78a2351568c4a268ee1112681e22bec. Message: Invalid argument: T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6885de38238442d6a9e5f1eb9fb569ff in current term 18: Already voted for candidate f78a2351568c4a268ee1112681e22bec in this term. W0312 20:10:12.378610 1153 leader_election.cc:347] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 18 pre-election: Vote denied by peer ded9b7f8677940fb91e1a8c8bceceaed with higher term. Message: Invalid argument: T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6885de38238442d6a9e5f1eb9fb569ff for earlier term 18. Current term is 19. I0312 20:10:12.378796 1153 leader_election.cc:350] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 18 pre-election: Cancelling election due to peer responding with higher term I0312 20:10:12.379252 1416 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff: Snoozing failure detection for 0.016s (election complete) I0312 20:10:12.379503 1416 raft_consensus.cc:2752] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 18 FOLLOWER]: Advancing to term 19 I0312 20:10:12.379586 1029 leader_election.cc:363] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 19 pre-election: Vote granted by peer 6885de38238442d6a9e5f1eb9fb569ff I0312 20:10:12.388430 1456 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 19 election: Requesting vote from peer f78a2351568c4a268ee1112681e22bec I0312 20:10:12.389469 1464 raft_consensus.cc:436] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0312 20:10:12.389780 1464 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec: Snoozing failure detection for 0.050s (starting election) I0312 20:10:12.390046 1345 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" candidate_term: 19 candidate_status { last_received { term: 17 index: 86 } } ignore_live_leader: false dest_uuid: "f78a2351568c4a268ee1112681e22bec" I0312 20:10:12.390084 1464 raft_consensus.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:12.390631 1456 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 19 election: Requesting vote from peer 6885de38238442d6a9e5f1eb9fb569ff I0312 20:10:12.391440 1345 raft_consensus.cc:2752] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 18 FOLLOWER]: Advancing to term 19 I0312 20:10:12.391687 1345 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec: Snoozing failure detection for 0.058s (vote granted) I0312 20:10:12.391866 1220 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" candidate_term: 19 candidate_status { last_received { term: 17 index: 86 } } ignore_live_leader: false dest_uuid: "6885de38238442d6a9e5f1eb9fb569ff" I0312 20:10:12.396924 1417 raft_consensus.cc:436] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0312 20:10:12.397191 1417 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed: Snoozing failure detection for 0.025s (starting election) I0312 20:10:12.397266 1345 raft_consensus.cc:2215] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ded9b7f8677940fb91e1a8c8bceceaed in term 19. I0312 20:10:12.397433 1417 raft_consensus.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:12.398453 1464 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 19 pre-election: Requesting pre-vote from peer 6885de38238442d6a9e5f1eb9fb569ff I0312 20:10:12.398564 1027 leader_election.cc:363] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 19 election: Vote granted by peer f78a2351568c4a268ee1112681e22bec I0312 20:10:12.398813 1027 leader_election.cc:258] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. I0312 20:10:12.399541 1417 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 20 pre-election: Requesting pre-vote from peer f78a2351568c4a268ee1112681e22bec I0312 20:10:12.399871 1219 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "f78a2351568c4a268ee1112681e22bec" candidate_term: 19 candidate_status { last_received { term: 17 index: 84 } } ignore_live_leader: false dest_uuid: "6885de38238442d6a9e5f1eb9fb569ff" is_pre_election: true I0312 20:10:12.400462 1464 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 19 pre-election: Requesting pre-vote from peer ded9b7f8677940fb91e1a8c8bceceaed I0312 20:10:12.400627 1417 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 20 pre-election: Requesting pre-vote from peer 6885de38238442d6a9e5f1eb9fb569ff I0312 20:10:12.401039 1417 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed: Snoozing failure detection for 0.016s (election complete) I0312 20:10:12.401273 1417 raft_consensus.cc:2507] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 19 FOLLOWER]: Leader election won for term 19 I0312 20:10:12.401895 1094 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "f78a2351568c4a268ee1112681e22bec" candidate_term: 19 candidate_status { last_received { term: 17 index: 84 } } ignore_live_leader: false dest_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" is_pre_election: true I0312 20:10:12.401532 1417 raft_consensus.cc:572] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 19 LEADER]: Becoming Leader. State: Replica: ded9b7f8677940fb91e1a8c8bceceaed, State: Running, Role: LEADER I0312 20:10:12.402884 1417 consensus_queue.cc:196] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 84, Committed index: 84, Last appended: 17.86, Last appended by leader: 86, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:12.404132 1345 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" candidate_term: 20 candidate_status { last_received { term: 17 index: 86 } } ignore_live_leader: false dest_uuid: "f78a2351568c4a268ee1112681e22bec" is_pre_election: true I0312 20:10:12.404588 1345 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec: Snoozing failure detection for 0.057s (vote granted) I0312 20:10:12.404871 1345 raft_consensus.cc:2215] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ded9b7f8677940fb91e1a8c8bceceaed in term 19. I0312 20:10:12.405761 1027 leader_election.cc:363] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 20 pre-election: Vote granted by peer f78a2351568c4a268ee1112681e22bec I0312 20:10:12.405966 1027 leader_election.cc:258] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. I0312 20:10:12.406414 1221 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" candidate_term: 20 candidate_status { last_received { term: 17 index: 86 } } ignore_live_leader: false dest_uuid: "6885de38238442d6a9e5f1eb9fb569ff" is_pre_election: true I0312 20:10:12.409641 1094 raft_consensus.cc:2176] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 19 LEADER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 19 because replica is either leader or believes a valid leader to be alive. I0312 20:10:12.410284 1345 raft_consensus.cc:1058] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 19 FOLLOWER]: Refusing update from remote peer ded9b7f8677940fb91e1a8c8bceceaed: Log matching property violated. Preceding OpId in replica: term: 17 index: 84. Preceding OpId from leader: term: 17 index: 86. (index mismatch) I0312 20:10:12.410730 1278 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 19 pre-election: Vote denied by peer ded9b7f8677940fb91e1a8c8bceceaed. Message: Invalid argument: T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 19 LEADER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 19 because replica is either leader or believes a valid leader to be alive. I0312 20:10:12.411494 1417 consensus_queue.cc:1017] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [LEADER]: Connected to new peer: Peer: permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 }, Status: LMP_MISMATCH, Last received: 17.84, Next index: 85, Last known committed idx: 80, Time since last communication: 0.000s I0312 20:10:12.412099 974 catalog_manager.cc:3592] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed reported cstate change: term changed from 17 to 19. New cstate: current_term: 19 leader_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } health_report { overall_health: HEALTHY } } } I0312 20:10:12.417289 1416 raft_consensus.cc:2455] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 19 FOLLOWER]: Leader pre-election lost for term 18. Reason: Vote denied by peer ded9b7f8677940fb91e1a8c8bceceaed with higher term. Message: Invalid argument: T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6885de38238442d6a9e5f1eb9fb569ff for earlier term 18. Current term is 19. I0312 20:10:12.417610 1220 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff: Snoozing failure detection for 0.073s (vote granted) I0312 20:10:12.419265 1418 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed: Snoozing failure detection for 0.015s (election complete) W0312 20:10:12.420480 1029 leader_election.cc:287] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 19 election: RPC error from VoteRequest() call to peer 6885de38238442d6a9e5f1eb9fb569ff: Timed out: RequestConsensusVote RPC to 127.0.53.2:51697 timed out after 0.029s (SENT) W0312 20:10:12.427829 1029 leader_election.cc:287] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 20 pre-election: RPC error from VoteRequest() call to peer 6885de38238442d6a9e5f1eb9fb569ff: Timed out: RequestConsensusVote RPC to 127.0.53.2:51697 timed out after 0.025s (SENT) I0312 20:10:12.437988 1464 raft_consensus.cc:436] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 19 FOLLOWER]: Starting pre-election (detected failure of leader ded9b7f8677940fb91e1a8c8bceceaed) I0312 20:10:12.438256 1464 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec: Snoozing failure detection for 0.019s (starting election) I0312 20:10:12.438509 1464 raft_consensus.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:12.447625 1464 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 20 pre-election: Requesting pre-vote from peer 6885de38238442d6a9e5f1eb9fb569ff I0312 20:10:12.449705 1217 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "f78a2351568c4a268ee1112681e22bec" candidate_term: 20 candidate_status { last_received { term: 19 index: 87 } } ignore_live_leader: false dest_uuid: "6885de38238442d6a9e5f1eb9fb569ff" is_pre_election: true W0312 20:10:12.450359 1279 leader_election.cc:287] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 19 pre-election: RPC error from VoteRequest() call to peer 6885de38238442d6a9e5f1eb9fb569ff: Timed out: RequestConsensusVote RPC to 127.0.53.2:51697 timed out after 0.050s (SENT) I0312 20:10:12.450577 1279 leader_election.cc:258] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. I0312 20:10:12.451177 1415 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec: Snoozing failure detection for 0.019s (election complete) I0312 20:10:12.451382 1415 raft_consensus.cc:2455] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority I0312 20:10:12.451707 1464 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 20 pre-election: Requesting pre-vote from peer ded9b7f8677940fb91e1a8c8bceceaed I0312 20:10:12.453510 1094 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "f78a2351568c4a268ee1112681e22bec" candidate_term: 20 candidate_status { last_received { term: 19 index: 87 } } ignore_live_leader: false dest_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" is_pre_election: true I0312 20:10:12.454041 1094 raft_consensus.cc:2176] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 19 LEADER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 20 because replica is either leader or believes a valid leader to be alive. I0312 20:10:12.455423 1278 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 20 pre-election: Vote denied by peer ded9b7f8677940fb91e1a8c8bceceaed. Message: Invalid argument: T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 19 LEADER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 20 because replica is either leader or believes a valid leader to be alive. W0312 20:10:12.467737 1279 leader_election.cc:287] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 20 pre-election: RPC error from VoteRequest() call to peer 6885de38238442d6a9e5f1eb9fb569ff: Timed out: RequestConsensusVote RPC to 127.0.53.2:51697 timed out after 0.019s (SENT) I0312 20:10:12.467957 1279 leader_election.cc:258] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. I0312 20:10:12.468569 1464 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec: Snoozing failure detection for 0.026s (election complete) I0312 20:10:12.468844 1464 raft_consensus.cc:2455] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority I0312 20:10:12.506953 1464 raft_consensus.cc:436] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 19 FOLLOWER]: Starting pre-election (detected failure of leader ded9b7f8677940fb91e1a8c8bceceaed) I0312 20:10:12.507608 1464 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec: Snoozing failure detection for 0.030s (starting election) I0312 20:10:12.509268 1464 raft_consensus.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:12.510469 1464 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 20 pre-election: Requesting pre-vote from peer 6885de38238442d6a9e5f1eb9fb569ff I0312 20:10:12.513404 1464 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 20 pre-election: Requesting pre-vote from peer ded9b7f8677940fb91e1a8c8bceceaed I0312 20:10:12.514559 1216 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "f78a2351568c4a268ee1112681e22bec" candidate_term: 20 candidate_status { last_received { term: 19 index: 89 } } ignore_live_leader: false dest_uuid: "6885de38238442d6a9e5f1eb9fb569ff" is_pre_election: true I0312 20:10:12.514670 1094 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "f78a2351568c4a268ee1112681e22bec" candidate_term: 20 candidate_status { last_received { term: 19 index: 89 } } ignore_live_leader: false dest_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" is_pre_election: true I0312 20:10:12.515193 1094 raft_consensus.cc:2176] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 19 LEADER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 20 because replica is either leader or believes a valid leader to be alive. I0312 20:10:12.517192 1278 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 20 pre-election: Vote denied by peer ded9b7f8677940fb91e1a8c8bceceaed. Message: Invalid argument: T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 19 LEADER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 20 because replica is either leader or believes a valid leader to be alive. I0312 20:10:12.532603 1464 raft_consensus.cc:436] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 19 FOLLOWER]: Starting pre-election (detected failure of leader ded9b7f8677940fb91e1a8c8bceceaed) I0312 20:10:12.532832 1464 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec: Snoozing failure detection for 0.029s (starting election) I0312 20:10:12.533053 1464 raft_consensus.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:12.534087 1464 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 20 pre-election: Requesting pre-vote from peer 6885de38238442d6a9e5f1eb9fb569ff I0312 20:10:12.534459 1464 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 20 pre-election: Requesting pre-vote from peer ded9b7f8677940fb91e1a8c8bceceaed I0312 20:10:12.535950 1215 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "f78a2351568c4a268ee1112681e22bec" candidate_term: 20 candidate_status { last_received { term: 19 index: 91 } } ignore_live_leader: false dest_uuid: "6885de38238442d6a9e5f1eb9fb569ff" is_pre_election: true W0312 20:10:12.544744 1279 leader_election.cc:287] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 20 pre-election: RPC error from VoteRequest() call to peer 6885de38238442d6a9e5f1eb9fb569ff: Timed out: RequestConsensusVote RPC to 127.0.53.2:51697 timed out after 0.030s (SENT) I0312 20:10:12.544968 1279 leader_election.cc:258] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. I0312 20:10:12.546213 1464 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec: Snoozing failure detection for 0.041s (election complete) I0312 20:10:12.546427 1464 raft_consensus.cc:2455] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority I0312 20:10:12.549106 1094 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "f78a2351568c4a268ee1112681e22bec" candidate_term: 20 candidate_status { last_received { term: 19 index: 91 } } ignore_live_leader: false dest_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" is_pre_election: true I0312 20:10:12.550276 1094 raft_consensus.cc:2176] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 19 LEADER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 20 because replica is either leader or believes a valid leader to be alive. I0312 20:10:12.555662 1278 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 20 pre-election: Vote denied by peer ded9b7f8677940fb91e1a8c8bceceaed. Message: Invalid argument: T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 19 LEADER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 20 because replica is either leader or believes a valid leader to be alive. I0312 20:10:12.555928 1220 raft_consensus.cc:2215] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ded9b7f8677940fb91e1a8c8bceceaed in term 19. W0312 20:10:12.556591 1220 rpcz_store.cc:251] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.53.1:43660 (request call id 75) took 165ms (client timeout 29). I0312 20:10:12.558174 1219 raft_consensus.cc:2190] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 19 because replica is already servicing an update from a current leader or another vote. W0312 20:10:12.558643 1219 rpcz_store.cc:251] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.53.3:38196 (request call id 71) took 159ms (client timeout 50). I0312 20:10:12.559875 1454 raft_consensus.cc:436] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0312 20:10:12.560120 1454 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff: Snoozing failure detection for 0.042s (starting election) I0312 20:10:12.560360 1454 raft_consensus.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:12.561525 1454 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 20 pre-election: Requesting pre-vote from peer f78a2351568c4a268ee1112681e22bec I0312 20:10:12.561888 1454 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 20 pre-election: Requesting pre-vote from peer ded9b7f8677940fb91e1a8c8bceceaed W0312 20:10:12.565053 1279 leader_election.cc:287] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 20 pre-election: RPC error from VoteRequest() call to peer 6885de38238442d6a9e5f1eb9fb569ff: Timed out: RequestConsensusVote RPC to 127.0.53.2:51697 timed out after 0.029s (SENT) I0312 20:10:12.565357 1279 leader_election.cc:258] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. I0312 20:10:12.568012 1464 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec: Snoozing failure detection for 0.069s (election complete) I0312 20:10:12.569454 1464 raft_consensus.cc:2455] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority I0312 20:10:12.563356 1221 raft_consensus.cc:2190] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ded9b7f8677940fb91e1a8c8bceceaed for term 20 because replica is already servicing an update from a current leader or another vote. W0312 20:10:12.577260 1221 rpcz_store.cc:251] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.53.1:43660 (request call id 76) took 171ms (client timeout 25). W0312 20:10:12.578811 1221 rpcz_store.cc:255] Trace: 0312 20:10:12.406180 (+ 0us) service_pool.cc:163] Inserting onto call queue 0312 20:10:12.406268 (+ 88us) service_pool.cc:222] Handling call 0312 20:10:12.577099 (+170831us) spinlock_profiling.cc:261] Waited 156 ms on lock 0x7b480003009c. stack: 00007efc5efb2c74 00007efc5efb2b70 00007efc5daa2314 00007efc66aef294 00007efc66aef229 00007efc65699606 00007efc66b2d56b 00007efc614e0b24 00007efc614e0ab1 00007efc614e09c4 00007efc60b3cc29 00007efc60b3c6ec 00007efc60b3dbcd 00007efc60b3f946 00007efc60b3f89a 00007efc60b3f823 0312 20:10:12.577205 (+ 106us) inbound_call.cc:157] Queueing success response Metrics: {"spinlock_wait_cycles":359749376} I0312 20:10:12.579288 1217 raft_consensus.cc:2190] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 20 because replica is already servicing an update from a current leader or another vote. W0312 20:10:12.579800 1217 rpcz_store.cc:251] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.53.3:38196 (request call id 72) took 131ms (client timeout 19). W0312 20:10:12.580422 1217 rpcz_store.cc:255] Trace: 0312 20:10:12.448710 (+ 0us) service_pool.cc:163] Inserting onto call queue 0312 20:10:12.448818 (+ 108us) service_pool.cc:222] Handling call 0312 20:10:12.579664 (+130846us) spinlock_profiling.cc:261] Waited 129 ms on lock 0x7b480003009c. stack: 00007efc5efb2c74 00007efc5efb2b70 00007efc5daa2314 00007efc66aef294 00007efc66aef229 00007efc65699606 00007efc66b2d56b 00007efc614e0b24 00007efc614e0ab1 00007efc614e09c4 00007efc60b3cc29 00007efc60b3c6ec 00007efc60b3dbcd 00007efc60b3f946 00007efc60b3f89a 00007efc60b3f823 0312 20:10:12.579753 (+ 89us) inbound_call.cc:157] Queueing success response Metrics: {"spinlock_wait_cycles":296874240} I0312 20:10:12.580914 1218 raft_consensus.cc:1058] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 19 FOLLOWER]: Refusing update from remote peer ded9b7f8677940fb91e1a8c8bceceaed: Log matching property violated. Preceding OpId in replica: term: 17 index: 85. Preceding OpId from leader: term: 19 index: 87. (index mismatch) I0312 20:10:12.581910 1215 raft_consensus.cc:2190] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 20 because replica is already servicing an update from a current leader or another vote. W0312 20:10:12.582370 1215 rpcz_store.cc:251] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.53.3:38196 (request call id 74) took 46ms (client timeout 29). W0312 20:10:12.583010 1215 rpcz_store.cc:255] Trace: 0312 20:10:12.535634 (+ 0us) service_pool.cc:163] Inserting onto call queue 0312 20:10:12.535730 (+ 96us) service_pool.cc:222] Handling call 0312 20:10:12.582246 (+ 46516us) spinlock_profiling.cc:261] Waited 45.4 ms on lock 0x7b480003009c. stack: 00007efc5efb2c74 00007efc5efb2b70 00007efc5daa2314 00007efc66aef294 00007efc66aef229 00007efc65699606 00007efc66b2d56b 00007efc614e0b24 00007efc614e0ab1 00007efc614e09c4 00007efc60b3cc29 00007efc60b3c6ec 00007efc60b3dbcd 00007efc60b3f946 00007efc60b3f89a 00007efc60b3f823 0312 20:10:12.582325 (+ 79us) inbound_call.cc:157] Queueing success response Metrics: {"spinlock_wait_cycles":104426880} I0312 20:10:12.583387 1216 raft_consensus.cc:2190] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 20 because replica is already servicing an update from a current leader or another vote. W0312 20:10:12.583822 1216 rpcz_store.cc:251] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.53.3:38196 (request call id 73) took 69ms (client timeout 30). W0312 20:10:12.584404 1216 rpcz_store.cc:255] Trace: 0312 20:10:12.514222 (+ 0us) service_pool.cc:163] Inserting onto call queue 0312 20:10:12.514335 (+ 113us) service_pool.cc:222] Handling call 0312 20:10:12.583697 (+ 69362us) spinlock_profiling.cc:261] Waited 68.3 ms on lock 0x7b480003009c. stack: 00007efc5efb2c74 00007efc5efb2b70 00007efc5daa2314 00007efc66aef294 00007efc66aef229 00007efc65699606 00007efc66b2d56b 00007efc614e0b24 00007efc614e0ab1 00007efc614e09c4 00007efc60b3cc29 00007efc60b3c6ec 00007efc60b3dbcd 00007efc60b3f946 00007efc60b3f89a 00007efc60b3f823 0312 20:10:12.583778 (+ 81us) inbound_call.cc:157] Queueing success response Metrics: {"spinlock_wait_cycles":157083008} I0312 20:10:12.586130 1094 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "6885de38238442d6a9e5f1eb9fb569ff" candidate_term: 20 candidate_status { last_received { term: 17 index: 85 } } ignore_live_leader: false dest_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" is_pre_election: true I0312 20:10:12.586664 1094 raft_consensus.cc:2176] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 19 LEADER]: Leader pre-election vote request: Denying vote to candidate 6885de38238442d6a9e5f1eb9fb569ff for term 20 because replica is either leader or believes a valid leader to be alive. I0312 20:10:12.587620 1153 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 20 pre-election: Vote denied by peer ded9b7f8677940fb91e1a8c8bceceaed. Message: Invalid argument: T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 19 LEADER]: Leader pre-election vote request: Denying vote to candidate 6885de38238442d6a9e5f1eb9fb569ff for term 20 because replica is either leader or believes a valid leader to be alive. W0312 20:10:12.559285 1219 rpcz_store.cc:255] Trace: 0312 20:10:12.399600 (+ 0us) service_pool.cc:163] Inserting onto call queue 0312 20:10:12.399722 (+ 122us) service_pool.cc:222] Handling call 0312 20:10:12.558510 (+158788us) spinlock_profiling.cc:261] Waited 157 ms on lock 0x7b480003009c. stack: 00007efc5efb2c74 00007efc5efb2b70 00007efc5daa2314 00007efc66aef294 00007efc66aef229 00007efc65699606 00007efc66b2d56b 00007efc614e0b24 00007efc614e0ab1 00007efc614e09c4 00007efc60b3cc29 00007efc60b3c6ec 00007efc60b3dbcd 00007efc60b3f946 00007efc60b3f89a 00007efc60b3f823 0312 20:10:12.558596 (+ 86us) inbound_call.cc:157] Queueing success response Metrics: {"spinlock_wait_cycles":360875264} I0312 20:10:12.589846 1464 raft_consensus.cc:436] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 19 FOLLOWER]: Starting pre-election (detected failure of leader ded9b7f8677940fb91e1a8c8bceceaed) I0312 20:10:12.590117 1464 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec: Snoozing failure detection for 0.101s (starting election) I0312 20:10:12.590354 1464 raft_consensus.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:12.591393 1464 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 20 pre-election: Requesting pre-vote from peer 6885de38238442d6a9e5f1eb9fb569ff I0312 20:10:12.591760 1464 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 20 pre-election: Requesting pre-vote from peer ded9b7f8677940fb91e1a8c8bceceaed W0312 20:10:12.564766 1220 rpcz_store.cc:255] Trace: 0312 20:10:12.391627 (+ 0us) service_pool.cc:163] Inserting onto call queue 0312 20:10:12.391724 (+ 97us) service_pool.cc:222] Handling call 0312 20:10:12.556452 (+164728us) spinlock_profiling.cc:261] Waited 25.3 ms on lock 0x7b480003009c. stack: 00007efc5efb2c74 00007efc5efb2b70 00007efc5daa2314 00007efc66aef294 00007efc66aef229 00007efc65699af3 00007efc66b2d56b 00007efc614e0b24 00007efc614e0ab1 00007efc614e09c4 00007efc60b3cc29 00007efc60b3c6ec 00007efc60b3dbcd 00007efc60b3f946 00007efc60b3f89a 00007efc60b3f823 0312 20:10:12.556540 (+ 88us) inbound_call.cc:157] Queueing success response Metrics: {"spinlock_wait_cycles":58122496} I0312 20:10:12.595469 1220 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "f78a2351568c4a268ee1112681e22bec" candidate_term: 20 candidate_status { last_received { term: 19 index: 93 } } ignore_live_leader: false dest_uuid: "6885de38238442d6a9e5f1eb9fb569ff" is_pre_election: true I0312 20:10:12.595919 1220 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff: Snoozing failure detection for 0.027s (vote granted) I0312 20:10:12.596133 1220 raft_consensus.cc:2215] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f78a2351568c4a268ee1112681e22bec in term 19. I0312 20:10:12.597312 1418 consensus_queue.cc:1017] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [LEADER]: Connected to new peer: Peer: permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 }, Status: LMP_MISMATCH, Last received: 17.85, Next index: 86, Last known committed idx: 84, Time since last communication: 0.000s I0312 20:10:12.597733 1279 leader_election.cc:363] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 20 pre-election: Vote granted by peer 6885de38238442d6a9e5f1eb9fb569ff I0312 20:10:12.597918 1279 leader_election.cc:258] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. I0312 20:10:12.598382 1464 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec: Snoozing failure detection for 0.057s (election complete) I0312 20:10:12.599808 1464 raft_consensus.cc:2507] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 19 FOLLOWER]: Leader pre-election won for term 20 I0312 20:10:12.600005 1464 raft_consensus.cc:436] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 19 FOLLOWER]: Starting leader election (detected failure of leader ded9b7f8677940fb91e1a8c8bceceaed) I0312 20:10:12.600204 1464 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec: Snoozing failure detection for 0.074s (starting election) I0312 20:10:12.600389 1464 raft_consensus.cc:2752] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 19 FOLLOWER]: Advancing to term 20 I0312 20:10:12.601034 1094 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "f78a2351568c4a268ee1112681e22bec" candidate_term: 20 candidate_status { last_received { term: 19 index: 93 } } ignore_live_leader: false dest_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" is_pre_election: true I0312 20:10:12.601547 1094 raft_consensus.cc:2176] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 19 LEADER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 20 because replica is either leader or believes a valid leader to be alive. I0312 20:10:12.603443 1278 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 20 pre-election: Vote denied by peer ded9b7f8677940fb91e1a8c8bceceaed. Message: Invalid argument: T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 19 LEADER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 20 because replica is either leader or believes a valid leader to be alive. I0312 20:10:12.605504 1464 raft_consensus.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } W0312 20:10:12.606217 1152 leader_election.cc:287] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 20 pre-election: RPC error from VoteRequest() call to peer f78a2351568c4a268ee1112681e22bec: Timed out: RequestConsensusVote RPC to 127.0.53.3:43274 timed out after 0.042s (SENT) I0312 20:10:12.606451 1152 leader_election.cc:258] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. I0312 20:10:12.609619 1344 raft_consensus.cc:1026] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 20 FOLLOWER]: Rejecting Update request from peer ded9b7f8677940fb91e1a8c8bceceaed for earlier term 19. Current term is 20. Ops: [] I0312 20:10:12.611312 1456 consensus_queue.cc:1028] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 }, Status: INVALID_TERM, Last received: 19.93, Next index: 94, Last known committed idx: 93, Time since last communication: 0.000s W0312 20:10:12.611544 1345 rpcz_store.cc:251] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.53.2:37068 (request call id 36) took 47ms (client timeout 42). I0312 20:10:12.611959 1418 raft_consensus.cc:2747] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 19 LEADER]: Stepping down as leader of term 19 I0312 20:10:12.612164 1418 raft_consensus.cc:605] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 19 LEADER]: Becoming Follower/Learner. State: Replica: ded9b7f8677940fb91e1a8c8bceceaed, State: Running, Role: LEADER I0312 20:10:12.612599 1418 consensus_queue.cc:219] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 93, Committed index: 93, Last appended: 19.95, Last appended by leader: 95, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:12.613242 1418 consensus_peers.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed -> Peer 6885de38238442d6a9e5f1eb9fb569ff (127.0.53.2:51697): Closing peer: 6885de38238442d6a9e5f1eb9fb569ff W0312 20:10:12.613421 1345 rpcz_store.cc:255] Trace: 0312 20:10:12.564418 (+ 0us) service_pool.cc:163] Inserting onto call queue 0312 20:10:12.611415 (+ 46997us) service_pool.cc:207] Skipping call since client already timed out 0312 20:10:12.611498 (+ 83us) inbound_call.cc:157] Queueing failure response Metrics: {} I0312 20:10:12.613420 1418 consensus_peers.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed -> Peer f78a2351568c4a268ee1112681e22bec (127.0.53.3:43274): Closing peer: f78a2351568c4a268ee1112681e22bec I0312 20:10:12.613595 1418 raft_consensus.cc:2752] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 19 FOLLOWER]: Advancing to term 20 I0312 20:10:12.618582 1464 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 20 election: Requesting vote from peer 6885de38238442d6a9e5f1eb9fb569ff I0312 20:10:12.619992 1219 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "f78a2351568c4a268ee1112681e22bec" candidate_term: 20 candidate_status { last_received { term: 19 index: 93 } } ignore_live_leader: false dest_uuid: "6885de38238442d6a9e5f1eb9fb569ff" I0312 20:10:12.620622 1464 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 20 election: Requesting vote from peer ded9b7f8677940fb91e1a8c8bceceaed I0312 20:10:12.621875 1094 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "f78a2351568c4a268ee1112681e22bec" candidate_term: 20 candidate_status { last_received { term: 19 index: 93 } } ignore_live_leader: false dest_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" I0312 20:10:12.622503 1094 raft_consensus.cc:2162] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 20 because replica has last-logged OpId of term: 19 index: 95, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 93. I0312 20:10:12.623569 1278 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 20 election: Vote denied by peer ded9b7f8677940fb91e1a8c8bceceaed. Message: Invalid argument: T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 20 because replica has last-logged OpId of term: 19 index: 95, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 93. I0312 20:10:12.629869 1418 raft_consensus.cc:436] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0312 20:10:12.630100 1418 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed: Snoozing failure detection for 0.015s (starting election) I0312 20:10:12.630316 1418 raft_consensus.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:12.631335 1418 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 21 pre-election: Requesting pre-vote from peer f78a2351568c4a268ee1112681e22bec I0312 20:10:12.631706 1418 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 21 pre-election: Requesting pre-vote from peer 6885de38238442d6a9e5f1eb9fb569ff I0312 20:10:12.633952 1216 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" candidate_term: 21 candidate_status { last_received { term: 19 index: 95 } } ignore_live_leader: false dest_uuid: "6885de38238442d6a9e5f1eb9fb569ff" is_pre_election: true I0312 20:10:12.634625 1345 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" candidate_term: 21 candidate_status { last_received { term: 19 index: 95 } } ignore_live_leader: false dest_uuid: "f78a2351568c4a268ee1112681e22bec" is_pre_election: true I0312 20:10:12.635169 1345 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec: Snoozing failure detection for 0.022s (vote granted) I0312 20:10:12.638387 1345 raft_consensus.cc:2215] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ded9b7f8677940fb91e1a8c8bceceaed in term 20. I0312 20:10:12.639351 1027 leader_election.cc:363] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 21 pre-election: Vote granted by peer f78a2351568c4a268ee1112681e22bec I0312 20:10:12.639535 1027 leader_election.cc:258] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. I0312 20:10:12.640372 1418 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed: Snoozing failure detection for 0.016s (election complete) I0312 20:10:12.640606 1418 raft_consensus.cc:2507] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 20 FOLLOWER]: Leader pre-election won for term 21 I0312 20:10:12.640776 1418 raft_consensus.cc:436] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0312 20:10:12.640972 1418 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed: Snoozing failure detection for 0.018s (starting election) I0312 20:10:12.641160 1418 raft_consensus.cc:2752] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 20 FOLLOWER]: Advancing to term 21 W0312 20:10:12.648344 1029 leader_election.cc:287] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 21 pre-election: RPC error from VoteRequest() call to peer 6885de38238442d6a9e5f1eb9fb569ff: Timed out: RequestConsensusVote RPC to 127.0.53.2:51697 timed out after 0.015s (SENT) I0312 20:10:12.658993 1219 raft_consensus.cc:2190] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 20 because replica is already servicing an update from a current leader or another vote. I0312 20:10:12.659670 1454 raft_consensus.cc:436] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 19 FOLLOWER]: Starting pre-election (detected failure of leader ded9b7f8677940fb91e1a8c8bceceaed) I0312 20:10:12.659904 1454 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff: Snoozing failure detection for 0.015s (starting election) I0312 20:10:12.660825 1279 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 20 election: Vote denied by peer 6885de38238442d6a9e5f1eb9fb569ff. Message: Service unavailable: T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 20 because replica is already servicing an update from a current leader or another vote. I0312 20:10:12.661057 1279 leader_election.cc:258] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. I0312 20:10:12.661166 1464 raft_consensus.cc:436] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0312 20:10:12.661411 1464 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec: Snoozing failure detection for 0.029s (starting election) I0312 20:10:12.660156 1454 raft_consensus.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:12.661653 1464 raft_consensus.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:12.662737 1464 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 21 pre-election: Requesting pre-vote from peer 6885de38238442d6a9e5f1eb9fb569ff I0312 20:10:12.662801 1454 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 20 pre-election: Requesting pre-vote from peer f78a2351568c4a268ee1112681e22bec I0312 20:10:12.662737 1415 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec: Snoozing failure detection for 0.039s (election complete) I0312 20:10:12.662994 1415 raft_consensus.cc:2455] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority I0312 20:10:12.663169 1454 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 20 pre-election: Requesting pre-vote from peer ded9b7f8677940fb91e1a8c8bceceaed I0312 20:10:12.663245 1464 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 21 pre-election: Requesting pre-vote from peer ded9b7f8677940fb91e1a8c8bceceaed I0312 20:10:12.663662 1216 raft_consensus.cc:2190] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ded9b7f8677940fb91e1a8c8bceceaed for term 21 because replica is already servicing an update from a current leader or another vote. W0312 20:10:12.664141 1216 rpcz_store.cc:251] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.53.1:43660 (request call id 79) took 30ms (client timeout 15). W0312 20:10:12.665911 1216 rpcz_store.cc:255] Trace: 0312 20:10:12.633697 (+ 0us) service_pool.cc:163] Inserting onto call queue 0312 20:10:12.633800 (+ 103us) service_pool.cc:222] Handling call 0312 20:10:12.663996 (+ 30196us) spinlock_profiling.cc:261] Waited 29.2 ms on lock 0x7b480003009c. stack: 00007efc5efb2c74 00007efc5efb2b70 00007efc5daa2314 00007efc66aef294 00007efc66aef229 00007efc65699606 00007efc66b2d56b 00007efc614e0b24 00007efc614e0ab1 00007efc614e09c4 00007efc60b3cc29 00007efc60b3c6ec 00007efc60b3dbcd 00007efc60b3f946 00007efc60b3f89a 00007efc60b3f823 0312 20:10:12.664094 (+ 98us) inbound_call.cc:157] Queueing success response Metrics: {"spinlock_wait_cycles":67145984} I0312 20:10:12.665766 1094 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "f78a2351568c4a268ee1112681e22bec" candidate_term: 21 candidate_status { last_received { term: 19 index: 93 } } ignore_live_leader: false dest_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" is_pre_election: true I0312 20:10:12.666612 1416 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff: Snoozing failure detection for 0.015s (election complete) I0312 20:10:12.666853 1416 raft_consensus.cc:2455] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority I0312 20:10:12.670801 1095 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "6885de38238442d6a9e5f1eb9fb569ff" candidate_term: 20 candidate_status { last_received { term: 19 index: 95 } } ignore_live_leader: false dest_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" is_pre_election: true I0312 20:10:12.675987 1216 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "f78a2351568c4a268ee1112681e22bec" candidate_term: 21 candidate_status { last_received { term: 19 index: 93 } } ignore_live_leader: false dest_uuid: "6885de38238442d6a9e5f1eb9fb569ff" is_pre_election: true I0312 20:10:12.676632 1216 raft_consensus.cc:2162] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 21 because replica has last-logged OpId of term: 19 index: 95, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 93. I0312 20:10:12.677696 1279 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 21 pre-election: Vote denied by peer 6885de38238442d6a9e5f1eb9fb569ff. Message: Invalid argument: T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 21 because replica has last-logged OpId of term: 19 index: 95, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 93. I0312 20:10:12.680390 1345 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "6885de38238442d6a9e5f1eb9fb569ff" candidate_term: 20 candidate_status { last_received { term: 19 index: 95 } } ignore_live_leader: false dest_uuid: "f78a2351568c4a268ee1112681e22bec" is_pre_election: true I0312 20:10:12.680974 1345 raft_consensus.cc:2145] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6885de38238442d6a9e5f1eb9fb569ff in current term 20: Already voted for candidate f78a2351568c4a268ee1112681e22bec in this term. W0312 20:10:12.681416 1153 leader_election.cc:287] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 20 pre-election: RPC error from VoteRequest() call to peer ded9b7f8677940fb91e1a8c8bceceaed: Timed out: RequestConsensusVote RPC to 127.0.53.1:35037 timed out after 0.015s (SENT) I0312 20:10:12.682297 1416 raft_consensus.cc:436] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 19 FOLLOWER]: Starting pre-election (detected failure of leader ded9b7f8677940fb91e1a8c8bceceaed) I0312 20:10:12.682639 1416 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff: Snoozing failure detection for 0.033s (starting election) I0312 20:10:12.682893 1416 raft_consensus.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:12.684002 1416 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 20 pre-election: Requesting pre-vote from peer f78a2351568c4a268ee1112681e22bec I0312 20:10:12.684367 1416 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 20 pre-election: Requesting pre-vote from peer ded9b7f8677940fb91e1a8c8bceceaed I0312 20:10:12.684522 1152 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 20 pre-election: Vote denied by peer f78a2351568c4a268ee1112681e22bec. Message: Invalid argument: T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6885de38238442d6a9e5f1eb9fb569ff in current term 20: Already voted for candidate f78a2351568c4a268ee1112681e22bec in this term. I0312 20:10:12.684834 1152 leader_election.cc:258] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. I0312 20:10:12.685276 1416 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff: Snoozing failure detection for 0.020s (election complete) I0312 20:10:12.685472 1416 raft_consensus.cc:2752] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 19 FOLLOWER]: Advancing to term 20 I0312 20:10:12.685691 1096 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "6885de38238442d6a9e5f1eb9fb569ff" candidate_term: 20 candidate_status { last_received { term: 19 index: 95 } } ignore_live_leader: false dest_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" is_pre_election: true I0312 20:10:12.690184 1416 raft_consensus.cc:2455] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority I0312 20:10:12.691550 1345 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "6885de38238442d6a9e5f1eb9fb569ff" candidate_term: 20 candidate_status { last_received { term: 19 index: 95 } } ignore_live_leader: false dest_uuid: "f78a2351568c4a268ee1112681e22bec" is_pre_election: true W0312 20:10:12.693433 1278 leader_election.cc:287] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 21 pre-election: RPC error from VoteRequest() call to peer ded9b7f8677940fb91e1a8c8bceceaed: Timed out: RequestConsensusVote RPC to 127.0.53.1:35037 timed out after 0.029s (SENT) I0312 20:10:12.693667 1278 leader_election.cc:258] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. I0312 20:10:12.694479 1345 raft_consensus.cc:2145] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6885de38238442d6a9e5f1eb9fb569ff in current term 20: Already voted for candidate f78a2351568c4a268ee1112681e22bec in this term. I0312 20:10:12.695384 1464 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec: Snoozing failure detection for 0.139s (election complete) I0312 20:10:12.695614 1464 raft_consensus.cc:2455] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority I0312 20:10:12.696538 1152 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 20 pre-election: Vote denied by peer f78a2351568c4a268ee1112681e22bec. Message: Invalid argument: T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6885de38238442d6a9e5f1eb9fb569ff in current term 20: Already voted for candidate f78a2351568c4a268ee1112681e22bec in this term. I0312 20:10:12.706660 1416 raft_consensus.cc:436] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0312 20:10:12.706929 1416 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff: Snoozing failure detection for 0.024s (starting election) I0312 20:10:12.707340 1416 raft_consensus.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:12.708458 1416 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 21 pre-election: Requesting pre-vote from peer f78a2351568c4a268ee1112681e22bec I0312 20:10:12.710083 1418 raft_consensus.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:12.711233 1418 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 21 election: Requesting vote from peer f78a2351568c4a268ee1112681e22bec I0312 20:10:12.711644 1418 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 21 election: Requesting vote from peer 6885de38238442d6a9e5f1eb9fb569ff I0312 20:10:12.711702 1096 raft_consensus.cc:2190] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6885de38238442d6a9e5f1eb9fb569ff for term 20 because replica is already servicing an update from a current leader or another vote. W0312 20:10:12.712813 1096 rpcz_store.cc:251] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.53.2:38849 (request call id 39) took 27ms (client timeout 33). W0312 20:10:12.714465 1096 rpcz_store.cc:255] Trace: 0312 20:10:12.685436 (+ 0us) service_pool.cc:163] Inserting onto call queue 0312 20:10:12.685544 (+ 108us) service_pool.cc:222] Handling call 0312 20:10:12.712674 (+ 27130us) spinlock_profiling.cc:261] Waited 25.5 ms on lock 0x7b480003c09c. stack: 00007f2232dd2c74 00007f2232dd2b70 00007f22318c2314 00007f223a90f294 00007f223a90f229 00007f22394b9606 00007f223a94d56b 00007f2235300b24 00007f2235300ab1 00007f22353009c4 00007f223495cc29 00007f223495c6ec 00007f223495dbcd 00007f223495f946 00007f223495f89a 00007f223495f823 0312 20:10:12.712765 (+ 91us) inbound_call.cc:157] Queueing success response Metrics: {"spinlock_wait_cycles":58562432} I0312 20:10:12.715932 1094 raft_consensus.cc:2145] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec in current term 21: Already voted for candidate ded9b7f8677940fb91e1a8c8bceceaed in this term. W0312 20:10:12.716446 1094 rpcz_store.cc:251] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.53.3:55789 (request call id 105) took 51ms (client timeout 29). I0312 20:10:12.717254 1095 raft_consensus.cc:2190] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6885de38238442d6a9e5f1eb9fb569ff for term 20 because replica is already servicing an update from a current leader or another vote. W0312 20:10:12.717707 1095 rpcz_store.cc:251] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.53.2:38849 (request call id 38) took 47ms (client timeout 15). W0312 20:10:12.718392 1095 rpcz_store.cc:255] Trace: 0312 20:10:12.670545 (+ 0us) service_pool.cc:163] Inserting onto call queue 0312 20:10:12.670658 (+ 113us) service_pool.cc:222] Handling call 0312 20:10:12.717580 (+ 46922us) spinlock_profiling.cc:261] Waited 45.9 ms on lock 0x7b480003c09c. stack: 00007f2232dd2c74 00007f2232dd2b70 00007f22318c2314 00007f223a90f294 00007f223a90f229 00007f22394b9606 00007f223a94d56b 00007f2235300b24 00007f2235300ab1 00007f22353009c4 00007f223495cc29 00007f223495c6ec 00007f223495dbcd 00007f223495f946 00007f223495f89a 00007f223495f823 0312 20:10:12.717664 (+ 84us) inbound_call.cc:157] Queueing success response Metrics: {"spinlock_wait_cycles":105552768} I0312 20:10:12.719516 1456 raft_consensus.cc:436] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0312 20:10:12.719740 1456 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed: Snoozing failure detection for 0.016s (starting election) I0312 20:10:12.719961 1456 raft_consensus.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:12.721364 1345 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" candidate_term: 21 candidate_status { last_received { term: 19 index: 95 } } ignore_live_leader: false dest_uuid: "f78a2351568c4a268ee1112681e22bec" I0312 20:10:12.721810 1345 raft_consensus.cc:2752] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 20 FOLLOWER]: Advancing to term 21 I0312 20:10:12.722074 1345 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec: Snoozing failure detection for 0.135s (vote granted) I0312 20:10:12.721056 1456 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 22 pre-election: Requesting pre-vote from peer f78a2351568c4a268ee1112681e22bec I0312 20:10:12.725594 1456 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 22 pre-election: Requesting pre-vote from peer 6885de38238442d6a9e5f1eb9fb569ff I0312 20:10:12.727461 1344 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" candidate_term: 22 candidate_status { last_received { term: 19 index: 95 } } ignore_live_leader: false dest_uuid: "f78a2351568c4a268ee1112681e22bec" is_pre_election: true W0312 20:10:12.728127 1094 rpcz_store.cc:255] Trace: 0312 20:10:12.665455 (+ 0us) service_pool.cc:163] Inserting onto call queue 0312 20:10:12.665577 (+ 122us) service_pool.cc:222] Handling call 0312 20:10:12.716306 (+ 50729us) spinlock_profiling.cc:261] Waited 49.5 ms on lock 0x7b480003c09c. stack: 00007f2232dd2c74 00007f2232dd2b70 00007f22318c2314 00007f223a90f294 00007f223a90f229 00007f22394b9af3 00007f223a94d56b 00007f2235300b24 00007f2235300ab1 00007f22353009c4 00007f223495cc29 00007f223495c6ec 00007f223495dbcd 00007f223495f946 00007f223495f89a 00007f223495f823 0312 20:10:12.716399 (+ 93us) inbound_call.cc:157] Queueing success response Metrics: {"spinlock_wait_cycles":113958656} W0312 20:10:12.730648 1029 leader_election.cc:287] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 21 election: RPC error from VoteRequest() call to peer 6885de38238442d6a9e5f1eb9fb569ff: Timed out: RequestConsensusVote RPC to 127.0.53.2:51697 timed out after 0.018s (SENT) W0312 20:10:12.732434 1027 leader_election.cc:287] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 21 election: RPC error from VoteRequest() call to peer f78a2351568c4a268ee1112681e22bec: Timed out: RequestConsensusVote RPC to 127.0.53.3:43274 timed out after 0.018s (SENT) I0312 20:10:12.732667 1027 leader_election.cc:258] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. I0312 20:10:12.733455 1418 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed: Snoozing failure detection for 0.015s (election complete) I0312 20:10:12.733659 1418 raft_consensus.cc:2455] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority W0312 20:10:12.715431 1153 leader_election.cc:347] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 20 pre-election: Vote denied by peer ded9b7f8677940fb91e1a8c8bceceaed with higher term. Message: Service unavailable: T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6885de38238442d6a9e5f1eb9fb569ff for term 20 because replica is already servicing an update from a current leader or another vote. I0312 20:10:12.737990 1153 leader_election.cc:350] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 20 pre-election: Cancelling election due to peer responding with higher term I0312 20:10:12.738890 1454 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff: Snoozing failure detection for 0.040s (election complete) I0312 20:10:12.739110 1454 raft_consensus.cc:2752] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 20 FOLLOWER]: Advancing to term 21 I0312 20:10:12.731518 1216 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" candidate_term: 21 candidate_status { last_received { term: 19 index: 95 } } ignore_live_leader: false dest_uuid: "6885de38238442d6a9e5f1eb9fb569ff" W0312 20:10:12.744063 1029 leader_election.cc:287] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer 6885de38238442d6a9e5f1eb9fb569ff: Timed out: RequestConsensusVote RPC to 127.0.53.2:51697 timed out after 0.016s (SENT) W0312 20:10:12.744766 1027 leader_election.cc:287] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer f78a2351568c4a268ee1112681e22bec: Timed out: RequestConsensusVote RPC to 127.0.53.3:43274 timed out after 0.016s (SENT) I0312 20:10:12.744956 1027 leader_election.cc:258] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. I0312 20:10:12.745607 1418 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed: Snoozing failure detection for 0.021s (election complete) I0312 20:10:12.746181 1418 raft_consensus.cc:2455] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority I0312 20:10:12.746711 1416 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 21 pre-election: Requesting pre-vote from peer ded9b7f8677940fb91e1a8c8bceceaed I0312 20:10:12.748827 1346 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "6885de38238442d6a9e5f1eb9fb569ff" candidate_term: 21 candidate_status { last_received { term: 19 index: 95 } } ignore_live_leader: false dest_uuid: "f78a2351568c4a268ee1112681e22bec" is_pre_election: true I0312 20:10:12.747354 1220 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" candidate_term: 22 candidate_status { last_received { term: 19 index: 95 } } ignore_live_leader: false dest_uuid: "6885de38238442d6a9e5f1eb9fb569ff" is_pre_election: true I0312 20:10:12.751687 1094 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "6885de38238442d6a9e5f1eb9fb569ff" candidate_term: 21 candidate_status { last_received { term: 19 index: 95 } } ignore_live_leader: false dest_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" is_pre_election: true I0312 20:10:12.752280 1094 raft_consensus.cc:2145] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6885de38238442d6a9e5f1eb9fb569ff in current term 21: Already voted for candidate ded9b7f8677940fb91e1a8c8bceceaed in this term. I0312 20:10:12.753338 1153 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 21 pre-election: Vote denied by peer ded9b7f8677940fb91e1a8c8bceceaed. Message: Invalid argument: T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6885de38238442d6a9e5f1eb9fb569ff in current term 21: Already voted for candidate ded9b7f8677940fb91e1a8c8bceceaed in this term. I0312 20:10:12.748797 1345 raft_consensus.cc:2215] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ded9b7f8677940fb91e1a8c8bceceaed in term 21. I0312 20:10:12.754123 1346 raft_consensus.cc:2190] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6885de38238442d6a9e5f1eb9fb569ff for term 21 because replica is already servicing an update from a current leader or another vote. I0312 20:10:12.754513 1344 raft_consensus.cc:2190] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ded9b7f8677940fb91e1a8c8bceceaed for term 22 because replica is already servicing an update from a current leader or another vote. W0312 20:10:12.755000 1344 rpcz_store.cc:251] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.53.1:47760 (request call id 80) took 27ms (client timeout 16). I0312 20:10:12.755312 1152 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 21 pre-election: Vote denied by peer f78a2351568c4a268ee1112681e22bec. Message: Service unavailable: T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6885de38238442d6a9e5f1eb9fb569ff for term 21 because replica is already servicing an update from a current leader or another vote. I0312 20:10:12.755524 1152 leader_election.cc:258] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. W0312 20:10:12.755658 1344 rpcz_store.cc:255] Trace: 0312 20:10:12.727213 (+ 0us) service_pool.cc:163] Inserting onto call queue 0312 20:10:12.727321 (+ 108us) service_pool.cc:222] Handling call 0312 20:10:12.754868 (+ 27547us) spinlock_profiling.cc:261] Waited 26.5 ms on lock 0x7b480003009c. stack: 00007f173be47c74 00007f173be47b70 00007f173a937314 00007f1743984294 00007f1743984229 00007f174252e606 00007f17439c256b 00007f173e375b24 00007f173e375ab1 00007f173e3759c4 00007f173d9d1c29 00007f173d9d16ec 00007f173d9d2bcd 00007f173d9d4946 00007f173d9d489a 00007f173d9d4823 0312 20:10:12.754954 (+ 86us) inbound_call.cc:157] Queueing success response Metrics: {"spinlock_wait_cycles":61033728} W0312 20:10:12.754060 1345 rpcz_store.cc:251] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.53.1:47760 (request call id 79) took 40ms (client timeout 18). W0312 20:10:12.756517 1345 rpcz_store.cc:255] Trace: 0312 20:10:12.713966 (+ 0us) service_pool.cc:163] Inserting onto call queue 0312 20:10:12.721194 (+ 7228us) service_pool.cc:222] Handling call 0312 20:10:12.754003 (+ 32809us) inbound_call.cc:157] Queueing success response Metrics: {} I0312 20:10:12.768506 1418 raft_consensus.cc:436] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0312 20:10:12.768745 1418 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed: Snoozing failure detection for 0.028s (starting election) I0312 20:10:12.769007 1418 raft_consensus.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:12.770241 1418 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 22 pre-election: Requesting pre-vote from peer f78a2351568c4a268ee1112681e22bec I0312 20:10:12.770628 1418 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 22 pre-election: Requesting pre-vote from peer 6885de38238442d6a9e5f1eb9fb569ff I0312 20:10:12.771695 1345 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" candidate_term: 22 candidate_status { last_received { term: 19 index: 95 } } ignore_live_leader: false dest_uuid: "f78a2351568c4a268ee1112681e22bec" is_pre_election: true I0312 20:10:12.772125 1345 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec: Snoozing failure detection for 0.154s (vote granted) I0312 20:10:12.772341 1345 raft_consensus.cc:2215] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ded9b7f8677940fb91e1a8c8bceceaed in term 21. I0312 20:10:12.773234 1027 leader_election.cc:363] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 22 pre-election: Vote granted by peer f78a2351568c4a268ee1112681e22bec I0312 20:10:12.773388 1027 leader_election.cc:258] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. I0312 20:10:12.773705 1418 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed: Snoozing failure detection for 0.028s (election complete) I0312 20:10:12.773950 1418 raft_consensus.cc:2507] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 21 FOLLOWER]: Leader pre-election won for term 22 I0312 20:10:12.774174 1418 raft_consensus.cc:436] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0312 20:10:12.774402 1418 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed: Snoozing failure detection for 0.039s (starting election) I0312 20:10:12.774358 1219 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" candidate_term: 22 candidate_status { last_received { term: 19 index: 95 } } ignore_live_leader: false dest_uuid: "6885de38238442d6a9e5f1eb9fb569ff" is_pre_election: true I0312 20:10:12.774663 1418 raft_consensus.cc:2752] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 21 FOLLOWER]: Advancing to term 22 I0312 20:10:12.779775 1418 raft_consensus.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:12.780884 1418 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 22 election: Requesting vote from peer f78a2351568c4a268ee1112681e22bec I0312 20:10:12.782202 1345 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" candidate_term: 22 candidate_status { last_received { term: 19 index: 95 } } ignore_live_leader: false dest_uuid: "f78a2351568c4a268ee1112681e22bec" I0312 20:10:12.782586 1345 raft_consensus.cc:2752] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 21 FOLLOWER]: Advancing to term 22 I0312 20:10:12.782800 1345 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec: Snoozing failure detection for 0.174s (vote granted) I0312 20:10:12.787358 1418 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 22 election: Requesting vote from peer 6885de38238442d6a9e5f1eb9fb569ff I0312 20:10:12.790870 1215 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" candidate_term: 22 candidate_status { last_received { term: 19 index: 95 } } ignore_live_leader: false dest_uuid: "6885de38238442d6a9e5f1eb9fb569ff" I0312 20:10:12.794687 1454 raft_consensus.cc:2455] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 21 FOLLOWER]: Leader pre-election lost for term 20. Reason: Vote denied by peer ded9b7f8677940fb91e1a8c8bceceaed with higher term. Message: Service unavailable: T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6885de38238442d6a9e5f1eb9fb569ff for term 20 because replica is already servicing an update from a current leader or another vote. I0312 20:10:12.795177 1220 raft_consensus.cc:2190] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ded9b7f8677940fb91e1a8c8bceceaed for term 22 because replica is already servicing an update from a current leader or another vote. W0312 20:10:12.795686 1220 rpcz_store.cc:251] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.53.1:43660 (request call id 81) took 57ms (client timeout 16). W0312 20:10:12.796360 1220 rpcz_store.cc:255] Trace: 0312 20:10:12.737864 (+ 0us) service_pool.cc:163] Inserting onto call queue 0312 20:10:12.747200 (+ 9336us) service_pool.cc:222] Handling call 0312 20:10:12.795539 (+ 48339us) spinlock_profiling.cc:261] Waited 44.8 ms on lock 0x7b480003009c. stack: 00007efc5efb2c74 00007efc5efb2b70 00007efc5daa2314 00007efc66aef294 00007efc66aef229 00007efc65699606 00007efc66b2d56b 00007efc614e0b24 00007efc614e0ab1 00007efc614e09c4 00007efc60b3cc29 00007efc60b3c6ec 00007efc60b3dbcd 00007efc60b3f946 00007efc60b3f89a 00007efc60b3f823 0312 20:10:12.795639 (+ 100us) inbound_call.cc:157] Queueing success response Metrics: {"spinlock_wait_cycles":103107840} I0312 20:10:12.796757 1467 raft_consensus.cc:436] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0312 20:10:12.796999 1467 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff: Snoozing failure detection for 0.054s (starting election) I0312 20:10:12.797260 1467 raft_consensus.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:12.798228 1215 raft_consensus.cc:2190] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate ded9b7f8677940fb91e1a8c8bceceaed for term 22 because replica is already servicing an update from a current leader or another vote. I0312 20:10:12.798230 1467 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 22 pre-election: Requesting pre-vote from peer f78a2351568c4a268ee1112681e22bec I0312 20:10:12.798905 1216 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff: Snoozing failure detection for 0.067s (vote granted) I0312 20:10:12.798614 1467 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 22 pre-election: Requesting pre-vote from peer ded9b7f8677940fb91e1a8c8bceceaed I0312 20:10:12.800645 1094 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "6885de38238442d6a9e5f1eb9fb569ff" candidate_term: 22 candidate_status { last_received { term: 19 index: 95 } } ignore_live_leader: false dest_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" is_pre_election: true I0312 20:10:12.801167 1094 raft_consensus.cc:2145] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6885de38238442d6a9e5f1eb9fb569ff in current term 22: Already voted for candidate ded9b7f8677940fb91e1a8c8bceceaed in this term. W0312 20:10:12.801275 1029 leader_election.cc:287] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer 6885de38238442d6a9e5f1eb9fb569ff: Timed out: RequestConsensusVote RPC to 127.0.53.2:51697 timed out after 0.028s (SENT) I0312 20:10:12.802201 1153 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 22 pre-election: Vote denied by peer ded9b7f8677940fb91e1a8c8bceceaed. Message: Invalid argument: T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6885de38238442d6a9e5f1eb9fb569ff in current term 22: Already voted for candidate ded9b7f8677940fb91e1a8c8bceceaed in this term. I0312 20:10:12.803614 1344 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "6885de38238442d6a9e5f1eb9fb569ff" candidate_term: 22 candidate_status { last_received { term: 19 index: 95 } } ignore_live_leader: false dest_uuid: "f78a2351568c4a268ee1112681e22bec" is_pre_election: true I0312 20:10:12.804247 1029 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 22 election: Vote denied by peer 6885de38238442d6a9e5f1eb9fb569ff. Message: Service unavailable: T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate ded9b7f8677940fb91e1a8c8bceceaed for term 22 because replica is already servicing an update from a current leader or another vote. I0312 20:10:12.815130 1418 raft_consensus.cc:436] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0312 20:10:12.815390 1418 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed: Snoozing failure detection for 0.038s (starting election) I0312 20:10:12.815802 1418 raft_consensus.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:12.816956 1418 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 23 pre-election: Requesting pre-vote from peer f78a2351568c4a268ee1112681e22bec I0312 20:10:12.818290 1346 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" candidate_term: 23 candidate_status { last_received { term: 19 index: 95 } } ignore_live_leader: false dest_uuid: "f78a2351568c4a268ee1112681e22bec" is_pre_election: true I0312 20:10:12.818548 1418 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 23 pre-election: Requesting pre-vote from peer 6885de38238442d6a9e5f1eb9fb569ff I0312 20:10:12.819878 1215 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" candidate_term: 23 candidate_status { last_received { term: 19 index: 95 } } ignore_live_leader: false dest_uuid: "6885de38238442d6a9e5f1eb9fb569ff" is_pre_election: true W0312 20:10:12.820542 1027 leader_election.cc:287] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 22 election: RPC error from VoteRequest() call to peer f78a2351568c4a268ee1112681e22bec: Timed out: RequestConsensusVote RPC to 127.0.53.3:43274 timed out after 0.039s (SENT) I0312 20:10:12.820772 1027 leader_election.cc:258] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. I0312 20:10:12.821421 1456 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed: Snoozing failure detection for 0.021s (election complete) I0312 20:10:12.821630 1456 raft_consensus.cc:2455] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority I0312 20:10:12.823781 1345 raft_consensus.cc:2215] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ded9b7f8677940fb91e1a8c8bceceaed in term 22. I0312 20:10:12.825268 1346 raft_consensus.cc:2190] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ded9b7f8677940fb91e1a8c8bceceaed for term 23 because replica is already servicing an update from a current leader or another vote. I0312 20:10:12.827394 1344 raft_consensus.cc:2190] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6885de38238442d6a9e5f1eb9fb569ff for term 22 because replica is already servicing an update from a current leader or another vote. I0312 20:10:12.828855 1152 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 22 pre-election: Vote denied by peer f78a2351568c4a268ee1112681e22bec. Message: Service unavailable: T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6885de38238442d6a9e5f1eb9fb569ff for term 22 because replica is already servicing an update from a current leader or another vote. I0312 20:10:12.829629 1027 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 23 pre-election: Vote denied by peer f78a2351568c4a268ee1112681e22bec. Message: Service unavailable: T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ded9b7f8677940fb91e1a8c8bceceaed for term 23 because replica is already servicing an update from a current leader or another vote. I0312 20:10:12.829988 1152 leader_election.cc:258] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. W0312 20:10:12.830852 1345 rpcz_store.cc:251] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.53.1:47760 (request call id 82) took 43ms (client timeout 39). W0312 20:10:12.831470 1345 rpcz_store.cc:255] Trace: 0312 20:10:12.781973 (+ 0us) service_pool.cc:163] Inserting onto call queue 0312 20:10:12.782073 (+ 100us) service_pool.cc:222] Handling call 0312 20:10:12.825029 (+ 42956us) inbound_call.cc:157] Queueing success response Metrics: {} I0312 20:10:12.843032 1418 raft_consensus.cc:436] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0312 20:10:12.843325 1418 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed: Snoozing failure detection for 0.018s (starting election) I0312 20:10:12.843559 1418 raft_consensus.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:12.845823 1216 raft_consensus.cc:2215] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ded9b7f8677940fb91e1a8c8bceceaed in term 21. W0312 20:10:12.846385 1216 rpcz_store.cc:251] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.53.1:43660 (request call id 80) took 115ms (client timeout 18). W0312 20:10:12.847025 1216 rpcz_store.cc:255] Trace: 0312 20:10:12.731221 (+ 0us) service_pool.cc:163] Inserting onto call queue 0312 20:10:12.731353 (+ 132us) service_pool.cc:222] Handling call 0312 20:10:12.846263 (+114910us) spinlock_profiling.cc:261] Waited 57.7 ms on lock 0x7b480003009c. stack: 00007efc5efb2c74 00007efc5efb2b70 00007efc5daa2314 00007efc66aef294 00007efc66aef229 00007efc65699af3 00007efc66b2d56b 00007efc614e0b24 00007efc614e0ab1 00007efc614e09c4 00007efc60b3cc29 00007efc60b3c6ec 00007efc60b3dbcd 00007efc60b3f946 00007efc60b3f89a 00007efc60b3f823 0312 20:10:12.846339 (+ 76us) inbound_call.cc:157] Queueing success response Metrics: {"spinlock_wait_cycles":132673792} I0312 20:10:12.847329 1467 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff: Snoozing failure detection for 0.054s (election complete) I0312 20:10:12.847566 1467 raft_consensus.cc:2752] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 21 FOLLOWER]: Advancing to term 22 I0312 20:10:12.854221 1418 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 23 pre-election: Requesting pre-vote from peer f78a2351568c4a268ee1112681e22bec I0312 20:10:12.855576 1345 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" candidate_term: 23 candidate_status { last_received { term: 19 index: 95 } } ignore_live_leader: false dest_uuid: "f78a2351568c4a268ee1112681e22bec" is_pre_election: true I0312 20:10:12.856132 1345 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec: Snoozing failure detection for 0.242s (vote granted) I0312 20:10:12.856374 1345 raft_consensus.cc:2215] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ded9b7f8677940fb91e1a8c8bceceaed in term 22. I0312 20:10:12.857340 1027 leader_election.cc:363] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 23 pre-election: Vote granted by peer f78a2351568c4a268ee1112681e22bec I0312 20:10:12.857559 1027 leader_election.cc:258] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. I0312 20:10:12.858135 1456 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed: Snoozing failure detection for 0.055s (election complete) I0312 20:10:12.858405 1456 raft_consensus.cc:2507] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 22 FOLLOWER]: Leader pre-election won for term 23 I0312 20:10:12.858613 1456 raft_consensus.cc:436] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0312 20:10:12.858847 1456 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed: Snoozing failure detection for 0.029s (starting election) I0312 20:10:12.859061 1456 raft_consensus.cc:2752] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 22 FOLLOWER]: Advancing to term 23 W0312 20:10:12.857517 1029 leader_election.cc:287] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 23 pre-election: RPC error from VoteRequest() call to peer 6885de38238442d6a9e5f1eb9fb569ff: Timed out: RequestConsensusVote RPC to 127.0.53.2:51697 timed out after 0.038s (SENT) I0312 20:10:12.865195 1029 leader_election.cc:258] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. I0312 20:10:12.864751 1456 raft_consensus.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:12.865855 1456 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 23 election: Requesting vote from peer f78a2351568c4a268ee1112681e22bec I0312 20:10:12.865906 1417 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed: Snoozing failure detection for 0.060s (election complete) I0312 20:10:12.866128 1417 raft_consensus.cc:2455] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority I0312 20:10:12.867280 1345 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" candidate_term: 23 candidate_status { last_received { term: 19 index: 95 } } ignore_live_leader: false dest_uuid: "f78a2351568c4a268ee1112681e22bec" I0312 20:10:12.867722 1345 raft_consensus.cc:2752] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 22 FOLLOWER]: Advancing to term 23 I0312 20:10:12.867940 1345 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec: Snoozing failure detection for 0.176s (vote granted) I0312 20:10:12.872313 1345 raft_consensus.cc:2215] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ded9b7f8677940fb91e1a8c8bceceaed in term 23. I0312 20:10:12.872627 1456 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 23 election: Requesting vote from peer 6885de38238442d6a9e5f1eb9fb569ff I0312 20:10:12.872886 1418 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 23 pre-election: Requesting pre-vote from peer 6885de38238442d6a9e5f1eb9fb569ff I0312 20:10:12.874739 1220 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" candidate_term: 23 candidate_status { last_received { term: 19 index: 95 } } ignore_live_leader: false dest_uuid: "6885de38238442d6a9e5f1eb9fb569ff" is_pre_election: true I0312 20:10:12.875481 1216 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" candidate_term: 23 candidate_status { last_received { term: 19 index: 95 } } ignore_live_leader: false dest_uuid: "6885de38238442d6a9e5f1eb9fb569ff" I0312 20:10:12.878115 1027 leader_election.cc:363] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 23 election: Vote granted by peer f78a2351568c4a268ee1112681e22bec I0312 20:10:12.878314 1027 leader_election.cc:258] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. I0312 20:10:12.879138 1418 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed: Snoozing failure detection for 0.047s (election complete) I0312 20:10:12.879361 1418 raft_consensus.cc:2507] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 23 FOLLOWER]: Leader election won for term 23 I0312 20:10:12.880271 1418 raft_consensus.cc:572] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 23 LEADER]: Becoming Leader. State: Replica: ded9b7f8677940fb91e1a8c8bceceaed, State: Running, Role: LEADER I0312 20:10:12.881275 1418 consensus_queue.cc:196] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 93, Committed index: 93, Last appended: 19.95, Last appended by leader: 95, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:12.886577 974 catalog_manager.cc:3592] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed reported cstate change: term changed from 19 to 23. New cstate: current_term: 23 leader_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } health_report { overall_health: HEALTHY } } } W0312 20:10:12.893476 1029 leader_election.cc:287] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 23 pre-election: RPC error from VoteRequest() call to peer 6885de38238442d6a9e5f1eb9fb569ff: Timed out: RequestConsensusVote RPC to 127.0.53.2:51697 timed out after 0.018s (SENT) I0312 20:10:12.900499 1467 raft_consensus.cc:2455] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority I0312 20:10:12.900796 1416 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff: Snoozing failure detection for 0.052s (election complete) I0312 20:10:12.901032 1416 raft_consensus.cc:2455] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 22 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority I0312 20:10:12.901643 1215 raft_consensus.cc:2190] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ded9b7f8677940fb91e1a8c8bceceaed for term 23 because replica is already servicing an update from a current leader or another vote. W0312 20:10:12.903023 1215 rpcz_store.cc:251] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.53.1:43660 (request call id 84) took 83ms (client timeout 38). I0312 20:10:12.903647 1220 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff: Snoozing failure detection for 0.081s (vote granted) I0312 20:10:12.903919 1220 raft_consensus.cc:2215] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ded9b7f8677940fb91e1a8c8bceceaed in term 22. W0312 20:10:12.906298 1220 rpcz_store.cc:251] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.53.1:43660 (request call id 86) took 31ms (client timeout 18). W0312 20:10:12.906949 1220 rpcz_store.cc:255] Trace: 0312 20:10:12.874448 (+ 0us) service_pool.cc:163] Inserting onto call queue 0312 20:10:12.874555 (+ 107us) service_pool.cc:222] Handling call 0312 20:10:12.906169 (+ 31614us) spinlock_profiling.cc:261] Waited 28.4 ms on lock 0x7b480003009c. stack: 00007efc5efb2c74 00007efc5efb2b70 00007efc5daa2314 00007efc66aef294 00007efc66aef229 00007efc65699af3 00007efc66b2d56b 00007efc614e0b24 00007efc614e0ab1 00007efc614e09c4 00007efc60b3cc29 00007efc60b3c6ec 00007efc60b3dbcd 00007efc60b3f946 00007efc60b3f89a 00007efc60b3f823 0312 20:10:12.906252 (+ 83us) inbound_call.cc:157] Queueing success response Metrics: {"spinlock_wait_cycles":65282432} I0312 20:10:12.907383 1219 raft_consensus.cc:2190] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ded9b7f8677940fb91e1a8c8bceceaed for term 22 because replica is already servicing an update from a current leader or another vote. W0312 20:10:12.907953 1219 rpcz_store.cc:251] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.53.1:43660 (request call id 82) took 133ms (client timeout 28). I0312 20:10:12.908849 1218 raft_consensus.cc:2752] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 22 FOLLOWER]: Advancing to term 23 W0312 20:10:12.920078 1219 rpcz_store.cc:255] Trace: 0312 20:10:12.774096 (+ 0us) service_pool.cc:163] Inserting onto call queue 0312 20:10:12.774210 (+ 114us) service_pool.cc:222] Handling call 0312 20:10:12.907745 (+133535us) spinlock_profiling.cc:261] Waited 133 ms on lock 0x7b480003009c. stack: 00007efc5efb2c74 00007efc5efb2b70 00007efc5daa2314 00007efc66aef294 00007efc66aef229 00007efc65699606 00007efc66b2d56b 00007efc614e0b24 00007efc614e0ab1 00007efc614e09c4 00007efc60b3cc29 00007efc60b3c6ec 00007efc60b3dbcd 00007efc60b3f946 00007efc60b3f89a 00007efc60b3f823 0312 20:10:12.907838 (+ 93us) inbound_call.cc:157] Queueing success response Metrics: {"spinlock_wait_cycles":304848768} W0312 20:10:12.903717 1215 rpcz_store.cc:255] Trace: 0312 20:10:12.819567 (+ 0us) service_pool.cc:163] Inserting onto call queue 0312 20:10:12.819676 (+ 109us) service_pool.cc:222] Handling call 0312 20:10:12.902021 (+ 82345us) spinlock_profiling.cc:261] Waited 81.2 ms on lock 0x7b480003009c. stack: 00007efc5efb2c74 00007efc5efb2b70 00007efc5daa2314 00007efc66aef294 00007efc66aef229 00007efc65699606 00007efc66b2d56b 00007efc614e0b24 00007efc614e0ab1 00007efc614e09c4 00007efc60b3cc29 00007efc60b3c6ec 00007efc60b3dbcd 00007efc60b3f946 00007efc60b3f89a 00007efc60b3f823 0312 20:10:12.902965 (+ 944us) inbound_call.cc:157] Queueing success response Metrics: {"spinlock_wait_cycles":186839424} W0312 20:10:12.924887 1029 leader_election.cc:287] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 23 election: RPC error from VoteRequest() call to peer 6885de38238442d6a9e5f1eb9fb569ff: Timed out: RequestConsensusVote RPC to 127.0.53.2:51697 timed out after 0.029s (SENT) I0312 20:10:12.927992 1218 raft_consensus.cc:1058] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 23 FOLLOWER]: Refusing update from remote peer ded9b7f8677940fb91e1a8c8bceceaed: Log matching property violated. Preceding OpId in replica: term: 19 index: 95. Preceding OpId from leader: term: 23 index: 96. (index mismatch) I0312 20:10:12.928927 1216 raft_consensus.cc:2190] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate ded9b7f8677940fb91e1a8c8bceceaed for term 23 because replica is already servicing an update from a current leader or another vote. W0312 20:10:12.929446 1216 rpcz_store.cc:251] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.53.1:43660 (request call id 85) took 55ms (client timeout 29). W0312 20:10:12.929977 1216 rpcz_store.cc:255] Trace: 0312 20:10:12.874065 (+ 0us) service_pool.cc:163] Inserting onto call queue 0312 20:10:12.875305 (+ 1240us) service_pool.cc:222] Handling call 0312 20:10:12.929333 (+ 54028us) spinlock_profiling.cc:261] Waited 53 ms on lock 0x7b480003009c. stack: 00007efc5efb2c74 00007efc5efb2b70 00007efc5daa2314 00007efc66aef294 00007efc66aef229 00007efc65699606 00007efc66b2d56b 00007efc614e0b24 00007efc614e0ab1 00007efc614e09c4 00007efc60b3cc29 00007efc60b3c6ec 00007efc60b3dbcd 00007efc60b3f946 00007efc60b3f89a 00007efc60b3f823 0312 20:10:12.929407 (+ 74us) inbound_call.cc:157] Queueing success response Metrics: {"spinlock_wait_cycles":121813760} I0312 20:10:12.931577 1345 raft_consensus.cc:1058] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 23 FOLLOWER]: Refusing update from remote peer ded9b7f8677940fb91e1a8c8bceceaed: Log matching property violated. Preceding OpId in replica: term: 19 index: 93. Preceding OpId from leader: term: 23 index: 96. (index mismatch) I0312 20:10:12.931897 1456 consensus_queue.cc:1017] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [LEADER]: Connected to new peer: Peer: permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 }, Status: LMP_MISMATCH, Last received: 19.95, Next index: 96, Last known committed idx: 93, Time since last communication: 0.001s I0312 20:10:12.932780 1417 consensus_queue.cc:1017] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [LEADER]: Connected to new peer: Peer: permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 }, Status: LMP_MISMATCH, Last received: 19.93, Next index: 94, Last known committed idx: 93, Time since last communication: 0.000s I0312 20:10:12.951432 1467 raft_consensus.cc:436] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 23 FOLLOWER]: Starting pre-election (detected failure of leader ded9b7f8677940fb91e1a8c8bceceaed) I0312 20:10:12.951728 1467 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff: Snoozing failure detection for 0.015s (starting election) I0312 20:10:12.952023 1467 raft_consensus.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:12.953054 1464 raft_consensus.cc:436] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 23 FOLLOWER]: Starting pre-election (detected failure of leader ded9b7f8677940fb91e1a8c8bceceaed) I0312 20:10:12.953308 1464 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec: Snoozing failure detection for 0.017s (starting election) I0312 20:10:12.953526 1464 raft_consensus.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:12.954504 1464 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 24 pre-election: Requesting pre-vote from peer 6885de38238442d6a9e5f1eb9fb569ff I0312 20:10:12.954861 1464 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 24 pre-election: Requesting pre-vote from peer ded9b7f8677940fb91e1a8c8bceceaed I0312 20:10:12.957132 1094 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "f78a2351568c4a268ee1112681e22bec" candidate_term: 24 candidate_status { last_received { term: 23 index: 96 } } ignore_live_leader: false dest_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" is_pre_election: true I0312 20:10:12.957696 1094 raft_consensus.cc:2176] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 23 LEADER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 24 because replica is either leader or believes a valid leader to be alive. I0312 20:10:12.958878 1278 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 24 pre-election: Vote denied by peer ded9b7f8677940fb91e1a8c8bceceaed. Message: Invalid argument: T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 23 LEADER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 24 because replica is either leader or believes a valid leader to be alive. W0312 20:10:12.974043 1279 leader_election.cc:287] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer 6885de38238442d6a9e5f1eb9fb569ff: Timed out: RequestConsensusVote RPC to 127.0.53.2:51697 timed out after 0.017s (SENT) I0312 20:10:12.975122 1279 leader_election.cc:258] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. I0312 20:10:12.976114 1464 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec: Snoozing failure detection for 0.018s (election complete) I0312 20:10:12.976330 1464 raft_consensus.cc:2455] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority I0312 20:10:12.977535 1467 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 24 pre-election: Requesting pre-vote from peer f78a2351568c4a268ee1112681e22bec I0312 20:10:12.979722 1344 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "6885de38238442d6a9e5f1eb9fb569ff" candidate_term: 24 candidate_status { last_received { term: 23 index: 96 } } ignore_live_leader: false dest_uuid: "f78a2351568c4a268ee1112681e22bec" is_pre_election: true I0312 20:10:12.980278 1344 raft_consensus.cc:2176] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6885de38238442d6a9e5f1eb9fb569ff for term 24 because replica is either leader or believes a valid leader to be alive. I0312 20:10:12.981228 1152 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 24 pre-election: Vote denied by peer f78a2351568c4a268ee1112681e22bec. Message: Invalid argument: T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6885de38238442d6a9e5f1eb9fb569ff for term 24 because replica is either leader or believes a valid leader to be alive. I0312 20:10:12.982522 1467 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 24 pre-election: Requesting pre-vote from peer ded9b7f8677940fb91e1a8c8bceceaed I0312 20:10:13.008275 1094 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "6885de38238442d6a9e5f1eb9fb569ff" candidate_term: 24 candidate_status { last_received { term: 23 index: 96 } } ignore_live_leader: false dest_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" is_pre_election: true I0312 20:10:13.008808 1094 raft_consensus.cc:2176] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 23 LEADER]: Leader pre-election vote request: Denying vote to candidate 6885de38238442d6a9e5f1eb9fb569ff for term 24 because replica is either leader or believes a valid leader to be alive. W0312 20:10:13.009258 1218 rpcz_store.cc:251] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.53.3:38196 (request call id 78) took 53ms (client timeout 17). I0312 20:10:13.009493 1464 raft_consensus.cc:436] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 23 FOLLOWER]: Starting pre-election (detected failure of leader ded9b7f8677940fb91e1a8c8bceceaed) I0312 20:10:13.009737 1464 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec: Snoozing failure detection for 0.030s (starting election) W0312 20:10:13.009783 1218 rpcz_store.cc:255] Trace: 0312 20:10:12.956275 (+ 0us) service_pool.cc:163] Inserting onto call queue 0312 20:10:13.009131 (+ 52856us) service_pool.cc:207] Skipping call since client already timed out 0312 20:10:13.009210 (+ 79us) inbound_call.cc:157] Queueing failure response Metrics: {} I0312 20:10:13.009979 1464 raft_consensus.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:13.011915 1464 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 24 pre-election: Requesting pre-vote from peer 6885de38238442d6a9e5f1eb9fb569ff I0312 20:10:13.012115 1153 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 24 pre-election: Vote denied by peer ded9b7f8677940fb91e1a8c8bceceaed. Message: Invalid argument: T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 23 LEADER]: Leader pre-election vote request: Denying vote to candidate 6885de38238442d6a9e5f1eb9fb569ff for term 24 because replica is either leader or believes a valid leader to be alive. I0312 20:10:13.012293 1464 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 24 pre-election: Requesting pre-vote from peer ded9b7f8677940fb91e1a8c8bceceaed I0312 20:10:13.012339 1153 leader_election.cc:258] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. I0312 20:10:13.013998 1216 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "f78a2351568c4a268ee1112681e22bec" candidate_term: 24 candidate_status { last_received { term: 23 index: 99 } } ignore_live_leader: false dest_uuid: "6885de38238442d6a9e5f1eb9fb569ff" is_pre_election: true I0312 20:10:13.014546 1216 raft_consensus.cc:2176] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 24 because replica is either leader or believes a valid leader to be alive. I0312 20:10:13.015408 1279 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 24 pre-election: Vote denied by peer 6885de38238442d6a9e5f1eb9fb569ff. Message: Invalid argument: T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 24 because replica is either leader or believes a valid leader to be alive. I0312 20:10:13.016041 1467 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff: Snoozing failure detection for 0.018s (election complete) I0312 20:10:13.017040 1467 raft_consensus.cc:2455] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority I0312 20:10:13.026515 1094 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "f78a2351568c4a268ee1112681e22bec" candidate_term: 24 candidate_status { last_received { term: 23 index: 99 } } ignore_live_leader: false dest_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" is_pre_election: true I0312 20:10:13.027051 1094 raft_consensus.cc:2176] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 23 LEADER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 24 because replica is either leader or believes a valid leader to be alive. I0312 20:10:13.036342 1464 raft_consensus.cc:436] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 23 FOLLOWER]: Starting pre-election (detected failure of leader ded9b7f8677940fb91e1a8c8bceceaed) I0312 20:10:13.036638 1464 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec: Snoozing failure detection for 0.018s (starting election) I0312 20:10:13.036886 1464 raft_consensus.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:13.037911 1464 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 24 pre-election: Requesting pre-vote from peer 6885de38238442d6a9e5f1eb9fb569ff I0312 20:10:13.038277 1464 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 24 pre-election: Requesting pre-vote from peer ded9b7f8677940fb91e1a8c8bceceaed I0312 20:10:13.039942 1216 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "f78a2351568c4a268ee1112681e22bec" candidate_term: 24 candidate_status { last_received { term: 23 index: 100 } } ignore_live_leader: false dest_uuid: "6885de38238442d6a9e5f1eb9fb569ff" is_pre_election: true I0312 20:10:13.040482 1216 raft_consensus.cc:2176] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 24 because replica is either leader or believes a valid leader to be alive. I0312 20:10:13.041373 1279 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 24 pre-election: Vote denied by peer 6885de38238442d6a9e5f1eb9fb569ff. Message: Invalid argument: T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 24 because replica is either leader or believes a valid leader to be alive. I0312 20:10:13.042807 1094 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "f78a2351568c4a268ee1112681e22bec" candidate_term: 24 candidate_status { last_received { term: 23 index: 100 } } ignore_live_leader: false dest_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" is_pre_election: true I0312 20:10:13.043334 1094 raft_consensus.cc:2176] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 23 LEADER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 24 because replica is either leader or believes a valid leader to be alive. W0312 20:10:13.043319 1278 leader_election.cc:287] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer ded9b7f8677940fb91e1a8c8bceceaed: Timed out: RequestConsensusVote RPC to 127.0.53.1:35037 timed out after 0.030s (SENT) I0312 20:10:13.043579 1278 leader_election.cc:258] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. I0312 20:10:13.044399 1464 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec: Snoozing failure detection for 0.028s (election complete) I0312 20:10:13.044612 1464 raft_consensus.cc:2455] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority I0312 20:10:13.051273 1278 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 24 pre-election: Vote denied by peer ded9b7f8677940fb91e1a8c8bceceaed. Message: Invalid argument: T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 23 LEADER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 24 because replica is either leader or believes a valid leader to be alive. I0312 20:10:13.051524 1278 leader_election.cc:258] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. I0312 20:10:13.055503 1464 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec: Snoozing failure detection for 0.032s (election complete) I0312 20:10:13.055719 1464 raft_consensus.cc:2455] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority I0312 20:10:13.088378 1464 raft_consensus.cc:436] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 23 FOLLOWER]: Starting pre-election (detected failure of leader ded9b7f8677940fb91e1a8c8bceceaed) I0312 20:10:13.088652 1464 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec: Snoozing failure detection for 0.029s (starting election) I0312 20:10:13.088896 1464 raft_consensus.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:13.089948 1464 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 24 pre-election: Requesting pre-vote from peer 6885de38238442d6a9e5f1eb9fb569ff I0312 20:10:13.090343 1464 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 24 pre-election: Requesting pre-vote from peer ded9b7f8677940fb91e1a8c8bceceaed I0312 20:10:13.092025 1218 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "f78a2351568c4a268ee1112681e22bec" candidate_term: 24 candidate_status { last_received { term: 23 index: 101 } } ignore_live_leader: false dest_uuid: "6885de38238442d6a9e5f1eb9fb569ff" is_pre_election: true I0312 20:10:13.092571 1218 raft_consensus.cc:2176] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 24 because replica is either leader or believes a valid leader to be alive. I0312 20:10:13.094825 1279 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 24 pre-election: Vote denied by peer 6885de38238442d6a9e5f1eb9fb569ff. Message: Invalid argument: T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 24 because replica is either leader or believes a valid leader to be alive. I0312 20:10:13.114856 1094 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "f78a2351568c4a268ee1112681e22bec" candidate_term: 24 candidate_status { last_received { term: 23 index: 101 } } ignore_live_leader: false dest_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" is_pre_election: true I0312 20:10:13.115463 1094 raft_consensus.cc:2176] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 23 LEADER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 24 because replica is either leader or believes a valid leader to be alive. I0312 20:10:13.116014 1464 raft_consensus.cc:436] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 23 FOLLOWER]: Starting pre-election (detected failure of leader ded9b7f8677940fb91e1a8c8bceceaed) I0312 20:10:13.116291 1464 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec: Snoozing failure detection for 0.072s (starting election) I0312 20:10:13.116533 1464 raft_consensus.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:13.117625 1464 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 24 pre-election: Requesting pre-vote from peer 6885de38238442d6a9e5f1eb9fb569ff I0312 20:10:13.117995 1464 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 24 pre-election: Requesting pre-vote from peer ded9b7f8677940fb91e1a8c8bceceaed I0312 20:10:13.120539 1218 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "f78a2351568c4a268ee1112681e22bec" candidate_term: 24 candidate_status { last_received { term: 23 index: 103 } } ignore_live_leader: false dest_uuid: "6885de38238442d6a9e5f1eb9fb569ff" is_pre_election: true W0312 20:10:13.121275 1278 leader_election.cc:287] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer ded9b7f8677940fb91e1a8c8bceceaed: Timed out: RequestConsensusVote RPC to 127.0.53.1:35037 timed out after 0.029s (SENT) I0312 20:10:13.121510 1278 leader_election.cc:258] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. I0312 20:10:13.122134 1218 raft_consensus.cc:2176] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 24 because replica is either leader or believes a valid leader to be alive. I0312 20:10:13.123188 1279 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 24 pre-election: Vote denied by peer 6885de38238442d6a9e5f1eb9fb569ff. Message: Invalid argument: T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 24 because replica is either leader or believes a valid leader to be alive. I0312 20:10:13.123585 1464 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec: Snoozing failure detection for 0.027s (election complete) I0312 20:10:13.123787 1464 raft_consensus.cc:2455] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority I0312 20:10:13.126687 1094 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "f78a2351568c4a268ee1112681e22bec" candidate_term: 24 candidate_status { last_received { term: 23 index: 103 } } ignore_live_leader: false dest_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" is_pre_election: true I0312 20:10:13.127323 1094 raft_consensus.cc:2176] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 23 LEADER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 24 because replica is either leader or believes a valid leader to be alive. I0312 20:10:13.131868 1278 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 24 pre-election: Vote denied by peer ded9b7f8677940fb91e1a8c8bceceaed. Message: Invalid argument: T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 23 LEADER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 24 because replica is either leader or believes a valid leader to be alive. I0312 20:10:13.132109 1278 leader_election.cc:258] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. I0312 20:10:13.138159 1464 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec: Snoozing failure detection for 0.029s (election complete) I0312 20:10:13.138360 1464 raft_consensus.cc:2455] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority I0312 20:10:13.162755 1464 raft_consensus.cc:436] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 23 FOLLOWER]: Starting pre-election (detected failure of leader ded9b7f8677940fb91e1a8c8bceceaed) I0312 20:10:13.162997 1464 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec: Snoozing failure detection for 0.126s (starting election) I0312 20:10:13.163209 1464 raft_consensus.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:13.164307 1464 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 24 pre-election: Requesting pre-vote from peer 6885de38238442d6a9e5f1eb9fb569ff I0312 20:10:13.164674 1464 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 24 pre-election: Requesting pre-vote from peer ded9b7f8677940fb91e1a8c8bceceaed I0312 20:10:13.166334 1216 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "f78a2351568c4a268ee1112681e22bec" candidate_term: 24 candidate_status { last_received { term: 23 index: 106 } } ignore_live_leader: false dest_uuid: "6885de38238442d6a9e5f1eb9fb569ff" is_pre_election: true I0312 20:10:13.166872 1216 raft_consensus.cc:2176] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 24 because replica is either leader or believes a valid leader to be alive. I0312 20:10:13.167903 1279 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 24 pre-election: Vote denied by peer 6885de38238442d6a9e5f1eb9fb569ff. Message: Invalid argument: T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 24 because replica is either leader or believes a valid leader to be alive. I0312 20:10:13.179033 1094 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "f78a2351568c4a268ee1112681e22bec" candidate_term: 24 candidate_status { last_received { term: 23 index: 106 } } ignore_live_leader: false dest_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" is_pre_election: true I0312 20:10:13.179558 1094 raft_consensus.cc:2176] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 23 LEADER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 24 because replica is either leader or believes a valid leader to be alive. I0312 20:10:13.193167 1278 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 24 pre-election: Vote denied by peer ded9b7f8677940fb91e1a8c8bceceaed. Message: Invalid argument: T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 23 LEADER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 24 because replica is either leader or believes a valid leader to be alive. I0312 20:10:13.194622 1278 leader_election.cc:258] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. I0312 20:10:13.195279 1464 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec: Snoozing failure detection for 0.088s (election complete) I0312 20:10:13.195508 1464 raft_consensus.cc:2455] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority I0312 20:10:13.224953 1464 raft_consensus.cc:436] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 23 FOLLOWER]: Starting pre-election (detected failure of leader ded9b7f8677940fb91e1a8c8bceceaed) I0312 20:10:13.225227 1464 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec: Snoozing failure detection for 0.070s (starting election) I0312 20:10:13.226496 1464 raft_consensus.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:13.227540 1464 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 24 pre-election: Requesting pre-vote from peer 6885de38238442d6a9e5f1eb9fb569ff I0312 20:10:13.228695 1464 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 24 pre-election: Requesting pre-vote from peer ded9b7f8677940fb91e1a8c8bceceaed I0312 20:10:13.228830 1218 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "f78a2351568c4a268ee1112681e22bec" candidate_term: 24 candidate_status { last_received { term: 23 index: 108 } } ignore_live_leader: false dest_uuid: "6885de38238442d6a9e5f1eb9fb569ff" is_pre_election: true I0312 20:10:13.229398 1218 raft_consensus.cc:2176] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 24 because replica is either leader or believes a valid leader to be alive. I0312 20:10:13.230628 1279 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 24 pre-election: Vote denied by peer 6885de38238442d6a9e5f1eb9fb569ff. Message: Invalid argument: T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 24 because replica is either leader or believes a valid leader to be alive. I0312 20:10:13.237610 1094 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "f78a2351568c4a268ee1112681e22bec" candidate_term: 24 candidate_status { last_received { term: 23 index: 108 } } ignore_live_leader: false dest_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" is_pre_election: true I0312 20:10:13.238176 1094 raft_consensus.cc:2176] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 23 LEADER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 24 because replica is either leader or believes a valid leader to be alive. I0312 20:10:13.239351 1278 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 24 pre-election: Vote denied by peer ded9b7f8677940fb91e1a8c8bceceaed. Message: Invalid argument: T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 23 LEADER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 24 because replica is either leader or believes a valid leader to be alive. I0312 20:10:13.239585 1278 leader_election.cc:258] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. I0312 20:10:13.240216 1464 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec: Snoozing failure detection for 0.078s (election complete) I0312 20:10:13.240429 1464 raft_consensus.cc:2455] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority I0312 20:10:13.343585 1464 raft_consensus.cc:436] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 23 FOLLOWER]: Starting pre-election (detected failure of leader ded9b7f8677940fb91e1a8c8bceceaed) I0312 20:10:13.343853 1464 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec: Snoozing failure detection for 0.313s (starting election) I0312 20:10:13.344079 1464 raft_consensus.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:13.345161 1464 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 24 pre-election: Requesting pre-vote from peer 6885de38238442d6a9e5f1eb9fb569ff I0312 20:10:13.345532 1464 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 24 pre-election: Requesting pre-vote from peer ded9b7f8677940fb91e1a8c8bceceaed I0312 20:10:13.347154 1218 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "f78a2351568c4a268ee1112681e22bec" candidate_term: 24 candidate_status { last_received { term: 23 index: 118 } } ignore_live_leader: false dest_uuid: "6885de38238442d6a9e5f1eb9fb569ff" is_pre_election: true I0312 20:10:13.349756 1094 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "f78a2351568c4a268ee1112681e22bec" candidate_term: 24 candidate_status { last_received { term: 23 index: 118 } } ignore_live_leader: false dest_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" is_pre_election: true I0312 20:10:13.350294 1094 raft_consensus.cc:2176] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 23 LEADER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 24 because replica is either leader or believes a valid leader to be alive. I0312 20:10:13.353492 1467 raft_consensus.cc:436] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 23 FOLLOWER]: Starting pre-election (detected failure of leader ded9b7f8677940fb91e1a8c8bceceaed) I0312 20:10:13.353744 1467 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff: Snoozing failure detection for 0.033s (starting election) I0312 20:10:13.354013 1467 raft_consensus.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:13.355880 1467 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 24 pre-election: Requesting pre-vote from peer f78a2351568c4a268ee1112681e22bec I0312 20:10:13.356273 1467 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 24 pre-election: Requesting pre-vote from peer ded9b7f8677940fb91e1a8c8bceceaed I0312 20:10:13.357309 1218 raft_consensus.cc:2190] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 24 because replica is already servicing an update from a current leader or another vote. I0312 20:10:13.357499 1278 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 24 pre-election: Vote denied by peer ded9b7f8677940fb91e1a8c8bceceaed. Message: Invalid argument: T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 23 LEADER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 24 because replica is either leader or believes a valid leader to be alive. I0312 20:10:13.358417 1279 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 24 pre-election: Vote denied by peer 6885de38238442d6a9e5f1eb9fb569ff. Message: Service unavailable: T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 24 because replica is already servicing an update from a current leader or another vote. I0312 20:10:13.358644 1279 leader_election.cc:258] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. I0312 20:10:13.358825 1094 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "6885de38238442d6a9e5f1eb9fb569ff" candidate_term: 24 candidate_status { last_received { term: 23 index: 118 } } ignore_live_leader: false dest_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" is_pre_election: true I0312 20:10:13.359381 1094 raft_consensus.cc:2176] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 23 LEADER]: Leader pre-election vote request: Denying vote to candidate 6885de38238442d6a9e5f1eb9fb569ff for term 24 because replica is either leader or believes a valid leader to be alive. I0312 20:10:13.360476 1153 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 24 pre-election: Vote denied by peer ded9b7f8677940fb91e1a8c8bceceaed. Message: Invalid argument: T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 23 LEADER]: Leader pre-election vote request: Denying vote to candidate 6885de38238442d6a9e5f1eb9fb569ff for term 24 because replica is either leader or believes a valid leader to be alive. I0312 20:10:13.361819 1344 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "6885de38238442d6a9e5f1eb9fb569ff" candidate_term: 24 candidate_status { last_received { term: 23 index: 118 } } ignore_live_leader: false dest_uuid: "f78a2351568c4a268ee1112681e22bec" is_pre_election: true I0312 20:10:13.376157 1344 raft_consensus.cc:2190] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6885de38238442d6a9e5f1eb9fb569ff for term 24 because replica is already servicing an update from a current leader or another vote. I0312 20:10:13.377346 1152 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 24 pre-election: Vote denied by peer f78a2351568c4a268ee1112681e22bec. Message: Service unavailable: T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6885de38238442d6a9e5f1eb9fb569ff for term 24 because replica is already servicing an update from a current leader or another vote. I0312 20:10:13.377346 1464 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec: Snoozing failure detection for 0.077s (election complete) I0312 20:10:13.377568 1152 leader_election.cc:258] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. I0312 20:10:13.377569 1464 raft_consensus.cc:2455] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority I0312 20:10:13.377961 1467 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff: Snoozing failure detection for 0.019s (election complete) I0312 20:10:13.378170 1467 raft_consensus.cc:2455] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority I0312 20:10:13.378657 1468 raft_consensus.cc:436] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 23 FOLLOWER]: Starting pre-election (detected failure of leader ded9b7f8677940fb91e1a8c8bceceaed) I0312 20:10:13.378921 1468 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec: Snoozing failure detection for 0.134s (starting election) I0312 20:10:13.379166 1468 raft_consensus.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:13.380955 1468 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 24 pre-election: Requesting pre-vote from peer 6885de38238442d6a9e5f1eb9fb569ff I0312 20:10:13.382400 1468 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 24 pre-election: Requesting pre-vote from peer ded9b7f8677940fb91e1a8c8bceceaed I0312 20:10:13.382189 1216 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "f78a2351568c4a268ee1112681e22bec" candidate_term: 24 candidate_status { last_received { term: 23 index: 119 } } ignore_live_leader: false dest_uuid: "6885de38238442d6a9e5f1eb9fb569ff" is_pre_election: true I0312 20:10:13.384655 1216 raft_consensus.cc:2176] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 24 because replica is either leader or believes a valid leader to be alive. I0312 20:10:13.385601 1279 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 24 pre-election: Vote denied by peer 6885de38238442d6a9e5f1eb9fb569ff. Message: Invalid argument: T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 24 because replica is either leader or believes a valid leader to be alive. I0312 20:10:13.387329 1094 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "f78a2351568c4a268ee1112681e22bec" candidate_term: 24 candidate_status { last_received { term: 23 index: 119 } } ignore_live_leader: false dest_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" is_pre_election: true I0312 20:10:13.387848 1094 raft_consensus.cc:2176] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 23 LEADER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 24 because replica is either leader or believes a valid leader to be alive. I0312 20:10:13.388906 1278 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 24 pre-election: Vote denied by peer ded9b7f8677940fb91e1a8c8bceceaed. Message: Invalid argument: T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 23 LEADER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 24 because replica is either leader or believes a valid leader to be alive. I0312 20:10:13.389145 1278 leader_election.cc:258] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. I0312 20:10:13.389745 1468 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec: Snoozing failure detection for 0.084s (election complete) I0312 20:10:13.389962 1468 raft_consensus.cc:2455] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority I0312 20:10:13.407733 1468 raft_consensus.cc:436] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 23 FOLLOWER]: Starting pre-election (detected failure of leader ded9b7f8677940fb91e1a8c8bceceaed) I0312 20:10:13.407981 1468 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec: Snoozing failure detection for 0.061s (starting election) I0312 20:10:13.408212 1468 raft_consensus.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:13.409255 1468 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 24 pre-election: Requesting pre-vote from peer 6885de38238442d6a9e5f1eb9fb569ff I0312 20:10:13.410502 1218 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "f78a2351568c4a268ee1112681e22bec" candidate_term: 24 candidate_status { last_received { term: 23 index: 121 } } ignore_live_leader: false dest_uuid: "6885de38238442d6a9e5f1eb9fb569ff" is_pre_election: true I0312 20:10:13.411028 1218 raft_consensus.cc:2176] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 24 because replica is either leader or believes a valid leader to be alive. I0312 20:10:13.411873 1279 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 24 pre-election: Vote denied by peer 6885de38238442d6a9e5f1eb9fb569ff. Message: Invalid argument: T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 24 because replica is either leader or believes a valid leader to be alive. I0312 20:10:13.412591 1468 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 24 pre-election: Requesting pre-vote from peer ded9b7f8677940fb91e1a8c8bceceaed I0312 20:10:13.434239 1094 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "f78a2351568c4a268ee1112681e22bec" candidate_term: 24 candidate_status { last_received { term: 23 index: 121 } } ignore_live_leader: false dest_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" is_pre_election: true I0312 20:10:13.434793 1094 raft_consensus.cc:2176] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 23 LEADER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 24 because replica is either leader or believes a valid leader to be alive. I0312 20:10:13.444924 1278 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 24 pre-election: Vote denied by peer ded9b7f8677940fb91e1a8c8bceceaed. Message: Invalid argument: T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 23 LEADER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 24 because replica is either leader or believes a valid leader to be alive. I0312 20:10:13.445183 1278 leader_election.cc:258] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. I0312 20:10:13.445583 1468 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec: Snoozing failure detection for 0.788s (election complete) I0312 20:10:13.445792 1468 raft_consensus.cc:2455] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority I0312 20:10:13.483060 1467 raft_consensus.cc:436] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 23 FOLLOWER]: Starting pre-election (detected failure of leader ded9b7f8677940fb91e1a8c8bceceaed) I0312 20:10:13.483348 1467 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff: Snoozing failure detection for 0.016s (starting election) I0312 20:10:13.483623 1467 raft_consensus.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:13.484776 1467 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 24 pre-election: Requesting pre-vote from peer f78a2351568c4a268ee1112681e22bec I0312 20:10:13.485555 1467 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 24 pre-election: Requesting pre-vote from peer ded9b7f8677940fb91e1a8c8bceceaed I0312 20:10:13.487138 1094 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "6885de38238442d6a9e5f1eb9fb569ff" candidate_term: 24 candidate_status { last_received { term: 23 index: 126 } } ignore_live_leader: false dest_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" is_pre_election: true I0312 20:10:13.487660 1094 raft_consensus.cc:2190] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 23 LEADER]: Leader pre-election vote request: Denying vote to candidate 6885de38238442d6a9e5f1eb9fb569ff for term 24 because replica is already servicing an update from a current leader or another vote. I0312 20:10:13.488804 1153 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 24 pre-election: Vote denied by peer ded9b7f8677940fb91e1a8c8bceceaed. Message: Service unavailable: T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 23 LEADER]: Leader pre-election vote request: Denying vote to candidate 6885de38238442d6a9e5f1eb9fb569ff for term 24 because replica is already servicing an update from a current leader or another vote. I0312 20:10:13.489552 1344 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "6885de38238442d6a9e5f1eb9fb569ff" candidate_term: 24 candidate_status { last_received { term: 23 index: 126 } } ignore_live_leader: false dest_uuid: "f78a2351568c4a268ee1112681e22bec" is_pre_election: true I0312 20:10:13.491626 1344 raft_consensus.cc:2190] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6885de38238442d6a9e5f1eb9fb569ff for term 24 because replica is already servicing an update from a current leader or another vote. I0312 20:10:13.493978 1152 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 24 pre-election: Vote denied by peer f78a2351568c4a268ee1112681e22bec. Message: Service unavailable: T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6885de38238442d6a9e5f1eb9fb569ff for term 24 because replica is already servicing an update from a current leader or another vote. I0312 20:10:13.494205 1152 leader_election.cc:258] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. I0312 20:10:13.494647 1467 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff: Snoozing failure detection for 0.015s (election complete) I0312 20:10:13.494848 1467 raft_consensus.cc:2455] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority I0312 20:10:13.538803 1468 raft_consensus.cc:436] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 23 FOLLOWER]: Starting pre-election (detected failure of leader ded9b7f8677940fb91e1a8c8bceceaed) I0312 20:10:13.539065 1468 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec: Snoozing failure detection for 0.167s (starting election) I0312 20:10:13.539317 1468 raft_consensus.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:13.540370 1468 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 24 pre-election: Requesting pre-vote from peer 6885de38238442d6a9e5f1eb9fb569ff I0312 20:10:13.540748 1468 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 24 pre-election: Requesting pre-vote from peer ded9b7f8677940fb91e1a8c8bceceaed I0312 20:10:13.542361 1216 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "f78a2351568c4a268ee1112681e22bec" candidate_term: 24 candidate_status { last_received { term: 23 index: 130 } } ignore_live_leader: false dest_uuid: "6885de38238442d6a9e5f1eb9fb569ff" is_pre_election: true I0312 20:10:13.542966 1216 raft_consensus.cc:2176] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 24 because replica is either leader or believes a valid leader to be alive. I0312 20:10:13.545825 1094 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "f78a2351568c4a268ee1112681e22bec" candidate_term: 24 candidate_status { last_received { term: 23 index: 130 } } ignore_live_leader: false dest_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" is_pre_election: true I0312 20:10:13.546386 1094 raft_consensus.cc:2176] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 23 LEADER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 24 because replica is either leader or believes a valid leader to be alive. I0312 20:10:13.547322 1279 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 24 pre-election: Vote denied by peer 6885de38238442d6a9e5f1eb9fb569ff. Message: Invalid argument: T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 24 because replica is either leader or believes a valid leader to be alive. I0312 20:10:13.552769 1278 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 24 pre-election: Vote denied by peer ded9b7f8677940fb91e1a8c8bceceaed. Message: Invalid argument: T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 23 LEADER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 24 because replica is either leader or believes a valid leader to be alive. I0312 20:10:13.552991 1278 leader_election.cc:258] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. I0312 20:10:13.553597 1468 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec: Snoozing failure detection for 1.157s (election complete) I0312 20:10:13.553798 1468 raft_consensus.cc:2455] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority I0312 20:10:13.595587 1468 raft_consensus.cc:436] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 23 FOLLOWER]: Starting pre-election (detected failure of leader ded9b7f8677940fb91e1a8c8bceceaed) I0312 20:10:13.595849 1468 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec: Snoozing failure detection for 0.252s (starting election) I0312 20:10:13.596091 1468 raft_consensus.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:13.597225 1468 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 24 pre-election: Requesting pre-vote from peer 6885de38238442d6a9e5f1eb9fb569ff I0312 20:10:13.597661 1468 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 24 pre-election: Requesting pre-vote from peer ded9b7f8677940fb91e1a8c8bceceaed I0312 20:10:13.598594 1216 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "f78a2351568c4a268ee1112681e22bec" candidate_term: 24 candidate_status { last_received { term: 23 index: 133 } } ignore_live_leader: false dest_uuid: "6885de38238442d6a9e5f1eb9fb569ff" is_pre_election: true I0312 20:10:13.599481 1216 raft_consensus.cc:2176] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 24 because replica is either leader or believes a valid leader to be alive. I0312 20:10:13.600400 1279 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 24 pre-election: Vote denied by peer 6885de38238442d6a9e5f1eb9fb569ff. Message: Invalid argument: T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 24 because replica is either leader or believes a valid leader to be alive. I0312 20:10:13.604801 1094 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "f78a2351568c4a268ee1112681e22bec" candidate_term: 24 candidate_status { last_received { term: 23 index: 133 } } ignore_live_leader: false dest_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" is_pre_election: true I0312 20:10:13.605366 1094 raft_consensus.cc:2176] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 23 LEADER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 24 because replica is either leader or believes a valid leader to be alive. I0312 20:10:13.612599 1278 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 24 pre-election: Vote denied by peer ded9b7f8677940fb91e1a8c8bceceaed. Message: Invalid argument: T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 23 LEADER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 24 because replica is either leader or believes a valid leader to be alive. I0312 20:10:13.612849 1278 leader_election.cc:258] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. I0312 20:10:13.613517 1468 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec: Snoozing failure detection for 0.095s (election complete) I0312 20:10:13.613755 1468 raft_consensus.cc:2455] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority I0312 20:10:13.716830 1468 raft_consensus.cc:436] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 23 FOLLOWER]: Starting pre-election (detected failure of leader ded9b7f8677940fb91e1a8c8bceceaed) I0312 20:10:13.717123 1468 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec: Snoozing failure detection for 2.255s (starting election) I0312 20:10:13.717388 1468 raft_consensus.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:13.718487 1468 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 24 pre-election: Requesting pre-vote from peer 6885de38238442d6a9e5f1eb9fb569ff I0312 20:10:13.718896 1468 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 24 pre-election: Requesting pre-vote from peer ded9b7f8677940fb91e1a8c8bceceaed I0312 20:10:13.720748 1216 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "f78a2351568c4a268ee1112681e22bec" candidate_term: 24 candidate_status { last_received { term: 23 index: 141 } } ignore_live_leader: false dest_uuid: "6885de38238442d6a9e5f1eb9fb569ff" is_pre_election: true I0312 20:10:13.723544 1216 raft_consensus.cc:2190] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 24 because replica is already servicing an update from a current leader or another vote. I0312 20:10:13.724815 1279 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 24 pre-election: Vote denied by peer 6885de38238442d6a9e5f1eb9fb569ff. Message: Service unavailable: T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 24 because replica is already servicing an update from a current leader or another vote. I0312 20:10:13.733312 1094 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "f78a2351568c4a268ee1112681e22bec" candidate_term: 24 candidate_status { last_received { term: 23 index: 141 } } ignore_live_leader: false dest_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" is_pre_election: true I0312 20:10:13.733865 1094 raft_consensus.cc:2176] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 23 LEADER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 24 because replica is either leader or believes a valid leader to be alive. I0312 20:10:13.738481 1278 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 24 pre-election: Vote denied by peer ded9b7f8677940fb91e1a8c8bceceaed. Message: Invalid argument: T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 23 LEADER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 24 because replica is either leader or believes a valid leader to be alive. I0312 20:10:13.738729 1278 leader_election.cc:258] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. I0312 20:10:13.742288 1468 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec: Snoozing failure detection for 1.373s (election complete) I0312 20:10:13.742509 1468 raft_consensus.cc:2455] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority I0312 20:10:13.795614 1467 raft_consensus.cc:436] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 23 FOLLOWER]: Starting pre-election (detected failure of leader ded9b7f8677940fb91e1a8c8bceceaed) I0312 20:10:13.795863 1467 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff: Snoozing failure detection for 0.070s (starting election) I0312 20:10:13.796090 1467 raft_consensus.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:13.797286 1467 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 24 pre-election: Requesting pre-vote from peer f78a2351568c4a268ee1112681e22bec I0312 20:10:13.797652 1467 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 24 pre-election: Requesting pre-vote from peer ded9b7f8677940fb91e1a8c8bceceaed I0312 20:10:13.799263 1094 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "6885de38238442d6a9e5f1eb9fb569ff" candidate_term: 24 candidate_status { last_received { term: 23 index: 147 } } ignore_live_leader: false dest_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" is_pre_election: true I0312 20:10:13.801403 1094 raft_consensus.cc:2176] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 23 LEADER]: Leader pre-election vote request: Denying vote to candidate 6885de38238442d6a9e5f1eb9fb569ff for term 24 because replica is either leader or believes a valid leader to be alive. I0312 20:10:13.802564 1153 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 24 pre-election: Vote denied by peer ded9b7f8677940fb91e1a8c8bceceaed. Message: Invalid argument: T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 23 LEADER]: Leader pre-election vote request: Denying vote to candidate 6885de38238442d6a9e5f1eb9fb569ff for term 24 because replica is either leader or believes a valid leader to be alive. I0312 20:10:13.813298 1344 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "6885de38238442d6a9e5f1eb9fb569ff" candidate_term: 24 candidate_status { last_received { term: 23 index: 147 } } ignore_live_leader: false dest_uuid: "f78a2351568c4a268ee1112681e22bec" is_pre_election: true I0312 20:10:13.813854 1344 raft_consensus.cc:2176] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6885de38238442d6a9e5f1eb9fb569ff for term 24 because replica is either leader or believes a valid leader to be alive. I0312 20:10:13.814795 1152 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 24 pre-election: Vote denied by peer f78a2351568c4a268ee1112681e22bec. Message: Invalid argument: T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6885de38238442d6a9e5f1eb9fb569ff for term 24 because replica is either leader or believes a valid leader to be alive. I0312 20:10:13.815018 1152 leader_election.cc:258] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. I0312 20:10:13.815661 1467 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff: Snoozing failure detection for 0.039s (election complete) I0312 20:10:13.815871 1467 raft_consensus.cc:2455] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority I0312 20:10:13.851811 1468 raft_consensus.cc:436] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 23 FOLLOWER]: Starting pre-election (detected failure of leader ded9b7f8677940fb91e1a8c8bceceaed) I0312 20:10:13.852087 1468 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec: Snoozing failure detection for 4.262s (starting election) I0312 20:10:13.852352 1468 raft_consensus.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:13.853173 1468 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 24 pre-election: Requesting pre-vote from peer 6885de38238442d6a9e5f1eb9fb569ff I0312 20:10:13.853508 1468 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 24 pre-election: Requesting pre-vote from peer ded9b7f8677940fb91e1a8c8bceceaed I0312 20:10:13.854457 1218 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "f78a2351568c4a268ee1112681e22bec" candidate_term: 24 candidate_status { last_received { term: 23 index: 153 } } ignore_live_leader: false dest_uuid: "6885de38238442d6a9e5f1eb9fb569ff" is_pre_election: true I0312 20:10:13.855660 1218 raft_consensus.cc:2176] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 24 because replica is either leader or believes a valid leader to be alive. I0312 20:10:13.857532 1279 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 24 pre-election: Vote denied by peer 6885de38238442d6a9e5f1eb9fb569ff. Message: Invalid argument: T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 24 because replica is either leader or believes a valid leader to be alive. I0312 20:10:13.858598 1094 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "f78a2351568c4a268ee1112681e22bec" candidate_term: 24 candidate_status { last_received { term: 23 index: 153 } } ignore_live_leader: false dest_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" is_pre_election: true I0312 20:10:13.859118 1094 raft_consensus.cc:2176] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 23 LEADER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 24 because replica is either leader or believes a valid leader to be alive. I0312 20:10:13.859930 1278 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 24 pre-election: Vote denied by peer ded9b7f8677940fb91e1a8c8bceceaed. Message: Invalid argument: T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 23 LEADER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 24 because replica is either leader or believes a valid leader to be alive. I0312 20:10:13.860129 1278 leader_election.cc:258] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. I0312 20:10:13.867770 1468 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec: Snoozing failure detection for 1.548s (election complete) I0312 20:10:13.867992 1468 raft_consensus.cc:2455] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority I0312 20:10:13.897341 1468 raft_consensus.cc:436] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 23 FOLLOWER]: Starting pre-election (detected failure of leader ded9b7f8677940fb91e1a8c8bceceaed) I0312 20:10:13.897624 1468 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec: Snoozing failure detection for 4.084s (starting election) I0312 20:10:13.897887 1468 raft_consensus.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:13.899108 1468 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 24 pre-election: Requesting pre-vote from peer 6885de38238442d6a9e5f1eb9fb569ff I0312 20:10:13.899499 1468 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 24 pre-election: Requesting pre-vote from peer ded9b7f8677940fb91e1a8c8bceceaed I0312 20:10:13.901623 1215 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "f78a2351568c4a268ee1112681e22bec" candidate_term: 24 candidate_status { last_received { term: 23 index: 156 } } ignore_live_leader: false dest_uuid: "6885de38238442d6a9e5f1eb9fb569ff" is_pre_election: true I0312 20:10:13.902166 1215 raft_consensus.cc:2176] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 24 because replica is either leader or believes a valid leader to be alive. I0312 20:10:13.903543 1279 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 24 pre-election: Vote denied by peer 6885de38238442d6a9e5f1eb9fb569ff. Message: Invalid argument: T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 24 because replica is either leader or believes a valid leader to be alive. I0312 20:10:13.905290 1094 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "f78a2351568c4a268ee1112681e22bec" candidate_term: 24 candidate_status { last_received { term: 23 index: 156 } } ignore_live_leader: false dest_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" is_pre_election: true I0312 20:10:13.906280 1094 raft_consensus.cc:2190] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 23 LEADER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 24 because replica is already servicing an update from a current leader or another vote. I0312 20:10:13.907148 1278 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 24 pre-election: Vote denied by peer ded9b7f8677940fb91e1a8c8bceceaed. Message: Service unavailable: T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 23 LEADER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 24 because replica is already servicing an update from a current leader or another vote. I0312 20:10:13.907375 1278 leader_election.cc:258] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. I0312 20:10:13.908006 1468 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec: Snoozing failure detection for 3.096s (election complete) I0312 20:10:13.908212 1468 raft_consensus.cc:2455] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority I0312 20:10:13.926262 1468 raft_consensus.cc:436] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 23 FOLLOWER]: Starting pre-election (detected failure of leader ded9b7f8677940fb91e1a8c8bceceaed) I0312 20:10:13.926477 1468 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec: Snoozing failure detection for 7.620s (starting election) I0312 20:10:13.926692 1468 raft_consensus.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:13.927642 1468 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 24 pre-election: Requesting pre-vote from peer 6885de38238442d6a9e5f1eb9fb569ff I0312 20:10:13.927983 1468 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 24 pre-election: Requesting pre-vote from peer ded9b7f8677940fb91e1a8c8bceceaed I0312 20:10:13.929605 1215 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "f78a2351568c4a268ee1112681e22bec" candidate_term: 24 candidate_status { last_received { term: 23 index: 160 } } ignore_live_leader: false dest_uuid: "6885de38238442d6a9e5f1eb9fb569ff" is_pre_election: true I0312 20:10:13.930132 1215 raft_consensus.cc:2176] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 24 because replica is either leader or believes a valid leader to be alive. I0312 20:10:13.931103 1279 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 24 pre-election: Vote denied by peer 6885de38238442d6a9e5f1eb9fb569ff. Message: Invalid argument: T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 24 because replica is either leader or believes a valid leader to be alive. I0312 20:10:13.931259 1094 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "f78a2351568c4a268ee1112681e22bec" candidate_term: 24 candidate_status { last_received { term: 23 index: 160 } } ignore_live_leader: false dest_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" is_pre_election: true I0312 20:10:13.931813 1094 raft_consensus.cc:2176] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 23 LEADER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 24 because replica is either leader or believes a valid leader to be alive. I0312 20:10:13.939651 1278 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 24 pre-election: Vote denied by peer ded9b7f8677940fb91e1a8c8bceceaed. Message: Invalid argument: T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 23 LEADER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 24 because replica is either leader or believes a valid leader to be alive. I0312 20:10:13.939879 1278 leader_election.cc:258] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. I0312 20:10:13.940491 1468 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec: Snoozing failure detection for 7.166s (election complete) I0312 20:10:13.940686 1468 raft_consensus.cc:2455] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority I0312 20:10:13.965401 1468 raft_consensus.cc:436] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 23 FOLLOWER]: Starting pre-election (detected failure of leader ded9b7f8677940fb91e1a8c8bceceaed) I0312 20:10:13.965703 1468 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec: Snoozing failure detection for 8.728s (starting election) I0312 20:10:13.965924 1468 raft_consensus.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:13.967015 1468 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 24 pre-election: Requesting pre-vote from peer 6885de38238442d6a9e5f1eb9fb569ff I0312 20:10:13.967360 1468 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 24 pre-election: Requesting pre-vote from peer ded9b7f8677940fb91e1a8c8bceceaed I0312 20:10:13.969924 1215 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "f78a2351568c4a268ee1112681e22bec" candidate_term: 24 candidate_status { last_received { term: 23 index: 162 } } ignore_live_leader: false dest_uuid: "6885de38238442d6a9e5f1eb9fb569ff" is_pre_election: true I0312 20:10:13.971441 1215 raft_consensus.cc:2190] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 24 because replica is already servicing an update from a current leader or another vote. I0312 20:10:13.972695 1279 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 24 pre-election: Vote denied by peer 6885de38238442d6a9e5f1eb9fb569ff. Message: Service unavailable: T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 24 because replica is already servicing an update from a current leader or another vote. I0312 20:10:13.976881 1094 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "f78a2351568c4a268ee1112681e22bec" candidate_term: 24 candidate_status { last_received { term: 23 index: 162 } } ignore_live_leader: false dest_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" is_pre_election: true I0312 20:10:13.977370 1094 raft_consensus.cc:2176] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 23 LEADER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 24 because replica is either leader or believes a valid leader to be alive. I0312 20:10:13.978436 1278 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 24 pre-election: Vote denied by peer ded9b7f8677940fb91e1a8c8bceceaed. Message: Invalid argument: T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 23 LEADER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 24 because replica is either leader or believes a valid leader to be alive. I0312 20:10:13.978739 1278 leader_election.cc:258] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. I0312 20:10:13.979499 1468 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec: Snoozing failure detection for 4.917s (election complete) I0312 20:10:13.979713 1468 raft_consensus.cc:2455] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority I0312 20:10:15.078951 1473 raft_consensus.cc:436] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 23 FOLLOWER]: Starting pre-election (detected failure of leader ded9b7f8677940fb91e1a8c8bceceaed) I0312 20:10:15.079213 1473 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec: Snoozing failure detection for 6.333s (starting election) I0312 20:10:15.079447 1473 raft_consensus.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:15.086210 1473 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 24 pre-election: Requesting pre-vote from peer 6885de38238442d6a9e5f1eb9fb569ff I0312 20:10:15.086560 1473 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 24 pre-election: Requesting pre-vote from peer ded9b7f8677940fb91e1a8c8bceceaed I0312 20:10:15.087412 1215 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "f78a2351568c4a268ee1112681e22bec" candidate_term: 24 candidate_status { last_received { term: 23 index: 322 } } ignore_live_leader: false dest_uuid: "6885de38238442d6a9e5f1eb9fb569ff" is_pre_election: true I0312 20:10:15.087858 1215 raft_consensus.cc:2176] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 24 because replica is either leader or believes a valid leader to be alive. I0312 20:10:15.088801 1279 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 24 pre-election: Vote denied by peer 6885de38238442d6a9e5f1eb9fb569ff. Message: Invalid argument: T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 24 because replica is either leader or believes a valid leader to be alive. I0312 20:10:15.089274 1094 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "f78a2351568c4a268ee1112681e22bec" candidate_term: 24 candidate_status { last_received { term: 23 index: 322 } } ignore_live_leader: false dest_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" is_pre_election: true I0312 20:10:15.090494 1094 raft_consensus.cc:2176] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 23 LEADER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 24 because replica is either leader or believes a valid leader to be alive. I0312 20:10:15.091478 1278 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 24 pre-election: Vote denied by peer ded9b7f8677940fb91e1a8c8bceceaed. Message: Invalid argument: T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 23 LEADER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 24 because replica is either leader or believes a valid leader to be alive. I0312 20:10:15.091709 1278 leader_election.cc:258] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. I0312 20:10:15.093253 1473 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec: Snoozing failure detection for 12.716s (election complete) I0312 20:10:15.093461 1473 raft_consensus.cc:2455] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority W0312 20:10:15.245105 1426 log.cc:662] Time spent T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed: Append to log took a long time: real 0.145s user 0.000s sys 0.000s W0312 20:10:15.245126 1425 log.cc:662] Time spent T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec: Append to log took a long time: real 0.147s user 0.000s sys 0.000s W0312 20:10:15.245126 1444 log.cc:662] Time spent T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff: Append to log took a long time: real 0.143s user 0.000s sys 0.000s I0312 20:10:15.267297 1474 raft_consensus.cc:436] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 23 FOLLOWER]: Starting pre-election (detected failure of leader ded9b7f8677940fb91e1a8c8bceceaed) I0312 20:10:15.267536 1474 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff: Snoozing failure detection for 0.047s (starting election) I0312 20:10:15.267750 1474 raft_consensus.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:15.269882 1474 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 24 pre-election: Requesting pre-vote from peer f78a2351568c4a268ee1112681e22bec I0312 20:10:15.270241 1474 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 24 pre-election: Requesting pre-vote from peer ded9b7f8677940fb91e1a8c8bceceaed I0312 20:10:15.271301 1094 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "6885de38238442d6a9e5f1eb9fb569ff" candidate_term: 24 candidate_status { last_received { term: 23 index: 329 } } ignore_live_leader: false dest_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" is_pre_election: true I0312 20:10:15.271687 1094 raft_consensus.cc:2176] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 23 LEADER]: Leader pre-election vote request: Denying vote to candidate 6885de38238442d6a9e5f1eb9fb569ff for term 24 because replica is either leader or believes a valid leader to be alive. I0312 20:10:15.272326 1153 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 24 pre-election: Vote denied by peer ded9b7f8677940fb91e1a8c8bceceaed. Message: Invalid argument: T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 23 LEADER]: Leader pre-election vote request: Denying vote to candidate 6885de38238442d6a9e5f1eb9fb569ff for term 24 because replica is either leader or believes a valid leader to be alive. I0312 20:10:15.274976 1345 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "6885de38238442d6a9e5f1eb9fb569ff" candidate_term: 24 candidate_status { last_received { term: 23 index: 329 } } ignore_live_leader: false dest_uuid: "f78a2351568c4a268ee1112681e22bec" is_pre_election: true I0312 20:10:15.276754 1345 raft_consensus.cc:2190] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6885de38238442d6a9e5f1eb9fb569ff for term 24 because replica is already servicing an update from a current leader or another vote. I0312 20:10:15.278769 1152 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 24 pre-election: Vote denied by peer f78a2351568c4a268ee1112681e22bec. Message: Service unavailable: T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6885de38238442d6a9e5f1eb9fb569ff for term 24 because replica is already servicing an update from a current leader or another vote. I0312 20:10:15.278990 1152 leader_election.cc:258] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. I0312 20:10:15.279399 1474 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff: Snoozing failure detection for 0.051s (election complete) I0312 20:10:15.279574 1474 raft_consensus.cc:2455] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority I0312 20:10:15.353251 1474 raft_consensus.cc:436] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 23 FOLLOWER]: Starting pre-election (detected failure of leader ded9b7f8677940fb91e1a8c8bceceaed) I0312 20:10:15.353440 1474 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff: Snoozing failure detection for 0.094s (starting election) I0312 20:10:15.353663 1474 raft_consensus.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:15.354461 1474 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 24 pre-election: Requesting pre-vote from peer f78a2351568c4a268ee1112681e22bec I0312 20:10:15.354809 1474 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 24 pre-election: Requesting pre-vote from peer ded9b7f8677940fb91e1a8c8bceceaed I0312 20:10:15.356554 1344 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "6885de38238442d6a9e5f1eb9fb569ff" candidate_term: 24 candidate_status { last_received { term: 23 index: 341 } } ignore_live_leader: false dest_uuid: "f78a2351568c4a268ee1112681e22bec" is_pre_election: true I0312 20:10:15.357127 1344 raft_consensus.cc:2176] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6885de38238442d6a9e5f1eb9fb569ff for term 24 because replica is either leader or believes a valid leader to be alive. I0312 20:10:15.357965 1152 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 24 pre-election: Vote denied by peer f78a2351568c4a268ee1112681e22bec. Message: Invalid argument: T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6885de38238442d6a9e5f1eb9fb569ff for term 24 because replica is either leader or believes a valid leader to be alive. I0312 20:10:15.358238 1094 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "6885de38238442d6a9e5f1eb9fb569ff" candidate_term: 24 candidate_status { last_received { term: 23 index: 341 } } ignore_live_leader: false dest_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" is_pre_election: true I0312 20:10:15.358814 1094 raft_consensus.cc:2176] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 23 LEADER]: Leader pre-election vote request: Denying vote to candidate 6885de38238442d6a9e5f1eb9fb569ff for term 24 because replica is either leader or believes a valid leader to be alive. I0312 20:10:15.359798 1153 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 24 pre-election: Vote denied by peer ded9b7f8677940fb91e1a8c8bceceaed. Message: Invalid argument: T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 23 LEADER]: Leader pre-election vote request: Denying vote to candidate 6885de38238442d6a9e5f1eb9fb569ff for term 24 because replica is either leader or believes a valid leader to be alive. I0312 20:10:15.360010 1153 leader_election.cc:258] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. I0312 20:10:15.364307 1474 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff: Snoozing failure detection for 0.092s (election complete) I0312 20:10:15.364507 1474 raft_consensus.cc:2455] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority I0312 20:10:15.980474 1475 raft_consensus.cc:436] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 23 FOLLOWER]: Starting pre-election (detected failure of leader ded9b7f8677940fb91e1a8c8bceceaed) I0312 20:10:15.980702 1475 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec: Snoozing failure detection for 9.184s (starting election) I0312 20:10:15.980890 1475 raft_consensus.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:15.981722 1475 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 24 pre-election: Requesting pre-vote from peer 6885de38238442d6a9e5f1eb9fb569ff I0312 20:10:15.982097 1475 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 24 pre-election: Requesting pre-vote from peer ded9b7f8677940fb91e1a8c8bceceaed I0312 20:10:15.982888 1218 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "f78a2351568c4a268ee1112681e22bec" candidate_term: 24 candidate_status { last_received { term: 23 index: 440 } } ignore_live_leader: false dest_uuid: "6885de38238442d6a9e5f1eb9fb569ff" is_pre_election: true I0312 20:10:15.983455 1218 raft_consensus.cc:2176] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 24 because replica is either leader or believes a valid leader to be alive. I0312 20:10:15.983296 1094 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "f78a2351568c4a268ee1112681e22bec" candidate_term: 24 candidate_status { last_received { term: 23 index: 440 } } ignore_live_leader: false dest_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" is_pre_election: true I0312 20:10:15.984159 1094 raft_consensus.cc:2190] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 23 LEADER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 24 because replica is already servicing an update from a current leader or another vote. I0312 20:10:15.984323 1279 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 24 pre-election: Vote denied by peer 6885de38238442d6a9e5f1eb9fb569ff. Message: Invalid argument: T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 24 because replica is either leader or believes a valid leader to be alive. I0312 20:10:15.985113 1278 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 24 pre-election: Vote denied by peer ded9b7f8677940fb91e1a8c8bceceaed. Message: Service unavailable: T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 23 LEADER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 24 because replica is already servicing an update from a current leader or another vote. I0312 20:10:15.985350 1278 leader_election.cc:258] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. I0312 20:10:15.985721 1475 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec: Snoozing failure detection for 4.638s (election complete) I0312 20:10:15.985925 1475 raft_consensus.cc:2455] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority I0312 20:10:16.008725 1475 raft_consensus.cc:436] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 23 FOLLOWER]: Starting pre-election (detected failure of leader ded9b7f8677940fb91e1a8c8bceceaed) I0312 20:10:16.008952 1475 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec: Snoozing failure detection for 10.019s (starting election) I0312 20:10:16.009189 1475 raft_consensus.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:16.017340 1475 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 24 pre-election: Requesting pre-vote from peer 6885de38238442d6a9e5f1eb9fb569ff I0312 20:10:16.017695 1475 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 24 pre-election: Requesting pre-vote from peer ded9b7f8677940fb91e1a8c8bceceaed I0312 20:10:16.018842 1216 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "f78a2351568c4a268ee1112681e22bec" candidate_term: 24 candidate_status { last_received { term: 23 index: 442 } } ignore_live_leader: false dest_uuid: "6885de38238442d6a9e5f1eb9fb569ff" is_pre_election: true I0312 20:10:16.019068 1094 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "f78a2351568c4a268ee1112681e22bec" candidate_term: 24 candidate_status { last_received { term: 23 index: 442 } } ignore_live_leader: false dest_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" is_pre_election: true I0312 20:10:16.019346 1216 raft_consensus.cc:2176] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 24 because replica is either leader or believes a valid leader to be alive. I0312 20:10:16.020057 1279 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 24 pre-election: Vote denied by peer 6885de38238442d6a9e5f1eb9fb569ff. Message: Invalid argument: T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 24 because replica is either leader or believes a valid leader to be alive. I0312 20:10:16.023077 1094 raft_consensus.cc:2176] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 23 LEADER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 24 because replica is either leader or believes a valid leader to be alive. I0312 20:10:16.024307 1278 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 24 pre-election: Vote denied by peer ded9b7f8677940fb91e1a8c8bceceaed. Message: Invalid argument: T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 23 LEADER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 24 because replica is either leader or believes a valid leader to be alive. I0312 20:10:16.024544 1278 leader_election.cc:258] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. I0312 20:10:16.029834 1475 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec: Snoozing failure detection for 16.142s (election complete) I0312 20:10:16.030238 1475 raft_consensus.cc:2455] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority I0312 20:10:16.887995 1478 raft_consensus.cc:436] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 23 FOLLOWER]: Starting pre-election (detected failure of leader ded9b7f8677940fb91e1a8c8bceceaed) I0312 20:10:16.888257 1478 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec: Snoozing failure detection for 18.272s (starting election) I0312 20:10:16.891332 1478 raft_consensus.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:16.898876 1478 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 24 pre-election: Requesting pre-vote from peer 6885de38238442d6a9e5f1eb9fb569ff I0312 20:10:16.910724 1478 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 24 pre-election: Requesting pre-vote from peer ded9b7f8677940fb91e1a8c8bceceaed I0312 20:10:16.912022 1216 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "f78a2351568c4a268ee1112681e22bec" candidate_term: 24 candidate_status { last_received { term: 23 index: 580 } } ignore_live_leader: false dest_uuid: "6885de38238442d6a9e5f1eb9fb569ff" is_pre_election: true I0312 20:10:16.912683 1216 raft_consensus.cc:2190] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 24 because replica is already servicing an update from a current leader or another vote. I0312 20:10:16.913677 1094 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "f78a2351568c4a268ee1112681e22bec" candidate_term: 24 candidate_status { last_received { term: 23 index: 580 } } ignore_live_leader: false dest_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" is_pre_election: true I0312 20:10:16.914678 1279 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 24 pre-election: Vote denied by peer 6885de38238442d6a9e5f1eb9fb569ff. Message: Service unavailable: T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 24 because replica is already servicing an update from a current leader or another vote. I0312 20:10:16.918427 1094 raft_consensus.cc:2176] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 23 LEADER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 24 because replica is either leader or believes a valid leader to be alive. I0312 20:10:16.919587 1278 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 24 pre-election: Vote denied by peer ded9b7f8677940fb91e1a8c8bceceaed. Message: Invalid argument: T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 23 LEADER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 24 because replica is either leader or believes a valid leader to be alive. I0312 20:10:16.919811 1278 leader_election.cc:258] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. I0312 20:10:16.931509 1478 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec: Snoozing failure detection for 1.711s (election complete) I0312 20:10:16.931748 1478 raft_consensus.cc:2455] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority I0312 20:10:17.043216 1480 raft_consensus.cc:436] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 23 FOLLOWER]: Starting pre-election (detected failure of leader ded9b7f8677940fb91e1a8c8bceceaed) I0312 20:10:17.044492 1480 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff: Snoozing failure detection for 0.070s (starting election) I0312 20:10:17.044800 1480 raft_consensus.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:17.048892 1480 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 24 pre-election: Requesting pre-vote from peer f78a2351568c4a268ee1112681e22bec I0312 20:10:17.049284 1480 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 24 pre-election: Requesting pre-vote from peer ded9b7f8677940fb91e1a8c8bceceaed I0312 20:10:17.050382 1344 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "6885de38238442d6a9e5f1eb9fb569ff" candidate_term: 24 candidate_status { last_received { term: 23 index: 604 } } ignore_live_leader: false dest_uuid: "f78a2351568c4a268ee1112681e22bec" is_pre_election: true I0312 20:10:17.050909 1344 raft_consensus.cc:2176] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6885de38238442d6a9e5f1eb9fb569ff for term 24 because replica is either leader or believes a valid leader to be alive. I0312 20:10:17.053509 1152 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 24 pre-election: Vote denied by peer f78a2351568c4a268ee1112681e22bec. Message: Invalid argument: T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6885de38238442d6a9e5f1eb9fb569ff for term 24 because replica is either leader or believes a valid leader to be alive. I0312 20:10:17.054764 1094 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "6885de38238442d6a9e5f1eb9fb569ff" candidate_term: 24 candidate_status { last_received { term: 23 index: 604 } } ignore_live_leader: false dest_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" is_pre_election: true I0312 20:10:17.055299 1094 raft_consensus.cc:2176] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 23 LEADER]: Leader pre-election vote request: Denying vote to candidate 6885de38238442d6a9e5f1eb9fb569ff for term 24 because replica is either leader or believes a valid leader to be alive. I0312 20:10:17.056241 1153 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 24 pre-election: Vote denied by peer ded9b7f8677940fb91e1a8c8bceceaed. Message: Invalid argument: T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 23 LEADER]: Leader pre-election vote request: Denying vote to candidate 6885de38238442d6a9e5f1eb9fb569ff for term 24 because replica is either leader or believes a valid leader to be alive. I0312 20:10:17.056460 1153 leader_election.cc:258] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. I0312 20:10:17.062105 1480 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff: Snoozing failure detection for 0.089s (election complete) I0312 20:10:17.062331 1480 raft_consensus.cc:2455] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority I0312 20:10:17.110813 1478 raft_consensus.cc:436] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 23 FOLLOWER]: Starting pre-election (detected failure of leader ded9b7f8677940fb91e1a8c8bceceaed) I0312 20:10:17.111063 1478 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec: Snoozing failure detection for 16.279s (starting election) I0312 20:10:17.111284 1478 raft_consensus.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:17.112293 1478 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 24 pre-election: Requesting pre-vote from peer 6885de38238442d6a9e5f1eb9fb569ff I0312 20:10:17.112664 1478 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 24 pre-election: Requesting pre-vote from peer ded9b7f8677940fb91e1a8c8bceceaed I0312 20:10:17.113447 1216 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "f78a2351568c4a268ee1112681e22bec" candidate_term: 24 candidate_status { last_received { term: 23 index: 613 } } ignore_live_leader: false dest_uuid: "6885de38238442d6a9e5f1eb9fb569ff" is_pre_election: true I0312 20:10:17.113806 1216 raft_consensus.cc:2176] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 24 because replica is either leader or believes a valid leader to be alive. I0312 20:10:17.114487 1279 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 24 pre-election: Vote denied by peer 6885de38238442d6a9e5f1eb9fb569ff. Message: Invalid argument: T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 24 because replica is either leader or believes a valid leader to be alive. I0312 20:10:17.122584 1094 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "f78a2351568c4a268ee1112681e22bec" candidate_term: 24 candidate_status { last_received { term: 23 index: 613 } } ignore_live_leader: false dest_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" is_pre_election: true I0312 20:10:17.123368 1094 raft_consensus.cc:2176] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 23 LEADER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 24 because replica is either leader or believes a valid leader to be alive. I0312 20:10:17.125175 1278 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 24 pre-election: Vote denied by peer ded9b7f8677940fb91e1a8c8bceceaed. Message: Invalid argument: T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 23 LEADER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 24 because replica is either leader or believes a valid leader to be alive. I0312 20:10:17.125373 1278 leader_election.cc:258] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. I0312 20:10:17.126031 1478 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec: Snoozing failure detection for 14.874s (election complete) I0312 20:10:17.126236 1478 raft_consensus.cc:2455] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority I0312 20:10:17.500850 1478 raft_consensus.cc:436] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 23 FOLLOWER]: Starting pre-election (detected failure of leader ded9b7f8677940fb91e1a8c8bceceaed) I0312 20:10:17.501102 1478 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec: Snoozing failure detection for 10.101s (starting election) I0312 20:10:17.501303 1478 raft_consensus.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:17.502079 1478 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 24 pre-election: Requesting pre-vote from peer 6885de38238442d6a9e5f1eb9fb569ff I0312 20:10:17.502372 1478 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 24 pre-election: Requesting pre-vote from peer ded9b7f8677940fb91e1a8c8bceceaed I0312 20:10:17.503237 1218 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "f78a2351568c4a268ee1112681e22bec" candidate_term: 24 candidate_status { last_received { term: 23 index: 672 } } ignore_live_leader: false dest_uuid: "6885de38238442d6a9e5f1eb9fb569ff" is_pre_election: true I0312 20:10:17.503618 1218 raft_consensus.cc:2176] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 24 because replica is either leader or believes a valid leader to be alive. I0312 20:10:17.504420 1279 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 24 pre-election: Vote denied by peer 6885de38238442d6a9e5f1eb9fb569ff. Message: Invalid argument: T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 24 because replica is either leader or believes a valid leader to be alive. I0312 20:10:17.505468 1480 raft_consensus.cc:436] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 23 FOLLOWER]: Starting pre-election (detected failure of leader ded9b7f8677940fb91e1a8c8bceceaed) I0312 20:10:17.505668 1480 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff: Snoozing failure detection for 0.107s (starting election) I0312 20:10:17.505842 1480 raft_consensus.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:17.506516 1480 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 24 pre-election: Requesting pre-vote from peer f78a2351568c4a268ee1112681e22bec I0312 20:10:17.506793 1480 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 24 pre-election: Requesting pre-vote from peer ded9b7f8677940fb91e1a8c8bceceaed I0312 20:10:17.507558 1345 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "6885de38238442d6a9e5f1eb9fb569ff" candidate_term: 24 candidate_status { last_received { term: 23 index: 672 } } ignore_live_leader: false dest_uuid: "f78a2351568c4a268ee1112681e22bec" is_pre_election: true I0312 20:10:17.507966 1345 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec: Snoozing failure detection for 13.326s (vote granted) I0312 20:10:17.508191 1345 raft_consensus.cc:2215] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6885de38238442d6a9e5f1eb9fb569ff in term 23. I0312 20:10:17.508301 1094 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "6885de38238442d6a9e5f1eb9fb569ff" candidate_term: 24 candidate_status { last_received { term: 23 index: 672 } } ignore_live_leader: false dest_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" is_pre_election: true I0312 20:10:17.508827 1094 raft_consensus.cc:2176] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 23 LEADER]: Leader pre-election vote request: Denying vote to candidate 6885de38238442d6a9e5f1eb9fb569ff for term 24 because replica is either leader or believes a valid leader to be alive. I0312 20:10:17.509896 1153 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 24 pre-election: Vote denied by peer ded9b7f8677940fb91e1a8c8bceceaed. Message: Invalid argument: T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 23 LEADER]: Leader pre-election vote request: Denying vote to candidate 6885de38238442d6a9e5f1eb9fb569ff for term 24 because replica is either leader or believes a valid leader to be alive. I0312 20:10:17.511242 1152 leader_election.cc:363] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 24 pre-election: Vote granted by peer f78a2351568c4a268ee1112681e22bec I0312 20:10:17.511394 1152 leader_election.cc:258] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. I0312 20:10:17.511705 1480 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff: Snoozing failure detection for 0.137s (election complete) I0312 20:10:17.511904 1480 raft_consensus.cc:2507] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 23 FOLLOWER]: Leader pre-election won for term 24 I0312 20:10:17.512073 1480 raft_consensus.cc:436] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 23 FOLLOWER]: Starting leader election (detected failure of leader ded9b7f8677940fb91e1a8c8bceceaed) I0312 20:10:17.512253 1480 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff: Snoozing failure detection for 0.118s (starting election) I0312 20:10:17.512421 1480 raft_consensus.cc:2752] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 23 FOLLOWER]: Advancing to term 24 I0312 20:10:17.514861 1094 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "f78a2351568c4a268ee1112681e22bec" candidate_term: 24 candidate_status { last_received { term: 23 index: 672 } } ignore_live_leader: false dest_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" is_pre_election: true I0312 20:10:17.515322 1094 raft_consensus.cc:2176] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 23 LEADER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 24 because replica is either leader or believes a valid leader to be alive. I0312 20:10:17.516882 1480 raft_consensus.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:17.517865 1480 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 24 election: Requesting vote from peer f78a2351568c4a268ee1112681e22bec I0312 20:10:17.518234 1480 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 24 election: Requesting vote from peer ded9b7f8677940fb91e1a8c8bceceaed I0312 20:10:17.518721 1218 raft_consensus.cc:1026] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 24 FOLLOWER]: Rejecting Update request from peer ded9b7f8677940fb91e1a8c8bceceaed for earlier term 23. Current term is 24. Ops: [] I0312 20:10:17.519855 1278 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 24 pre-election: Vote denied by peer ded9b7f8677940fb91e1a8c8bceceaed. Message: Invalid argument: T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 23 LEADER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 24 because replica is either leader or believes a valid leader to be alive. I0312 20:10:17.520072 1278 leader_election.cc:258] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. I0312 20:10:17.520021 1345 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "6885de38238442d6a9e5f1eb9fb569ff" candidate_term: 24 candidate_status { last_received { term: 23 index: 672 } } ignore_live_leader: false dest_uuid: "f78a2351568c4a268ee1112681e22bec" I0312 20:10:17.520601 1345 raft_consensus.cc:2176] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 6885de38238442d6a9e5f1eb9fb569ff for term 24 because replica is either leader or believes a valid leader to be alive. I0312 20:10:17.521128 1094 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "6885de38238442d6a9e5f1eb9fb569ff" candidate_term: 24 candidate_status { last_received { term: 23 index: 672 } } ignore_live_leader: false dest_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" I0312 20:10:17.521622 1094 raft_consensus.cc:2176] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 23 LEADER]: Leader election vote request: Denying vote to candidate 6885de38238442d6a9e5f1eb9fb569ff for term 24 because replica is either leader or believes a valid leader to be alive. I0312 20:10:17.520822 1478 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec: Snoozing failure detection for 12.458s (election complete) I0312 20:10:17.521824 1478 raft_consensus.cc:2455] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority I0312 20:10:17.522318 1417 consensus_queue.cc:1028] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 }, Status: INVALID_TERM, Last received: 23.672, Next index: 673, Last known committed idx: 672, Time since last communication: 0.000s I0312 20:10:17.522670 1153 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 24 election: Vote denied by peer ded9b7f8677940fb91e1a8c8bceceaed. Message: Invalid argument: T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 23 LEADER]: Leader election vote request: Denying vote to candidate 6885de38238442d6a9e5f1eb9fb569ff for term 24 because replica is either leader or believes a valid leader to be alive. I0312 20:10:17.522821 1417 raft_consensus.cc:2747] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 23 LEADER]: Stepping down as leader of term 23 I0312 20:10:17.522986 1417 raft_consensus.cc:605] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 23 LEADER]: Becoming Follower/Learner. State: Replica: ded9b7f8677940fb91e1a8c8bceceaed, State: Running, Role: LEADER I0312 20:10:17.523367 1417 consensus_queue.cc:219] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 672, Committed index: 672, Last appended: 23.673, Last appended by leader: 673, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:17.523952 1417 consensus_peers.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed -> Peer 6885de38238442d6a9e5f1eb9fb569ff (127.0.53.2:51697): Closing peer: 6885de38238442d6a9e5f1eb9fb569ff I0312 20:10:17.524113 1417 consensus_peers.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed -> Peer f78a2351568c4a268ee1112681e22bec (127.0.53.3:43274): Closing peer: f78a2351568c4a268ee1112681e22bec I0312 20:10:17.524371 1417 raft_consensus.cc:2752] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 23 FOLLOWER]: Advancing to term 24 I0312 20:10:17.529374 1152 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 24 election: Vote denied by peer f78a2351568c4a268ee1112681e22bec. Message: Invalid argument: T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 6885de38238442d6a9e5f1eb9fb569ff for term 24 because replica is either leader or believes a valid leader to be alive. I0312 20:10:17.529721 1152 leader_election.cc:258] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. I0312 20:10:17.530334 1480 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff: Snoozing failure detection for 0.107s (election complete) I0312 20:10:17.530537 1480 raft_consensus.cc:2455] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority I0312 20:10:17.535002 1477 raft_consensus.cc:436] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0312 20:10:17.535182 1477 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed: Snoozing failure detection for 0.019s (starting election) I0312 20:10:17.535404 1477 raft_consensus.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:17.536229 1477 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 25 pre-election: Requesting pre-vote from peer f78a2351568c4a268ee1112681e22bec I0312 20:10:17.536501 1477 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 25 pre-election: Requesting pre-vote from peer 6885de38238442d6a9e5f1eb9fb569ff I0312 20:10:17.537272 1345 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" candidate_term: 25 candidate_status { last_received { term: 23 index: 673 } } ignore_live_leader: false dest_uuid: "f78a2351568c4a268ee1112681e22bec" is_pre_election: true I0312 20:10:17.537817 1345 raft_consensus.cc:2176] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ded9b7f8677940fb91e1a8c8bceceaed for term 25 because replica is either leader or believes a valid leader to be alive. I0312 20:10:17.537719 1218 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" candidate_term: 25 candidate_status { last_received { term: 23 index: 673 } } ignore_live_leader: false dest_uuid: "6885de38238442d6a9e5f1eb9fb569ff" is_pre_election: true I0312 20:10:17.538159 1218 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff: Snoozing failure detection for 0.089s (vote granted) I0312 20:10:17.538390 1218 raft_consensus.cc:2215] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ded9b7f8677940fb91e1a8c8bceceaed in term 24. I0312 20:10:17.538769 1027 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 25 pre-election: Vote denied by peer f78a2351568c4a268ee1112681e22bec. Message: Invalid argument: T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ded9b7f8677940fb91e1a8c8bceceaed for term 25 because replica is either leader or believes a valid leader to be alive. I0312 20:10:17.539131 1029 leader_election.cc:363] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 25 pre-election: Vote granted by peer 6885de38238442d6a9e5f1eb9fb569ff I0312 20:10:17.539281 1029 leader_election.cc:258] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. I0312 20:10:17.539618 1477 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed: Snoozing failure detection for 0.020s (election complete) I0312 20:10:17.539860 1477 raft_consensus.cc:2507] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 24 FOLLOWER]: Leader pre-election won for term 25 I0312 20:10:17.540019 1477 raft_consensus.cc:436] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0312 20:10:17.540159 1477 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed: Snoozing failure detection for 0.017s (starting election) I0312 20:10:17.540311 1477 raft_consensus.cc:2752] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 24 FOLLOWER]: Advancing to term 25 I0312 20:10:17.540773 1478 raft_consensus.cc:436] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 23 FOLLOWER]: Starting pre-election (detected failure of leader ded9b7f8677940fb91e1a8c8bceceaed) I0312 20:10:17.540979 1478 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec: Snoozing failure detection for 12.761s (starting election) I0312 20:10:17.541215 1478 raft_consensus.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:17.542196 1478 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 24 pre-election: Requesting pre-vote from peer 6885de38238442d6a9e5f1eb9fb569ff I0312 20:10:17.542541 1478 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 24 pre-election: Requesting pre-vote from peer ded9b7f8677940fb91e1a8c8bceceaed I0312 20:10:17.543428 1218 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "f78a2351568c4a268ee1112681e22bec" candidate_term: 24 candidate_status { last_received { term: 23 index: 673 } } ignore_live_leader: false dest_uuid: "6885de38238442d6a9e5f1eb9fb569ff" is_pre_election: true I0312 20:10:17.543817 1094 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "f78a2351568c4a268ee1112681e22bec" candidate_term: 24 candidate_status { last_received { term: 23 index: 673 } } ignore_live_leader: false dest_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" is_pre_election: true I0312 20:10:17.543956 1218 raft_consensus.cc:2145] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec in current term 24: Already voted for candidate 6885de38238442d6a9e5f1eb9fb569ff in this term. I0312 20:10:17.545531 1279 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 24 pre-election: Vote denied by peer 6885de38238442d6a9e5f1eb9fb569ff. Message: Invalid argument: T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec in current term 24: Already voted for candidate 6885de38238442d6a9e5f1eb9fb569ff in this term. I0312 20:10:17.628437 1480 raft_consensus.cc:436] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0312 20:10:17.628621 1480 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff: Snoozing failure detection for 0.485s (starting election) I0312 20:10:17.628787 1480 raft_consensus.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:17.629528 1480 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 25 pre-election: Requesting pre-vote from peer f78a2351568c4a268ee1112681e22bec I0312 20:10:17.629823 1480 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 25 pre-election: Requesting pre-vote from peer ded9b7f8677940fb91e1a8c8bceceaed I0312 20:10:17.630677 1345 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "6885de38238442d6a9e5f1eb9fb569ff" candidate_term: 25 candidate_status { last_received { term: 23 index: 672 } } ignore_live_leader: false dest_uuid: "f78a2351568c4a268ee1112681e22bec" is_pre_election: true I0312 20:10:17.630879 1095 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "6885de38238442d6a9e5f1eb9fb569ff" candidate_term: 25 candidate_status { last_received { term: 23 index: 672 } } ignore_live_leader: false dest_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" is_pre_election: true I0312 20:10:17.631171 1345 raft_consensus.cc:2162] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6885de38238442d6a9e5f1eb9fb569ff for term 25 because replica has last-logged OpId of term: 23 index: 673, which is greater than that of the candidate, which has last-logged OpId of term: 23 index: 672. I0312 20:10:17.631794 1152 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 25 pre-election: Vote denied by peer f78a2351568c4a268ee1112681e22bec. Message: Invalid argument: T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6885de38238442d6a9e5f1eb9fb569ff for term 25 because replica has last-logged OpId of term: 23 index: 673, which is greater than that of the candidate, which has last-logged OpId of term: 23 index: 672. I0312 20:10:18.113939 1480 raft_consensus.cc:436] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0312 20:10:18.114140 1480 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff: Snoozing failure detection for 0.039s (starting election) I0312 20:10:18.114346 1480 raft_consensus.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:18.115265 1480 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 25 pre-election: Requesting pre-vote from peer f78a2351568c4a268ee1112681e22bec I0312 20:10:18.115572 1480 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 25 pre-election: Requesting pre-vote from peer ded9b7f8677940fb91e1a8c8bceceaed W0312 20:10:18.115689 1153 leader_election.cc:287] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer ded9b7f8677940fb91e1a8c8bceceaed: Timed out: RequestConsensusVote RPC to 127.0.53.1:35037 timed out after 0.485s (SENT) I0312 20:10:18.115937 1153 leader_election.cc:258] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. I0312 20:10:18.116282 1480 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff: Snoozing failure detection for 0.399s (election complete) I0312 20:10:18.116451 1480 raft_consensus.cc:2455] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority I0312 20:10:18.117203 1345 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "6885de38238442d6a9e5f1eb9fb569ff" candidate_term: 25 candidate_status { last_received { term: 23 index: 672 } } ignore_live_leader: false dest_uuid: "f78a2351568c4a268ee1112681e22bec" is_pre_election: true I0312 20:10:18.117198 1096 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "6885de38238442d6a9e5f1eb9fb569ff" candidate_term: 25 candidate_status { last_received { term: 23 index: 672 } } ignore_live_leader: false dest_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" is_pre_election: true I0312 20:10:18.117668 1345 raft_consensus.cc:2162] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6885de38238442d6a9e5f1eb9fb569ff for term 25 because replica has last-logged OpId of term: 23 index: 673, which is greater than that of the candidate, which has last-logged OpId of term: 23 index: 672. I0312 20:10:18.118397 1152 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 25 pre-election: Vote denied by peer f78a2351568c4a268ee1112681e22bec. Message: Invalid argument: T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6885de38238442d6a9e5f1eb9fb569ff for term 25 because replica has last-logged OpId of term: 23 index: 673, which is greater than that of the candidate, which has last-logged OpId of term: 23 index: 672. W0312 20:10:18.156066 1153 leader_election.cc:287] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer ded9b7f8677940fb91e1a8c8bceceaed: Timed out: RequestConsensusVote RPC to 127.0.53.1:35037 timed out after 0.039s (SENT) I0312 20:10:18.156227 1153 leader_election.cc:258] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. I0312 20:10:18.156536 1480 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff: Snoozing failure detection for 0.289s (election complete) I0312 20:10:18.156698 1480 raft_consensus.cc:2455] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority I0312 20:10:18.446696 1480 raft_consensus.cc:436] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0312 20:10:18.446918 1480 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff: Snoozing failure detection for 0.996s (starting election) I0312 20:10:18.447106 1480 raft_consensus.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:18.447919 1480 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 25 pre-election: Requesting pre-vote from peer f78a2351568c4a268ee1112681e22bec I0312 20:10:18.448233 1480 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 25 pre-election: Requesting pre-vote from peer ded9b7f8677940fb91e1a8c8bceceaed I0312 20:10:18.449167 1345 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "6885de38238442d6a9e5f1eb9fb569ff" candidate_term: 25 candidate_status { last_received { term: 23 index: 672 } } ignore_live_leader: false dest_uuid: "f78a2351568c4a268ee1112681e22bec" is_pre_election: true I0312 20:10:18.449656 1345 raft_consensus.cc:2162] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6885de38238442d6a9e5f1eb9fb569ff for term 25 because replica has last-logged OpId of term: 23 index: 673, which is greater than that of the candidate, which has last-logged OpId of term: 23 index: 672. I0312 20:10:18.449400 1093 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "6885de38238442d6a9e5f1eb9fb569ff" candidate_term: 25 candidate_status { last_received { term: 23 index: 672 } } ignore_live_leader: false dest_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" is_pre_election: true I0312 20:10:18.450384 1152 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 25 pre-election: Vote denied by peer f78a2351568c4a268ee1112681e22bec. Message: Invalid argument: T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6885de38238442d6a9e5f1eb9fb569ff for term 25 because replica has last-logged OpId of term: 23 index: 673, which is greater than that of the candidate, which has last-logged OpId of term: 23 index: 672. W0312 20:10:18.777276 1477 consensus_meta.cc:216] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed: Time spent flushing consensus metadata: real 1.237s user 0.000s sys 0.004s I0312 20:10:18.777810 1477 raft_consensus.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:18.778939 1477 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 25 election: Requesting vote from peer f78a2351568c4a268ee1112681e22bec I0312 20:10:18.779325 1477 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 25 election: Requesting vote from peer 6885de38238442d6a9e5f1eb9fb569ff I0312 20:10:18.779742 1093 raft_consensus.cc:2190] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6885de38238442d6a9e5f1eb9fb569ff for term 25 because replica is already servicing an update from a current leader or another vote. I0312 20:10:18.780464 1345 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" candidate_term: 25 candidate_status { last_received { term: 23 index: 673 } } ignore_live_leader: false dest_uuid: "f78a2351568c4a268ee1112681e22bec" I0312 20:10:18.780504 1054 rpcz_store.cc:265] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:39390 (ReqId={client: 88cdfc08b6194c19b1581ef853d0ff5a, seq_no=661, attempt_no=3}) took 1223ms. Request Metrics: {"child_traces":[["txn",{"spinlock_wait_cycles":2811355520}]]} I0312 20:10:18.780920 1345 raft_consensus.cc:2752] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 23 FOLLOWER]: Advancing to term 25 I0312 20:10:18.781302 1345 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec: Snoozing failure detection for 9.010s (vote granted) I0312 20:10:18.781404 1153 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 25 pre-election: Vote denied by peer ded9b7f8677940fb91e1a8c8bceceaed. Message: Service unavailable: T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6885de38238442d6a9e5f1eb9fb569ff for term 25 because replica is already servicing an update from a current leader or another vote. I0312 20:10:18.781307 1218 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" candidate_term: 25 candidate_status { last_received { term: 23 index: 673 } } ignore_live_leader: false dest_uuid: "6885de38238442d6a9e5f1eb9fb569ff" I0312 20:10:18.781654 1153 leader_election.cc:258] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. I0312 20:10:18.781787 1095 raft_consensus.cc:2190] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6885de38238442d6a9e5f1eb9fb569ff for term 25 because replica is already servicing an update from a current leader or another vote. I0312 20:10:18.781733 1218 raft_consensus.cc:2752] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 24 FOLLOWER]: Advancing to term 25 I0312 20:10:18.781970 1218 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff: Snoozing failure detection for 0.446s (vote granted) W0312 20:10:18.782258 1095 rpcz_store.cc:251] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.53.2:38849 (request call id 51) took 1151ms (client timeout 485). W0312 20:10:18.782745 1095 rpcz_store.cc:255] Trace: 0312 20:10:17.630653 (+ 0us) service_pool.cc:163] Inserting onto call queue 0312 20:10:17.630759 (+ 106us) service_pool.cc:222] Handling call 0312 20:10:18.782100 (+1151341us) spinlock_profiling.cc:261] Waited 1.15 s on lock 0x7b480003c09c. stack: 00007f2232dd2c74 00007f2232dd2b70 00007f22318c2314 00007f223a90f294 00007f223a90f229 00007f22394b9606 00007f223a94d56b 00007f2235300b24 00007f2235300ab1 00007f22353009c4 00007f223495cc29 00007f223495c6ec 00007f223495dbcd 00007f223495f946 00007f223495f89a 00007f223495f823 0312 20:10:18.782210 (+ 110us) inbound_call.cc:157] Queueing success response Metrics: {"spinlock_wait_cycles":2644739968} I0312 20:10:18.783229 1096 raft_consensus.cc:2190] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6885de38238442d6a9e5f1eb9fb569ff for term 25 because replica is already servicing an update from a current leader or another vote. W0312 20:10:18.783623 1096 rpcz_store.cc:251] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.53.2:38849 (request call id 52) took 666ms (client timeout 39). W0312 20:10:18.784021 1096 rpcz_store.cc:255] Trace: 0312 20:10:18.116931 (+ 0us) service_pool.cc:163] Inserting onto call queue 0312 20:10:18.117037 (+ 106us) service_pool.cc:222] Handling call 0312 20:10:18.783521 (+666484us) spinlock_profiling.cc:261] Waited 666 ms on lock 0x7b480003c09c. stack: 00007f2232dd2c74 00007f2232dd2b70 00007f22318c2314 00007f223a90f294 00007f223a90f229 00007f22394b9606 00007f223a94d56b 00007f2235300b24 00007f2235300ab1 00007f22353009c4 00007f223495cc29 00007f223495c6ec 00007f223495dbcd 00007f223495f946 00007f223495f89a 00007f223495f823 0312 20:10:18.783588 (+ 67us) inbound_call.cc:157] Queueing success response Metrics: {"spinlock_wait_cycles":1531142912} I0312 20:10:18.784490 1094 raft_consensus.cc:2120] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for earlier term 24. Current term is 25. I0312 20:10:18.784914 1094 rpcz_store.cc:265] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.53.3:55789 (request call id 129) took 1241ms. Request Metrics: {"spinlock_wait_cycles":2853130112} I0312 20:10:18.785378 1417 raft_consensus.cc:436] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0312 20:10:18.785542 1417 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed: Snoozing failure detection for 0.017s (starting election) I0312 20:10:18.785732 1417 raft_consensus.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:18.785904 1345 raft_consensus.cc:2215] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ded9b7f8677940fb91e1a8c8bceceaed in term 25. I0312 20:10:18.786098 1218 raft_consensus.cc:2215] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ded9b7f8677940fb91e1a8c8bceceaed in term 25. I0312 20:10:18.786424 1480 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff: Snoozing failure detection for 0.289s (election complete) I0312 20:10:18.786664 1480 raft_consensus.cc:2455] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 25 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority I0312 20:10:18.786880 1417 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 26 pre-election: Requesting pre-vote from peer f78a2351568c4a268ee1112681e22bec I0312 20:10:18.787122 1029 leader_election.cc:363] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 25 election: Vote granted by peer 6885de38238442d6a9e5f1eb9fb569ff I0312 20:10:18.787225 1417 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 26 pre-election: Requesting pre-vote from peer 6885de38238442d6a9e5f1eb9fb569ff I0312 20:10:18.787263 1029 leader_election.cc:258] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. I0312 20:10:18.787629 1477 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed: Snoozing failure detection for 0.020s (election complete) I0312 20:10:18.787843 1477 raft_consensus.cc:2507] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 25 FOLLOWER]: Leader election won for term 25 I0312 20:10:18.788146 1477 raft_consensus.cc:572] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 25 LEADER]: Becoming Leader. State: Replica: ded9b7f8677940fb91e1a8c8bceceaed, State: Running, Role: LEADER I0312 20:10:18.788866 1027 leader_election.cc:363] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 25 election: Vote granted by peer f78a2351568c4a268ee1112681e22bec I0312 20:10:18.788727 1477 consensus_queue.cc:196] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 672, Committed index: 672, Last appended: 23.673, Last appended by leader: 673, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:18.789151 1218 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" candidate_term: 26 candidate_status { last_received { term: 23 index: 673 } } ignore_live_leader: false dest_uuid: "6885de38238442d6a9e5f1eb9fb569ff" is_pre_election: true I0312 20:10:18.789536 1218 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff: Snoozing failure detection for 0.324s (vote granted) I0312 20:10:18.789666 1218 raft_consensus.cc:2215] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ded9b7f8677940fb91e1a8c8bceceaed in term 25. I0312 20:10:18.789904 1345 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" candidate_term: 26 candidate_status { last_received { term: 23 index: 673 } } ignore_live_leader: false dest_uuid: "f78a2351568c4a268ee1112681e22bec" is_pre_election: true I0312 20:10:18.790292 1345 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec: Snoozing failure detection for 3.416s (vote granted) I0312 20:10:18.790385 1029 leader_election.cc:363] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 26 pre-election: Vote granted by peer 6885de38238442d6a9e5f1eb9fb569ff I0312 20:10:18.790526 1029 leader_election.cc:258] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. I0312 20:10:18.790467 1345 raft_consensus.cc:2215] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ded9b7f8677940fb91e1a8c8bceceaed in term 25. W0312 20:10:18.791265 1278 leader_election.cc:347] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 24 pre-election: Vote denied by peer ded9b7f8677940fb91e1a8c8bceceaed with higher term. Message: Invalid argument: T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for earlier term 24. Current term is 25. I0312 20:10:18.791409 1278 leader_election.cc:350] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 24 pre-election: Cancelling election due to peer responding with higher term I0312 20:10:18.791997 1027 leader_election.cc:363] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 26 pre-election: Vote granted by peer f78a2351568c4a268ee1112681e22bec I0312 20:10:18.792356 1417 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed: Snoozing failure detection for 0.021s (election complete) I0312 20:10:18.792536 1484 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec: Snoozing failure detection for 14.963s (election complete) I0312 20:10:18.792753 1484 raft_consensus.cc:2455] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 25 FOLLOWER]: Leader pre-election lost for term 24. Reason: Vote denied by peer ded9b7f8677940fb91e1a8c8bceceaed with higher term. Message: Invalid argument: T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for earlier term 24. Current term is 25. I0312 20:10:18.793571 974 catalog_manager.cc:3592] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed reported cstate change: term changed from 23 to 25. New cstate: current_term: 25 leader_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } health_report { overall_health: HEALTHY } } } I0312 20:10:18.794693 1218 raft_consensus.cc:1058] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 25 FOLLOWER]: Refusing update from remote peer ded9b7f8677940fb91e1a8c8bceceaed: Log matching property violated. Preceding OpId in replica: term: 23 index: 672. Preceding OpId from leader: term: 25 index: 674. (index mismatch) I0312 20:10:18.795117 1345 raft_consensus.cc:1058] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 25 FOLLOWER]: Refusing update from remote peer ded9b7f8677940fb91e1a8c8bceceaed: Log matching property violated. Preceding OpId in replica: term: 23 index: 673. Preceding OpId from leader: term: 25 index: 674. (index mismatch) I0312 20:10:18.795626 1417 consensus_queue.cc:1017] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [LEADER]: Connected to new peer: Peer: permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 }, Status: LMP_MISMATCH, Last received: 23.672, Next index: 673, Last known committed idx: 672, Time since last communication: 0.000s I0312 20:10:18.798008 1417 consensus_queue.cc:1017] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [LEADER]: Connected to new peer: Peer: permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 }, Status: LMP_MISMATCH, Last received: 23.673, Next index: 674, Last known committed idx: 672, Time since last communication: 0.000s I0312 20:10:18.805835 1491 rpcz_store.cc:265] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:39390 (ReqId={client: 88cdfc08b6194c19b1581ef853d0ff5a, seq_no=660, attempt_no=0}) took 1293ms. Request Metrics: {"child_traces":[["txn",{"apply.queue_time_us":1572,"num_ops":1,"prepare.queue_time_us":91,"prepare.run_cpu_time_us":860,"prepare.run_wall_time_us":860,"replication_time_us":1288264,"thread_start_us":1698,"threads_started":1}]]} I0312 20:10:19.053530 1484 raft_consensus.cc:436] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 25 FOLLOWER]: Starting pre-election (detected failure of leader ded9b7f8677940fb91e1a8c8bceceaed) I0312 20:10:19.053791 1484 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec: Snoozing failure detection for 0.017s (starting election) I0312 20:10:19.054033 1484 raft_consensus.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:19.055039 1484 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 26 pre-election: Requesting pre-vote from peer 6885de38238442d6a9e5f1eb9fb569ff I0312 20:10:19.055414 1484 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 26 pre-election: Requesting pre-vote from peer ded9b7f8677940fb91e1a8c8bceceaed I0312 20:10:19.057297 1218 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "f78a2351568c4a268ee1112681e22bec" candidate_term: 26 candidate_status { last_received { term: 25 index: 710 } } ignore_live_leader: false dest_uuid: "6885de38238442d6a9e5f1eb9fb569ff" is_pre_election: true I0312 20:10:19.057299 1094 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "f78a2351568c4a268ee1112681e22bec" candidate_term: 26 candidate_status { last_received { term: 25 index: 710 } } ignore_live_leader: false dest_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" is_pre_election: true I0312 20:10:19.057778 1218 raft_consensus.cc:2176] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 26 because replica is either leader or believes a valid leader to be alive. I0312 20:10:19.058594 1279 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 26 pre-election: Vote denied by peer 6885de38238442d6a9e5f1eb9fb569ff. Message: Invalid argument: T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 26 because replica is either leader or believes a valid leader to be alive. I0312 20:10:19.061868 1094 raft_consensus.cc:2190] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 25 LEADER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 26 because replica is already servicing an update from a current leader or another vote. I0312 20:10:19.063433 1278 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 26 pre-election: Vote denied by peer ded9b7f8677940fb91e1a8c8bceceaed. Message: Service unavailable: T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 25 LEADER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 26 because replica is already servicing an update from a current leader or another vote. I0312 20:10:19.063679 1278 leader_election.cc:258] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. I0312 20:10:19.064095 1484 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec: Snoozing failure detection for 0.021s (election complete) I0312 20:10:19.064261 1484 raft_consensus.cc:2455] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority I0312 20:10:19.085593 1480 raft_consensus.cc:436] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 25 FOLLOWER]: Starting pre-election (detected failure of leader ded9b7f8677940fb91e1a8c8bceceaed) I0312 20:10:19.085862 1480 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff: Snoozing failure detection for 0.020s (starting election) I0312 20:10:19.086081 1480 raft_consensus.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:19.086941 1480 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 26 pre-election: Requesting pre-vote from peer f78a2351568c4a268ee1112681e22bec I0312 20:10:19.087255 1480 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 26 pre-election: Requesting pre-vote from peer ded9b7f8677940fb91e1a8c8bceceaed I0312 20:10:19.088137 1344 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "6885de38238442d6a9e5f1eb9fb569ff" candidate_term: 26 candidate_status { last_received { term: 25 index: 714 } } ignore_live_leader: false dest_uuid: "f78a2351568c4a268ee1112681e22bec" is_pre_election: true I0312 20:10:19.088666 1344 raft_consensus.cc:2176] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6885de38238442d6a9e5f1eb9fb569ff for term 26 because replica is either leader or believes a valid leader to be alive. I0312 20:10:19.088941 1094 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "6885de38238442d6a9e5f1eb9fb569ff" candidate_term: 26 candidate_status { last_received { term: 25 index: 714 } } ignore_live_leader: false dest_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" is_pre_election: true I0312 20:10:19.090062 1152 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 26 pre-election: Vote denied by peer f78a2351568c4a268ee1112681e22bec. Message: Invalid argument: T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6885de38238442d6a9e5f1eb9fb569ff for term 26 because replica is either leader or believes a valid leader to be alive. I0312 20:10:19.091531 1094 raft_consensus.cc:2176] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 25 LEADER]: Leader pre-election vote request: Denying vote to candidate 6885de38238442d6a9e5f1eb9fb569ff for term 26 because replica is either leader or believes a valid leader to be alive. I0312 20:10:19.092653 1153 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 26 pre-election: Vote denied by peer ded9b7f8677940fb91e1a8c8bceceaed. Message: Invalid argument: T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 25 LEADER]: Leader pre-election vote request: Denying vote to candidate 6885de38238442d6a9e5f1eb9fb569ff for term 26 because replica is either leader or believes a valid leader to be alive. I0312 20:10:19.092871 1153 leader_election.cc:258] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. I0312 20:10:19.093631 1480 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff: Snoozing failure detection for 0.019s (election complete) I0312 20:10:19.093837 1480 raft_consensus.cc:2455] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority I0312 20:10:19.624471 1502 raft_consensus.cc:436] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 25 FOLLOWER]: Starting pre-election (detected failure of leader ded9b7f8677940fb91e1a8c8bceceaed) I0312 20:10:19.624680 1502 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec: Snoozing failure detection for 0.016s (starting election) I0312 20:10:19.624883 1502 raft_consensus.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:19.625814 1502 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 26 pre-election: Requesting pre-vote from peer 6885de38238442d6a9e5f1eb9fb569ff I0312 20:10:19.626231 1502 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 26 pre-election: Requesting pre-vote from peer ded9b7f8677940fb91e1a8c8bceceaed I0312 20:10:19.627408 1216 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "f78a2351568c4a268ee1112681e22bec" candidate_term: 26 candidate_status { last_received { term: 25 index: 793 } } ignore_live_leader: false dest_uuid: "6885de38238442d6a9e5f1eb9fb569ff" is_pre_election: true I0312 20:10:19.627962 1216 raft_consensus.cc:2176] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 26 because replica is either leader or believes a valid leader to be alive. I0312 20:10:19.629317 1279 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 26 pre-election: Vote denied by peer 6885de38238442d6a9e5f1eb9fb569ff. Message: Invalid argument: T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 26 because replica is either leader or believes a valid leader to be alive. I0312 20:10:19.630126 1094 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "f78a2351568c4a268ee1112681e22bec" candidate_term: 26 candidate_status { last_received { term: 25 index: 793 } } ignore_live_leader: false dest_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" is_pre_election: true I0312 20:10:19.630656 1094 raft_consensus.cc:2176] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 25 LEADER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 26 because replica is either leader or believes a valid leader to be alive. I0312 20:10:19.632156 1278 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 26 pre-election: Vote denied by peer ded9b7f8677940fb91e1a8c8bceceaed. Message: Invalid argument: T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 25 LEADER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 26 because replica is either leader or believes a valid leader to be alive. I0312 20:10:19.632405 1278 leader_election.cc:258] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. I0312 20:10:19.651942 1502 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec: Snoozing failure detection for 0.016s (election complete) I0312 20:10:19.652180 1502 raft_consensus.cc:2455] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority I0312 20:10:19.652379 1503 raft_consensus.cc:436] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 25 FOLLOWER]: Starting pre-election (detected failure of leader ded9b7f8677940fb91e1a8c8bceceaed) I0312 20:10:19.652637 1503 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec: Snoozing failure detection for 0.028s (starting election) I0312 20:10:19.652932 1503 raft_consensus.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:19.654160 1503 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 26 pre-election: Requesting pre-vote from peer 6885de38238442d6a9e5f1eb9fb569ff I0312 20:10:19.654544 1503 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 26 pre-election: Requesting pre-vote from peer ded9b7f8677940fb91e1a8c8bceceaed I0312 20:10:19.655638 1218 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "f78a2351568c4a268ee1112681e22bec" candidate_term: 26 candidate_status { last_received { term: 25 index: 796 } } ignore_live_leader: false dest_uuid: "6885de38238442d6a9e5f1eb9fb569ff" is_pre_election: true I0312 20:10:19.656186 1218 raft_consensus.cc:2176] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 26 because replica is either leader or believes a valid leader to be alive. I0312 20:10:19.657022 1279 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 26 pre-election: Vote denied by peer 6885de38238442d6a9e5f1eb9fb569ff. Message: Invalid argument: T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 26 because replica is either leader or believes a valid leader to be alive. I0312 20:10:19.666872 1094 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "f78a2351568c4a268ee1112681e22bec" candidate_term: 26 candidate_status { last_received { term: 25 index: 796 } } ignore_live_leader: false dest_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" is_pre_election: true I0312 20:10:19.668395 1094 raft_consensus.cc:2176] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 25 LEADER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 26 because replica is either leader or believes a valid leader to be alive. I0312 20:10:19.670331 1278 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 26 pre-election: Vote denied by peer ded9b7f8677940fb91e1a8c8bceceaed. Message: Invalid argument: T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 25 LEADER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 26 because replica is either leader or believes a valid leader to be alive. I0312 20:10:19.670574 1278 leader_election.cc:258] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. I0312 20:10:19.671725 1503 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec: Snoozing failure detection for 0.038s (election complete) I0312 20:10:19.671962 1503 raft_consensus.cc:2455] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority I0312 20:10:19.703927 1503 raft_consensus.cc:436] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 25 FOLLOWER]: Starting pre-election (detected failure of leader ded9b7f8677940fb91e1a8c8bceceaed) I0312 20:10:19.704232 1503 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec: Snoozing failure detection for 0.036s (starting election) I0312 20:10:19.704461 1503 raft_consensus.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:19.705437 1503 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 26 pre-election: Requesting pre-vote from peer 6885de38238442d6a9e5f1eb9fb569ff I0312 20:10:19.705771 1503 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 26 pre-election: Requesting pre-vote from peer ded9b7f8677940fb91e1a8c8bceceaed I0312 20:10:19.706889 1216 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "f78a2351568c4a268ee1112681e22bec" candidate_term: 26 candidate_status { last_received { term: 25 index: 803 } } ignore_live_leader: false dest_uuid: "6885de38238442d6a9e5f1eb9fb569ff" is_pre_election: true I0312 20:10:19.707432 1216 raft_consensus.cc:2176] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 26 because replica is either leader or believes a valid leader to be alive. I0312 20:10:19.708432 1279 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 26 pre-election: Vote denied by peer 6885de38238442d6a9e5f1eb9fb569ff. Message: Invalid argument: T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 26 because replica is either leader or believes a valid leader to be alive. I0312 20:10:19.710105 1094 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "f78a2351568c4a268ee1112681e22bec" candidate_term: 26 candidate_status { last_received { term: 25 index: 803 } } ignore_live_leader: false dest_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" is_pre_election: true I0312 20:10:19.710614 1094 raft_consensus.cc:2176] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 25 LEADER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 26 because replica is either leader or believes a valid leader to be alive. I0312 20:10:19.712244 1278 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 26 pre-election: Vote denied by peer ded9b7f8677940fb91e1a8c8bceceaed. Message: Invalid argument: T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 25 LEADER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 26 because replica is either leader or believes a valid leader to be alive. I0312 20:10:19.712538 1278 leader_election.cc:258] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. I0312 20:10:19.714293 1503 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec: Snoozing failure detection for 0.030s (election complete) I0312 20:10:19.714507 1503 raft_consensus.cc:2455] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority I0312 20:10:20.010948 1503 raft_consensus.cc:436] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 25 FOLLOWER]: Starting pre-election (detected failure of leader ded9b7f8677940fb91e1a8c8bceceaed) I0312 20:10:20.011205 1503 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec: Snoozing failure detection for 0.108s (starting election) I0312 20:10:20.011442 1503 raft_consensus.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:20.012485 1503 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 26 pre-election: Requesting pre-vote from peer 6885de38238442d6a9e5f1eb9fb569ff I0312 20:10:20.012846 1503 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 26 pre-election: Requesting pre-vote from peer ded9b7f8677940fb91e1a8c8bceceaed I0312 20:10:20.015664 1094 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "f78a2351568c4a268ee1112681e22bec" candidate_term: 26 candidate_status { last_received { term: 25 index: 851 } } ignore_live_leader: false dest_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" is_pre_election: true I0312 20:10:20.016489 1094 raft_consensus.cc:2190] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 25 LEADER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 26 because replica is already servicing an update from a current leader or another vote. I0312 20:10:20.017248 1278 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 26 pre-election: Vote denied by peer ded9b7f8677940fb91e1a8c8bceceaed. Message: Service unavailable: T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 25 LEADER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 26 because replica is already servicing an update from a current leader or another vote. I0312 20:10:20.021728 1216 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "f78a2351568c4a268ee1112681e22bec" candidate_term: 26 candidate_status { last_received { term: 25 index: 851 } } ignore_live_leader: false dest_uuid: "6885de38238442d6a9e5f1eb9fb569ff" is_pre_election: true I0312 20:10:20.022280 1216 raft_consensus.cc:2176] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 26 because replica is either leader or believes a valid leader to be alive. I0312 20:10:20.023042 1279 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 26 pre-election: Vote denied by peer 6885de38238442d6a9e5f1eb9fb569ff. Message: Invalid argument: T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 26 because replica is either leader or believes a valid leader to be alive. I0312 20:10:20.023226 1279 leader_election.cc:258] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. I0312 20:10:20.023850 1503 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec: Snoozing failure detection for 0.074s (election complete) I0312 20:10:20.024111 1503 raft_consensus.cc:2455] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority I0312 20:10:20.065887 1505 raft_consensus.cc:436] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 25 FOLLOWER]: Starting pre-election (detected failure of leader ded9b7f8677940fb91e1a8c8bceceaed) I0312 20:10:20.066192 1505 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff: Snoozing failure detection for 0.030s (starting election) I0312 20:10:20.066462 1505 raft_consensus.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:20.082101 1503 raft_consensus.cc:436] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 25 FOLLOWER]: Starting pre-election (detected failure of leader ded9b7f8677940fb91e1a8c8bceceaed) I0312 20:10:20.082353 1503 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec: Snoozing failure detection for 0.097s (starting election) I0312 20:10:20.082607 1503 raft_consensus.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:20.083631 1503 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 26 pre-election: Requesting pre-vote from peer 6885de38238442d6a9e5f1eb9fb569ff I0312 20:10:20.084013 1503 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 26 pre-election: Requesting pre-vote from peer ded9b7f8677940fb91e1a8c8bceceaed I0312 20:10:20.085355 1218 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "f78a2351568c4a268ee1112681e22bec" candidate_term: 26 candidate_status { last_received { term: 25 index: 859 } } ignore_live_leader: false dest_uuid: "6885de38238442d6a9e5f1eb9fb569ff" is_pre_election: true I0312 20:10:20.085844 1218 raft_consensus.cc:2176] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 26 because replica is either leader or believes a valid leader to be alive. I0312 20:10:20.086850 1279 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 26 pre-election: Vote denied by peer 6885de38238442d6a9e5f1eb9fb569ff. Message: Invalid argument: T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 26 because replica is either leader or believes a valid leader to be alive. I0312 20:10:20.088002 1094 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "f78a2351568c4a268ee1112681e22bec" candidate_term: 26 candidate_status { last_received { term: 25 index: 859 } } ignore_live_leader: false dest_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" is_pre_election: true I0312 20:10:20.088522 1094 raft_consensus.cc:2176] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 25 LEADER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 26 because replica is either leader or believes a valid leader to be alive. I0312 20:10:20.091722 1278 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 26 pre-election: Vote denied by peer ded9b7f8677940fb91e1a8c8bceceaed. Message: Invalid argument: T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 25 LEADER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 26 because replica is either leader or believes a valid leader to be alive. I0312 20:10:20.091902 1278 leader_election.cc:258] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. I0312 20:10:20.092226 1503 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec: Snoozing failure detection for 0.164s (election complete) I0312 20:10:20.092438 1503 raft_consensus.cc:2455] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority I0312 20:10:20.093262 1505 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 26 pre-election: Requesting pre-vote from peer f78a2351568c4a268ee1112681e22bec I0312 20:10:20.093657 1505 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 26 pre-election: Requesting pre-vote from peer ded9b7f8677940fb91e1a8c8bceceaed I0312 20:10:20.094858 1094 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "6885de38238442d6a9e5f1eb9fb569ff" candidate_term: 26 candidate_status { last_received { term: 25 index: 855 } } ignore_live_leader: false dest_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" is_pre_election: true I0312 20:10:20.095340 1094 raft_consensus.cc:2176] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 25 LEADER]: Leader pre-election vote request: Denying vote to candidate 6885de38238442d6a9e5f1eb9fb569ff for term 26 because replica is either leader or believes a valid leader to be alive. I0312 20:10:20.095052 1344 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "6885de38238442d6a9e5f1eb9fb569ff" candidate_term: 26 candidate_status { last_received { term: 25 index: 855 } } ignore_live_leader: false dest_uuid: "f78a2351568c4a268ee1112681e22bec" is_pre_election: true I0312 20:10:20.096200 1153 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 26 pre-election: Vote denied by peer ded9b7f8677940fb91e1a8c8bceceaed. Message: Invalid argument: T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 25 LEADER]: Leader pre-election vote request: Denying vote to candidate 6885de38238442d6a9e5f1eb9fb569ff for term 26 because replica is either leader or believes a valid leader to be alive. I0312 20:10:20.099542 1344 raft_consensus.cc:2190] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6885de38238442d6a9e5f1eb9fb569ff for term 26 because replica is already servicing an update from a current leader or another vote. I0312 20:10:20.100628 1152 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 26 pre-election: Vote denied by peer f78a2351568c4a268ee1112681e22bec. Message: Service unavailable: T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6885de38238442d6a9e5f1eb9fb569ff for term 26 because replica is already servicing an update from a current leader or another vote. I0312 20:10:20.101418 1152 leader_election.cc:258] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. I0312 20:10:20.101759 1505 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff: Snoozing failure detection for 0.032s (election complete) I0312 20:10:20.101971 1505 raft_consensus.cc:2455] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority W0312 20:10:20.185588 1483 log.cc:662] Time spent T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed: Append to log took a long time: real 0.088s user 0.000s sys 0.000s W0312 20:10:20.674599 1483 log.cc:662] Time spent T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed: Append to log took a long time: real 0.489s user 0.008s sys 0.000s W0312 20:10:20.674999 1489 log.cc:662] Time spent T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec: Append to log took a long time: real 0.575s user 0.000s sys 0.000s W0312 20:10:20.676991 1493 log.cc:662] Time spent T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff: Append to log took a long time: real 0.573s user 0.000s sys 0.000s I0312 20:10:20.700227 1508 raft_consensus.cc:436] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 25 FOLLOWER]: Starting pre-election (detected failure of leader ded9b7f8677940fb91e1a8c8bceceaed) I0312 20:10:20.700481 1508 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff: Snoozing failure detection for 0.031s (starting election) I0312 20:10:20.700590 1507 raft_consensus.cc:436] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 25 FOLLOWER]: Starting pre-election (detected failure of leader ded9b7f8677940fb91e1a8c8bceceaed) I0312 20:10:20.700877 1507 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec: Snoozing failure detection for 0.048s (starting election) I0312 20:10:20.700726 1508 raft_consensus.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:20.701150 1507 raft_consensus.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:20.701812 1508 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 26 pre-election: Requesting pre-vote from peer f78a2351568c4a268ee1112681e22bec I0312 20:10:20.702188 1508 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 26 pre-election: Requesting pre-vote from peer ded9b7f8677940fb91e1a8c8bceceaed I0312 20:10:20.702205 1507 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 26 pre-election: Requesting pre-vote from peer 6885de38238442d6a9e5f1eb9fb569ff I0312 20:10:20.702625 1507 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 26 pre-election: Requesting pre-vote from peer ded9b7f8677940fb91e1a8c8bceceaed I0312 20:10:20.703373 1094 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "6885de38238442d6a9e5f1eb9fb569ff" candidate_term: 26 candidate_status { last_received { term: 25 index: 862 } } ignore_live_leader: false dest_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" is_pre_election: true I0312 20:10:20.703526 1218 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "f78a2351568c4a268ee1112681e22bec" candidate_term: 26 candidate_status { last_received { term: 25 index: 861 } } ignore_live_leader: false dest_uuid: "6885de38238442d6a9e5f1eb9fb569ff" is_pre_election: true I0312 20:10:20.707156 1218 raft_consensus.cc:2190] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 26 because replica is already servicing an update from a current leader or another vote. I0312 20:10:20.708272 1279 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 26 pre-election: Vote denied by peer 6885de38238442d6a9e5f1eb9fb569ff. Message: Service unavailable: T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 26 because replica is already servicing an update from a current leader or another vote. I0312 20:10:20.710211 1096 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "f78a2351568c4a268ee1112681e22bec" candidate_term: 26 candidate_status { last_received { term: 25 index: 861 } } ignore_live_leader: false dest_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" is_pre_election: true I0312 20:10:20.710551 1345 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "6885de38238442d6a9e5f1eb9fb569ff" candidate_term: 26 candidate_status { last_received { term: 25 index: 862 } } ignore_live_leader: false dest_uuid: "f78a2351568c4a268ee1112681e22bec" is_pre_election: true I0312 20:10:20.710992 1345 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec: Snoozing failure detection for 0.075s (vote granted) I0312 20:10:20.711166 1345 raft_consensus.cc:2215] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6885de38238442d6a9e5f1eb9fb569ff in term 25. I0312 20:10:20.714995 1152 leader_election.cc:363] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 26 pre-election: Vote granted by peer f78a2351568c4a268ee1112681e22bec I0312 20:10:20.715643 1152 leader_election.cc:258] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. I0312 20:10:20.716316 1508 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff: Snoozing failure detection for 0.041s (election complete) I0312 20:10:20.716516 1508 raft_consensus.cc:2507] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 25 FOLLOWER]: Leader pre-election won for term 26 I0312 20:10:20.716676 1508 raft_consensus.cc:436] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 25 FOLLOWER]: Starting leader election (detected failure of leader ded9b7f8677940fb91e1a8c8bceceaed) I0312 20:10:20.716850 1508 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff: Snoozing failure detection for 0.032s (starting election) I0312 20:10:20.717011 1508 raft_consensus.cc:2752] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 25 FOLLOWER]: Advancing to term 26 I0312 20:10:20.721743 1508 raft_consensus.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:20.723270 1508 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 26 election: Requesting vote from peer f78a2351568c4a268ee1112681e22bec I0312 20:10:20.723429 1216 raft_consensus.cc:1026] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 26 FOLLOWER]: Rejecting Update request from peer ded9b7f8677940fb91e1a8c8bceceaed for earlier term 25. Current term is 26. Ops: [] I0312 20:10:20.723687 1508 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 26 election: Requesting vote from peer ded9b7f8677940fb91e1a8c8bceceaed I0312 20:10:20.725018 1095 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "6885de38238442d6a9e5f1eb9fb569ff" candidate_term: 26 candidate_status { last_received { term: 25 index: 863 } } ignore_live_leader: false dest_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" I0312 20:10:20.729768 1510 consensus_queue.cc:1028] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 }, Status: INVALID_TERM, Last received: 25.863, Next index: 864, Last known committed idx: 863, Time since last communication: 0.000s I0312 20:10:20.730044 1344 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "6885de38238442d6a9e5f1eb9fb569ff" candidate_term: 26 candidate_status { last_received { term: 25 index: 863 } } ignore_live_leader: false dest_uuid: "f78a2351568c4a268ee1112681e22bec" I0312 20:10:20.731999 1507 raft_consensus.cc:436] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 25 FOLLOWER]: Starting pre-election (detected failure of leader ded9b7f8677940fb91e1a8c8bceceaed) I0312 20:10:20.732221 1507 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec: Snoozing failure detection for 0.055s (starting election) I0312 20:10:20.732435 1507 raft_consensus.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:20.733486 1095 raft_consensus.cc:2190] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 25 LEADER]: Leader election vote request: Denying vote to candidate 6885de38238442d6a9e5f1eb9fb569ff for term 26 because replica is already servicing an update from a current leader or another vote. I0312 20:10:20.733539 1507 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 26 pre-election: Requesting pre-vote from peer 6885de38238442d6a9e5f1eb9fb569ff I0312 20:10:20.733558 1216 raft_consensus.cc:1026] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 26 FOLLOWER]: Rejecting Update request from peer ded9b7f8677940fb91e1a8c8bceceaed for earlier term 25. Current term is 26. Ops: [] I0312 20:10:20.733898 1507 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 26 pre-election: Requesting pre-vote from peer ded9b7f8677940fb91e1a8c8bceceaed W0312 20:10:20.734155 1153 leader_election.cc:287] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer ded9b7f8677940fb91e1a8c8bceceaed: Timed out: RequestConsensusVote RPC to 127.0.53.1:35037 timed out after 0.031s (SENT) I0312 20:10:20.734876 1344 raft_consensus.cc:2190] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 6885de38238442d6a9e5f1eb9fb569ff for term 26 because replica is already servicing an update from a current leader or another vote. I0312 20:10:20.735298 1094 raft_consensus.cc:2176] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 25 LEADER]: Leader pre-election vote request: Denying vote to candidate 6885de38238442d6a9e5f1eb9fb569ff for term 26 because replica is either leader or believes a valid leader to be alive. I0312 20:10:20.735374 1509 consensus_queue.cc:1028] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 }, Status: INVALID_TERM, Last received: 25.863, Next index: 864, Last known committed idx: 863, Time since last communication: 0.000s W0312 20:10:20.735800 1094 rpcz_store.cc:251] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.53.2:38849 (request call id 56) took 32ms (client timeout 31). W0312 20:10:20.736424 1094 rpcz_store.cc:255] Trace: 0312 20:10:20.703106 (+ 0us) service_pool.cc:163] Inserting onto call queue 0312 20:10:20.703213 (+ 107us) service_pool.cc:222] Handling call 0312 20:10:20.735669 (+ 32456us) spinlock_profiling.cc:261] Waited 31.4 ms on lock 0x7b480003c09c. stack: 00007f2232dd2c74 00007f2232dd2b70 00007f22318c2314 00007f223a90f294 00007f223a90f229 00007f22394b9af3 00007f223a94d56b 00007f2235300b24 00007f2235300ab1 00007f22353009c4 00007f223495cc29 00007f223495c6ec 00007f223495dbcd 00007f223495f946 00007f223495f89a 00007f223495f823 0312 20:10:20.735756 (+ 87us) inbound_call.cc:157] Queueing success response Metrics: {"spinlock_wait_cycles":72146304} I0312 20:10:20.736917 1096 raft_consensus.cc:2190] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 25 LEADER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 26 because replica is already servicing an update from a current leader or another vote. I0312 20:10:20.737957 1218 raft_consensus.cc:1026] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 26 FOLLOWER]: Rejecting Update request from peer ded9b7f8677940fb91e1a8c8bceceaed for earlier term 25. Current term is 26. Ops: [] I0312 20:10:20.738587 1278 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 26 pre-election: Vote denied by peer ded9b7f8677940fb91e1a8c8bceceaed. Message: Service unavailable: T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 25 LEADER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 26 because replica is already servicing an update from a current leader or another vote. I0312 20:10:20.738718 1152 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 26 election: Vote denied by peer f78a2351568c4a268ee1112681e22bec. Message: Service unavailable: T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 6885de38238442d6a9e5f1eb9fb569ff for term 26 because replica is already servicing an update from a current leader or another vote. I0312 20:10:20.738811 1278 leader_election.cc:258] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. I0312 20:10:20.739509 1507 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec: Snoozing failure detection for 0.056s (election complete) I0312 20:10:20.739708 1507 raft_consensus.cc:2455] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority I0312 20:10:20.739677 1509 consensus_queue.cc:1028] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 }, Status: INVALID_TERM, Last received: 25.863, Next index: 864, Last known committed idx: 863, Time since last communication: 0.000s I0312 20:10:20.739918 1153 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 26 election: Vote denied by peer ded9b7f8677940fb91e1a8c8bceceaed. Message: Service unavailable: T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 25 LEADER]: Leader election vote request: Denying vote to candidate 6885de38238442d6a9e5f1eb9fb569ff for term 26 because replica is already servicing an update from a current leader or another vote. I0312 20:10:20.740129 1153 leader_election.cc:258] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. I0312 20:10:20.741111 1508 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff: Snoozing failure detection for 0.025s (election complete) I0312 20:10:20.741304 1508 raft_consensus.cc:2455] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority I0312 20:10:20.741817 1216 raft_consensus.cc:1026] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 26 FOLLOWER]: Rejecting Update request from peer ded9b7f8677940fb91e1a8c8bceceaed for earlier term 25. Current term is 26. Ops: [] I0312 20:10:20.741739 1096 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "f78a2351568c4a268ee1112681e22bec" candidate_term: 26 candidate_status { last_received { term: 25 index: 863 } } ignore_live_leader: false dest_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" is_pre_election: true I0312 20:10:20.742228 1096 raft_consensus.cc:2176] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 25 LEADER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 26 because replica is either leader or believes a valid leader to be alive. I0312 20:10:20.742758 1218 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "f78a2351568c4a268ee1112681e22bec" candidate_term: 26 candidate_status { last_received { term: 25 index: 863 } } ignore_live_leader: false dest_uuid: "6885de38238442d6a9e5f1eb9fb569ff" is_pre_election: true I0312 20:10:20.743013 1509 consensus_queue.cc:1028] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 }, Status: INVALID_TERM, Last received: 25.863, Next index: 864, Last known committed idx: 863, Time since last communication: 0.000s I0312 20:10:20.743266 1218 raft_consensus.cc:2145] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec in current term 26: Already voted for candidate 6885de38238442d6a9e5f1eb9fb569ff in this term. I0312 20:10:20.744477 1279 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 26 pre-election: Vote denied by peer 6885de38238442d6a9e5f1eb9fb569ff. Message: Invalid argument: T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec in current term 26: Already voted for candidate 6885de38238442d6a9e5f1eb9fb569ff in this term. I0312 20:10:20.745009 1278 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 26 pre-election: Vote denied by peer ded9b7f8677940fb91e1a8c8bceceaed. Message: Invalid argument: T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 25 LEADER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 26 because replica is either leader or believes a valid leader to be alive. I0312 20:10:20.745241 1278 leader_election.cc:258] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. I0312 20:10:20.746201 1506 raft_consensus.cc:2747] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 25 LEADER]: Stepping down as leader of term 25 I0312 20:10:20.746390 1506 raft_consensus.cc:605] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 25 LEADER]: Becoming Follower/Learner. State: Replica: ded9b7f8677940fb91e1a8c8bceceaed, State: Running, Role: LEADER I0312 20:10:20.746359 1218 raft_consensus.cc:1026] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 26 FOLLOWER]: Rejecting Update request from peer ded9b7f8677940fb91e1a8c8bceceaed for earlier term 25. Current term is 26. Ops: [] I0312 20:10:20.746875 1506 consensus_queue.cc:219] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 863, Committed index: 863, Last appended: 25.863, Last appended by leader: 863, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:20.747557 1506 consensus_peers.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed -> Peer 6885de38238442d6a9e5f1eb9fb569ff (127.0.53.2:51697): Closing peer: 6885de38238442d6a9e5f1eb9fb569ff I0312 20:10:20.747738 1506 consensus_peers.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed -> Peer f78a2351568c4a268ee1112681e22bec (127.0.53.3:43274): Closing peer: f78a2351568c4a268ee1112681e22bec I0312 20:10:20.748005 1506 raft_consensus.cc:2752] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 25 FOLLOWER]: Advancing to term 26 I0312 20:10:20.751333 1507 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec: Snoozing failure detection for 0.251s (election complete) I0312 20:10:20.751538 1507 raft_consensus.cc:2752] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 25 FOLLOWER]: Advancing to term 26 I0312 20:10:20.756525 1507 raft_consensus.cc:2455] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 26 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority I0312 20:10:20.765620 1506 raft_consensus.cc:436] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0312 20:10:20.765920 1506 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed: Snoozing failure detection for 0.018s (starting election) I0312 20:10:20.766166 1506 raft_consensus.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:20.767055 1508 raft_consensus.cc:436] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0312 20:10:20.767257 1508 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff: Snoozing failure detection for 0.032s (starting election) I0312 20:10:20.767472 1508 raft_consensus.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:20.768507 1508 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 27 pre-election: Requesting pre-vote from peer f78a2351568c4a268ee1112681e22bec I0312 20:10:20.768877 1508 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 27 pre-election: Requesting pre-vote from peer ded9b7f8677940fb91e1a8c8bceceaed I0312 20:10:20.771994 1096 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "6885de38238442d6a9e5f1eb9fb569ff" candidate_term: 27 candidate_status { last_received { term: 25 index: 863 } } ignore_live_leader: false dest_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" is_pre_election: true I0312 20:10:20.773891 1345 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "6885de38238442d6a9e5f1eb9fb569ff" candidate_term: 27 candidate_status { last_received { term: 25 index: 863 } } ignore_live_leader: false dest_uuid: "f78a2351568c4a268ee1112681e22bec" is_pre_election: true I0312 20:10:20.774291 1345 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec: Snoozing failure detection for 0.189s (vote granted) I0312 20:10:20.774493 1345 raft_consensus.cc:2215] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6885de38238442d6a9e5f1eb9fb569ff in term 26. I0312 20:10:20.775451 1152 leader_election.cc:363] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 27 pre-election: Vote granted by peer f78a2351568c4a268ee1112681e22bec I0312 20:10:20.775645 1152 leader_election.cc:258] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. I0312 20:10:20.776131 1508 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff: Snoozing failure detection for 0.064s (election complete) I0312 20:10:20.776330 1508 raft_consensus.cc:2507] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 26 FOLLOWER]: Leader pre-election won for term 27 I0312 20:10:20.776501 1508 raft_consensus.cc:436] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0312 20:10:20.776665 1508 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff: Snoozing failure detection for 0.045s (starting election) I0312 20:10:20.776826 1508 raft_consensus.cc:2752] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 26 FOLLOWER]: Advancing to term 27 I0312 20:10:20.782629 1096 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed: Snoozing failure detection for 0.021s (vote granted) I0312 20:10:20.782835 1096 raft_consensus.cc:2215] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6885de38238442d6a9e5f1eb9fb569ff in term 26. I0312 20:10:20.785176 1153 leader_election.cc:363] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 27 pre-election: Vote granted by peer ded9b7f8677940fb91e1a8c8bceceaed I0312 20:10:20.785605 1506 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 27 pre-election: Requesting pre-vote from peer f78a2351568c4a268ee1112681e22bec I0312 20:10:20.785996 1506 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 27 pre-election: Requesting pre-vote from peer 6885de38238442d6a9e5f1eb9fb569ff I0312 20:10:20.787230 1218 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" candidate_term: 27 candidate_status { last_received { term: 25 index: 863 } } ignore_live_leader: false dest_uuid: "6885de38238442d6a9e5f1eb9fb569ff" is_pre_election: true I0312 20:10:20.788444 1345 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" candidate_term: 27 candidate_status { last_received { term: 25 index: 863 } } ignore_live_leader: false dest_uuid: "f78a2351568c4a268ee1112681e22bec" is_pre_election: true I0312 20:10:20.788826 1345 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec: Snoozing failure detection for 0.309s (vote granted) I0312 20:10:20.789001 1345 raft_consensus.cc:2215] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ded9b7f8677940fb91e1a8c8bceceaed in term 26. I0312 20:10:20.789922 1027 leader_election.cc:363] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 27 pre-election: Vote granted by peer f78a2351568c4a268ee1112681e22bec I0312 20:10:20.790119 1027 leader_election.cc:258] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. I0312 20:10:20.790520 1506 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed: Snoozing failure detection for 0.020s (election complete) I0312 20:10:20.790755 1506 raft_consensus.cc:2507] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 26 FOLLOWER]: Leader pre-election won for term 27 I0312 20:10:20.790946 1506 raft_consensus.cc:436] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0312 20:10:20.791129 1506 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed: Snoozing failure detection for 0.019s (starting election) I0312 20:10:20.791308 1506 raft_consensus.cc:2752] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 26 FOLLOWER]: Advancing to term 27 W0312 20:10:20.805055 1029 leader_election.cc:287] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer 6885de38238442d6a9e5f1eb9fb569ff: Timed out: RequestConsensusVote RPC to 127.0.53.2:51697 timed out after 0.018s (SENT) I0312 20:10:21.099068 1507 raft_consensus.cc:436] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0312 20:10:21.099268 1507 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec: Snoozing failure detection for 0.520s (starting election) I0312 20:10:21.099440 1507 raft_consensus.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:21.100200 1507 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 27 pre-election: Requesting pre-vote from peer 6885de38238442d6a9e5f1eb9fb569ff I0312 20:10:21.100492 1507 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 27 pre-election: Requesting pre-vote from peer ded9b7f8677940fb91e1a8c8bceceaed I0312 20:10:21.101320 1216 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "f78a2351568c4a268ee1112681e22bec" candidate_term: 27 candidate_status { last_received { term: 25 index: 863 } } ignore_live_leader: false dest_uuid: "6885de38238442d6a9e5f1eb9fb569ff" is_pre_election: true I0312 20:10:21.101572 1096 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "f78a2351568c4a268ee1112681e22bec" candidate_term: 27 candidate_status { last_received { term: 25 index: 863 } } ignore_live_leader: false dest_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" is_pre_election: true I0312 20:10:21.620640 1519 raft_consensus.cc:436] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0312 20:10:21.620832 1519 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec: Snoozing failure detection for 0.528s (starting election) I0312 20:10:21.621011 1519 raft_consensus.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } W0312 20:10:21.621347 1279 leader_election.cc:287] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer 6885de38238442d6a9e5f1eb9fb569ff: Timed out: RequestConsensusVote RPC to 127.0.53.2:51697 timed out after 0.520s (SENT) W0312 20:10:21.621657 1278 leader_election.cc:287] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer ded9b7f8677940fb91e1a8c8bceceaed: Timed out: RequestConsensusVote RPC to 127.0.53.1:35037 timed out after 0.520s (SENT) I0312 20:10:21.621878 1278 leader_election.cc:258] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. I0312 20:10:21.622005 1519 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 27 pre-election: Requesting pre-vote from peer 6885de38238442d6a9e5f1eb9fb569ff I0312 20:10:21.622402 1519 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 27 pre-election: Requesting pre-vote from peer ded9b7f8677940fb91e1a8c8bceceaed I0312 20:10:21.622880 1519 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec: Snoozing failure detection for 0.100s (election complete) I0312 20:10:21.623121 1519 raft_consensus.cc:2455] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority I0312 20:10:21.623416 1215 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "f78a2351568c4a268ee1112681e22bec" candidate_term: 27 candidate_status { last_received { term: 25 index: 863 } } ignore_live_leader: false dest_uuid: "6885de38238442d6a9e5f1eb9fb569ff" is_pre_election: true I0312 20:10:21.623859 1094 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "f78a2351568c4a268ee1112681e22bec" candidate_term: 27 candidate_status { last_received { term: 25 index: 863 } } ignore_live_leader: false dest_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" is_pre_election: true I0312 20:10:21.723987 1519 raft_consensus.cc:436] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0312 20:10:21.724171 1519 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec: Snoozing failure detection for 0.088s (starting election) I0312 20:10:21.724354 1519 raft_consensus.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:21.725136 1519 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 27 pre-election: Requesting pre-vote from peer 6885de38238442d6a9e5f1eb9fb569ff I0312 20:10:21.725430 1519 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 27 pre-election: Requesting pre-vote from peer ded9b7f8677940fb91e1a8c8bceceaed I0312 20:10:21.726380 1219 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "f78a2351568c4a268ee1112681e22bec" candidate_term: 27 candidate_status { last_received { term: 25 index: 863 } } ignore_live_leader: false dest_uuid: "6885de38238442d6a9e5f1eb9fb569ff" is_pre_election: true I0312 20:10:21.726487 1095 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "f78a2351568c4a268ee1112681e22bec" candidate_term: 27 candidate_status { last_received { term: 25 index: 863 } } ignore_live_leader: false dest_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" is_pre_election: true I0312 20:10:21.813171 1519 raft_consensus.cc:436] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0312 20:10:21.813396 1519 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec: Snoozing failure detection for 0.328s (starting election) I0312 20:10:21.813601 1519 raft_consensus.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } W0312 20:10:21.814236 1279 leader_election.cc:287] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer 6885de38238442d6a9e5f1eb9fb569ff: Timed out: RequestConsensusVote RPC to 127.0.53.2:51697 timed out after 0.088s (SENT) I0312 20:10:21.814481 1519 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 27 pre-election: Requesting pre-vote from peer 6885de38238442d6a9e5f1eb9fb569ff W0312 20:10:21.814658 1278 leader_election.cc:287] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer ded9b7f8677940fb91e1a8c8bceceaed: Timed out: RequestConsensusVote RPC to 127.0.53.1:35037 timed out after 0.088s (SENT) I0312 20:10:21.814858 1519 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 27 pre-election: Requesting pre-vote from peer ded9b7f8677940fb91e1a8c8bceceaed I0312 20:10:21.814872 1278 leader_election.cc:258] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. I0312 20:10:21.815666 1519 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec: Snoozing failure detection for 0.305s (election complete) I0312 20:10:21.815865 1519 raft_consensus.cc:2455] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority I0312 20:10:21.815904 1220 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "f78a2351568c4a268ee1112681e22bec" candidate_term: 27 candidate_status { last_received { term: 25 index: 863 } } ignore_live_leader: false dest_uuid: "6885de38238442d6a9e5f1eb9fb569ff" is_pre_election: true I0312 20:10:21.816650 1093 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "f78a2351568c4a268ee1112681e22bec" candidate_term: 27 candidate_status { last_received { term: 25 index: 863 } } ignore_live_leader: false dest_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" is_pre_election: true I0312 20:10:22.121369 1519 raft_consensus.cc:436] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0312 20:10:22.121577 1519 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec: Snoozing failure detection for 0.591s (starting election) I0312 20:10:22.121748 1519 raft_consensus.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:22.122613 1519 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 27 pre-election: Requesting pre-vote from peer 6885de38238442d6a9e5f1eb9fb569ff I0312 20:10:22.123024 1519 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 27 pre-election: Requesting pre-vote from peer ded9b7f8677940fb91e1a8c8bceceaed I0312 20:10:22.124060 1217 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "f78a2351568c4a268ee1112681e22bec" candidate_term: 27 candidate_status { last_received { term: 25 index: 863 } } ignore_live_leader: false dest_uuid: "6885de38238442d6a9e5f1eb9fb569ff" is_pre_election: true I0312 20:10:22.124794 1092 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "f78a2351568c4a268ee1112681e22bec" candidate_term: 27 candidate_status { last_received { term: 25 index: 863 } } ignore_live_leader: false dest_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" is_pre_election: true W0312 20:10:22.143892 1279 leader_election.cc:287] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer 6885de38238442d6a9e5f1eb9fb569ff: Timed out: RequestConsensusVote RPC to 127.0.53.2:51697 timed out after 0.328s (SENT) W0312 20:10:22.144644 1278 leader_election.cc:287] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer ded9b7f8677940fb91e1a8c8bceceaed: Timed out: RequestConsensusVote RPC to 127.0.53.1:35037 timed out after 0.328s (SENT) I0312 20:10:22.144812 1278 leader_election.cc:258] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. I0312 20:10:22.145128 1519 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec: Snoozing failure detection for 0.830s (election complete) I0312 20:10:22.145292 1519 raft_consensus.cc:2455] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority W0312 20:10:22.151403 1279 leader_election.cc:287] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer 6885de38238442d6a9e5f1eb9fb569ff: Timed out: RequestConsensusVote RPC to 127.0.53.2:51697 timed out after 0.528s (SENT) W0312 20:10:22.152429 1278 leader_election.cc:287] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer ded9b7f8677940fb91e1a8c8bceceaed: Timed out: RequestConsensusVote RPC to 127.0.53.1:35037 timed out after 0.528s (SENT) I0312 20:10:22.152596 1278 leader_election.cc:258] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. I0312 20:10:22.152894 1519 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec: Snoozing failure detection for 0.539s (election complete) I0312 20:10:22.153053 1519 raft_consensus.cc:2455] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority W0312 20:10:22.426286 1506 consensus_meta.cc:216] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed: Time spent flushing consensus metadata: real 1.635s user 0.000s sys 0.004s W0312 20:10:22.426295 1508 consensus_meta.cc:216] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff: Time spent flushing consensus metadata: real 1.649s user 0.004s sys 0.000s I0312 20:10:22.426502 1508 raft_consensus.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:22.426514 1506 raft_consensus.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:22.427297 1508 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 27 election: Requesting vote from peer f78a2351568c4a268ee1112681e22bec I0312 20:10:22.427320 1506 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 27 election: Requesting vote from peer f78a2351568c4a268ee1112681e22bec I0312 20:10:22.427450 1218 raft_consensus.cc:2145] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ded9b7f8677940fb91e1a8c8bceceaed in current term 27: Already voted for candidate 6885de38238442d6a9e5f1eb9fb569ff in this term. I0312 20:10:22.427495 1096 raft_consensus.cc:2145] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec in current term 27: Already voted for candidate ded9b7f8677940fb91e1a8c8bceceaed in this term. I0312 20:10:22.427695 1508 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 27 election: Requesting vote from peer ded9b7f8677940fb91e1a8c8bceceaed I0312 20:10:22.427693 1506 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 27 election: Requesting vote from peer 6885de38238442d6a9e5f1eb9fb569ff W0312 20:10:22.427917 1218 rpcz_store.cc:251] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.53.1:43660 (request call id 1236) took 1640ms (client timeout 18). W0312 20:10:22.427956 1096 rpcz_store.cc:251] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.53.3:55789 (request call id 138) took 1326ms (client timeout 520). W0312 20:10:22.428527 1218 rpcz_store.cc:255] Trace: 0312 20:10:20.787008 (+ 0us) service_pool.cc:163] Inserting onto call queue 0312 20:10:20.787101 (+ 93us) service_pool.cc:222] Handling call 0312 20:10:22.427761 (+1640660us) spinlock_profiling.cc:261] Waited 1.64 s on lock 0x7b480003009c. stack: 00007efc5efb2c74 00007efc5efb2b70 00007efc5daa2314 00007efc66aef294 00007efc66aef229 00007efc65699af3 00007efc66b2d56b 00007efc614e0b24 00007efc614e0ab1 00007efc614e09c4 00007efc60b3cc29 00007efc60b3c6ec 00007efc60b3dbcd 00007efc60b3f946 00007efc60b3f89a 00007efc60b3f823 0312 20:10:22.427868 (+ 107us) inbound_call.cc:157] Queueing success response Metrics: {"spinlock_wait_cycles":3772372224} W0312 20:10:22.428583 1096 rpcz_store.cc:255] Trace: 0312 20:10:21.101335 (+ 0us) service_pool.cc:163] Inserting onto call queue 0312 20:10:21.101436 (+ 101us) service_pool.cc:222] Handling call 0312 20:10:22.427816 (+1326380us) spinlock_profiling.cc:261] Waited 1.33 s on lock 0x7b480003c09c. stack: 00007f2232dd2c74 00007f2232dd2b70 00007f22318c2314 00007f223a90f294 00007f223a90f229 00007f22394b9af3 00007f223a94d56b 00007f2235300b24 00007f2235300ab1 00007f22353009c4 00007f223495cc29 00007f223495c6ec 00007f223495dbcd 00007f223495f946 00007f223495f89a 00007f223495f823 0312 20:10:22.427909 (+ 93us) inbound_call.cc:157] Queueing success response Metrics: {"spinlock_wait_cycles":3049441536} I0312 20:10:22.428856 1517 raft_consensus.cc:436] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0312 20:10:22.428947 1510 raft_consensus.cc:436] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0312 20:10:22.429106 1517 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff: Snoozing failure detection for 0.016s (starting election) I0312 20:10:22.429183 1510 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed: Snoozing failure detection for 0.015s (starting election) I0312 20:10:22.429325 1517 raft_consensus.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:22.429409 1510 raft_consensus.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:22.429745 1096 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "6885de38238442d6a9e5f1eb9fb569ff" candidate_term: 27 candidate_status { last_received { term: 25 index: 863 } } ignore_live_leader: false dest_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" I0312 20:10:22.429734 1345 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "6885de38238442d6a9e5f1eb9fb569ff" candidate_term: 27 candidate_status { last_received { term: 25 index: 863 } } ignore_live_leader: false dest_uuid: "f78a2351568c4a268ee1112681e22bec" I0312 20:10:22.430188 1345 raft_consensus.cc:2752] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 26 FOLLOWER]: Advancing to term 27 I0312 20:10:22.430405 1345 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec: Snoozing failure detection for 2.103s (vote granted) I0312 20:10:22.430433 1517 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 28 pre-election: Requesting pre-vote from peer f78a2351568c4a268ee1112681e22bec I0312 20:10:22.430562 1095 raft_consensus.cc:2190] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 27 because replica is already servicing an update from a current leader or another vote. I0312 20:10:22.430567 1510 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 28 pre-election: Requesting pre-vote from peer f78a2351568c4a268ee1112681e22bec I0312 20:10:22.430795 1517 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 28 pre-election: Requesting pre-vote from peer ded9b7f8677940fb91e1a8c8bceceaed I0312 20:10:22.430979 1510 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 28 pre-election: Requesting pre-vote from peer 6885de38238442d6a9e5f1eb9fb569ff W0312 20:10:22.431051 1095 rpcz_store.cc:251] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.53.3:55789 (request call id 140) took 704ms (client timeout 88). I0312 20:10:22.432008 1344 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" candidate_term: 27 candidate_status { last_received { term: 25 index: 863 } } ignore_live_leader: false dest_uuid: "f78a2351568c4a268ee1112681e22bec" W0312 20:10:22.432410 1095 rpcz_store.cc:255] Trace: 0312 20:10:21.726227 (+ 0us) service_pool.cc:163] Inserting onto call queue 0312 20:10:21.726327 (+ 100us) service_pool.cc:222] Handling call 0312 20:10:22.430915 (+704588us) spinlock_profiling.cc:261] Waited 704 ms on lock 0x7b480003c09c. stack: 00007f2232dd2c74 00007f2232dd2b70 00007f22318c2314 00007f223a90f294 00007f223a90f229 00007f22394b9606 00007f223a94d56b 00007f2235300b24 00007f2235300ab1 00007f22353009c4 00007f223495cc29 00007f223495c6ec 00007f223495dbcd 00007f223495f946 00007f223495f89a 00007f223495f823 0312 20:10:22.431002 (+ 87us) inbound_call.cc:157] Queueing success response Metrics: {"spinlock_wait_cycles":1618621696} I0312 20:10:22.432348 1181 rpcz_store.cc:265] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:40835 (ReqId={client: 88cdfc08b6194c19b1581ef853d0ff5a, seq_no=851, attempt_no=1}) took 1639ms. Request Metrics: {"child_traces":[["txn",{"spinlock_wait_cycles":3767817984}]]} I0312 20:10:22.432801 1093 raft_consensus.cc:2190] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 27 because replica is already servicing an update from a current leader or another vote. I0312 20:10:22.433152 1092 raft_consensus.cc:2190] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 27 because replica is already servicing an update from a current leader or another vote. W0312 20:10:22.433261 1093 rpcz_store.cc:251] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.53.3:55789 (request call id 141) took 616ms (client timeout 328). I0312 20:10:22.433938 1217 raft_consensus.cc:2190] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 27 because replica is already servicing an update from a current leader or another vote. W0312 20:10:22.433985 1093 rpcz_store.cc:255] Trace: 0312 20:10:21.816404 (+ 0us) service_pool.cc:163] Inserting onto call queue 0312 20:10:21.816505 (+ 101us) service_pool.cc:222] Handling call 0312 20:10:22.433121 (+616616us) spinlock_profiling.cc:261] Waited 615 ms on lock 0x7b480003c09c. stack: 00007f2232dd2c74 00007f2232dd2b70 00007f22318c2314 00007f223a90f294 00007f223a90f229 00007f22394b9606 00007f223a94d56b 00007f2235300b24 00007f2235300ab1 00007f22353009c4 00007f223495cc29 00007f223495c6ec 00007f223495dbcd 00007f223495f946 00007f223495f89a 00007f223495f823 0312 20:10:22.433201 (+ 80us) inbound_call.cc:157] Queueing success response Metrics: {"spinlock_wait_cycles":1413850368} I0312 20:10:22.434147 1180 rpcz_store.cc:265] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:40835 (ReqId={client: 88cdfc08b6194c19b1581ef853d0ff5a, seq_no=850, attempt_no=2}) took 1652ms. Request Metrics: {"child_traces":[["txn",{"spinlock_wait_cycles":3796954368}]]} I0312 20:10:22.434816 1220 raft_consensus.cc:2190] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 27 because replica is already servicing an update from a current leader or another vote. I0312 20:10:22.434962 1217 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" candidate_term: 27 candidate_status { last_received { term: 25 index: 863 } } ignore_live_leader: false dest_uuid: "6885de38238442d6a9e5f1eb9fb569ff" W0312 20:10:22.435261 1220 rpcz_store.cc:251] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.53.3:38196 (request call id 113) took 619ms (client timeout 328). I0312 20:10:22.435492 1217 raft_consensus.cc:2145] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate ded9b7f8677940fb91e1a8c8bceceaed in current term 27: Already voted for candidate 6885de38238442d6a9e5f1eb9fb569ff in this term. I0312 20:10:22.435427 1345 raft_consensus.cc:2215] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6885de38238442d6a9e5f1eb9fb569ff in term 27. I0312 20:10:22.435758 1344 raft_consensus.cc:2190] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate ded9b7f8677940fb91e1a8c8bceceaed for term 27 because replica is already servicing an update from a current leader or another vote. W0312 20:10:22.435912 1220 rpcz_store.cc:255] Trace: 0312 20:10:21.815620 (+ 0us) service_pool.cc:163] Inserting onto call queue 0312 20:10:21.815729 (+ 109us) service_pool.cc:222] Handling call 0312 20:10:22.435136 (+619407us) spinlock_profiling.cc:261] Waited 619 ms on lock 0x7b480003009c. stack: 00007efc5efb2c74 00007efc5efb2b70 00007efc5daa2314 00007efc66aef294 00007efc66aef229 00007efc65699606 00007efc66b2d56b 00007efc614e0b24 00007efc614e0ab1 00007efc614e09c4 00007efc60b3cc29 00007efc60b3c6ec 00007efc60b3dbcd 00007efc60b3f946 00007efc60b3f89a 00007efc60b3f823 0312 20:10:22.435214 (+ 78us) inbound_call.cc:157] Queueing success response Metrics: {"spinlock_wait_cycles":1422923520} I0312 20:10:22.436277 1215 raft_consensus.cc:2190] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 27 because replica is already servicing an update from a current leader or another vote. W0312 20:10:22.436739 1215 rpcz_store.cc:251] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.53.3:38196 (request call id 111) took 813ms (client timeout 528). W0312 20:10:22.437325 1215 rpcz_store.cc:255] Trace: 0312 20:10:21.623091 (+ 0us) service_pool.cc:163] Inserting onto call queue 0312 20:10:21.623209 (+ 118us) service_pool.cc:222] Handling call 0312 20:10:22.436616 (+813407us) spinlock_profiling.cc:261] Waited 813 ms on lock 0x7b480003009c. stack: 00007efc5efb2c74 00007efc5efb2b70 00007efc5daa2314 00007efc66aef294 00007efc66aef229 00007efc65699606 00007efc66b2d56b 00007efc614e0b24 00007efc614e0ab1 00007efc614e09c4 00007efc60b3cc29 00007efc60b3c6ec 00007efc60b3dbcd 00007efc60b3f946 00007efc60b3f89a 00007efc60b3f823 0312 20:10:22.436695 (+ 79us) inbound_call.cc:157] Queueing success response Metrics: {"spinlock_wait_cycles":1869090560} I0312 20:10:22.437433 1029 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 27 election: Vote denied by peer 6885de38238442d6a9e5f1eb9fb569ff. Message: Invalid argument: T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate ded9b7f8677940fb91e1a8c8bceceaed in current term 27: Already voted for candidate 6885de38238442d6a9e5f1eb9fb569ff in this term. I0312 20:10:22.437664 1216 raft_consensus.cc:2190] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 27 because replica is already servicing an update from a current leader or another vote. I0312 20:10:22.437856 1093 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "6885de38238442d6a9e5f1eb9fb569ff" candidate_term: 28 candidate_status { last_received { term: 25 index: 863 } } ignore_live_leader: false dest_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" is_pre_election: true I0312 20:10:22.438082 1152 leader_election.cc:363] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 27 election: Vote granted by peer f78a2351568c4a268ee1112681e22bec W0312 20:10:22.438079 1216 rpcz_store.cc:251] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.53.3:38196 (request call id 110) took 1337ms (client timeout 520). I0312 20:10:22.438386 1094 raft_consensus.cc:2190] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 27 because replica is already servicing an update from a current leader or another vote. I0312 20:10:22.438442 1219 raft_consensus.cc:2190] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 27 because replica is already servicing an update from a current leader or another vote. W0312 20:10:22.438768 1216 rpcz_store.cc:255] Trace: 0312 20:10:21.101046 (+ 0us) service_pool.cc:163] Inserting onto call queue 0312 20:10:21.101164 (+ 118us) service_pool.cc:222] Handling call 0312 20:10:22.437964 (+1336800us) spinlock_profiling.cc:261] Waited 1.34 s on lock 0x7b480003009c. stack: 00007efc5efb2c74 00007efc5efb2b70 00007efc5daa2314 00007efc66aef294 00007efc66aef229 00007efc65699606 00007efc66b2d56b 00007efc614e0b24 00007efc614e0ab1 00007efc614e09c4 00007efc60b3cc29 00007efc60b3c6ec 00007efc60b3dbcd 00007efc60b3f946 00007efc60b3f89a 00007efc60b3f823 0312 20:10:22.438038 (+ 74us) inbound_call.cc:157] Queueing success response Metrics: {"spinlock_wait_cycles":3073463552} W0312 20:10:22.438815 1094 rpcz_store.cc:251] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.53.3:55789 (request call id 139) took 815ms (client timeout 528). W0312 20:10:22.438926 1219 rpcz_store.cc:251] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.53.3:38196 (request call id 112) took 712ms (client timeout 88). I0312 20:10:22.439033 1345 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "6885de38238442d6a9e5f1eb9fb569ff" candidate_term: 28 candidate_status { last_received { term: 25 index: 863 } } ignore_live_leader: false dest_uuid: "f78a2351568c4a268ee1112681e22bec" is_pre_election: true I0312 20:10:22.439236 1152 leader_election.cc:258] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. W0312 20:10:22.439409 1094 rpcz_store.cc:255] Trace: 0312 20:10:21.623585 (+ 0us) service_pool.cc:163] Inserting onto call queue 0312 20:10:21.623702 (+ 117us) service_pool.cc:222] Handling call 0312 20:10:22.438699 (+814997us) spinlock_profiling.cc:261] Waited 810 ms on lock 0x7b480003c09c. stack: 00007f2232dd2c74 00007f2232dd2b70 00007f22318c2314 00007f223a90f294 00007f223a90f229 00007f22394b9606 00007f223a94d56b 00007f2235300b24 00007f2235300ab1 00007f22353009c4 00007f223495cc29 00007f223495c6ec 00007f223495dbcd 00007f223495f946 00007f223495f89a 00007f223495f823 0312 20:10:22.438775 (+ 76us) inbound_call.cc:157] Queueing success response Metrics: {"spinlock_wait_cycles":1862622080} I0312 20:10:22.439411 1345 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec: Snoozing failure detection for 0.025s (vote granted) W0312 20:10:22.439534 1219 rpcz_store.cc:255] Trace: 0312 20:10:21.726025 (+ 0us) service_pool.cc:163] Inserting onto call queue 0312 20:10:21.726161 (+ 136us) service_pool.cc:222] Handling call 0312 20:10:22.438795 (+712634us) spinlock_profiling.cc:261] Waited 712 ms on lock 0x7b480003009c. stack: 00007efc5efb2c74 00007efc5efb2b70 00007efc5daa2314 00007efc66aef294 00007efc66aef229 00007efc65699606 00007efc66b2d56b 00007efc614e0b24 00007efc614e0ab1 00007efc614e09c4 00007efc60b3cc29 00007efc60b3c6ec 00007efc60b3dbcd 00007efc60b3f946 00007efc60b3f89a 00007efc60b3f823 0312 20:10:22.438884 (+ 89us) inbound_call.cc:157] Queueing success response Metrics: {"spinlock_wait_cycles":1636902144} I0312 20:10:22.439589 1345 raft_consensus.cc:2215] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6885de38238442d6a9e5f1eb9fb569ff in term 27. I0312 20:10:22.439646 1517 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff: Snoozing failure detection for 0.015s (election complete) I0312 20:10:22.439795 1096 raft_consensus.cc:2145] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate 6885de38238442d6a9e5f1eb9fb569ff in current term 27: Already voted for candidate ded9b7f8677940fb91e1a8c8bceceaed in this term. I0312 20:10:22.439826 1517 raft_consensus.cc:2507] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 27 FOLLOWER]: Leader election won for term 27 I0312 20:10:22.440104 1093 raft_consensus.cc:2190] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6885de38238442d6a9e5f1eb9fb569ff for term 28 because replica is already servicing an update from a current leader or another vote. I0312 20:10:22.440042 1517 raft_consensus.cc:572] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 27 LEADER]: Becoming Leader. State: Replica: 6885de38238442d6a9e5f1eb9fb569ff, State: Running, Role: LEADER I0312 20:10:22.440443 1517 consensus_queue.cc:196] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 863, Committed index: 863, Last appended: 25.863, Last appended by leader: 863, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:22.441383 1279 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 27 pre-election: Vote denied by peer 6885de38238442d6a9e5f1eb9fb569ff. Message: Service unavailable: T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 27 because replica is already servicing an update from a current leader or another vote. I0312 20:10:22.441596 1153 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 27 election: Vote denied by peer ded9b7f8677940fb91e1a8c8bceceaed. Message: Invalid argument: T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate 6885de38238442d6a9e5f1eb9fb569ff in current term 27: Already voted for candidate ded9b7f8677940fb91e1a8c8bceceaed in this term. I0312 20:10:22.442126 1153 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 28 pre-election: Vote denied by peer ded9b7f8677940fb91e1a8c8bceceaed. Message: Service unavailable: T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6885de38238442d6a9e5f1eb9fb569ff for term 28 because replica is already servicing an update from a current leader or another vote. I0312 20:10:22.443236 1152 leader_election.cc:363] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 28 pre-election: Vote granted by peer f78a2351568c4a268ee1112681e22bec I0312 20:10:22.443480 1152 leader_election.cc:258] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. I0312 20:10:22.443905 1517 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff: Snoozing failure detection for 0.020s (election complete) I0312 20:10:22.444797 1278 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 27 pre-election: Vote denied by peer ded9b7f8677940fb91e1a8c8bceceaed. Message: Service unavailable: T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 27 because replica is already servicing an update from a current leader or another vote. I0312 20:10:22.445035 1278 leader_election.cc:258] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. I0312 20:10:22.445325 1220 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" candidate_term: 28 candidate_status { last_received { term: 25 index: 863 } } ignore_live_leader: false dest_uuid: "6885de38238442d6a9e5f1eb9fb569ff" is_pre_election: true I0312 20:10:22.445837 1220 raft_consensus.cc:2176] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 27 LEADER]: Leader pre-election vote request: Denying vote to candidate ded9b7f8677940fb91e1a8c8bceceaed for term 28 because replica is either leader or believes a valid leader to be alive. I0312 20:10:22.445405 973 catalog_manager.cc:3592] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff reported cstate change: term changed from 25 to 27, leader changed from ded9b7f8677940fb91e1a8c8bceceaed (127.0.53.1) to 6885de38238442d6a9e5f1eb9fb569ff (127.0.53.2). New cstate: current_term: 27 leader_uuid: "6885de38238442d6a9e5f1eb9fb569ff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } health_report { overall_health: UNKNOWN } } } I0312 20:10:22.447088 1345 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" candidate_term: 28 candidate_status { last_received { term: 25 index: 863 } } ignore_live_leader: false dest_uuid: "f78a2351568c4a268ee1112681e22bec" is_pre_election: true W0312 20:10:22.447244 1027 leader_election.cc:287] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 27 election: RPC error from VoteRequest() call to peer f78a2351568c4a268ee1112681e22bec: Timed out: RequestConsensusVote RPC to 127.0.53.3:43274 timed out after 0.019s (SENT) I0312 20:10:22.447458 1027 leader_election.cc:258] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. I0312 20:10:22.447468 1345 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec: Snoozing failure detection for 0.274s (vote granted) I0312 20:10:22.447639 1510 raft_consensus.cc:436] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0312 20:10:22.447662 1345 raft_consensus.cc:2215] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ded9b7f8677940fb91e1a8c8bceceaed in term 27. I0312 20:10:22.447854 1510 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed: Snoozing failure detection for 0.018s (starting election) I0312 20:10:22.448071 1510 raft_consensus.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } W0312 20:10:22.448601 1027 leader_election.cc:287] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer f78a2351568c4a268ee1112681e22bec: Timed out: RequestConsensusVote RPC to 127.0.53.3:43274 timed out after 0.015s (SENT) I0312 20:10:22.449041 1510 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 28 pre-election: Requesting pre-vote from peer f78a2351568c4a268ee1112681e22bec I0312 20:10:22.449095 1506 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed: Snoozing failure detection for 0.018s (election complete) I0312 20:10:22.449345 1506 raft_consensus.cc:2455] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority I0312 20:10:22.449426 1510 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 28 pre-election: Requesting pre-vote from peer 6885de38238442d6a9e5f1eb9fb569ff W0312 20:10:22.450070 1029 leader_election.cc:287] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer 6885de38238442d6a9e5f1eb9fb569ff: Timed out: RequestConsensusVote RPC to 127.0.53.2:51697 timed out after 0.015s (SENT) I0312 20:10:22.450259 1029 leader_election.cc:258] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. I0312 20:10:22.451113 1510 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed: Snoozing failure detection for 0.028s (election complete) I0312 20:10:22.451323 1510 raft_consensus.cc:2455] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority I0312 20:10:22.452677 1519 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec: Snoozing failure detection for 0.335s (election complete) I0312 20:10:22.452906 1519 raft_consensus.cc:2455] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 27 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority I0312 20:10:22.453428 1093 raft_consensus.cc:1058] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 27 FOLLOWER]: Refusing update from remote peer 6885de38238442d6a9e5f1eb9fb569ff: Log matching property violated. Preceding OpId in replica: term: 25 index: 863. Preceding OpId from leader: term: 27 index: 864. (index mismatch) I0312 20:10:22.453769 1345 raft_consensus.cc:1058] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 27 FOLLOWER]: Refusing update from remote peer 6885de38238442d6a9e5f1eb9fb569ff: Log matching property violated. Preceding OpId in replica: term: 25 index: 863. Preceding OpId from leader: term: 27 index: 864. (index mismatch) I0312 20:10:22.454015 1220 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" candidate_term: 28 candidate_status { last_received { term: 25 index: 863 } } ignore_live_leader: false dest_uuid: "6885de38238442d6a9e5f1eb9fb569ff" is_pre_election: true I0312 20:10:22.454494 1220 raft_consensus.cc:2176] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 27 LEADER]: Leader pre-election vote request: Denying vote to candidate ded9b7f8677940fb91e1a8c8bceceaed for term 28 because replica is either leader or believes a valid leader to be alive. I0312 20:10:22.454875 1345 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" candidate_term: 28 candidate_status { last_received { term: 25 index: 863 } } ignore_live_leader: false dest_uuid: "f78a2351568c4a268ee1112681e22bec" is_pre_election: true I0312 20:10:22.454828 1517 consensus_queue.cc:1017] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 }, Status: LMP_MISMATCH, Last received: 25.863, Next index: 864, Last known committed idx: 863, Time since last communication: 0.000s I0312 20:10:22.455248 1345 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec: Snoozing failure detection for 2.067s (vote granted) I0312 20:10:22.455436 1345 raft_consensus.cc:2215] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ded9b7f8677940fb91e1a8c8bceceaed in term 27. I0312 20:10:22.455958 1029 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 28 pre-election: Vote denied by peer 6885de38238442d6a9e5f1eb9fb569ff. Message: Invalid argument: T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 27 LEADER]: Leader pre-election vote request: Denying vote to candidate ded9b7f8677940fb91e1a8c8bceceaed for term 28 because replica is either leader or believes a valid leader to be alive. I0312 20:10:22.455793 1508 consensus_queue.cc:1017] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 }, Status: LMP_MISMATCH, Last received: 25.863, Next index: 864, Last known committed idx: 863, Time since last communication: 0.000s I0312 20:10:22.456269 1027 leader_election.cc:363] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 28 pre-election: Vote granted by peer f78a2351568c4a268ee1112681e22bec I0312 20:10:22.456487 1027 leader_election.cc:258] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. I0312 20:10:22.457084 1510 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed: Snoozing failure detection for 0.031s (election complete) I0312 20:10:22.457298 1510 raft_consensus.cc:2507] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 27 FOLLOWER]: Leader pre-election won for term 28 I0312 20:10:22.457473 1510 raft_consensus.cc:436] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0312 20:10:22.457640 1510 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed: Snoozing failure detection for 0.032s (starting election) I0312 20:10:22.457808 1510 raft_consensus.cc:2752] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 27 FOLLOWER]: Advancing to term 28 I0312 20:10:22.462198 1510 raft_consensus.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:22.463063 1510 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 28 election: Requesting vote from peer f78a2351568c4a268ee1112681e22bec I0312 20:10:22.463162 1093 raft_consensus.cc:1026] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 28 FOLLOWER]: Rejecting Update request from peer 6885de38238442d6a9e5f1eb9fb569ff for earlier term 27. Current term is 28. Ops: [] I0312 20:10:22.463415 1510 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 28 election: Requesting vote from peer 6885de38238442d6a9e5f1eb9fb569ff I0312 20:10:22.464205 1345 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" candidate_term: 28 candidate_status { last_received { term: 25 index: 863 } } ignore_live_leader: false dest_uuid: "f78a2351568c4a268ee1112681e22bec" I0312 20:10:22.464586 1345 raft_consensus.cc:2752] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 27 FOLLOWER]: Advancing to term 28 I0312 20:10:22.464809 1345 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec: Snoozing failure detection for 2.516s (vote granted) I0312 20:10:22.466009 1220 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" candidate_term: 28 candidate_status { last_received { term: 25 index: 863 } } ignore_live_leader: false dest_uuid: "6885de38238442d6a9e5f1eb9fb569ff" I0312 20:10:22.466481 1220 raft_consensus.cc:2176] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 27 LEADER]: Leader election vote request: Denying vote to candidate ded9b7f8677940fb91e1a8c8bceceaed for term 28 because replica is either leader or believes a valid leader to be alive. I0312 20:10:22.467118 1029 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 28 election: Vote denied by peer 6885de38238442d6a9e5f1eb9fb569ff. Message: Invalid argument: T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 27 LEADER]: Leader election vote request: Denying vote to candidate ded9b7f8677940fb91e1a8c8bceceaed for term 28 because replica is either leader or believes a valid leader to be alive. I0312 20:10:22.469045 1517 consensus_queue.cc:1028] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 }, Status: INVALID_TERM, Last received: 25.863, Next index: 864, Last known committed idx: 863, Time since last communication: 0.000s I0312 20:10:22.469518 1508 raft_consensus.cc:2747] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 27 LEADER]: Stepping down as leader of term 27 I0312 20:10:22.469672 1508 raft_consensus.cc:605] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 27 LEADER]: Becoming Follower/Learner. State: Replica: 6885de38238442d6a9e5f1eb9fb569ff, State: Running, Role: LEADER I0312 20:10:22.470024 1508 consensus_queue.cc:219] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 863, Committed index: 863, Last appended: 27.864, Last appended by leader: 864, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:22.470633 1508 consensus_peers.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff -> Peer ded9b7f8677940fb91e1a8c8bceceaed (127.0.53.1:35037): Closing peer: ded9b7f8677940fb91e1a8c8bceceaed I0312 20:10:22.470804 1508 consensus_peers.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff -> Peer f78a2351568c4a268ee1112681e22bec (127.0.53.3:43274): Closing peer: f78a2351568c4a268ee1112681e22bec I0312 20:10:22.471005 1508 raft_consensus.cc:2752] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 27 FOLLOWER]: Advancing to term 28 I0312 20:10:22.482002 1508 raft_consensus.cc:436] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0312 20:10:22.482153 1508 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff: Snoozing failure detection for 0.016s (starting election) I0312 20:10:22.482303 1508 raft_consensus.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:22.482969 1508 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 29 pre-election: Requesting pre-vote from peer f78a2351568c4a268ee1112681e22bec I0312 20:10:22.483222 1508 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 29 pre-election: Requesting pre-vote from peer ded9b7f8677940fb91e1a8c8bceceaed I0312 20:10:22.484017 1346 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "6885de38238442d6a9e5f1eb9fb569ff" candidate_term: 29 candidate_status { last_received { term: 27 index: 864 } } ignore_live_leader: false dest_uuid: "f78a2351568c4a268ee1112681e22bec" is_pre_election: true I0312 20:10:22.484184 1093 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "6885de38238442d6a9e5f1eb9fb569ff" candidate_term: 29 candidate_status { last_received { term: 27 index: 864 } } ignore_live_leader: false dest_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" is_pre_election: true I0312 20:10:22.484604 1093 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed: Snoozing failure detection for 0.044s (vote granted) I0312 20:10:22.484757 1093 raft_consensus.cc:2215] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6885de38238442d6a9e5f1eb9fb569ff in term 28. I0312 20:10:22.485391 1153 leader_election.cc:363] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 29 pre-election: Vote granted by peer ded9b7f8677940fb91e1a8c8bceceaed I0312 20:10:22.485532 1153 leader_election.cc:258] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. I0312 20:10:22.485852 1508 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff: Snoozing failure detection for 0.019s (election complete) I0312 20:10:22.486045 1508 raft_consensus.cc:2507] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 28 FOLLOWER]: Leader pre-election won for term 29 I0312 20:10:22.486165 1508 raft_consensus.cc:436] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0312 20:10:22.486271 1508 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff: Snoozing failure detection for 0.018s (starting election) I0312 20:10:22.486414 1508 raft_consensus.cc:2752] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 28 FOLLOWER]: Advancing to term 29 W0312 20:10:22.496317 1027 leader_election.cc:287] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 28 election: RPC error from VoteRequest() call to peer f78a2351568c4a268ee1112681e22bec: Timed out: RequestConsensusVote RPC to 127.0.53.3:43274 timed out after 0.032s (SENT) I0312 20:10:22.496472 1027 leader_election.cc:258] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. I0312 20:10:22.496783 1510 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed: Snoozing failure detection for 0.043s (election complete) I0312 20:10:22.496961 1510 raft_consensus.cc:2455] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority W0312 20:10:22.499653 1152 leader_election.cc:287] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer f78a2351568c4a268ee1112681e22bec: Timed out: RequestConsensusVote RPC to 127.0.53.3:43274 timed out after 0.016s (SENT) I0312 20:10:22.540832 1510 raft_consensus.cc:436] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0312 20:10:22.540987 1510 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed: Snoozing failure detection for 0.023s (starting election) I0312 20:10:22.541151 1510 raft_consensus.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:22.541810 1510 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 29 pre-election: Requesting pre-vote from peer f78a2351568c4a268ee1112681e22bec I0312 20:10:22.542073 1510 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 29 pre-election: Requesting pre-vote from peer 6885de38238442d6a9e5f1eb9fb569ff I0312 20:10:22.542945 1343 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" candidate_term: 29 candidate_status { last_received { term: 25 index: 863 } } ignore_live_leader: false dest_uuid: "f78a2351568c4a268ee1112681e22bec" is_pre_election: true I0312 20:10:22.543089 1220 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" candidate_term: 29 candidate_status { last_received { term: 25 index: 863 } } ignore_live_leader: false dest_uuid: "6885de38238442d6a9e5f1eb9fb569ff" is_pre_election: true I0312 20:10:22.564445 1510 raft_consensus.cc:436] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0312 20:10:22.564613 1510 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed: Snoozing failure detection for 0.047s (starting election) I0312 20:10:22.564774 1510 raft_consensus.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:22.565459 1510 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 29 pre-election: Requesting pre-vote from peer f78a2351568c4a268ee1112681e22bec W0312 20:10:22.565716 1027 leader_election.cc:287] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer f78a2351568c4a268ee1112681e22bec: Timed out: RequestConsensusVote RPC to 127.0.53.3:43274 timed out after 0.023s (SENT) I0312 20:10:22.565771 1510 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 29 pre-election: Requesting pre-vote from peer 6885de38238442d6a9e5f1eb9fb569ff W0312 20:10:22.565976 1029 leader_election.cc:287] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer 6885de38238442d6a9e5f1eb9fb569ff: Timed out: RequestConsensusVote RPC to 127.0.53.2:51697 timed out after 0.023s (SENT) I0312 20:10:22.566179 1029 leader_election.cc:258] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. I0312 20:10:22.566557 1510 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed: Snoozing failure detection for 0.047s (election complete) I0312 20:10:22.566710 1510 raft_consensus.cc:2455] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority I0312 20:10:22.566866 1342 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" candidate_term: 29 candidate_status { last_received { term: 25 index: 863 } } ignore_live_leader: false dest_uuid: "f78a2351568c4a268ee1112681e22bec" is_pre_election: true I0312 20:10:22.567356 1219 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" candidate_term: 29 candidate_status { last_received { term: 25 index: 863 } } ignore_live_leader: false dest_uuid: "6885de38238442d6a9e5f1eb9fb569ff" is_pre_election: true W0312 20:10:22.613664 1027 leader_election.cc:287] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer f78a2351568c4a268ee1112681e22bec: Timed out: RequestConsensusVote RPC to 127.0.53.3:43274 timed out after 0.047s (SENT) W0312 20:10:22.614346 1029 leader_election.cc:287] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer 6885de38238442d6a9e5f1eb9fb569ff: Timed out: RequestConsensusVote RPC to 127.0.53.2:51697 timed out after 0.047s (SENT) I0312 20:10:22.614507 1029 leader_election.cc:258] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. I0312 20:10:22.614742 1510 raft_consensus.cc:436] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0312 20:10:22.614903 1510 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed: Snoozing failure detection for 0.053s (starting election) I0312 20:10:22.615072 1510 raft_consensus.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:22.615800 1510 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 29 pre-election: Requesting pre-vote from peer f78a2351568c4a268ee1112681e22bec I0312 20:10:22.615826 1506 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed: Snoozing failure detection for 0.113s (election complete) I0312 20:10:22.616011 1506 raft_consensus.cc:2455] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority I0312 20:10:22.616099 1510 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 29 pre-election: Requesting pre-vote from peer 6885de38238442d6a9e5f1eb9fb569ff I0312 20:10:22.616950 1341 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" candidate_term: 29 candidate_status { last_received { term: 25 index: 863 } } ignore_live_leader: false dest_uuid: "f78a2351568c4a268ee1112681e22bec" is_pre_election: true I0312 20:10:22.617141 1216 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" candidate_term: 29 candidate_status { last_received { term: 25 index: 863 } } ignore_live_leader: false dest_uuid: "6885de38238442d6a9e5f1eb9fb569ff" is_pre_election: true W0312 20:10:22.669662 1027 leader_election.cc:287] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer f78a2351568c4a268ee1112681e22bec: Timed out: RequestConsensusVote RPC to 127.0.53.3:43274 timed out after 0.053s (SENT) W0312 20:10:22.669914 1029 leader_election.cc:287] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer 6885de38238442d6a9e5f1eb9fb569ff: Timed out: RequestConsensusVote RPC to 127.0.53.2:51697 timed out after 0.053s (SENT) I0312 20:10:22.670044 1029 leader_election.cc:258] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. I0312 20:10:22.670322 1510 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed: Snoozing failure detection for 0.170s (election complete) I0312 20:10:22.670480 1510 raft_consensus.cc:2455] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority I0312 20:10:22.842211 1510 raft_consensus.cc:436] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0312 20:10:22.842391 1510 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed: Snoozing failure detection for 0.221s (starting election) I0312 20:10:22.842591 1510 raft_consensus.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:22.843345 1510 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 29 pre-election: Requesting pre-vote from peer f78a2351568c4a268ee1112681e22bec I0312 20:10:22.843621 1510 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 29 pre-election: Requesting pre-vote from peer 6885de38238442d6a9e5f1eb9fb569ff I0312 20:10:22.844439 1340 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" candidate_term: 29 candidate_status { last_received { term: 25 index: 863 } } ignore_live_leader: false dest_uuid: "f78a2351568c4a268ee1112681e22bec" is_pre_election: true I0312 20:10:22.844604 1215 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" candidate_term: 29 candidate_status { last_received { term: 25 index: 863 } } ignore_live_leader: false dest_uuid: "6885de38238442d6a9e5f1eb9fb569ff" is_pre_election: true I0312 20:10:23.064203 1510 raft_consensus.cc:436] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0312 20:10:23.064415 1510 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed: Snoozing failure detection for 0.120s (starting election) I0312 20:10:23.064595 1510 raft_consensus.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:23.065402 1510 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 29 pre-election: Requesting pre-vote from peer f78a2351568c4a268ee1112681e22bec W0312 20:10:23.065604 1027 leader_election.cc:287] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer f78a2351568c4a268ee1112681e22bec: Timed out: RequestConsensusVote RPC to 127.0.53.3:43274 timed out after 0.221s (SENT) I0312 20:10:23.065675 1510 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 29 pre-election: Requesting pre-vote from peer 6885de38238442d6a9e5f1eb9fb569ff W0312 20:10:23.065863 1029 leader_election.cc:287] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer 6885de38238442d6a9e5f1eb9fb569ff: Timed out: RequestConsensusVote RPC to 127.0.53.2:51697 timed out after 0.221s (SENT) I0312 20:10:23.066079 1029 leader_election.cc:258] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. I0312 20:10:23.066469 1510 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed: Snoozing failure detection for 0.194s (election complete) I0312 20:10:23.066658 1510 raft_consensus.cc:2455] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority I0312 20:10:23.066782 1339 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" candidate_term: 29 candidate_status { last_received { term: 25 index: 863 } } ignore_live_leader: false dest_uuid: "f78a2351568c4a268ee1112681e22bec" is_pre_election: true I0312 20:10:23.067312 1217 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" candidate_term: 29 candidate_status { last_received { term: 25 index: 863 } } ignore_live_leader: false dest_uuid: "6885de38238442d6a9e5f1eb9fb569ff" is_pre_election: true W0312 20:10:23.186806 1027 leader_election.cc:287] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer f78a2351568c4a268ee1112681e22bec: Timed out: RequestConsensusVote RPC to 127.0.53.3:43274 timed out after 0.120s (SENT) W0312 20:10:23.187412 1029 leader_election.cc:287] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer 6885de38238442d6a9e5f1eb9fb569ff: Timed out: RequestConsensusVote RPC to 127.0.53.2:51697 timed out after 0.120s (SENT) I0312 20:10:23.187554 1029 leader_election.cc:258] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. I0312 20:10:23.187885 1510 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed: Snoozing failure detection for 0.250s (election complete) I0312 20:10:23.188081 1510 raft_consensus.cc:2455] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority I0312 20:10:23.439280 1510 raft_consensus.cc:436] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0312 20:10:23.439533 1510 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed: Snoozing failure detection for 0.364s (starting election) I0312 20:10:23.439724 1510 raft_consensus.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:23.440534 1510 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 29 pre-election: Requesting pre-vote from peer f78a2351568c4a268ee1112681e22bec I0312 20:10:23.440846 1510 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 29 pre-election: Requesting pre-vote from peer 6885de38238442d6a9e5f1eb9fb569ff I0312 20:10:23.441951 1338 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" candidate_term: 29 candidate_status { last_received { term: 25 index: 863 } } ignore_live_leader: false dest_uuid: "f78a2351568c4a268ee1112681e22bec" is_pre_election: true I0312 20:10:23.442277 1218 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" candidate_term: 29 candidate_status { last_received { term: 25 index: 863 } } ignore_live_leader: false dest_uuid: "6885de38238442d6a9e5f1eb9fb569ff" is_pre_election: true I0312 20:10:23.804385 1510 raft_consensus.cc:436] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0312 20:10:23.804584 1510 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed: Snoozing failure detection for 0.448s (starting election) I0312 20:10:23.804774 1510 raft_consensus.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } W0312 20:10:23.805373 1027 leader_election.cc:287] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer f78a2351568c4a268ee1112681e22bec: Timed out: RequestConsensusVote RPC to 127.0.53.3:43274 timed out after 0.364s (SENT) W0312 20:10:23.805634 1029 leader_election.cc:287] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer 6885de38238442d6a9e5f1eb9fb569ff: Timed out: RequestConsensusVote RPC to 127.0.53.2:51697 timed out after 0.364s (SENT) I0312 20:10:23.805634 1510 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 29 pre-election: Requesting pre-vote from peer f78a2351568c4a268ee1112681e22bec I0312 20:10:23.805883 1029 leader_election.cc:258] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. I0312 20:10:23.806108 1510 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 29 pre-election: Requesting pre-vote from peer 6885de38238442d6a9e5f1eb9fb569ff I0312 20:10:23.806527 1510 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed: Snoozing failure detection for 0.070s (election complete) I0312 20:10:23.806780 1510 raft_consensus.cc:2455] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority I0312 20:10:23.807742 1337 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" candidate_term: 29 candidate_status { last_received { term: 25 index: 863 } } ignore_live_leader: false dest_uuid: "f78a2351568c4a268ee1112681e22bec" is_pre_election: true I0312 20:10:23.807862 1221 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" candidate_term: 29 candidate_status { last_received { term: 25 index: 863 } } ignore_live_leader: false dest_uuid: "6885de38238442d6a9e5f1eb9fb569ff" is_pre_election: true I0312 20:10:23.877946 1510 raft_consensus.cc:436] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0312 20:10:23.878111 1510 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed: Snoozing failure detection for 0.476s (starting election) I0312 20:10:23.878269 1510 raft_consensus.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:23.878995 1510 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 29 pre-election: Requesting pre-vote from peer f78a2351568c4a268ee1112681e22bec I0312 20:10:23.879252 1510 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 29 pre-election: Requesting pre-vote from peer 6885de38238442d6a9e5f1eb9fb569ff I0312 20:10:23.880228 1336 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" candidate_term: 29 candidate_status { last_received { term: 25 index: 863 } } ignore_live_leader: false dest_uuid: "f78a2351568c4a268ee1112681e22bec" is_pre_election: true I0312 20:10:23.880370 1214 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" candidate_term: 29 candidate_status { last_received { term: 25 index: 863 } } ignore_live_leader: false dest_uuid: "6885de38238442d6a9e5f1eb9fb569ff" is_pre_election: true W0312 20:10:23.974486 1345 consensus_meta.cc:216] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec: Time spent flushing consensus metadata: real 1.509s user 0.004s sys 0.000s I0312 20:10:23.974704 1345 raft_consensus.cc:2215] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ded9b7f8677940fb91e1a8c8bceceaed in term 28. W0312 20:10:23.974961 1508 consensus_meta.cc:216] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff: Time spent flushing consensus metadata: real 1.488s user 0.004s sys 0.000s W0312 20:10:23.975047 1345 rpcz_store.cc:251] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.53.1:47760 (request call id 967) took 1511ms (client timeout 32). I0312 20:10:23.975121 1344 raft_consensus.cc:1026] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 28 FOLLOWER]: Rejecting Update request from peer 6885de38238442d6a9e5f1eb9fb569ff for earlier term 27. Current term is 28. Ops: [] W0312 20:10:23.975564 1345 rpcz_store.cc:255] Trace: 0312 20:10:22.464009 (+ 0us) service_pool.cc:163] Inserting onto call queue 0312 20:10:22.464094 (+ 85us) service_pool.cc:222] Handling call 0312 20:10:23.974997 (+1510903us) inbound_call.cc:157] Queueing success response Metrics: {} I0312 20:10:23.975219 1508 raft_consensus.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:23.975718 1344 rpcz_store.cc:265] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.53.2:37068 (request call id 61) took 1510ms. Request Metrics: {"spinlock_wait_cycles":3472135552} I0312 20:10:23.976011 1340 raft_consensus.cc:2190] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ded9b7f8677940fb91e1a8c8bceceaed for term 29 because replica is already servicing an update from a current leader or another vote. I0312 20:10:23.976287 1508 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 29 election: Requesting vote from peer f78a2351568c4a268ee1112681e22bec I0312 20:10:23.976646 1508 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 29 election: Requesting vote from peer ded9b7f8677940fb91e1a8c8bceceaed W0312 20:10:23.976544 1340 rpcz_store.cc:251] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.53.1:47760 (request call id 971) took 1132ms (client timeout 221). I0312 20:10:23.977242 1221 raft_consensus.cc:2190] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ded9b7f8677940fb91e1a8c8bceceaed for term 29 because replica is already servicing an update from a current leader or another vote. W0312 20:10:23.977337 1340 rpcz_store.cc:255] Trace: 0312 20:10:22.844136 (+ 0us) service_pool.cc:163] Inserting onto call queue 0312 20:10:22.844255 (+ 119us) service_pool.cc:222] Handling call 0312 20:10:23.976409 (+1132154us) spinlock_profiling.cc:261] Waited 1.13 s on lock 0x7b480003009c. stack: 00007f173be47c74 00007f173be47b70 00007f173a937314 00007f1743984294 00007f1743984229 00007f174252e606 00007f17439c256b 00007f173e375b24 00007f173e375ab1 00007f173e3759c4 00007f173d9d1c29 00007f173d9d16ec 00007f173d9d2bcd 00007f173d9d4946 00007f173d9d489a 00007f173d9d4823 0312 20:10:23.976496 (+ 87us) inbound_call.cc:157] Queueing success response Metrics: {"spinlock_wait_cycles":2602353408} I0312 20:10:23.977828 1340 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "6885de38238442d6a9e5f1eb9fb569ff" candidate_term: 29 candidate_status { last_received { term: 27 index: 864 } } ignore_live_leader: false dest_uuid: "f78a2351568c4a268ee1112681e22bec" I0312 20:10:23.977917 1517 raft_consensus.cc:436] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0312 20:10:23.978127 1517 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff: Snoozing failure detection for 0.017s (starting election) I0312 20:10:23.978332 1029 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 29 pre-election: Vote denied by peer 6885de38238442d6a9e5f1eb9fb569ff. Message: Service unavailable: T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ded9b7f8677940fb91e1a8c8bceceaed for term 29 because replica is already servicing an update from a current leader or another vote. I0312 20:10:23.978351 1517 raft_consensus.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:23.979279 1517 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 30 pre-election: Requesting pre-vote from peer f78a2351568c4a268ee1112681e22bec I0312 20:10:23.979305 1219 raft_consensus.cc:2190] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ded9b7f8677940fb91e1a8c8bceceaed for term 29 because replica is already servicing an update from a current leader or another vote. I0312 20:10:23.979554 1517 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 30 pre-election: Requesting pre-vote from peer ded9b7f8677940fb91e1a8c8bceceaed W0312 20:10:23.979742 1219 rpcz_store.cc:251] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.53.1:43660 (request call id 1242) took 1412ms (client timeout 47). W0312 20:10:23.980350 1219 rpcz_store.cc:255] Trace: 0312 20:10:22.567155 (+ 0us) service_pool.cc:163] Inserting onto call queue 0312 20:10:22.567249 (+ 94us) service_pool.cc:222] Handling call 0312 20:10:23.979601 (+1412352us) spinlock_profiling.cc:261] Waited 1.41 s on lock 0x7b480003009c. stack: 00007efc5efb2c74 00007efc5efb2b70 00007efc5daa2314 00007efc66aef294 00007efc66aef229 00007efc65699606 00007efc66b2d56b 00007efc614e0b24 00007efc614e0ab1 00007efc614e09c4 00007efc60b3cc29 00007efc60b3c6ec 00007efc60b3dbcd 00007efc60b3f946 00007efc60b3f89a 00007efc60b3f823 0312 20:10:23.979686 (+ 85us) inbound_call.cc:157] Queueing success response Metrics: {"spinlock_wait_cycles":3247576832} I0312 20:10:23.980478 1346 raft_consensus.cc:2190] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6885de38238442d6a9e5f1eb9fb569ff for term 29 because replica is already servicing an update from a current leader or another vote. I0312 20:10:23.980504 1181 rpcz_store.cc:265] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:40835 (ReqId={client: 88cdfc08b6194c19b1581ef853d0ff5a, seq_no=850, attempt_no=6}) took 1491ms. Request Metrics: {"child_traces":[["txn",{"spinlock_wait_cycles":3427699584}]]} I0312 20:10:23.980829 1305 rpcz_store.cc:265] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:51178 (ReqId={client: 88cdfc08b6194c19b1581ef853d0ff5a, seq_no=851, attempt_no=5}) took 1493ms. Request Metrics: {"child_traces":[["txn",{"spinlock_wait_cycles":3427028352}]]} I0312 20:10:23.981469 1216 raft_consensus.cc:2190] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ded9b7f8677940fb91e1a8c8bceceaed for term 29 because replica is already servicing an update from a current leader or another vote. I0312 20:10:23.981758 1218 raft_consensus.cc:2190] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ded9b7f8677940fb91e1a8c8bceceaed for term 29 because replica is already servicing an update from a current leader or another vote. I0312 20:10:23.981935 1338 raft_consensus.cc:2190] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ded9b7f8677940fb91e1a8c8bceceaed for term 29 because replica is already servicing an update from a current leader or another vote. W0312 20:10:23.981876 1216 rpcz_store.cc:251] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.53.1:43660 (request call id 1243) took 1365ms (client timeout 53). W0312 20:10:23.982154 1218 rpcz_store.cc:251] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.53.1:43660 (request call id 1246) took 540ms (client timeout 364). I0312 20:10:23.982264 1339 raft_consensus.cc:2190] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ded9b7f8677940fb91e1a8c8bceceaed for term 29 because replica is already servicing an update from a current leader or another vote. W0312 20:10:23.982470 1216 rpcz_store.cc:255] Trace: 0312 20:10:22.616917 (+ 0us) service_pool.cc:163] Inserting onto call queue 0312 20:10:22.617011 (+ 94us) service_pool.cc:222] Handling call 0312 20:10:23.981750 (+1364739us) spinlock_profiling.cc:261] Waited 1.36 s on lock 0x7b480003009c. stack: 00007efc5efb2c74 00007efc5efb2b70 00007efc5daa2314 00007efc66aef294 00007efc66aef229 00007efc65699606 00007efc66b2d56b 00007efc614e0b24 00007efc614e0ab1 00007efc614e09c4 00007efc60b3cc29 00007efc60b3c6ec 00007efc60b3dbcd 00007efc60b3f946 00007efc60b3f89a 00007efc60b3f823 0312 20:10:23.981831 (+ 81us) inbound_call.cc:157] Queueing success response Metrics: {"spinlock_wait_cycles":3137883008} W0312 20:10:23.982388 1338 rpcz_store.cc:251] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.53.1:47760 (request call id 973) took 540ms (client timeout 364). W0312 20:10:23.982724 1218 rpcz_store.cc:255] Trace: 0312 20:10:23.442009 (+ 0us) service_pool.cc:163] Inserting onto call queue 0312 20:10:23.442111 (+ 102us) service_pool.cc:222] Handling call 0312 20:10:23.982042 (+539931us) spinlock_profiling.cc:261] Waited 539 ms on lock 0x7b480003009c. stack: 00007efc5efb2c74 00007efc5efb2b70 00007efc5daa2314 00007efc66aef294 00007efc66aef229 00007efc65699606 00007efc66b2d56b 00007efc614e0b24 00007efc614e0ab1 00007efc614e09c4 00007efc60b3cc29 00007efc60b3c6ec 00007efc60b3dbcd 00007efc60b3f946 00007efc60b3f89a 00007efc60b3f823 0312 20:10:23.982112 (+ 70us) inbound_call.cc:157] Queueing success response Metrics: {"spinlock_wait_cycles":1239955712} W0312 20:10:23.980921 1346 rpcz_store.cc:251] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.53.2:37068 (request call id 62) took 1497ms (client timeout 16). W0312 20:10:23.982708 1339 rpcz_store.cc:251] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.53.1:47760 (request call id 972) took 916ms (client timeout 120). I0312 20:10:23.983080 1217 raft_consensus.cc:2190] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ded9b7f8677940fb91e1a8c8bceceaed for term 29 because replica is already servicing an update from a current leader or another vote. W0312 20:10:23.983369 1338 rpcz_store.cc:255] Trace: 0312 20:10:23.441571 (+ 0us) service_pool.cc:163] Inserting onto call queue 0312 20:10:23.441695 (+ 124us) service_pool.cc:222] Handling call 0312 20:10:23.982261 (+540566us) spinlock_profiling.cc:261] Waited 539 ms on lock 0x7b480003009c. stack: 00007f173be47c74 00007f173be47b70 00007f173a937314 00007f1743984294 00007f1743984229 00007f174252e606 00007f17439c256b 00007f173e375b24 00007f173e375ab1 00007f173e3759c4 00007f173d9d1c29 00007f173d9d16ec 00007f173d9d2bcd 00007f173d9d4946 00007f173d9d489a 00007f173d9d4823 0312 20:10:23.982342 (+ 81us) inbound_call.cc:157] Queueing success response Metrics: {"spinlock_wait_cycles":1239199488} W0312 20:10:23.983507 1217 rpcz_store.cc:251] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.53.1:43660 (request call id 1245) took 916ms (client timeout 120). I0312 20:10:23.983727 1341 raft_consensus.cc:2190] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ded9b7f8677940fb91e1a8c8bceceaed for term 29 because replica is already servicing an update from a current leader or another vote. W0312 20:10:23.984093 1217 rpcz_store.cc:255] Trace: 0312 20:10:23.067086 (+ 0us) service_pool.cc:163] Inserting onto call queue 0312 20:10:23.067183 (+ 97us) service_pool.cc:222] Handling call 0312 20:10:23.983381 (+916198us) spinlock_profiling.cc:261] Waited 916 ms on lock 0x7b480003009c. stack: 00007efc5efb2c74 00007efc5efb2b70 00007efc5daa2314 00007efc66aef294 00007efc66aef229 00007efc65699606 00007efc66b2d56b 00007efc614e0b24 00007efc614e0ab1 00007efc614e09c4 00007efc60b3cc29 00007efc60b3c6ec 00007efc60b3dbcd 00007efc60b3f946 00007efc60b3f89a 00007efc60b3f823 0312 20:10:23.983463 (+ 82us) inbound_call.cc:157] Queueing success response Metrics: {"spinlock_wait_cycles":2105938304} W0312 20:10:23.984171 1341 rpcz_store.cc:251] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.53.1:47760 (request call id 970) took 1367ms (client timeout 53). I0312 20:10:23.984140 1093 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "6885de38238442d6a9e5f1eb9fb569ff" candidate_term: 29 candidate_status { last_received { term: 27 index: 864 } } ignore_live_leader: false dest_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" I0312 20:10:23.984449 1215 raft_consensus.cc:2190] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ded9b7f8677940fb91e1a8c8bceceaed for term 29 because replica is already servicing an update from a current leader or another vote. I0312 20:10:23.984561 1093 raft_consensus.cc:2752] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 28 FOLLOWER]: Advancing to term 29 I0312 20:10:23.984781 1093 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed: Snoozing failure detection for 0.780s (vote granted) W0312 20:10:23.984889 1215 rpcz_store.cc:251] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.53.1:43660 (request call id 1244) took 1140ms (client timeout 221). I0312 20:10:23.984356 1338 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "6885de38238442d6a9e5f1eb9fb569ff" candidate_term: 30 candidate_status { last_received { term: 27 index: 864 } } ignore_live_leader: false dest_uuid: "f78a2351568c4a268ee1112681e22bec" is_pre_election: true W0312 20:10:23.985466 1215 rpcz_store.cc:255] Trace: 0312 20:10:22.844383 (+ 0us) service_pool.cc:163] Inserting onto call queue 0312 20:10:22.844476 (+ 93us) service_pool.cc:222] Handling call 0312 20:10:23.984768 (+1140292us) spinlock_profiling.cc:261] Waited 1.14 s on lock 0x7b480003009c. stack: 00007efc5efb2c74 00007efc5efb2b70 00007efc5daa2314 00007efc66aef294 00007efc66aef229 00007efc65699606 00007efc66b2d56b 00007efc614e0b24 00007efc614e0ab1 00007efc614e09c4 00007efc60b3cc29 00007efc60b3c6ec 00007efc60b3dbcd 00007efc60b3f946 00007efc60b3f89a 00007efc60b3f823 0312 20:10:23.984845 (+ 77us) inbound_call.cc:157] Queueing success response Metrics: {"spinlock_wait_cycles":2621369216} W0312 20:10:23.985524 1341 rpcz_store.cc:255] Trace: 0312 20:10:22.616607 (+ 0us) service_pool.cc:163] Inserting onto call queue 0312 20:10:22.616738 (+ 131us) service_pool.cc:222] Handling call 0312 20:10:23.984044 (+1367306us) spinlock_profiling.cc:261] Waited 1.37 s on lock 0x7b480003009c. stack: 00007f173be47c74 00007f173be47b70 00007f173a937314 00007f1743984294 00007f1743984229 00007f174252e606 00007f17439c256b 00007f173e375b24 00007f173e375ab1 00007f173e3759c4 00007f173d9d1c29 00007f173d9d16ec 00007f173d9d2bcd 00007f173d9d4946 00007f173d9d489a 00007f173d9d4823 0312 20:10:23.984121 (+ 77us) inbound_call.cc:157] Queueing success response Metrics: {"spinlock_wait_cycles":3141762304} I0312 20:10:23.985530 1096 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "6885de38238442d6a9e5f1eb9fb569ff" candidate_term: 30 candidate_status { last_received { term: 27 index: 864 } } ignore_live_leader: false dest_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" is_pre_election: true I0312 20:10:23.985841 1214 raft_consensus.cc:2190] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ded9b7f8677940fb91e1a8c8bceceaed for term 29 because replica is already servicing an update from a current leader or another vote. I0312 20:10:23.985854 1343 raft_consensus.cc:2190] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ded9b7f8677940fb91e1a8c8bceceaed for term 29 because replica is already servicing an update from a current leader or another vote. W0312 20:10:23.986315 1343 rpcz_store.cc:251] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.53.1:47760 (request call id 968) took 1443ms (client timeout 23). I0312 20:10:23.986546 1220 raft_consensus.cc:2145] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ded9b7f8677940fb91e1a8c8bceceaed in current term 29: Already voted for candidate 6885de38238442d6a9e5f1eb9fb569ff in this term. I0312 20:10:23.986913 1029 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 29 pre-election: Vote denied by peer 6885de38238442d6a9e5f1eb9fb569ff. Message: Service unavailable: T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ded9b7f8677940fb91e1a8c8bceceaed for term 29 because replica is already servicing an update from a current leader or another vote. W0312 20:10:23.986976 1220 rpcz_store.cc:251] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.53.1:43660 (request call id 1241) took 1444ms (client timeout 23). W0312 20:10:23.987291 1343 rpcz_store.cc:255] Trace: 0312 20:10:22.542683 (+ 0us) service_pool.cc:163] Inserting onto call queue 0312 20:10:22.542800 (+ 117us) service_pool.cc:222] Handling call 0312 20:10:23.986174 (+1443374us) spinlock_profiling.cc:261] Waited 1.44 s on lock 0x7b480003009c. stack: 00007f173be47c74 00007f173be47b70 00007f173a937314 00007f1743984294 00007f1743984229 00007f174252e606 00007f17439c256b 00007f173e375b24 00007f173e375ab1 00007f173e3759c4 00007f173d9d1c29 00007f173d9d16ec 00007f173d9d2bcd 00007f173d9d4946 00007f173d9d489a 00007f173d9d4823 0312 20:10:23.986269 (+ 95us) inbound_call.cc:157] Queueing success response Metrics: {"spinlock_wait_cycles":3315939584} W0312 20:10:23.987491 1220 rpcz_store.cc:255] Trace: 0312 20:10:22.542862 (+ 0us) service_pool.cc:163] Inserting onto call queue 0312 20:10:22.542960 (+ 98us) service_pool.cc:222] Handling call 0312 20:10:23.986846 (+1443886us) spinlock_profiling.cc:261] Waited 1.44 s on lock 0x7b480003009c. stack: 00007efc5efb2c74 00007efc5efb2b70 00007efc5daa2314 00007efc66aef294 00007efc66aef229 00007efc65699af3 00007efc66b2d56b 00007efc614e0b24 00007efc614e0ab1 00007efc614e09c4 00007efc60b3cc29 00007efc60b3c6ec 00007efc60b3dbcd 00007efc60b3f946 00007efc60b3f89a 00007efc60b3f823 0312 20:10:23.986930 (+ 84us) inbound_call.cc:157] Queueing success response Metrics: {"spinlock_wait_cycles":3319771392} I0312 20:10:23.987773 1336 raft_consensus.cc:2190] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ded9b7f8677940fb91e1a8c8bceceaed for term 29 because replica is already servicing an update from a current leader or another vote. I0312 20:10:23.988463 1337 raft_consensus.cc:2190] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ded9b7f8677940fb91e1a8c8bceceaed for term 29 because replica is already servicing an update from a current leader or another vote. I0312 20:10:23.988766 1340 raft_consensus.cc:2752] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 28 FOLLOWER]: Advancing to term 29 W0312 20:10:23.988970 1339 rpcz_store.cc:255] Trace: 0312 20:10:23.066478 (+ 0us) service_pool.cc:163] Inserting onto call queue 0312 20:10:23.066607 (+ 129us) service_pool.cc:222] Handling call 0312 20:10:23.982589 (+915982us) spinlock_profiling.cc:261] Waited 915 ms on lock 0x7b480003009c. stack: 00007f173be47c74 00007f173be47b70 00007f173a937314 00007f1743984294 00007f1743984229 00007f174252e606 00007f17439c256b 00007f173e375b24 00007f173e375ab1 00007f173e3759c4 00007f173d9d1c29 00007f173d9d16ec 00007f173d9d2bcd 00007f173d9d4946 00007f173d9d489a 00007f173d9d4823 0312 20:10:23.982664 (+ 75us) inbound_call.cc:157] Queueing success response Metrics: {"spinlock_wait_cycles":2105050496} I0312 20:10:23.989002 1340 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec: Snoozing failure detection for 3.575s (vote granted) W0312 20:10:23.989753 1346 rpcz_store.cc:255] Trace: 0312 20:10:22.483732 (+ 0us) service_pool.cc:163] Inserting onto call queue 0312 20:10:22.483853 (+ 121us) service_pool.cc:222] Handling call 0312 20:10:23.980797 (+1496944us) spinlock_profiling.cc:261] Waited 1.5 s on lock 0x7b480003009c. stack: 00007f173be47c74 00007f173be47b70 00007f173a937314 00007f1743984294 00007f1743984229 00007f174252e606 00007f17439c256b 00007f173e375b24 00007f173e375ab1 00007f173e3759c4 00007f173d9d1c29 00007f173d9d16ec 00007f173d9d2bcd 00007f173d9d4946 00007f173d9d489a 00007f173d9d4823 0312 20:10:23.980879 (+ 82us) inbound_call.cc:157] Queueing success response Metrics: {"spinlock_wait_cycles":3441832320} I0312 20:10:23.989825 1093 raft_consensus.cc:2215] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6885de38238442d6a9e5f1eb9fb569ff in term 29. I0312 20:10:23.990371 1096 raft_consensus.cc:2190] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6885de38238442d6a9e5f1eb9fb569ff for term 30 because replica is already servicing an update from a current leader or another vote. I0312 20:10:23.990694 1153 leader_election.cc:363] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 29 election: Vote granted by peer ded9b7f8677940fb91e1a8c8bceceaed I0312 20:10:23.990876 1153 leader_election.cc:258] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. I0312 20:10:23.991526 1153 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 30 pre-election: Vote denied by peer ded9b7f8677940fb91e1a8c8bceceaed. Message: Service unavailable: T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6885de38238442d6a9e5f1eb9fb569ff for term 30 because replica is already servicing an update from a current leader or another vote. I0312 20:10:23.991883 1517 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff: Snoozing failure detection for 0.017s (election complete) I0312 20:10:23.992100 1517 raft_consensus.cc:2507] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 29 FOLLOWER]: Leader election won for term 29 I0312 20:10:23.992369 1517 raft_consensus.cc:572] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 29 LEADER]: Becoming Leader. State: Replica: 6885de38238442d6a9e5f1eb9fb569ff, State: Running, Role: LEADER I0312 20:10:23.992513 1340 raft_consensus.cc:2215] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6885de38238442d6a9e5f1eb9fb569ff in term 29. I0312 20:10:23.992900 1342 raft_consensus.cc:2190] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ded9b7f8677940fb91e1a8c8bceceaed for term 29 because replica is already servicing an update from a current leader or another vote. W0312 20:10:23.993362 1342 rpcz_store.cc:251] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.53.1:47760 (request call id 969) took 1426ms (client timeout 47). I0312 20:10:23.993602 1027 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 29 pre-election: Vote denied by peer f78a2351568c4a268ee1112681e22bec. Message: Service unavailable: T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ded9b7f8677940fb91e1a8c8bceceaed for term 29 because replica is already servicing an update from a current leader or another vote. I0312 20:10:23.992746 1517 consensus_queue.cc:196] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 863, Committed index: 863, Last appended: 27.864, Last appended by leader: 864, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:23.993827 1027 leader_election.cc:258] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. W0312 20:10:23.993991 1342 rpcz_store.cc:255] Trace: 0312 20:10:22.566511 (+ 0us) service_pool.cc:163] Inserting onto call queue 0312 20:10:22.566645 (+ 134us) service_pool.cc:222] Handling call 0312 20:10:23.993231 (+1426586us) spinlock_profiling.cc:261] Waited 1.43 s on lock 0x7b480003009c. stack: 00007f173be47c74 00007f173be47b70 00007f173a937314 00007f1743984294 00007f1743984229 00007f174252e606 00007f17439c256b 00007f173e375b24 00007f173e375ab1 00007f173e3759c4 00007f173d9d1c29 00007f173d9d16ec 00007f173d9d2bcd 00007f173d9d4946 00007f173d9d489a 00007f173d9d4823 0312 20:10:23.993315 (+ 84us) inbound_call.cc:157] Queueing success response Metrics: {"spinlock_wait_cycles":3279737088} I0312 20:10:23.994355 1338 raft_consensus.cc:2190] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6885de38238442d6a9e5f1eb9fb569ff for term 30 because replica is already servicing an update from a current leader or another vote. I0312 20:10:23.994781 1027 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 29 pre-election: Vote denied by peer f78a2351568c4a268ee1112681e22bec. Message: Service unavailable: T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ded9b7f8677940fb91e1a8c8bceceaed for term 29 because replica is already servicing an update from a current leader or another vote. I0312 20:10:23.994971 1027 leader_election.cc:258] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. W0312 20:10:23.995458 1340 rpcz_store.cc:251] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.53.2:37068 (request call id 63) took 18ms (client timeout 18). I0312 20:10:23.995698 1510 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed: Snoozing failure detection for 0.320s (election complete) I0312 20:10:23.995899 1510 raft_consensus.cc:2455] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 29 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority W0312 20:10:23.996053 1340 rpcz_store.cc:255] Trace: 0312 20:10:23.977624 (+ 0us) service_pool.cc:163] Inserting onto call queue 0312 20:10:23.977717 (+ 93us) service_pool.cc:222] Handling call 0312 20:10:23.995352 (+ 17635us) spinlock_profiling.cc:261] Waited 10.5 ms on lock 0x7b480003009c. stack: 00007f173be47c74 00007f173be47b70 00007f173a937314 00007f1743984294 00007f1743984229 00007f174252eaf3 00007f17439c256b 00007f173e375b24 00007f173e375ab1 00007f173e3759c4 00007f173d9d1c29 00007f173d9d16ec 00007f173d9d2bcd 00007f173d9d4946 00007f173d9d489a 00007f173d9d4823 0312 20:10:23.995417 (+ 65us) inbound_call.cc:157] Queueing success response Metrics: {"spinlock_wait_cycles":24229760} I0312 20:10:23.996145 1510 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed: Snoozing failure detection for 0.069s (election complete) I0312 20:10:23.996314 1510 raft_consensus.cc:2455] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 29 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority W0312 20:10:23.998817 1152 leader_election.cc:287] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 29 election: RPC error from VoteRequest() call to peer f78a2351568c4a268ee1112681e22bec: Timed out: RequestConsensusVote RPC to 127.0.53.3:43274 timed out after 0.018s (SENT) W0312 20:10:23.999227 1152 leader_election.cc:287] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 30 pre-election: RPC error from VoteRequest() call to peer f78a2351568c4a268ee1112681e22bec: Timed out: RequestConsensusVote RPC to 127.0.53.3:43274 timed out after 0.017s (SENT) I0312 20:10:23.999418 1152 leader_election.cc:258] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. I0312 20:10:24.001263 1518 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff: Snoozing failure detection for 0.015s (election complete) I0312 20:10:24.001514 1518 raft_consensus.cc:2455] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 29 LEADER]: Leader pre-election lost for term 30. Reason: could not achieve majority I0312 20:10:24.001687 1340 raft_consensus.cc:1058] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 29 FOLLOWER]: Refusing update from remote peer 6885de38238442d6a9e5f1eb9fb569ff: Log matching property violated. Preceding OpId in replica: term: 25 index: 863. Preceding OpId from leader: term: 27 index: 864. (index mismatch) I0312 20:10:24.002763 1518 consensus_queue.cc:1017] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 }, Status: LMP_MISMATCH, Last received: 25.863, Next index: 864, Last known committed idx: 863, Time since last communication: 0.000s I0312 20:10:24.003808 1096 raft_consensus.cc:1058] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 29 FOLLOWER]: Refusing update from remote peer 6885de38238442d6a9e5f1eb9fb569ff: Log matching property violated. Preceding OpId in replica: term: 25 index: 863. Preceding OpId from leader: term: 29 index: 865. (index mismatch) I0312 20:10:24.003156 973 catalog_manager.cc:3592] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff reported cstate change: term changed from 27 to 29. New cstate: current_term: 29 leader_uuid: "6885de38238442d6a9e5f1eb9fb569ff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } health_report { overall_health: UNKNOWN } } } I0312 20:10:24.005023 1518 consensus_queue.cc:1017] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 }, Status: LMP_MISMATCH, Last received: 25.863, Next index: 864, Last known committed idx: 863, Time since last communication: 0.000s I0312 20:10:24.052368 1527 raft_consensus.cc:436] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 6885de38238442d6a9e5f1eb9fb569ff) I0312 20:10:24.052636 1527 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed: Snoozing failure detection for 0.021s (starting election) I0312 20:10:24.052889 1527 raft_consensus.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:24.054039 1527 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 30 pre-election: Requesting pre-vote from peer f78a2351568c4a268ee1112681e22bec I0312 20:10:24.054399 1527 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 30 pre-election: Requesting pre-vote from peer 6885de38238442d6a9e5f1eb9fb569ff I0312 20:10:24.061358 1220 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" candidate_term: 30 candidate_status { last_received { term: 29 index: 866 } } ignore_live_leader: false dest_uuid: "6885de38238442d6a9e5f1eb9fb569ff" is_pre_election: true I0312 20:10:24.061980 1220 raft_consensus.cc:2176] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 29 LEADER]: Leader pre-election vote request: Denying vote to candidate ded9b7f8677940fb91e1a8c8bceceaed for term 30 because replica is either leader or believes a valid leader to be alive. I0312 20:10:24.062599 1338 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" candidate_term: 30 candidate_status { last_received { term: 29 index: 866 } } ignore_live_leader: false dest_uuid: "f78a2351568c4a268ee1112681e22bec" is_pre_election: true I0312 20:10:24.062902 1029 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 30 pre-election: Vote denied by peer 6885de38238442d6a9e5f1eb9fb569ff. Message: Invalid argument: T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 29 LEADER]: Leader pre-election vote request: Denying vote to candidate ded9b7f8677940fb91e1a8c8bceceaed for term 30 because replica is either leader or believes a valid leader to be alive. I0312 20:10:24.063133 1338 raft_consensus.cc:2176] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ded9b7f8677940fb91e1a8c8bceceaed for term 30 because replica is either leader or believes a valid leader to be alive. I0312 20:10:24.063985 1027 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 30 pre-election: Vote denied by peer f78a2351568c4a268ee1112681e22bec. Message: Invalid argument: T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ded9b7f8677940fb91e1a8c8bceceaed for term 30 because replica is either leader or believes a valid leader to be alive. I0312 20:10:24.064188 1027 leader_election.cc:258] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. I0312 20:10:24.064775 1527 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed: Snoozing failure detection for 0.015s (election complete) I0312 20:10:24.064968 1527 raft_consensus.cc:2455] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority I0312 20:10:24.090037 1530 raft_consensus.cc:436] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 6885de38238442d6a9e5f1eb9fb569ff) I0312 20:10:24.090299 1530 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec: Snoozing failure detection for 0.018s (starting election) I0312 20:10:24.090528 1530 raft_consensus.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:24.091630 1530 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 30 pre-election: Requesting pre-vote from peer 6885de38238442d6a9e5f1eb9fb569ff I0312 20:10:24.092021 1530 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 30 pre-election: Requesting pre-vote from peer ded9b7f8677940fb91e1a8c8bceceaed I0312 20:10:24.094028 1220 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "f78a2351568c4a268ee1112681e22bec" candidate_term: 30 candidate_status { last_received { term: 29 index: 871 } } ignore_live_leader: false dest_uuid: "6885de38238442d6a9e5f1eb9fb569ff" is_pre_election: true I0312 20:10:24.094542 1220 raft_consensus.cc:2176] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 29 LEADER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 30 because replica is either leader or believes a valid leader to be alive. I0312 20:10:24.095459 1279 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 30 pre-election: Vote denied by peer 6885de38238442d6a9e5f1eb9fb569ff. Message: Invalid argument: T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 29 LEADER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 30 because replica is either leader or believes a valid leader to be alive. I0312 20:10:24.095937 1096 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "f78a2351568c4a268ee1112681e22bec" candidate_term: 30 candidate_status { last_received { term: 29 index: 871 } } ignore_live_leader: false dest_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" is_pre_election: true I0312 20:10:24.096477 1096 raft_consensus.cc:2176] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 30 because replica is either leader or believes a valid leader to be alive. I0312 20:10:24.097358 1278 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 30 pre-election: Vote denied by peer ded9b7f8677940fb91e1a8c8bceceaed. Message: Invalid argument: T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 30 because replica is either leader or believes a valid leader to be alive. I0312 20:10:24.097579 1278 leader_election.cc:258] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. I0312 20:10:24.098213 1530 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec: Snoozing failure detection for 0.021s (election complete) I0312 20:10:24.098410 1530 raft_consensus.cc:2455] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority I0312 20:10:24.350478 1530 raft_consensus.cc:436] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 6885de38238442d6a9e5f1eb9fb569ff) I0312 20:10:24.350742 1530 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec: Snoozing failure detection for 0.031s (starting election) I0312 20:10:24.350960 1530 raft_consensus.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:24.351959 1530 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 30 pre-election: Requesting pre-vote from peer 6885de38238442d6a9e5f1eb9fb569ff I0312 20:10:24.352325 1530 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 30 pre-election: Requesting pre-vote from peer ded9b7f8677940fb91e1a8c8bceceaed I0312 20:10:24.353224 1220 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "f78a2351568c4a268ee1112681e22bec" candidate_term: 30 candidate_status { last_received { term: 29 index: 915 } } ignore_live_leader: false dest_uuid: "6885de38238442d6a9e5f1eb9fb569ff" is_pre_election: true I0312 20:10:24.353735 1220 raft_consensus.cc:2176] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 29 LEADER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 30 because replica is either leader or believes a valid leader to be alive. I0312 20:10:24.353543 1093 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "f78a2351568c4a268ee1112681e22bec" candidate_term: 30 candidate_status { last_received { term: 29 index: 915 } } ignore_live_leader: false dest_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" is_pre_election: true I0312 20:10:24.354609 1093 raft_consensus.cc:2190] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 30 because replica is already servicing an update from a current leader or another vote. I0312 20:10:24.354727 1279 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 30 pre-election: Vote denied by peer 6885de38238442d6a9e5f1eb9fb569ff. Message: Invalid argument: T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 29 LEADER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 30 because replica is either leader or believes a valid leader to be alive. I0312 20:10:24.355437 1278 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 30 pre-election: Vote denied by peer ded9b7f8677940fb91e1a8c8bceceaed. Message: Service unavailable: T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 30 because replica is already servicing an update from a current leader or another vote. I0312 20:10:24.355666 1278 leader_election.cc:258] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. I0312 20:10:24.365264 1530 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec: Snoozing failure detection for 0.015s (election complete) I0312 20:10:24.365481 1530 raft_consensus.cc:2455] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority I0312 20:10:24.722302 1530 raft_consensus.cc:436] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 6885de38238442d6a9e5f1eb9fb569ff) I0312 20:10:24.722564 1530 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec: Snoozing failure detection for 0.049s (starting election) I0312 20:10:24.722792 1530 raft_consensus.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:24.723791 1530 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 30 pre-election: Requesting pre-vote from peer 6885de38238442d6a9e5f1eb9fb569ff I0312 20:10:24.724169 1530 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 30 pre-election: Requesting pre-vote from peer ded9b7f8677940fb91e1a8c8bceceaed I0312 20:10:24.724969 1220 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "f78a2351568c4a268ee1112681e22bec" candidate_term: 30 candidate_status { last_received { term: 29 index: 975 } } ignore_live_leader: false dest_uuid: "6885de38238442d6a9e5f1eb9fb569ff" is_pre_election: true I0312 20:10:24.725489 1220 raft_consensus.cc:2176] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 29 LEADER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 30 because replica is either leader or believes a valid leader to be alive. I0312 20:10:24.725711 1094 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "f78a2351568c4a268ee1112681e22bec" candidate_term: 30 candidate_status { last_received { term: 29 index: 975 } } ignore_live_leader: false dest_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" is_pre_election: true I0312 20:10:24.726229 1094 raft_consensus.cc:2176] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 30 because replica is either leader or believes a valid leader to be alive. I0312 20:10:24.726397 1279 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 30 pre-election: Vote denied by peer 6885de38238442d6a9e5f1eb9fb569ff. Message: Invalid argument: T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 29 LEADER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 30 because replica is either leader or believes a valid leader to be alive. I0312 20:10:24.727352 1278 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 30 pre-election: Vote denied by peer ded9b7f8677940fb91e1a8c8bceceaed. Message: Invalid argument: T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 30 because replica is either leader or believes a valid leader to be alive. I0312 20:10:24.727586 1278 leader_election.cc:258] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. I0312 20:10:24.728998 1530 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec: Snoozing failure detection for 0.027s (election complete) I0312 20:10:24.729233 1530 raft_consensus.cc:2455] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority I0312 20:10:24.875350 1546 raft_consensus.cc:436] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 6885de38238442d6a9e5f1eb9fb569ff) I0312 20:10:24.875676 1546 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed: Snoozing failure detection for 0.019s (starting election) I0312 20:10:24.875876 1546 raft_consensus.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:24.876654 1546 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 30 pre-election: Requesting pre-vote from peer f78a2351568c4a268ee1112681e22bec I0312 20:10:24.876935 1546 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 30 pre-election: Requesting pre-vote from peer 6885de38238442d6a9e5f1eb9fb569ff I0312 20:10:24.877899 1340 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" candidate_term: 30 candidate_status { last_received { term: 29 index: 1000 } } ignore_live_leader: false dest_uuid: "f78a2351568c4a268ee1112681e22bec" is_pre_election: true I0312 20:10:24.878422 1340 raft_consensus.cc:2176] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ded9b7f8677940fb91e1a8c8bceceaed for term 30 because replica is either leader or believes a valid leader to be alive. I0312 20:10:24.879623 1027 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 30 pre-election: Vote denied by peer f78a2351568c4a268ee1112681e22bec. Message: Invalid argument: T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ded9b7f8677940fb91e1a8c8bceceaed for term 30 because replica is either leader or believes a valid leader to be alive. I0312 20:10:24.891090 1220 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" candidate_term: 30 candidate_status { last_received { term: 29 index: 1000 } } ignore_live_leader: false dest_uuid: "6885de38238442d6a9e5f1eb9fb569ff" is_pre_election: true I0312 20:10:24.891587 1220 raft_consensus.cc:2190] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 29 LEADER]: Leader pre-election vote request: Denying vote to candidate ded9b7f8677940fb91e1a8c8bceceaed for term 30 because replica is already servicing an update from a current leader or another vote. I0312 20:10:24.892864 1029 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 30 pre-election: Vote denied by peer 6885de38238442d6a9e5f1eb9fb569ff. Message: Service unavailable: T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 29 LEADER]: Leader pre-election vote request: Denying vote to candidate ded9b7f8677940fb91e1a8c8bceceaed for term 30 because replica is already servicing an update from a current leader or another vote. I0312 20:10:24.893103 1029 leader_election.cc:258] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. I0312 20:10:24.895867 1546 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed: Snoozing failure detection for 0.029s (election complete) I0312 20:10:24.896065 1546 raft_consensus.cc:2455] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority Connected to the Master Fetched info from all 3 Tablet Servers Table test-workload is HEALTHY (1 tablet(s) checked) Table Summary Name | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable ---------------+---------+---------------+---------+------------+------------------+------------- test-workload | HEALTHY | 1 | 1 | 0 | 0 | 0 The metadata for 1 table(s) is HEALTHY I0312 20:10:25.383020 1546 raft_consensus.cc:436] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 6885de38238442d6a9e5f1eb9fb569ff) I0312 20:10:25.383265 1546 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed: Snoozing failure detection for 0.044s (starting election) I0312 20:10:25.383473 1546 raft_consensus.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:25.384507 1546 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 30 pre-election: Requesting pre-vote from peer f78a2351568c4a268ee1112681e22bec I0312 20:10:25.384863 1546 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 30 pre-election: Requesting pre-vote from peer 6885de38238442d6a9e5f1eb9fb569ff I0312 20:10:25.385980 1338 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" candidate_term: 30 candidate_status { last_received { term: 29 index: 1018 } } ignore_live_leader: false dest_uuid: "f78a2351568c4a268ee1112681e22bec" is_pre_election: true I0312 20:10:25.386499 1338 raft_consensus.cc:2176] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ded9b7f8677940fb91e1a8c8bceceaed for term 30 because replica is either leader or believes a valid leader to be alive. I0312 20:10:25.386432 1220 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" candidate_term: 30 candidate_status { last_received { term: 29 index: 1018 } } ignore_live_leader: false dest_uuid: "6885de38238442d6a9e5f1eb9fb569ff" is_pre_election: true I0312 20:10:25.386936 1220 raft_consensus.cc:2176] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 29 LEADER]: Leader pre-election vote request: Denying vote to candidate ded9b7f8677940fb91e1a8c8bceceaed for term 30 because replica is either leader or believes a valid leader to be alive. I0312 20:10:25.387588 1027 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 30 pre-election: Vote denied by peer f78a2351568c4a268ee1112681e22bec. Message: Invalid argument: T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ded9b7f8677940fb91e1a8c8bceceaed for term 30 because replica is either leader or believes a valid leader to be alive. I0312 20:10:25.390383 1029 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 30 pre-election: Vote denied by peer 6885de38238442d6a9e5f1eb9fb569ff. Message: Invalid argument: T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 29 LEADER]: Leader pre-election vote request: Denying vote to candidate ded9b7f8677940fb91e1a8c8bceceaed for term 30 because replica is either leader or believes a valid leader to be alive. I0312 20:10:25.390611 1029 leader_election.cc:258] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. I0312 20:10:25.391419 1546 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed: Snoozing failure detection for 0.048s (election complete) I0312 20:10:25.391620 1546 raft_consensus.cc:2455] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority Checksum finished in 0s: 0/3 replicas remaining (0B from disk, 3.01k rows summed) ----------------------- test-workload ----------------------- T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed (127.0.53.1:35037): Checksum: 2185295419668 T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec (127.0.53.3:43274): Checksum: 2185295419668 T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff (127.0.53.2:51697): Checksum: 2185295419668 I0312 20:10:25.686372 212 log_verifier.cc:121] Checking tablet 92318a1b744e4baa9c7d3f3b9758739b I0312 20:10:26.041658 1572 raft_consensus.cc:436] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 6885de38238442d6a9e5f1eb9fb569ff) I0312 20:10:26.042047 1572 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec: Snoozing failure detection for 0.019s (starting election) I0312 20:10:26.042356 1572 raft_consensus.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:26.043467 1572 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 30 pre-election: Requesting pre-vote from peer 6885de38238442d6a9e5f1eb9fb569ff I0312 20:10:26.044826 1220 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "f78a2351568c4a268ee1112681e22bec" candidate_term: 30 candidate_status { last_received { term: 29 index: 1018 } } ignore_live_leader: false dest_uuid: "6885de38238442d6a9e5f1eb9fb569ff" is_pre_election: true I0312 20:10:26.045399 1220 raft_consensus.cc:2176] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 29 LEADER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 30 because replica is either leader or believes a valid leader to be alive. I0312 20:10:26.046252 1573 raft_consensus.cc:436] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 6885de38238442d6a9e5f1eb9fb569ff) I0312 20:10:26.046409 1279 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 30 pre-election: Vote denied by peer 6885de38238442d6a9e5f1eb9fb569ff. Message: Invalid argument: T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 29 LEADER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 30 because replica is either leader or believes a valid leader to be alive. I0312 20:10:26.046542 1573 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed: Snoozing failure detection for 0.067s (starting election) I0312 20:10:26.046758 1572 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 30 pre-election: Requesting pre-vote from peer ded9b7f8677940fb91e1a8c8bceceaed I0312 20:10:26.046775 1573 raft_consensus.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:26.048032 1094 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "f78a2351568c4a268ee1112681e22bec" candidate_term: 30 candidate_status { last_received { term: 29 index: 1018 } } ignore_live_leader: false dest_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" is_pre_election: true I0312 20:10:26.057622 1573 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 30 pre-election: Requesting pre-vote from peer f78a2351568c4a268ee1112681e22bec I0312 20:10:26.057966 1573 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 30 pre-election: Requesting pre-vote from peer 6885de38238442d6a9e5f1eb9fb569ff I0312 20:10:26.058861 1094 raft_consensus.cc:2190] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 30 because replica is already servicing an update from a current leader or another vote. I0312 20:10:26.060194 1278 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 30 pre-election: Vote denied by peer ded9b7f8677940fb91e1a8c8bceceaed. Message: Service unavailable: T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f78a2351568c4a268ee1112681e22bec for term 30 because replica is already servicing an update from a current leader or another vote. I0312 20:10:26.060420 1278 leader_election.cc:258] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. I0312 20:10:26.061542 1340 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" candidate_term: 30 candidate_status { last_received { term: 29 index: 1018 } } ignore_live_leader: false dest_uuid: "f78a2351568c4a268ee1112681e22bec" is_pre_election: true I0312 20:10:26.062085 1340 raft_consensus.cc:2176] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ded9b7f8677940fb91e1a8c8bceceaed for term 30 because replica is either leader or believes a valid leader to be alive. I0312 20:10:26.063457 1027 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 30 pre-election: Vote denied by peer f78a2351568c4a268ee1112681e22bec. Message: Invalid argument: T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ded9b7f8677940fb91e1a8c8bceceaed for term 30 because replica is either leader or believes a valid leader to be alive. I0312 20:10:26.063905 1220 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "92318a1b744e4baa9c7d3f3b9758739b" candidate_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" candidate_term: 30 candidate_status { last_received { term: 29 index: 1018 } } ignore_live_leader: false dest_uuid: "6885de38238442d6a9e5f1eb9fb569ff" is_pre_election: true I0312 20:10:26.064440 1220 raft_consensus.cc:2176] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 29 LEADER]: Leader pre-election vote request: Denying vote to candidate ded9b7f8677940fb91e1a8c8bceceaed for term 30 because replica is either leader or believes a valid leader to be alive. I0312 20:10:26.065407 1029 leader_election.cc:377] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 30 pre-election: Vote denied by peer 6885de38238442d6a9e5f1eb9fb569ff. Message: Invalid argument: T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff [term 29 LEADER]: Leader pre-election vote request: Denying vote to candidate ded9b7f8677940fb91e1a8c8bceceaed for term 30 because replica is either leader or believes a valid leader to be alive. I0312 20:10:26.065629 1029 leader_election.cc:258] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. I0312 20:10:26.066251 1573 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed: Snoozing failure detection for 0.065s (election complete) I0312 20:10:26.066464 1573 raft_consensus.cc:2455] T 92318a1b744e4baa9c7d3f3b9758739b P ded9b7f8677940fb91e1a8c8bceceaed [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority I0312 20:10:26.068702 1572 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec: Snoozing failure detection for 0.075s (election complete) I0312 20:10:26.068928 1572 raft_consensus.cc:2455] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority I0312 20:10:26.774121 212 log_verifier.cc:172] Verified matching terms for 1018 ops in tablet 92318a1b744e4baa9c7d3f3b9758739b I0312 20:10:26.822338 212 external_mini_cluster.cc:997] Killing /tmp/dist-test-task3bwPeE/build/tsan/bin/kudu-tserver with pid 1008 W0312 20:10:26.868767 1153 connection.cc:513] client connection to 127.0.53.1:35037 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0312 20:10:26.868808 956 connection.cc:513] client connection to 127.0.53.1:35037 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0312 20:10:26.869024 1153 consensus_peers.cc:436] T 92318a1b744e4baa9c7d3f3b9758739b P 6885de38238442d6a9e5f1eb9fb569ff -> Peer ded9b7f8677940fb91e1a8c8bceceaed (127.0.53.1:35037): Couldn't send request to peer ded9b7f8677940fb91e1a8c8bceceaed for tablet 92318a1b744e4baa9c7d3f3b9758739b. Status: Network error: failed to read from TLS socket: Connection reset by peer (error 104). Retrying in the next heartbeat period. Already tried 1 times. W0312 20:10:26.869210 1278 connection.cc:513] client connection to 127.0.53.1:35037 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) I0312 20:10:26.870548 212 external_mini_cluster.cc:997] Killing /tmp/dist-test-task3bwPeE/build/tsan/bin/kudu-tserver with pid 1143 W0312 20:10:26.870839 959 connection.cc:513] server connection from 127.0.53.1:39786 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) I0312 20:10:26.885673 1583 raft_consensus.cc:436] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 6885de38238442d6a9e5f1eb9fb569ff) I0312 20:10:26.885990 1583 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec: Snoozing failure detection for 0.071s (starting election) I0312 20:10:26.886266 1583 raft_consensus.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:26.887440 1583 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 30 pre-election: Requesting pre-vote from peer 6885de38238442d6a9e5f1eb9fb569ff I0312 20:10:26.888299 1583 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 30 pre-election: Requesting pre-vote from peer ded9b7f8677940fb91e1a8c8bceceaed W0312 20:10:26.889320 1277 connection.cc:513] server connection from 127.0.53.1:47760 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0312 20:10:26.890846 1278 leader_election.cc:287] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 30 pre-election: RPC error from VoteRequest() call to peer ded9b7f8677940fb91e1a8c8bceceaed: Network error: Client connection negotiation failed: client connection to 127.0.53.1:35037: connect: Connection refused (error 111) I0312 20:10:26.902688 1583 raft_consensus.cc:436] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 6885de38238442d6a9e5f1eb9fb569ff) I0312 20:10:26.902994 1583 raft_consensus.cc:2700] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec: Snoozing failure detection for 0.088s (starting election) I0312 20:10:26.903282 1583 raft_consensus.cc:458] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f78a2351568c4a268ee1112681e22bec" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 43274 } } peers { permanent_uuid: "6885de38238442d6a9e5f1eb9fb569ff" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 51697 } } peers { permanent_uuid: "ded9b7f8677940fb91e1a8c8bceceaed" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 35037 } } I0312 20:10:26.904378 1583 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 30 pre-election: Requesting pre-vote from peer 6885de38238442d6a9e5f1eb9fb569ff I0312 20:10:26.905184 1583 leader_election.cc:231] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 30 pre-election: Requesting pre-vote from peer ded9b7f8677940fb91e1a8c8bceceaed W0312 20:10:26.906471 1278 leader_election.cc:287] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 30 pre-election: RPC error from VoteRequest() call to peer ded9b7f8677940fb91e1a8c8bceceaed: Network error: Client connection negotiation failed: client connection to 127.0.53.1:35037: connect: Connection refused (error 111) W0312 20:10:26.929247 957 connection.cc:513] client connection to 127.0.53.2:51697 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0312 20:10:26.930372 957 connection.cc:513] server connection from 127.0.53.2:43315 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0312 20:10:26.931008 1279 connection.cc:513] client connection to 127.0.53.2:51697 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0312 20:10:26.931293 1279 leader_election.cc:287] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 30 pre-election: RPC error from VoteRequest() call to peer 6885de38238442d6a9e5f1eb9fb569ff: Network error: failed to read from TLS socket: Connection reset by peer (error 104) I0312 20:10:26.931505 1279 leader_election.cc:258] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. W0312 20:10:26.932147 1279 leader_election.cc:287] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 30 pre-election: RPC error from VoteRequest() call to peer 6885de38238442d6a9e5f1eb9fb569ff: Network error: failed to read from TLS socket: Connection reset by peer (error 104) I0312 20:10:26.932359 1279 leader_election.cc:258] T 92318a1b744e4baa9c7d3f3b9758739b P f78a2351568c4a268ee1112681e22bec [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. W0312 20:10:26.933250 1281 connection.cc:513] server connection from 127.0.53.2:37068 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) I0312 20:10:26.937258 212 external_mini_cluster.cc:997] Killing /tmp/dist-test-task3bwPeE/build/tsan/bin/kudu-tserver with pid 1268 W0312 20:10:26.980691 956 connection.cc:513] server connection from 127.0.53.3:40529 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0312 20:10:26.980762 955 connection.cc:513] client connection to 127.0.53.3:43274 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) I0312 20:10:26.981988 212 external_mini_cluster.cc:997] Killing /tmp/dist-test-task3bwPeE/build/tsan/bin/kudu-master with pid 946 W0312 20:10:27.004690 945 connection.cc:513] client connection to 127.0.53.62:57865 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) [ OK ] RaftConsensusElectionITest.ChurnyElections (19096 ms) [ RUN ] RaftConsensusElectionITest.ChurnyElections_WithNotificationLatency I0312 20:10:27.019536 212 test_util.cc:192] Using random seed: -1147199753 I0312 20:10:27.022351 212 ts_itest-base.cc:113] Starting cluster with: I0312 20:10:27.022451 212 ts_itest-base.cc:114] -------------- I0312 20:10:27.022541 212 ts_itest-base.cc:115] 3 tablet servers I0312 20:10:27.022624 212 ts_itest-base.cc:116] 3 replicas per TS I0312 20:10:27.022706 212 ts_itest-base.cc:117] -------------- I0312 20:10:27.024713 212 external_mini_cluster.cc:805] Running /tmp/dist-test-task3bwPeE/build/tsan/bin/kudu-master /tmp/dist-test-task3bwPeE/build/tsan/bin/kudu-master --fs_wal_dir=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ChurnyElections_WithNotificationLatency.1520885401735786-212/raft_consensus-itest-cluster/master-0/wal --fs_data_dirs=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ChurnyElections_WithNotificationLatency.1520885401735786-212/raft_consensus-itest-cluster/master-0/data --block_manager=log --webserver_interface=localhost --ipki_ca_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.0.53.62:0 --webserver_interface=127.0.53.62 --webserver_port=0 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ChurnyElections_WithNotificationLatency.1520885401735786-212/raft_consensus-itest-cluster/master-0/logs --server_dump_info_path=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ChurnyElections_WithNotificationLatency.1520885401735786-212/raft_consensus-itest-cluster/master-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W0312 20:10:27.147408 1591 flags.cc:412] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0312 20:10:27.147733 1591 flags.cc:412] Enabled unsafe flag: --never_fsync=true W0312 20:10:27.163089 1591 flags.cc:412] Enabled experimental flag: --ipki_ca_key_size=1024 W0312 20:10:27.163224 1591 flags.cc:412] Enabled experimental flag: --ipki_server_key_size=1024 W0312 20:10:27.163336 1591 flags.cc:412] Enabled experimental flag: --tsk_num_rsa_bits=512 I0312 20:10:27.183596 1591 master_main.cc:66] Master server non-default flags: --fs_data_dirs=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ChurnyElections_WithNotificationLatency.1520885401735786-212/raft_consensus-itest-cluster/master-0/data --fs_wal_dir=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ChurnyElections_WithNotificationLatency.1520885401735786-212/raft_consensus-itest-cluster/master-0/wal --ipki_ca_key_size=1024 --ipki_server_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.0.53.62:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ChurnyElections_WithNotificationLatency.1520885401735786-212/raft_consensus-itest-cluster/master-0/data/info.pb --webserver_interface=127.0.53.62 --webserver_port=0 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --log_dir=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ChurnyElections_WithNotificationLatency.1520885401735786-212/raft_consensus-itest-cluster/master-0/logs --logbuflevel=-1 --logtostderr=true Master server version: kudu 1.7.0-SNAPSHOT revision 13e10c63c46de30ac85f34b543cdd6fb44881f61 build type FASTDEBUG built by None at 12 Mar 2018 20:06:40 UTC on jenkins-slave-vn0g build id 12388 TSAN enabled I0312 20:10:27.193240 1591 master_main.cc:73] Initializing master server... I0312 20:10:27.194795 1591 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 1080776us I0312 20:10:27.195549 1591 fs_manager.cc:260] Metadata directory not provided I0312 20:10:27.195704 1591 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0312 20:10:27.195935 1591 server_base.cc:434] Could not load existing FS layout: Not found: /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ChurnyElections_WithNotificationLatency.1520885401735786-212/raft_consensus-itest-cluster/master-0/data/instance: No such file or directory (error 2) I0312 20:10:27.196085 1591 server_base.cc:435] Attempting to create new FS layout instead I0312 20:10:27.203543 1591 fs_manager.cc:596] Generated new instance metadata in path /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ChurnyElections_WithNotificationLatency.1520885401735786-212/raft_consensus-itest-cluster/master-0/data/instance: uuid: "2fa7fa54299047a7b5aadb36fe0936ae" format_stamp: "Formatted at 2018-03-12 20:10:27 on dist-test-slave-dist-test-slave-f27p" I0312 20:10:27.204210 1591 fs_manager.cc:596] Generated new instance metadata in path /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ChurnyElections_WithNotificationLatency.1520885401735786-212/raft_consensus-itest-cluster/master-0/wal/instance: uuid: "2fa7fa54299047a7b5aadb36fe0936ae" format_stamp: "Formatted at 2018-03-12 20:10:27 on dist-test-slave-dist-test-slave-f27p" I0312 20:10:27.212527 1591 fs_manager.cc:496] Time spent creating directory manager: real 0.008s user 0.004s sys 0.008s I0312 20:10:27.213701 1591 env_posix.cc:1620] Not raising this process' open files per process limit of 65536; it is already as high as it can go I0312 20:10:27.213963 1591 file_cache.cc:470] Constructed file cache lbm with capacity 26214 I0312 20:10:27.220484 1591 fs_manager.cc:418] Time spent opening block manager: real 0.002s user 0.000s sys 0.004s I0312 20:10:27.220593 1591 fs_manager.cc:429] Opened local filesystem: /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ChurnyElections_WithNotificationLatency.1520885401735786-212/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ChurnyElections_WithNotificationLatency.1520885401735786-212/raft_consensus-itest-cluster/master-0/wal uuid: "2fa7fa54299047a7b5aadb36fe0936ae" format_stamp: "Formatted at 2018-03-12 20:10:27 on dist-test-slave-dist-test-slave-f27p" I0312 20:10:27.220789 1591 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ChurnyElections_WithNotificationLatency.1520885401735786-212/raft_consensus-itest-cluster/master-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0312 20:10:27.391805 1591 env_posix.cc:1620] Not raising this process' running threads per effective uid limit of 60284; it is already as high as it can go I0312 20:10:27.392417 1591 master_main.cc:76] Starting Master server... I0312 20:10:27.393649 1608 process_memory.cc:182] Process hard memory limit is 11.777631 GB I0312 20:10:27.393791 1608 process_memory.cc:184] Process soft memory limit is 9.422105 GB I0312 20:10:27.393890 1608 process_memory.cc:187] Process memory pressure threshold is 7.066578 GB I0312 20:10:27.433392 1591 rpc_server.cc:201] RPC server started. Bound to: 127.0.53.62:44334 I0312 20:10:27.459005 1591 webserver.cc:173] Starting webserver on 127.0.53.62:0 I0312 20:10:27.459112 1591 webserver.cc:184] Document root disabled I0312 20:10:27.460346 1591 webserver.cc:311] Webserver started. Bound to: http://127.0.53.62:58895/ I0312 20:10:27.461100 1591 server_base.cc:607] Dumped server information to /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ChurnyElections_WithNotificationLatency.1520885401735786-212/raft_consensus-itest-cluster/master-0/data/info.pb I0312 20:10:27.463775 1652 data_dirs.cc:931] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0312 20:10:27.468503 212 external_mini_cluster.cc:867] Started /tmp/dist-test-task3bwPeE/build/tsan/bin/kudu-master as pid 1591 I0312 20:10:27.469890 212 external_mini_cluster.cc:805] Running /tmp/dist-test-task3bwPeE/build/tsan/bin/kudu-tserver /tmp/dist-test-task3bwPeE/build/tsan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ChurnyElections_WithNotificationLatency.1520885401735786-212/raft_consensus-itest-cluster/ts-0/wal --fs_data_dirs=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ChurnyElections_WithNotificationLatency.1520885401735786-212/raft_consensus-itest-cluster/ts-0/data --block_manager=log --rpc_bind_addresses=127.0.53.1:0 --local_ip_for_outbound_sockets=127.0.53.1 --webserver_interface=127.0.53.1 --webserver_port=0 --tserver_master_addrs=127.0.53.62:44334 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ChurnyElections_WithNotificationLatency.1520885401735786-212/raft_consensus-itest-cluster/ts-0/logs --server_dump_info_path=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ChurnyElections_WithNotificationLatency.1520885401735786-212/raft_consensus-itest-cluster/ts-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_heartbeat_interval_ms=5 --consensus_inject_latency_ms_in_notifications=50 with env {} I0312 20:10:27.480166 1652 tablet_bootstrap.cc:436] T 00000000000000000000000000000000 P 2fa7fa54299047a7b5aadb36fe0936ae: Bootstrap starting. I0312 20:10:27.482527 1652 tablet_bootstrap.cc:581] T 00000000000000000000000000000000 P 2fa7fa54299047a7b5aadb36fe0936ae: No blocks or log segments found. Creating new log. I0312 20:10:27.483700 1652 log.cc:520] T 00000000000000000000000000000000 P 2fa7fa54299047a7b5aadb36fe0936ae: Log is configured to *not* fsync() on all Append() calls WARNING: Logging before InitGoogleLogging() is written to STDERR W0312 20:10:27.587111 1653 flags.cc:412] Enabled unsafe flag: --consensus_inject_latency_ms_in_notifications=50 W0312 20:10:27.587476 1653 flags.cc:412] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0312 20:10:27.587662 1653 flags.cc:412] Enabled unsafe flag: --never_fsync=true W0312 20:10:27.601759 1653 flags.cc:412] Enabled experimental flag: --ipki_server_key_size=1024 W0312 20:10:27.602063 1653 flags.cc:412] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.53.1 I0312 20:10:27.621031 1653 tablet_server_main.cc:72] Tablet server non-default flags: --consensus_inject_latency_ms_in_notifications=50 --raft_heartbeat_interval_ms=5 --fs_data_dirs=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ChurnyElections_WithNotificationLatency.1520885401735786-212/raft_consensus-itest-cluster/ts-0/data --fs_wal_dir=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ChurnyElections_WithNotificationLatency.1520885401735786-212/raft_consensus-itest-cluster/ts-0/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.0.53.1:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ChurnyElections_WithNotificationLatency.1520885401735786-212/raft_consensus-itest-cluster/ts-0/data/info.pb --webserver_interface=127.0.53.1 --webserver_port=0 --tserver_master_addrs=127.0.53.62:44334 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.0.53.1 --log_dir=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ChurnyElections_WithNotificationLatency.1520885401735786-212/raft_consensus-itest-cluster/ts-0/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.7.0-SNAPSHOT revision 13e10c63c46de30ac85f34b543cdd6fb44881f61 build type FASTDEBUG built by None at 12 Mar 2018 20:06:40 UTC on jenkins-slave-vn0g build id 12388 TSAN enabled I0312 20:10:27.630964 1653 tablet_server_main.cc:79] Initializing tablet server... I0312 20:10:27.632589 1653 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 1080776us I0312 20:10:27.633018 1653 fs_manager.cc:260] Metadata directory not provided I0312 20:10:27.633146 1653 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0312 20:10:27.633332 1653 server_base.cc:434] Could not load existing FS layout: Not found: /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ChurnyElections_WithNotificationLatency.1520885401735786-212/raft_consensus-itest-cluster/ts-0/data/instance: No such file or directory (error 2) I0312 20:10:27.633437 1653 server_base.cc:435] Attempting to create new FS layout instead I0312 20:10:27.640543 1653 fs_manager.cc:596] Generated new instance metadata in path /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ChurnyElections_WithNotificationLatency.1520885401735786-212/raft_consensus-itest-cluster/ts-0/data/instance: uuid: "9da22e5afdce42078fe8f94ad166ac3c" format_stamp: "Formatted at 2018-03-12 20:10:27 on dist-test-slave-dist-test-slave-f27p" I0312 20:10:27.641233 1653 fs_manager.cc:596] Generated new instance metadata in path /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ChurnyElections_WithNotificationLatency.1520885401735786-212/raft_consensus-itest-cluster/ts-0/wal/instance: uuid: "9da22e5afdce42078fe8f94ad166ac3c" format_stamp: "Formatted at 2018-03-12 20:10:27 on dist-test-slave-dist-test-slave-f27p" I0312 20:10:27.649286 1653 fs_manager.cc:496] Time spent creating directory manager: real 0.008s user 0.004s sys 0.004s I0312 20:10:27.650405 1653 env_posix.cc:1620] Not raising this process' open files per process limit of 65536; it is already as high as it can go I0312 20:10:27.650658 1653 file_cache.cc:470] Constructed file cache lbm with capacity 26214 I0312 20:10:27.656920 1653 fs_manager.cc:418] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s I0312 20:10:27.657034 1653 fs_manager.cc:429] Opened local filesystem: /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ChurnyElections_WithNotificationLatency.1520885401735786-212/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ChurnyElections_WithNotificationLatency.1520885401735786-212/raft_consensus-itest-cluster/ts-0/wal uuid: "9da22e5afdce42078fe8f94ad166ac3c" format_stamp: "Formatted at 2018-03-12 20:10:27 on dist-test-slave-dist-test-slave-f27p" I0312 20:10:27.657290 1653 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ChurnyElections_WithNotificationLatency.1520885401735786-212/raft_consensus-itest-cluster/ts-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0312 20:10:27.658066 1652 tablet_bootstrap.cc:436] T 00000000000000000000000000000000 P 2fa7fa54299047a7b5aadb36fe0936ae: No bootstrap required, opened a new log I0312 20:10:27.670745 1652 raft_consensus.cc:304] T 00000000000000000000000000000000 P 2fa7fa54299047a7b5aadb36fe0936ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fa7fa54299047a7b5aadb36fe0936ae" member_type: VOTER } I0312 20:10:27.671083 1652 raft_consensus.cc:330] T 00000000000000000000000000000000 P 2fa7fa54299047a7b5aadb36fe0936ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0312 20:10:27.671229 1652 raft_consensus.cc:605] T 00000000000000000000000000000000 P 2fa7fa54299047a7b5aadb36fe0936ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2fa7fa54299047a7b5aadb36fe0936ae, State: Initialized, Role: FOLLOWER I0312 20:10:27.671761 1652 consensus_queue.cc:219] T 00000000000000000000000000000000 P 2fa7fa54299047a7b5aadb36fe0936ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fa7fa54299047a7b5aadb36fe0936ae" member_type: VOTER } I0312 20:10:27.672124 1652 raft_consensus.cc:344] T 00000000000000000000000000000000 P 2fa7fa54299047a7b5aadb36fe0936ae [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0312 20:10:27.672293 1652 raft_consensus.cc:436] T 00000000000000000000000000000000 P 2fa7fa54299047a7b5aadb36fe0936ae [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0312 20:10:27.672451 1652 raft_consensus.cc:2700] T 00000000000000000000000000000000 P 2fa7fa54299047a7b5aadb36fe0936ae: Snoozing failure detection for 1.730s (starting election) I0312 20:10:27.672611 1652 raft_consensus.cc:2752] T 00000000000000000000000000000000 P 2fa7fa54299047a7b5aadb36fe0936ae [term 0 FOLLOWER]: Advancing to term 1 I0312 20:10:27.675271 1652 raft_consensus.cc:458] T 00000000000000000000000000000000 P 2fa7fa54299047a7b5aadb36fe0936ae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fa7fa54299047a7b5aadb36fe0936ae" member_type: VOTER } I0312 20:10:27.675725 1652 leader_election.cc:258] T 00000000000000000000000000000000 P 2fa7fa54299047a7b5aadb36fe0936ae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0312 20:10:27.682826 1666 raft_consensus.cc:2700] T 00000000000000000000000000000000 P 2fa7fa54299047a7b5aadb36fe0936ae: Snoozing failure detection for 1.920s (election complete) I0312 20:10:27.683661 1666 raft_consensus.cc:2507] T 00000000000000000000000000000000 P 2fa7fa54299047a7b5aadb36fe0936ae [term 1 FOLLOWER]: Leader election won for term 1 I0312 20:10:27.687788 1666 raft_consensus.cc:572] T 00000000000000000000000000000000 P 2fa7fa54299047a7b5aadb36fe0936ae [term 1 LEADER]: Becoming Leader. State: Replica: 2fa7fa54299047a7b5aadb36fe0936ae, State: Running, Role: LEADER I0312 20:10:27.688158 1666 consensus_queue.cc:196] T 00000000000000000000000000000000 P 2fa7fa54299047a7b5aadb36fe0936ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fa7fa54299047a7b5aadb36fe0936ae" member_type: VOTER } I0312 20:10:27.694917 1669 sys_catalog.cc:335] T 00000000000000000000000000000000 P 2fa7fa54299047a7b5aadb36fe0936ae [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2fa7fa54299047a7b5aadb36fe0936ae. Latest consensus state: current_term: 1 leader_uuid: "2fa7fa54299047a7b5aadb36fe0936ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fa7fa54299047a7b5aadb36fe0936ae" member_type: VOTER } } I0312 20:10:27.695050 1652 sys_catalog.cc:335] T 00000000000000000000000000000000 P 2fa7fa54299047a7b5aadb36fe0936ae [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 1 leader_uuid: "2fa7fa54299047a7b5aadb36fe0936ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fa7fa54299047a7b5aadb36fe0936ae" member_type: VOTER } } I0312 20:10:27.695384 1669 sys_catalog.cc:338] T 00000000000000000000000000000000 P 2fa7fa54299047a7b5aadb36fe0936ae [sys.catalog]: This master's current role is: LEADER I0312 20:10:27.695525 1652 sys_catalog.cc:338] T 00000000000000000000000000000000 P 2fa7fa54299047a7b5aadb36fe0936ae [sys.catalog]: This master's current role is: LEADER I0312 20:10:27.695870 1668 sys_catalog.cc:335] T 00000000000000000000000000000000 P 2fa7fa54299047a7b5aadb36fe0936ae [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2fa7fa54299047a7b5aadb36fe0936ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fa7fa54299047a7b5aadb36fe0936ae" member_type: VOTER } } I0312 20:10:27.696889 1668 sys_catalog.cc:338] T 00000000000000000000000000000000 P 2fa7fa54299047a7b5aadb36fe0936ae [sys.catalog]: This master's current role is: LEADER I0312 20:10:27.697443 1652 sys_catalog.cc:422] T 00000000000000000000000000000000 P 2fa7fa54299047a7b5aadb36fe0936ae [sys.catalog]: configured and running, proceeding with master startup. I0312 20:10:27.702533 1672 catalog_manager.cc:502] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: 1 I0312 20:10:27.703125 1671 catalog_manager.cc:972] Loading table and tablet metadata into memory... I0312 20:10:27.703200 1591 master_main.cc:79] Master server successfully started. I0312 20:10:27.713368 1671 catalog_manager.cc:983] Initializing Kudu internal certificate authority... I0312 20:10:27.821264 1671 catalog_manager.cc:870] Generated new certificate authority record I0312 20:10:27.822820 1671 catalog_manager.cc:992] Loading token signing keys... I0312 20:10:27.862284 1671 catalog_manager.cc:3868] T 00000000000000000000000000000000 P 2fa7fa54299047a7b5aadb36fe0936ae: Generated new TSK 0 I0312 20:10:27.913863 1653 env_posix.cc:1620] Not raising this process' running threads per effective uid limit of 60284; it is already as high as it can go I0312 20:10:27.915133 1653 ts_tablet_manager.cc:349] Loaded tablet metadata (0 live tablets) I0312 20:10:27.917572 1653 tablet_server_main.cc:84] Starting tablet server... I0312 20:10:28.308464 1653 rpc_server.cc:201] RPC server started. Bound to: 127.0.53.1:39175 I0312 20:10:28.309831 1653 webserver.cc:173] Starting webserver on 127.0.53.1:0 I0312 20:10:28.309986 1653 webserver.cc:184] Document root disabled I0312 20:10:28.311498 1653 webserver.cc:311] Webserver started. Bound to: http://127.0.53.1:46081/ I0312 20:10:28.312446 1653 server_base.cc:607] Dumped server information to /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ChurnyElections_WithNotificationLatency.1520885401735786-212/raft_consensus-itest-cluster/ts-0/data/info.pb I0312 20:10:28.320868 212 external_mini_cluster.cc:867] Started /tmp/dist-test-task3bwPeE/build/tsan/bin/kudu-tserver as pid 1653 I0312 20:10:28.322304 212 external_mini_cluster.cc:805] Running /tmp/dist-test-task3bwPeE/build/tsan/bin/kudu-tserver /tmp/dist-test-task3bwPeE/build/tsan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ChurnyElections_WithNotificationLatency.1520885401735786-212/raft_consensus-itest-cluster/ts-1/wal --fs_data_dirs=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ChurnyElections_WithNotificationLatency.1520885401735786-212/raft_consensus-itest-cluster/ts-1/data --block_manager=log --rpc_bind_addresses=127.0.53.2:0 --local_ip_for_outbound_sockets=127.0.53.2 --webserver_interface=127.0.53.2 --webserver_port=0 --tserver_master_addrs=127.0.53.62:44334 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ChurnyElections_WithNotificationLatency.1520885401735786-212/raft_consensus-itest-cluster/ts-1/logs --server_dump_info_path=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ChurnyElections_WithNotificationLatency.1520885401735786-212/raft_consensus-itest-cluster/ts-1/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_heartbeat_interval_ms=5 --consensus_inject_latency_ms_in_notifications=50 with env {} I0312 20:10:28.329859 1785 process_memory.cc:182] Process hard memory limit is 11.777631 GB I0312 20:10:28.330442 1785 process_memory.cc:184] Process soft memory limit is 9.422105 GB I0312 20:10:28.330595 1785 process_memory.cc:187] Process memory pressure threshold is 7.066578 GB I0312 20:10:28.342222 1784 heartbeater.cc:346] Connected to a master server at 127.0.53.62:44334 I0312 20:10:28.342519 1784 heartbeater.cc:426] Registering TS with master... I0312 20:10:28.343080 1784 heartbeater.cc:475] Master 127.0.53.62:44334 requested a full tablet report, sending... I0312 20:10:28.345161 1619 ts_manager.cc:81] Registered new tserver with Master: 9da22e5afdce42078fe8f94ad166ac3c (127.0.53.1:39175) I0312 20:10:28.347784 1619 master_service.cc:243] Signed X509 certificate for tserver {username='slave'} at 127.0.53.1:50514 I0312 20:10:28.330272 1653 tablet_server_main.cc:87] Tablet server successfully started. WARNING: Logging before InitGoogleLogging() is written to STDERR W0312 20:10:28.515568 1788 flags.cc:412] Enabled unsafe flag: --consensus_inject_latency_ms_in_notifications=50 W0312 20:10:28.515933 1788 flags.cc:412] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0312 20:10:28.516129 1788 flags.cc:412] Enabled unsafe flag: --never_fsync=true W0312 20:10:28.530210 1788 flags.cc:412] Enabled experimental flag: --ipki_server_key_size=1024 W0312 20:10:28.530524 1788 flags.cc:412] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.53.2 I0312 20:10:28.553544 1788 tablet_server_main.cc:72] Tablet server non-default flags: --consensus_inject_latency_ms_in_notifications=50 --raft_heartbeat_interval_ms=5 --fs_data_dirs=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ChurnyElections_WithNotificationLatency.1520885401735786-212/raft_consensus-itest-cluster/ts-1/data --fs_wal_dir=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ChurnyElections_WithNotificationLatency.1520885401735786-212/raft_consensus-itest-cluster/ts-1/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.0.53.2:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ChurnyElections_WithNotificationLatency.1520885401735786-212/raft_consensus-itest-cluster/ts-1/data/info.pb --webserver_interface=127.0.53.2 --webserver_port=0 --tserver_master_addrs=127.0.53.62:44334 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.0.53.2 --log_dir=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ChurnyElections_WithNotificationLatency.1520885401735786-212/raft_consensus-itest-cluster/ts-1/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.7.0-SNAPSHOT revision 13e10c63c46de30ac85f34b543cdd6fb44881f61 build type FASTDEBUG built by None at 12 Mar 2018 20:06:40 UTC on jenkins-slave-vn0g build id 12388 TSAN enabled I0312 20:10:28.577600 1788 tablet_server_main.cc:79] Initializing tablet server... I0312 20:10:28.579424 1788 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 1081276us I0312 20:10:28.579877 1788 fs_manager.cc:260] Metadata directory not provided I0312 20:10:28.580009 1788 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0312 20:10:28.580217 1788 server_base.cc:434] Could not load existing FS layout: Not found: /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ChurnyElections_WithNotificationLatency.1520885401735786-212/raft_consensus-itest-cluster/ts-1/data/instance: No such file or directory (error 2) I0312 20:10:28.580333 1788 server_base.cc:435] Attempting to create new FS layout instead I0312 20:10:28.587621 1788 fs_manager.cc:596] Generated new instance metadata in path /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ChurnyElections_WithNotificationLatency.1520885401735786-212/raft_consensus-itest-cluster/ts-1/data/instance: uuid: "012786a8f1b64995847cf7dacbc029a4" format_stamp: "Formatted at 2018-03-12 20:10:28 on dist-test-slave-dist-test-slave-f27p" I0312 20:10:28.588337 1788 fs_manager.cc:596] Generated new instance metadata in path /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ChurnyElections_WithNotificationLatency.1520885401735786-212/raft_consensus-itest-cluster/ts-1/wal/instance: uuid: "012786a8f1b64995847cf7dacbc029a4" format_stamp: "Formatted at 2018-03-12 20:10:28 on dist-test-slave-dist-test-slave-f27p" I0312 20:10:28.598534 1788 fs_manager.cc:496] Time spent creating directory manager: real 0.010s user 0.008s sys 0.004s I0312 20:10:28.599673 1788 env_posix.cc:1620] Not raising this process' open files per process limit of 65536; it is already as high as it can go I0312 20:10:28.599972 1788 file_cache.cc:470] Constructed file cache lbm with capacity 26214 I0312 20:10:28.614161 1788 fs_manager.cc:418] Time spent opening block manager: real 0.009s user 0.000s sys 0.012s I0312 20:10:28.614279 1788 fs_manager.cc:429] Opened local filesystem: /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ChurnyElections_WithNotificationLatency.1520885401735786-212/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ChurnyElections_WithNotificationLatency.1520885401735786-212/raft_consensus-itest-cluster/ts-1/wal uuid: "012786a8f1b64995847cf7dacbc029a4" format_stamp: "Formatted at 2018-03-12 20:10:28 on dist-test-slave-dist-test-slave-f27p" I0312 20:10:28.614473 1788 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ChurnyElections_WithNotificationLatency.1520885401735786-212/raft_consensus-itest-cluster/ts-1/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0312 20:10:28.857637 1788 env_posix.cc:1620] Not raising this process' running threads per effective uid limit of 60284; it is already as high as it can go I0312 20:10:28.859439 1788 ts_tablet_manager.cc:349] Loaded tablet metadata (0 live tablets) I0312 20:10:28.860606 1788 tablet_server_main.cc:84] Starting tablet server... I0312 20:10:29.090601 1788 rpc_server.cc:201] RPC server started. Bound to: 127.0.53.2:43059 I0312 20:10:29.091506 1788 webserver.cc:173] Starting webserver on 127.0.53.2:0 I0312 20:10:29.091614 1788 webserver.cc:184] Document root disabled I0312 20:10:29.092857 1788 webserver.cc:311] Webserver started. Bound to: http://127.0.53.2:37004/ I0312 20:10:29.093618 1788 server_base.cc:607] Dumped server information to /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ChurnyElections_WithNotificationLatency.1520885401735786-212/raft_consensus-itest-cluster/ts-1/data/info.pb I0312 20:10:29.094418 212 external_mini_cluster.cc:867] Started /tmp/dist-test-task3bwPeE/build/tsan/bin/kudu-tserver as pid 1788 I0312 20:10:29.095706 212 external_mini_cluster.cc:805] Running /tmp/dist-test-task3bwPeE/build/tsan/bin/kudu-tserver /tmp/dist-test-task3bwPeE/build/tsan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ChurnyElections_WithNotificationLatency.1520885401735786-212/raft_consensus-itest-cluster/ts-2/wal --fs_data_dirs=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ChurnyElections_WithNotificationLatency.1520885401735786-212/raft_consensus-itest-cluster/ts-2/data --block_manager=log --rpc_bind_addresses=127.0.53.3:0 --local_ip_for_outbound_sockets=127.0.53.3 --webserver_interface=127.0.53.3 --webserver_port=0 --tserver_master_addrs=127.0.53.62:44334 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ChurnyElections_WithNotificationLatency.1520885401735786-212/raft_consensus-itest-cluster/ts-2/logs --server_dump_info_path=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ChurnyElections_WithNotificationLatency.1520885401735786-212/raft_consensus-itest-cluster/ts-2/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_heartbeat_interval_ms=5 --consensus_inject_latency_ms_in_notifications=50 with env {} I0312 20:10:29.100224 1910 process_memory.cc:182] Process hard memory limit is 11.777631 GB I0312 20:10:29.100757 1910 process_memory.cc:184] Process soft memory limit is 9.422105 GB I0312 20:10:29.100917 1910 process_memory.cc:187] Process memory pressure threshold is 7.066578 GB I0312 20:10:29.123667 1909 heartbeater.cc:346] Connected to a master server at 127.0.53.62:44334 I0312 20:10:29.123966 1909 heartbeater.cc:426] Registering TS with master... I0312 20:10:29.124441 1909 heartbeater.cc:475] Master 127.0.53.62:44334 requested a full tablet report, sending... I0312 20:10:29.125831 1619 ts_manager.cc:81] Registered new tserver with Master: 012786a8f1b64995847cf7dacbc029a4 (127.0.53.2:43059) I0312 20:10:29.126947 1619 master_service.cc:243] Signed X509 certificate for tserver {username='slave'} at 127.0.53.2:48706 I0312 20:10:29.100679 1788 tablet_server_main.cc:87] Tablet server successfully started. WARNING: Logging before InitGoogleLogging() is written to STDERR W0312 20:10:29.223054 1913 flags.cc:412] Enabled unsafe flag: --consensus_inject_latency_ms_in_notifications=50 W0312 20:10:29.223400 1913 flags.cc:412] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0312 20:10:29.223584 1913 flags.cc:412] Enabled unsafe flag: --never_fsync=true W0312 20:10:29.237866 1913 flags.cc:412] Enabled experimental flag: --ipki_server_key_size=1024 W0312 20:10:29.238167 1913 flags.cc:412] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.53.3 I0312 20:10:29.257169 1913 tablet_server_main.cc:72] Tablet server non-default flags: --consensus_inject_latency_ms_in_notifications=50 --raft_heartbeat_interval_ms=5 --fs_data_dirs=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ChurnyElections_WithNotificationLatency.1520885401735786-212/raft_consensus-itest-cluster/ts-2/data --fs_wal_dir=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ChurnyElections_WithNotificationLatency.1520885401735786-212/raft_consensus-itest-cluster/ts-2/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.0.53.3:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ChurnyElections_WithNotificationLatency.1520885401735786-212/raft_consensus-itest-cluster/ts-2/data/info.pb --webserver_interface=127.0.53.3 --webserver_port=0 --tserver_master_addrs=127.0.53.62:44334 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.0.53.3 --log_dir=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ChurnyElections_WithNotificationLatency.1520885401735786-212/raft_consensus-itest-cluster/ts-2/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.7.0-SNAPSHOT revision 13e10c63c46de30ac85f34b543cdd6fb44881f61 build type FASTDEBUG built by None at 12 Mar 2018 20:06:40 UTC on jenkins-slave-vn0g build id 12388 TSAN enabled I0312 20:10:29.266790 1913 tablet_server_main.cc:79] Initializing tablet server... I0312 20:10:29.268357 1913 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 1081776us I0312 20:10:29.268786 1913 fs_manager.cc:260] Metadata directory not provided I0312 20:10:29.268908 1913 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0312 20:10:29.269112 1913 server_base.cc:434] Could not load existing FS layout: Not found: /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ChurnyElections_WithNotificationLatency.1520885401735786-212/raft_consensus-itest-cluster/ts-2/data/instance: No such file or directory (error 2) I0312 20:10:29.269225 1913 server_base.cc:435] Attempting to create new FS layout instead I0312 20:10:29.276370 1913 fs_manager.cc:596] Generated new instance metadata in path /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ChurnyElections_WithNotificationLatency.1520885401735786-212/raft_consensus-itest-cluster/ts-2/data/instance: uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" format_stamp: "Formatted at 2018-03-12 20:10:29 on dist-test-slave-dist-test-slave-f27p" I0312 20:10:29.277052 1913 fs_manager.cc:596] Generated new instance metadata in path /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ChurnyElections_WithNotificationLatency.1520885401735786-212/raft_consensus-itest-cluster/ts-2/wal/instance: uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" format_stamp: "Formatted at 2018-03-12 20:10:29 on dist-test-slave-dist-test-slave-f27p" I0312 20:10:29.285130 1913 fs_manager.cc:496] Time spent creating directory manager: real 0.008s user 0.008s sys 0.000s I0312 20:10:29.286285 1913 env_posix.cc:1620] Not raising this process' open files per process limit of 65536; it is already as high as it can go I0312 20:10:29.286556 1913 file_cache.cc:470] Constructed file cache lbm with capacity 26214 I0312 20:10:29.292825 1913 fs_manager.cc:418] Time spent opening block manager: real 0.001s user 0.004s sys 0.000s I0312 20:10:29.292939 1913 fs_manager.cc:429] Opened local filesystem: /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ChurnyElections_WithNotificationLatency.1520885401735786-212/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ChurnyElections_WithNotificationLatency.1520885401735786-212/raft_consensus-itest-cluster/ts-2/wal uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" format_stamp: "Formatted at 2018-03-12 20:10:29 on dist-test-slave-dist-test-slave-f27p" I0312 20:10:29.293145 1913 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ChurnyElections_WithNotificationLatency.1520885401735786-212/raft_consensus-itest-cluster/ts-2/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0312 20:10:29.350975 1784 heartbeater.cc:467] Master 127.0.53.62:44334 was elected leader, sending a full tablet report... I0312 20:10:29.477694 1913 env_posix.cc:1620] Not raising this process' running threads per effective uid limit of 60284; it is already as high as it can go I0312 20:10:29.478946 1913 ts_tablet_manager.cc:349] Loaded tablet metadata (0 live tablets) I0312 20:10:29.479925 1913 tablet_server_main.cc:84] Starting tablet server... I0312 20:10:29.566803 1913 rpc_server.cc:201] RPC server started. Bound to: 127.0.53.3:49316 I0312 20:10:29.567672 1913 webserver.cc:173] Starting webserver on 127.0.53.3:0 I0312 20:10:29.567772 1913 webserver.cc:184] Document root disabled I0312 20:10:29.568977 1913 webserver.cc:311] Webserver started. Bound to: http://127.0.53.3:47345/ I0312 20:10:29.569715 1913 server_base.cc:607] Dumped server information to /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ChurnyElections_WithNotificationLatency.1520885401735786-212/raft_consensus-itest-cluster/ts-2/data/info.pb I0312 20:10:29.572420 2035 process_memory.cc:182] Process hard memory limit is 11.777631 GB I0312 20:10:29.572929 2035 process_memory.cc:184] Process soft memory limit is 9.422105 GB I0312 20:10:29.573096 2035 process_memory.cc:187] Process memory pressure threshold is 7.066578 GB I0312 20:10:29.579370 212 external_mini_cluster.cc:867] Started /tmp/dist-test-task3bwPeE/build/tsan/bin/kudu-tserver as pid 1913 I0312 20:10:29.572878 1913 tablet_server_main.cc:87] Tablet server successfully started. I0312 20:10:29.617242 2034 heartbeater.cc:346] Connected to a master server at 127.0.53.62:44334 I0312 20:10:29.617465 2034 heartbeater.cc:426] Registering TS with master... I0312 20:10:29.617874 2034 heartbeater.cc:475] Master 127.0.53.62:44334 requested a full tablet report, sending... I0312 20:10:29.619256 1619 ts_manager.cc:81] Registered new tserver with Master: e3aa58d3045c4a05b94b94eb8bf58ae1 (127.0.53.3:49316) I0312 20:10:29.620705 1619 master_service.cc:243] Signed X509 certificate for tserver {username='slave'} at 127.0.53.3:49501 I0312 20:10:29.621232 212 external_mini_cluster.cc:498] 3 TS(s) registered with all masters I0312 20:10:29.624038 212 test_util.cc:192] Using random seed: -1144595250 I0312 20:10:29.639926 1619 catalog_manager.cc:1294] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44783: name: "test-workload" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 3 split_rows_range_bounds { } partition_schema { range_schema { columns { name: "key" } } } W0312 20:10:29.648362 1619 catalog_manager.cc:1450] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a replica failure: 4 tablet servers are needed, 3 are alive. Consider bringing up additional tablet server(s) or running both the masters and all tablet servers with --raft_prepare_replacement_before_eviction=false flag (not recommended). I0312 20:10:29.676000 1846 tablet_service.cc:755] Processing CreateTablet for tablet 7581a0a77de54ef8943c1521124c7728 (table=test-workload [id=e00bf54237784b2dba71cc550cbd7cca]), partition=RANGE (key) PARTITION UNBOUNDED I0312 20:10:29.676913 1846 data_dirs.cc:931] Could only allocate 1 dirs of requested 3 for tablet 7581a0a77de54ef8943c1521124c7728. 1 dirs total, 0 dirs full, 0 dirs failed I0312 20:10:29.691620 1846 ts_tablet_manager.cc:1163] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4: Registered tablet (data state: TABLET_DATA_READY) I0312 20:10:29.692845 2052 ts_tablet_manager.cc:1018] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4: Bootstrapping tablet I0312 20:10:29.693248 2052 tablet_bootstrap.cc:436] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4: Bootstrap starting. I0312 20:10:29.695842 2052 tablet_bootstrap.cc:581] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4: No blocks or log segments found. Creating new log. I0312 20:10:29.697265 2052 log.cc:520] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4: Log is configured to *not* fsync() on all Append() calls I0312 20:10:29.698782 1971 tablet_service.cc:755] Processing CreateTablet for tablet 7581a0a77de54ef8943c1521124c7728 (table=test-workload [id=e00bf54237784b2dba71cc550cbd7cca]), partition=RANGE (key) PARTITION UNBOUNDED I0312 20:10:29.699682 1971 data_dirs.cc:931] Could only allocate 1 dirs of requested 3 for tablet 7581a0a77de54ef8943c1521124c7728. 1 dirs total, 0 dirs full, 0 dirs failed I0312 20:10:29.702391 2052 tablet_bootstrap.cc:436] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4: No bootstrap required, opened a new log I0312 20:10:29.702783 2052 ts_tablet_manager.cc:1035] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.000s I0312 20:10:29.707536 1721 tablet_service.cc:755] Processing CreateTablet for tablet 7581a0a77de54ef8943c1521124c7728 (table=test-workload [id=e00bf54237784b2dba71cc550cbd7cca]), partition=RANGE (key) PARTITION UNBOUNDED I0312 20:10:29.708242 1721 data_dirs.cc:931] Could only allocate 1 dirs of requested 3 for tablet 7581a0a77de54ef8943c1521124c7728. 1 dirs total, 0 dirs full, 0 dirs failed I0312 20:10:29.716351 1971 ts_tablet_manager.cc:1163] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1: Registered tablet (data state: TABLET_DATA_READY) I0312 20:10:29.718230 2055 ts_tablet_manager.cc:1018] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1: Bootstrapping tablet I0312 20:10:29.718726 2055 tablet_bootstrap.cc:436] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1: Bootstrap starting. I0312 20:10:29.719511 1721 ts_tablet_manager.cc:1163] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c: Registered tablet (data state: TABLET_DATA_READY) I0312 20:10:29.721160 2052 raft_consensus.cc:304] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "012786a8f1b64995847cf7dacbc029a4" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 43059 } } peers { permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 } } peers { permanent_uuid: "9da22e5afdce42078fe8f94ad166ac3c" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 39175 } } I0312 20:10:29.721730 2052 raft_consensus.cc:330] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0312 20:10:29.721827 2055 tablet_bootstrap.cc:581] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1: No blocks or log segments found. Creating new log. I0312 20:10:29.721889 2052 raft_consensus.cc:605] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 012786a8f1b64995847cf7dacbc029a4, State: Initialized, Role: FOLLOWER I0312 20:10:29.723322 2055 log.cc:520] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1: Log is configured to *not* fsync() on all Append() calls I0312 20:10:29.725404 2056 ts_tablet_manager.cc:1018] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c: Bootstrapping tablet I0312 20:10:29.725011 2052 consensus_queue.cc:219] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "012786a8f1b64995847cf7dacbc029a4" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 43059 } } peers { permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 } } peers { permanent_uuid: "9da22e5afdce42078fe8f94ad166ac3c" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 39175 } } I0312 20:10:29.725864 2056 tablet_bootstrap.cc:436] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c: Bootstrap starting. I0312 20:10:29.726977 2059 raft_consensus.cc:436] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0312 20:10:29.727198 2059 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4: Snoozing failure detection for 0.017s (starting election) I0312 20:10:29.727438 2059 raft_consensus.cc:458] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "012786a8f1b64995847cf7dacbc029a4" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 43059 } } peers { permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 } } peers { permanent_uuid: "9da22e5afdce42078fe8f94ad166ac3c" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 39175 } } I0312 20:10:29.727718 2055 tablet_bootstrap.cc:436] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1: No bootstrap required, opened a new log I0312 20:10:29.728173 2055 ts_tablet_manager.cc:1035] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s I0312 20:10:29.728564 2059 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer e3aa58d3045c4a05b94b94eb8bf58ae1 I0312 20:10:29.729001 2059 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 9da22e5afdce42078fe8f94ad166ac3c I0312 20:10:29.729768 1909 heartbeater.cc:467] Master 127.0.53.62:44334 was elected leader, sending a full tablet report... I0312 20:10:29.730321 2056 tablet_bootstrap.cc:581] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c: No blocks or log segments found. Creating new log. I0312 20:10:29.731917 2056 log.cc:520] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c: Log is configured to *not* fsync() on all Append() calls I0312 20:10:29.733860 2052 ts_tablet_manager.cc:1063] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4: Time spent starting tablet: real 0.031s user 0.024s sys 0.000s I0312 20:10:29.739354 2056 tablet_bootstrap.cc:436] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c: No bootstrap required, opened a new log I0312 20:10:29.739761 2056 ts_tablet_manager.cc:1035] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c: Time spent bootstrapping tablet: real 0.014s user 0.008s sys 0.004s I0312 20:10:29.743943 2055 raft_consensus.cc:304] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "012786a8f1b64995847cf7dacbc029a4" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 43059 } } peers { permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 } } peers { permanent_uuid: "9da22e5afdce42078fe8f94ad166ac3c" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 39175 } } I0312 20:10:29.744925 2059 raft_consensus.cc:436] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0312 20:10:29.745151 2059 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4: Snoozing failure detection for 0.018s (starting election) I0312 20:10:29.745424 2059 raft_consensus.cc:458] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "012786a8f1b64995847cf7dacbc029a4" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 43059 } } peers { permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 } } peers { permanent_uuid: "9da22e5afdce42078fe8f94ad166ac3c" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 39175 } } I0312 20:10:29.746270 2055 raft_consensus.cc:330] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0312 20:10:29.746484 2059 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer e3aa58d3045c4a05b94b94eb8bf58ae1 I0312 20:10:29.746498 2055 raft_consensus.cc:605] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3aa58d3045c4a05b94b94eb8bf58ae1, State: Initialized, Role: FOLLOWER I0312 20:10:29.746841 2059 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 9da22e5afdce42078fe8f94ad166ac3c I0312 20:10:29.747119 2055 consensus_queue.cc:219] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "012786a8f1b64995847cf7dacbc029a4" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 43059 } } peers { permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 } } peers { permanent_uuid: "9da22e5afdce42078fe8f94ad166ac3c" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 39175 } } I0312 20:10:29.755103 2056 raft_consensus.cc:304] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "012786a8f1b64995847cf7dacbc029a4" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 43059 } } peers { permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 } } peers { permanent_uuid: "9da22e5afdce42078fe8f94ad166ac3c" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 39175 } } I0312 20:10:29.755898 2056 raft_consensus.cc:330] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0312 20:10:29.756108 2056 raft_consensus.cc:605] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9da22e5afdce42078fe8f94ad166ac3c, State: Initialized, Role: FOLLOWER I0312 20:10:29.756772 2056 consensus_queue.cc:219] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "012786a8f1b64995847cf7dacbc029a4" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 43059 } } peers { permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 } } peers { permanent_uuid: "9da22e5afdce42078fe8f94ad166ac3c" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 39175 } } I0312 20:10:29.758747 2034 heartbeater.cc:467] Master 127.0.53.62:44334 was elected leader, sending a full tablet report... I0312 20:10:29.761312 2064 raft_consensus.cc:436] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0312 20:10:29.761582 2064 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1: Snoozing failure detection for 0.021s (starting election) I0312 20:10:29.761886 2064 raft_consensus.cc:458] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "012786a8f1b64995847cf7dacbc029a4" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 43059 } } peers { permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 } } peers { permanent_uuid: "9da22e5afdce42078fe8f94ad166ac3c" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 39175 } } W0312 20:10:29.763541 1797 leader_election.cc:287] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer e3aa58d3045c4a05b94b94eb8bf58ae1: Timed out: RequestConsensusVote RPC to 127.0.53.3:49316 timed out after 0.017s (ON_OUTBOUND_QUEUE) I0312 20:10:29.763629 2064 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 012786a8f1b64995847cf7dacbc029a4 W0312 20:10:29.763952 1797 leader_election.cc:287] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 9da22e5afdce42078fe8f94ad166ac3c: Timed out: connection negotiation to 127.0.53.1:39175 for RPC RequestConsensusVote timed out after 0.017s (ON_OUTBOUND_QUEUE) I0312 20:10:29.764215 1797 leader_election.cc:258] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. I0312 20:10:29.764199 2059 raft_consensus.cc:436] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0312 20:10:29.764492 2059 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4: Snoozing failure detection for 0.019s (starting election) I0312 20:10:29.764806 2055 ts_tablet_manager.cc:1063] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1: Time spent starting tablet: real 0.036s user 0.012s sys 0.004s I0312 20:10:29.764709 2059 raft_consensus.cc:458] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "012786a8f1b64995847cf7dacbc029a4" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 43059 } } peers { permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 } } peers { permanent_uuid: "9da22e5afdce42078fe8f94ad166ac3c" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 39175 } } I0312 20:10:29.765311 2064 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 9da22e5afdce42078fe8f94ad166ac3c I0312 20:10:29.765576 2059 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer e3aa58d3045c4a05b94b94eb8bf58ae1 I0312 20:10:29.765897 2059 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 9da22e5afdce42078fe8f94ad166ac3c I0312 20:10:29.766686 2059 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4: Snoozing failure detection for 0.017s (election complete) I0312 20:10:29.766919 2059 raft_consensus.cc:2455] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority I0312 20:10:29.770184 2065 raft_consensus.cc:436] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0312 20:10:29.770882 2065 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c: Snoozing failure detection for 0.018s (starting election) I0312 20:10:29.771210 2065 raft_consensus.cc:458] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "012786a8f1b64995847cf7dacbc029a4" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 43059 } } peers { permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 } } peers { permanent_uuid: "9da22e5afdce42078fe8f94ad166ac3c" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 39175 } } I0312 20:10:29.772572 2065 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 012786a8f1b64995847cf7dacbc029a4 I0312 20:10:29.773099 2065 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer e3aa58d3045c4a05b94b94eb8bf58ae1 I0312 20:10:29.770685 2056 ts_tablet_manager.cc:1063] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c: Time spent starting tablet: real 0.031s user 0.020s sys 0.000s I0312 20:10:29.780405 2064 raft_consensus.cc:436] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0312 20:10:29.780625 2064 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1: Snoozing failure detection for 0.015s (starting election) I0312 20:10:29.780840 2064 raft_consensus.cc:458] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "012786a8f1b64995847cf7dacbc029a4" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 43059 } } peers { permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 } } peers { permanent_uuid: "9da22e5afdce42078fe8f94ad166ac3c" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 39175 } } I0312 20:10:29.781867 2064 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 012786a8f1b64995847cf7dacbc029a4 I0312 20:10:29.781735 1866 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "012786a8f1b64995847cf7dacbc029a4" is_pre_election: true I0312 20:10:29.782251 2064 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 9da22e5afdce42078fe8f94ad166ac3c I0312 20:10:29.782224 1866 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4: Snoozing failure detection for 0.019s (vote granted) I0312 20:10:29.782454 1866 raft_consensus.cc:2215] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3aa58d3045c4a05b94b94eb8bf58ae1 in term 0. I0312 20:10:29.783164 1991 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "012786a8f1b64995847cf7dacbc029a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" is_pre_election: true I0312 20:10:29.783684 1991 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1: Snoozing failure detection for 0.015s (vote granted) I0312 20:10:29.783562 1866 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "012786a8f1b64995847cf7dacbc029a4" is_pre_election: true I0312 20:10:29.783944 1866 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4: Snoozing failure detection for 0.018s (vote granted) I0312 20:10:29.783896 1991 raft_consensus.cc:2215] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 012786a8f1b64995847cf7dacbc029a4 in term 0. I0312 20:10:29.784144 1866 raft_consensus.cc:2215] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3aa58d3045c4a05b94b94eb8bf58ae1 in term 0. I0312 20:10:29.784353 1926 leader_election.cc:363] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 1 pre-election: Vote granted by peer 012786a8f1b64995847cf7dacbc029a4 I0312 20:10:29.784646 1926 leader_election.cc:258] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. I0312 20:10:29.785054 1991 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "012786a8f1b64995847cf7dacbc029a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" is_pre_election: true W0312 20:10:29.785194 1797 leader_election.cc:287] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer e3aa58d3045c4a05b94b94eb8bf58ae1: Timed out: RequestConsensusVote RPC to 127.0.53.3:49316 timed out after 0.018s (SENT) I0312 20:10:29.785347 1926 leader_election.cc:363] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 1 pre-election: Vote granted by peer 012786a8f1b64995847cf7dacbc029a4 I0312 20:10:29.785475 1991 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1: Snoozing failure detection for 0.015s (vote granted) I0312 20:10:29.785542 1926 leader_election.cc:258] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. W0312 20:10:29.785576 1797 leader_election.cc:287] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 9da22e5afdce42078fe8f94ad166ac3c: Timed out: connection negotiation to 127.0.53.1:39175 for RPC RequestConsensusVote timed out after 0.018s (ON_OUTBOUND_QUEUE) I0312 20:10:29.785682 1991 raft_consensus.cc:2215] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 012786a8f1b64995847cf7dacbc029a4 in term 0. I0312 20:10:29.785753 1797 leader_election.cc:258] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. I0312 20:10:29.786252 2064 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1: Snoozing failure detection for 0.020s (election complete) I0312 20:10:29.786456 2064 raft_consensus.cc:2507] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 0 FOLLOWER]: Leader pre-election won for term 1 I0312 20:10:29.786635 2064 raft_consensus.cc:436] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0312 20:10:29.786806 2064 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1: Snoozing failure detection for 0.019s (starting election) I0312 20:10:29.786978 2064 raft_consensus.cc:2752] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 0 FOLLOWER]: Advancing to term 1 I0312 20:10:29.786995 2059 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4: Snoozing failure detection for 0.019s (election complete) I0312 20:10:29.787187 2059 raft_consensus.cc:2455] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority W0312 20:10:29.787751 1797 leader_election.cc:287] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer e3aa58d3045c4a05b94b94eb8bf58ae1: Timed out: RequestConsensusVote RPC to 127.0.53.3:49316 timed out after 0.019s (SENT) W0312 20:10:29.788058 1797 leader_election.cc:287] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 9da22e5afdce42078fe8f94ad166ac3c: Timed out: connection negotiation to 127.0.53.1:39175 for RPC RequestConsensusVote timed out after 0.019s (ON_OUTBOUND_QUEUE) I0312 20:10:29.788228 1797 leader_election.cc:258] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. I0312 20:10:29.788609 2059 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4: Snoozing failure detection for 0.045s (election complete) I0312 20:10:29.788803 2059 raft_consensus.cc:2455] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority W0312 20:10:29.789413 1922 leader_election.cc:287] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 9da22e5afdce42078fe8f94ad166ac3c: Timed out: connection negotiation to 127.0.53.1:39175 for RPC RequestConsensusVote timed out after 0.021s (ON_OUTBOUND_QUEUE) I0312 20:10:29.789657 2065 raft_consensus.cc:436] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0312 20:10:29.789855 2065 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c: Snoozing failure detection for 0.020s (starting election) I0312 20:10:29.790102 2065 raft_consensus.cc:458] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "012786a8f1b64995847cf7dacbc029a4" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 43059 } } peers { permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 } } peers { permanent_uuid: "9da22e5afdce42078fe8f94ad166ac3c" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 39175 } } I0312 20:10:29.791483 2065 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 012786a8f1b64995847cf7dacbc029a4 I0312 20:10:29.792136 2065 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer e3aa58d3045c4a05b94b94eb8bf58ae1 I0312 20:10:29.791899 2064 raft_consensus.cc:458] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "012786a8f1b64995847cf7dacbc029a4" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 43059 } } peers { permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 } } peers { permanent_uuid: "9da22e5afdce42078fe8f94ad166ac3c" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 39175 } } W0312 20:10:29.792409 1676 leader_election.cc:287] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 012786a8f1b64995847cf7dacbc029a4: Timed out: connection negotiation to 127.0.53.2:43059 for RPC RequestConsensusVote timed out after 0.018s (ON_OUTBOUND_QUEUE) W0312 20:10:29.793607 1663 leader_election.cc:287] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer e3aa58d3045c4a05b94b94eb8bf58ae1: Timed out: connection negotiation to 127.0.53.3:49316 for RPC RequestConsensusVote timed out after 0.018s (ON_OUTBOUND_QUEUE) I0312 20:10:29.793732 2064 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 1 election: Requesting vote from peer 012786a8f1b64995847cf7dacbc029a4 I0312 20:10:29.794045 2064 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 1 election: Requesting vote from peer 9da22e5afdce42078fe8f94ad166ac3c I0312 20:10:29.794881 2063 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1: Snoozing failure detection for 0.019s (election complete) I0312 20:10:29.795114 2063 raft_consensus.cc:2470] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won I0312 20:10:29.795076 1866 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "012786a8f1b64995847cf7dacbc029a4" I0312 20:10:29.795563 1866 raft_consensus.cc:2752] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 0 FOLLOWER]: Advancing to term 1 I0312 20:10:29.794646 1663 leader_election.cc:258] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. I0312 20:10:29.795771 1866 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4: Snoozing failure detection for 0.058s (vote granted) W0312 20:10:29.799811 1922 leader_election.cc:287] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 9da22e5afdce42078fe8f94ad166ac3c: Timed out: RequestConsensusVote RPC to 127.0.53.1:39175 timed out after 0.015s (ON_OUTBOUND_QUEUE) I0312 20:10:29.799942 1866 raft_consensus.cc:2215] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3aa58d3045c4a05b94b94eb8bf58ae1 in term 1. I0312 20:10:29.800715 1926 leader_election.cc:363] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 1 election: Vote granted by peer 012786a8f1b64995847cf7dacbc029a4 I0312 20:10:29.801112 1926 leader_election.cc:258] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0312 20:10:29.801224 1741 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9da22e5afdce42078fe8f94ad166ac3c" I0312 20:10:29.801584 2064 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1: Snoozing failure detection for 0.022s (election complete) I0312 20:10:29.801854 2064 raft_consensus.cc:2507] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 1 FOLLOWER]: Leader election won for term 1 I0312 20:10:29.802150 2064 raft_consensus.cc:572] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 1 LEADER]: Becoming Leader. State: Replica: e3aa58d3045c4a05b94b94eb8bf58ae1, State: Running, Role: LEADER I0312 20:10:29.802711 2064 consensus_queue.cc:196] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "012786a8f1b64995847cf7dacbc029a4" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 43059 } } peers { permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 } } peers { permanent_uuid: "9da22e5afdce42078fe8f94ad166ac3c" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 39175 } } I0312 20:10:29.800423 2065 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c: Snoozing failure detection for 0.019s (election complete) I0312 20:10:29.804149 2065 raft_consensus.cc:2455] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority I0312 20:10:29.805182 1741 raft_consensus.cc:2752] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 0 FOLLOWER]: Advancing to term 1 I0312 20:10:29.805419 1741 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c: Snoozing failure detection for 0.032s (vote granted) I0312 20:10:29.806555 1866 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "9da22e5afdce42078fe8f94ad166ac3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "012786a8f1b64995847cf7dacbc029a4" is_pre_election: true I0312 20:10:29.807123 1866 raft_consensus.cc:2145] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9da22e5afdce42078fe8f94ad166ac3c in current term 1: Already voted for candidate e3aa58d3045c4a05b94b94eb8bf58ae1 in this term. I0312 20:10:29.808385 1676 leader_election.cc:377] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [CANDIDATE]: Term 1 pre-election: Vote denied by peer 012786a8f1b64995847cf7dacbc029a4. Message: Invalid argument: T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9da22e5afdce42078fe8f94ad166ac3c in current term 1: Already voted for candidate e3aa58d3045c4a05b94b94eb8bf58ae1 in this term. I0312 20:10:29.808992 1741 raft_consensus.cc:2215] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3aa58d3045c4a05b94b94eb8bf58ae1 in term 1. I0312 20:10:29.810289 1922 leader_election.cc:363] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 1 election: Vote granted by peer 9da22e5afdce42078fe8f94ad166ac3c I0312 20:10:29.811234 1866 raft_consensus.cc:1058] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 1 FOLLOWER]: Refusing update from remote peer e3aa58d3045c4a05b94b94eb8bf58ae1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch) I0312 20:10:29.811362 1618 catalog_manager.cc:3592] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 reported cstate change: term changed from 0 to 1, leader changed from to e3aa58d3045c4a05b94b94eb8bf58ae1 (127.0.53.3). New cstate: current_term: 1 leader_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "012786a8f1b64995847cf7dacbc029a4" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 43059 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9da22e5afdce42078fe8f94ad166ac3c" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 39175 } health_report { overall_health: UNKNOWN } } } I0312 20:10:29.812270 2064 consensus_queue.cc:1017] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "012786a8f1b64995847cf7dacbc029a4" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 43059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0312 20:10:29.814577 1741 raft_consensus.cc:1058] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 1 FOLLOWER]: Refusing update from remote peer e3aa58d3045c4a05b94b94eb8bf58ae1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch) I0312 20:10:29.815850 2064 consensus_queue.cc:1017] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9da22e5afdce42078fe8f94ad166ac3c" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 39175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s W0312 20:10:29.816269 1663 leader_election.cc:287] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer e3aa58d3045c4a05b94b94eb8bf58ae1: Timed out: RequestConsensusVote RPC to 127.0.53.3:49316 timed out after 0.020s (ON_OUTBOUND_QUEUE) I0312 20:10:29.816470 1663 leader_election.cc:258] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. I0312 20:10:29.816838 2065 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c: Snoozing failure detection for 0.027s (election complete) I0312 20:10:29.817024 2065 raft_consensus.cc:2455] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority W0312 20:10:29.823451 2035 tablet.cc:1679] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1: Can't schedule compaction. Clean time has not been advanced past its initial value. W0312 20:10:29.828938 2075 fault_injection.cc:41] FAULT INJECTION ENABLED! W0312 20:10:29.829128 2075 fault_injection.cc:42] THIS SERVER MAY CRASH! I0312 20:10:29.830446 2059 raft_consensus.cc:436] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e3aa58d3045c4a05b94b94eb8bf58ae1) I0312 20:10:29.830699 2059 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4: Snoozing failure detection for 0.021s (starting election) I0312 20:10:29.830921 2059 raft_consensus.cc:458] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "012786a8f1b64995847cf7dacbc029a4" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 43059 } } peers { permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 } } peers { permanent_uuid: "9da22e5afdce42078fe8f94ad166ac3c" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 39175 } } W0312 20:10:29.831558 1785 tablet.cc:1679] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c: Can't schedule compaction. Clean time has not been advanced past its initial value. I0312 20:10:29.831912 2059 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer e3aa58d3045c4a05b94b94eb8bf58ae1 I0312 20:10:29.832280 2059 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 9da22e5afdce42078fe8f94ad166ac3c I0312 20:10:29.834187 1991 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "012786a8f1b64995847cf7dacbc029a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" is_pre_election: true I0312 20:10:29.834725 1991 raft_consensus.cc:2176] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate 012786a8f1b64995847cf7dacbc029a4 for term 2 because replica is either leader or believes a valid leader to be alive. I0312 20:10:29.835144 1740 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "012786a8f1b64995847cf7dacbc029a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9da22e5afdce42078fe8f94ad166ac3c" is_pre_election: true I0312 20:10:29.835729 1740 raft_consensus.cc:2190] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 012786a8f1b64995847cf7dacbc029a4 for term 2 because replica is already servicing an update from a current leader or another vote. I0312 20:10:29.835817 1797 leader_election.cc:377] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [CANDIDATE]: Term 2 pre-election: Vote denied by peer e3aa58d3045c4a05b94b94eb8bf58ae1. Message: Invalid argument: T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate 012786a8f1b64995847cf7dacbc029a4 for term 2 because replica is either leader or believes a valid leader to be alive. I0312 20:10:29.836704 1797 leader_election.cc:377] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 9da22e5afdce42078fe8f94ad166ac3c. Message: Service unavailable: T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 012786a8f1b64995847cf7dacbc029a4 for term 2 because replica is already servicing an update from a current leader or another vote. I0312 20:10:29.836922 1797 leader_election.cc:258] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. I0312 20:10:29.839957 2059 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4: Snoozing failure detection for 0.017s (election complete) I0312 20:10:29.840165 2059 raft_consensus.cc:2455] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority W0312 20:10:29.851691 1910 tablet.cc:1679] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4: Can't schedule compaction. Clean time has not been advanced past its initial value. I0312 20:10:29.881681 2059 raft_consensus.cc:436] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e3aa58d3045c4a05b94b94eb8bf58ae1) I0312 20:10:29.881904 2059 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4: Snoozing failure detection for 0.016s (starting election) I0312 20:10:29.882130 2059 raft_consensus.cc:458] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "012786a8f1b64995847cf7dacbc029a4" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 43059 } } peers { permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 } } peers { permanent_uuid: "9da22e5afdce42078fe8f94ad166ac3c" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 39175 } } I0312 20:10:29.883277 2059 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer e3aa58d3045c4a05b94b94eb8bf58ae1 I0312 20:10:29.883638 2059 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 9da22e5afdce42078fe8f94ad166ac3c I0312 20:10:29.884896 1741 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "012786a8f1b64995847cf7dacbc029a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "9da22e5afdce42078fe8f94ad166ac3c" is_pre_election: true I0312 20:10:29.884974 1991 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "012786a8f1b64995847cf7dacbc029a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" is_pre_election: true I0312 20:10:29.885473 1741 raft_consensus.cc:2176] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 012786a8f1b64995847cf7dacbc029a4 for term 2 because replica is either leader or believes a valid leader to be alive. I0312 20:10:29.885545 1991 raft_consensus.cc:2176] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate 012786a8f1b64995847cf7dacbc029a4 for term 2 because replica is either leader or believes a valid leader to be alive. I0312 20:10:29.886812 1797 leader_election.cc:377] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 9da22e5afdce42078fe8f94ad166ac3c. Message: Invalid argument: T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 012786a8f1b64995847cf7dacbc029a4 for term 2 because replica is either leader or believes a valid leader to be alive. I0312 20:10:29.887228 1797 leader_election.cc:377] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [CANDIDATE]: Term 2 pre-election: Vote denied by peer e3aa58d3045c4a05b94b94eb8bf58ae1. Message: Invalid argument: T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate 012786a8f1b64995847cf7dacbc029a4 for term 2 because replica is either leader or believes a valid leader to be alive. I0312 20:10:29.887420 1797 leader_election.cc:258] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. I0312 20:10:29.888181 2059 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4: Snoozing failure detection for 0.026s (election complete) I0312 20:10:29.888383 2059 raft_consensus.cc:2455] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority I0312 20:10:30.109333 2065 raft_consensus.cc:436] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e3aa58d3045c4a05b94b94eb8bf58ae1) I0312 20:10:30.109642 2065 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c: Snoozing failure detection for 0.017s (starting election) I0312 20:10:30.109879 2065 raft_consensus.cc:458] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "012786a8f1b64995847cf7dacbc029a4" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 43059 } } peers { permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 } } peers { permanent_uuid: "9da22e5afdce42078fe8f94ad166ac3c" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 39175 } } I0312 20:10:30.110919 2065 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 012786a8f1b64995847cf7dacbc029a4 I0312 20:10:30.111281 2065 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer e3aa58d3045c4a05b94b94eb8bf58ae1 I0312 20:10:30.112720 1991 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "9da22e5afdce42078fe8f94ad166ac3c" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" is_pre_election: true I0312 20:10:30.112735 1865 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "9da22e5afdce42078fe8f94ad166ac3c" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "012786a8f1b64995847cf7dacbc029a4" is_pre_election: true I0312 20:10:30.113152 1991 raft_consensus.cc:2176] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate 9da22e5afdce42078fe8f94ad166ac3c for term 2 because replica is either leader or believes a valid leader to be alive. I0312 20:10:30.113179 1865 raft_consensus.cc:2190] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9da22e5afdce42078fe8f94ad166ac3c for term 2 because replica is already servicing an update from a current leader or another vote. I0312 20:10:30.114274 1676 leader_election.cc:377] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [CANDIDATE]: Term 2 pre-election: Vote denied by peer 012786a8f1b64995847cf7dacbc029a4. Message: Service unavailable: T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9da22e5afdce42078fe8f94ad166ac3c for term 2 because replica is already servicing an update from a current leader or another vote. I0312 20:10:30.121341 1663 leader_election.cc:377] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [CANDIDATE]: Term 2 pre-election: Vote denied by peer e3aa58d3045c4a05b94b94eb8bf58ae1. Message: Invalid argument: T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate 9da22e5afdce42078fe8f94ad166ac3c for term 2 because replica is either leader or believes a valid leader to be alive. I0312 20:10:30.121520 1663 leader_election.cc:258] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. I0312 20:10:30.121858 2065 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c: Snoozing failure detection for 0.020s (election complete) I0312 20:10:30.122058 2065 raft_consensus.cc:2455] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority I0312 20:10:30.125715 2059 raft_consensus.cc:436] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e3aa58d3045c4a05b94b94eb8bf58ae1) I0312 20:10:30.125929 2059 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4: Snoozing failure detection for 0.022s (starting election) I0312 20:10:30.126127 2059 raft_consensus.cc:458] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "012786a8f1b64995847cf7dacbc029a4" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 43059 } } peers { permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 } } peers { permanent_uuid: "9da22e5afdce42078fe8f94ad166ac3c" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 39175 } } I0312 20:10:30.126966 2059 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer e3aa58d3045c4a05b94b94eb8bf58ae1 I0312 20:10:30.127280 2059 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 9da22e5afdce42078fe8f94ad166ac3c I0312 20:10:30.128545 1741 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "012786a8f1b64995847cf7dacbc029a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "9da22e5afdce42078fe8f94ad166ac3c" is_pre_election: true I0312 20:10:30.129092 1741 raft_consensus.cc:2176] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 012786a8f1b64995847cf7dacbc029a4 for term 2 because replica is either leader or believes a valid leader to be alive. I0312 20:10:30.130125 1797 leader_election.cc:377] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 9da22e5afdce42078fe8f94ad166ac3c. Message: Invalid argument: T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 012786a8f1b64995847cf7dacbc029a4 for term 2 because replica is either leader or believes a valid leader to be alive. I0312 20:10:30.135601 1991 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "012786a8f1b64995847cf7dacbc029a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" is_pre_election: true I0312 20:10:30.141330 1991 raft_consensus.cc:2176] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate 012786a8f1b64995847cf7dacbc029a4 for term 2 because replica is either leader or believes a valid leader to be alive. I0312 20:10:30.143265 1797 leader_election.cc:377] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [CANDIDATE]: Term 2 pre-election: Vote denied by peer e3aa58d3045c4a05b94b94eb8bf58ae1. Message: Invalid argument: T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate 012786a8f1b64995847cf7dacbc029a4 for term 2 because replica is either leader or believes a valid leader to be alive. I0312 20:10:30.143496 1797 leader_election.cc:258] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. I0312 20:10:30.144265 2059 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4: Snoozing failure detection for 0.031s (election complete) I0312 20:10:30.144471 2059 raft_consensus.cc:2455] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority W0312 20:10:30.384552 2072 log.cc:662] Time spent T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1: Append to log took a long time: real 0.127s user 0.004s sys 0.000s W0312 20:10:30.644258 2072 log.cc:662] Time spent T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1: Append to log took a long time: real 0.259s user 0.000s sys 0.000s W0312 20:10:30.644307 2073 log.cc:662] Time spent T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4: Append to log took a long time: real 0.384s user 0.000s sys 0.000s W0312 20:10:30.645185 2074 log.cc:662] Time spent T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c: Append to log took a long time: real 0.385s user 0.000s sys 0.000s I0312 20:10:30.913683 2098 raft_consensus.cc:436] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e3aa58d3045c4a05b94b94eb8bf58ae1) I0312 20:10:30.914021 2098 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c: Snoozing failure detection for 0.027s (starting election) I0312 20:10:30.914294 2098 raft_consensus.cc:458] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "012786a8f1b64995847cf7dacbc029a4" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 43059 } } peers { permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 } } peers { permanent_uuid: "9da22e5afdce42078fe8f94ad166ac3c" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 39175 } } I0312 20:10:30.922595 2098 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 012786a8f1b64995847cf7dacbc029a4 I0312 20:10:30.926154 1866 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "9da22e5afdce42078fe8f94ad166ac3c" candidate_term: 2 candidate_status { last_received { term: 1 index: 31 } } ignore_live_leader: false dest_uuid: "012786a8f1b64995847cf7dacbc029a4" is_pre_election: true I0312 20:10:30.926779 1866 raft_consensus.cc:2176] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9da22e5afdce42078fe8f94ad166ac3c for term 2 because replica is either leader or believes a valid leader to be alive. I0312 20:10:30.927685 2098 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer e3aa58d3045c4a05b94b94eb8bf58ae1 I0312 20:10:30.930568 1991 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "9da22e5afdce42078fe8f94ad166ac3c" candidate_term: 2 candidate_status { last_received { term: 1 index: 31 } } ignore_live_leader: false dest_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" is_pre_election: true I0312 20:10:30.931125 1991 raft_consensus.cc:2176] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate 9da22e5afdce42078fe8f94ad166ac3c for term 2 because replica is either leader or believes a valid leader to be alive. I0312 20:10:30.933306 1676 leader_election.cc:377] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [CANDIDATE]: Term 2 pre-election: Vote denied by peer 012786a8f1b64995847cf7dacbc029a4. Message: Invalid argument: T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9da22e5afdce42078fe8f94ad166ac3c for term 2 because replica is either leader or believes a valid leader to be alive. I0312 20:10:30.944038 2098 raft_consensus.cc:436] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e3aa58d3045c4a05b94b94eb8bf58ae1) I0312 20:10:30.944310 2098 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c: Snoozing failure detection for 0.025s (starting election) I0312 20:10:30.944583 2098 raft_consensus.cc:458] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "012786a8f1b64995847cf7dacbc029a4" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 43059 } } peers { permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 } } peers { permanent_uuid: "9da22e5afdce42078fe8f94ad166ac3c" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 39175 } } I0312 20:10:30.945569 2098 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 012786a8f1b64995847cf7dacbc029a4 I0312 20:10:30.946444 2098 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer e3aa58d3045c4a05b94b94eb8bf58ae1 I0312 20:10:30.948084 1663 leader_election.cc:377] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [CANDIDATE]: Term 2 pre-election: Vote denied by peer e3aa58d3045c4a05b94b94eb8bf58ae1. Message: Invalid argument: T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate 9da22e5afdce42078fe8f94ad166ac3c for term 2 because replica is either leader or believes a valid leader to be alive. I0312 20:10:30.948376 1663 leader_election.cc:258] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. I0312 20:10:30.949196 2098 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c: Snoozing failure detection for 0.028s (election complete) I0312 20:10:30.949425 2098 raft_consensus.cc:2455] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority I0312 20:10:30.951846 1991 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "9da22e5afdce42078fe8f94ad166ac3c" candidate_term: 2 candidate_status { last_received { term: 1 index: 32 } } ignore_live_leader: false dest_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" is_pre_election: true I0312 20:10:30.952358 1991 raft_consensus.cc:2176] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate 9da22e5afdce42078fe8f94ad166ac3c for term 2 because replica is either leader or believes a valid leader to be alive. I0312 20:10:30.954365 1865 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "9da22e5afdce42078fe8f94ad166ac3c" candidate_term: 2 candidate_status { last_received { term: 1 index: 32 } } ignore_live_leader: false dest_uuid: "012786a8f1b64995847cf7dacbc029a4" is_pre_election: true I0312 20:10:30.954916 1865 raft_consensus.cc:2176] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9da22e5afdce42078fe8f94ad166ac3c for term 2 because replica is either leader or believes a valid leader to be alive. I0312 20:10:30.955812 1676 leader_election.cc:377] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [CANDIDATE]: Term 2 pre-election: Vote denied by peer 012786a8f1b64995847cf7dacbc029a4. Message: Invalid argument: T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9da22e5afdce42078fe8f94ad166ac3c for term 2 because replica is either leader or believes a valid leader to be alive. I0312 20:10:30.956833 1663 leader_election.cc:377] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [CANDIDATE]: Term 2 pre-election: Vote denied by peer e3aa58d3045c4a05b94b94eb8bf58ae1. Message: Invalid argument: T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate 9da22e5afdce42078fe8f94ad166ac3c for term 2 because replica is either leader or believes a valid leader to be alive. I0312 20:10:30.957093 1663 leader_election.cc:258] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. I0312 20:10:30.957478 2098 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c: Snoozing failure detection for 0.028s (election complete) I0312 20:10:30.957789 2098 raft_consensus.cc:2455] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority I0312 20:10:30.961699 2099 raft_consensus.cc:436] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e3aa58d3045c4a05b94b94eb8bf58ae1) I0312 20:10:30.969172 2099 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4: Snoozing failure detection for 0.070s (starting election) I0312 20:10:30.969431 2099 raft_consensus.cc:458] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "012786a8f1b64995847cf7dacbc029a4" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 43059 } } peers { permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 } } peers { permanent_uuid: "9da22e5afdce42078fe8f94ad166ac3c" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 39175 } } I0312 20:10:30.977125 2099 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer e3aa58d3045c4a05b94b94eb8bf58ae1 I0312 20:10:30.977988 2099 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 9da22e5afdce42078fe8f94ad166ac3c I0312 20:10:30.978420 1991 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "012786a8f1b64995847cf7dacbc029a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 33 } } ignore_live_leader: false dest_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" is_pre_election: true I0312 20:10:30.978932 1991 raft_consensus.cc:2176] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate 012786a8f1b64995847cf7dacbc029a4 for term 2 because replica is either leader or believes a valid leader to be alive. I0312 20:10:30.979297 1741 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "012786a8f1b64995847cf7dacbc029a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 33 } } ignore_live_leader: false dest_uuid: "9da22e5afdce42078fe8f94ad166ac3c" is_pre_election: true I0312 20:10:30.980134 1797 leader_election.cc:377] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [CANDIDATE]: Term 2 pre-election: Vote denied by peer e3aa58d3045c4a05b94b94eb8bf58ae1. Message: Invalid argument: T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate 012786a8f1b64995847cf7dacbc029a4 for term 2 because replica is either leader or believes a valid leader to be alive. I0312 20:10:30.981891 1741 raft_consensus.cc:2190] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 012786a8f1b64995847cf7dacbc029a4 for term 2 because replica is already servicing an update from a current leader or another vote. I0312 20:10:30.983085 1797 leader_election.cc:377] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 9da22e5afdce42078fe8f94ad166ac3c. Message: Service unavailable: T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 012786a8f1b64995847cf7dacbc029a4 for term 2 because replica is already servicing an update from a current leader or another vote. I0312 20:10:30.983260 1797 leader_election.cc:258] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. I0312 20:10:30.983577 2099 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4: Snoozing failure detection for 0.043s (election complete) I0312 20:10:30.983788 2099 raft_consensus.cc:2455] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority I0312 20:10:31.506639 2100 raft_consensus.cc:436] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e3aa58d3045c4a05b94b94eb8bf58ae1) I0312 20:10:31.506969 2100 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4: Snoozing failure detection for 0.019s (starting election) I0312 20:10:31.507239 2100 raft_consensus.cc:458] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "012786a8f1b64995847cf7dacbc029a4" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 43059 } } peers { permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 } } peers { permanent_uuid: "9da22e5afdce42078fe8f94ad166ac3c" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 39175 } } I0312 20:10:31.517805 2100 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer e3aa58d3045c4a05b94b94eb8bf58ae1 I0312 20:10:31.519769 1991 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "012786a8f1b64995847cf7dacbc029a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 53 } } ignore_live_leader: false dest_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" is_pre_election: true I0312 20:10:31.520320 1991 raft_consensus.cc:2176] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate 012786a8f1b64995847cf7dacbc029a4 for term 2 because replica is either leader or believes a valid leader to be alive. I0312 20:10:31.521440 1797 leader_election.cc:377] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [CANDIDATE]: Term 2 pre-election: Vote denied by peer e3aa58d3045c4a05b94b94eb8bf58ae1. Message: Invalid argument: T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate 012786a8f1b64995847cf7dacbc029a4 for term 2 because replica is either leader or believes a valid leader to be alive. I0312 20:10:31.522485 2100 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 9da22e5afdce42078fe8f94ad166ac3c I0312 20:10:31.525887 1740 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "012786a8f1b64995847cf7dacbc029a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 53 } } ignore_live_leader: false dest_uuid: "9da22e5afdce42078fe8f94ad166ac3c" is_pre_election: true I0312 20:10:31.526506 1740 raft_consensus.cc:2176] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 012786a8f1b64995847cf7dacbc029a4 for term 2 because replica is either leader or believes a valid leader to be alive. I0312 20:10:31.527621 1797 leader_election.cc:377] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 9da22e5afdce42078fe8f94ad166ac3c. Message: Invalid argument: T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 012786a8f1b64995847cf7dacbc029a4 for term 2 because replica is either leader or believes a valid leader to be alive. I0312 20:10:31.527848 1797 leader_election.cc:258] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. I0312 20:10:31.528487 2100 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4: Snoozing failure detection for 0.058s (election complete) I0312 20:10:31.528702 2100 raft_consensus.cc:2455] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority I0312 20:10:31.709786 2101 raft_consensus.cc:436] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e3aa58d3045c4a05b94b94eb8bf58ae1) I0312 20:10:31.713173 2101 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c: Snoozing failure detection for 0.030s (starting election) I0312 20:10:31.713526 2101 raft_consensus.cc:458] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "012786a8f1b64995847cf7dacbc029a4" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 43059 } } peers { permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 } } peers { permanent_uuid: "9da22e5afdce42078fe8f94ad166ac3c" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 39175 } } I0312 20:10:31.714617 2101 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 012786a8f1b64995847cf7dacbc029a4 I0312 20:10:31.717797 1866 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "9da22e5afdce42078fe8f94ad166ac3c" candidate_term: 2 candidate_status { last_received { term: 1 index: 60 } } ignore_live_leader: false dest_uuid: "012786a8f1b64995847cf7dacbc029a4" is_pre_election: true I0312 20:10:31.718219 2101 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer e3aa58d3045c4a05b94b94eb8bf58ae1 I0312 20:10:31.718603 1866 raft_consensus.cc:2176] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9da22e5afdce42078fe8f94ad166ac3c for term 2 because replica is either leader or believes a valid leader to be alive. I0312 20:10:31.719969 1991 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "9da22e5afdce42078fe8f94ad166ac3c" candidate_term: 2 candidate_status { last_received { term: 1 index: 60 } } ignore_live_leader: false dest_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" is_pre_election: true I0312 20:10:31.720540 1991 raft_consensus.cc:2176] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate 9da22e5afdce42078fe8f94ad166ac3c for term 2 because replica is either leader or believes a valid leader to be alive. I0312 20:10:31.721746 1663 leader_election.cc:377] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [CANDIDATE]: Term 2 pre-election: Vote denied by peer e3aa58d3045c4a05b94b94eb8bf58ae1. Message: Invalid argument: T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate 9da22e5afdce42078fe8f94ad166ac3c for term 2 because replica is either leader or believes a valid leader to be alive. I0312 20:10:31.722728 1676 leader_election.cc:377] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [CANDIDATE]: Term 2 pre-election: Vote denied by peer 012786a8f1b64995847cf7dacbc029a4. Message: Invalid argument: T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9da22e5afdce42078fe8f94ad166ac3c for term 2 because replica is either leader or believes a valid leader to be alive. I0312 20:10:31.722954 1676 leader_election.cc:258] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. I0312 20:10:31.734174 2101 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c: Snoozing failure detection for 0.023s (election complete) I0312 20:10:31.734397 2101 raft_consensus.cc:2455] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority I0312 20:10:31.734935 2101 raft_consensus.cc:436] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e3aa58d3045c4a05b94b94eb8bf58ae1) I0312 20:10:31.735150 2101 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c: Snoozing failure detection for 0.045s (starting election) I0312 20:10:31.735378 2101 raft_consensus.cc:458] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "012786a8f1b64995847cf7dacbc029a4" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 43059 } } peers { permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 } } peers { permanent_uuid: "9da22e5afdce42078fe8f94ad166ac3c" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 39175 } } I0312 20:10:31.736376 2101 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 012786a8f1b64995847cf7dacbc029a4 I0312 20:10:31.736726 2101 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer e3aa58d3045c4a05b94b94eb8bf58ae1 I0312 20:10:31.738345 1991 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "9da22e5afdce42078fe8f94ad166ac3c" candidate_term: 2 candidate_status { last_received { term: 1 index: 61 } } ignore_live_leader: false dest_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" is_pre_election: true I0312 20:10:31.738865 1991 raft_consensus.cc:2176] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate 9da22e5afdce42078fe8f94ad166ac3c for term 2 because replica is either leader or believes a valid leader to be alive. I0312 20:10:31.739789 1663 leader_election.cc:377] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [CANDIDATE]: Term 2 pre-election: Vote denied by peer e3aa58d3045c4a05b94b94eb8bf58ae1. Message: Invalid argument: T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate 9da22e5afdce42078fe8f94ad166ac3c for term 2 because replica is either leader or believes a valid leader to be alive. I0312 20:10:31.746376 1866 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "9da22e5afdce42078fe8f94ad166ac3c" candidate_term: 2 candidate_status { last_received { term: 1 index: 61 } } ignore_live_leader: false dest_uuid: "012786a8f1b64995847cf7dacbc029a4" is_pre_election: true I0312 20:10:31.747426 1866 raft_consensus.cc:2176] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9da22e5afdce42078fe8f94ad166ac3c for term 2 because replica is either leader or believes a valid leader to be alive. I0312 20:10:31.748848 1676 leader_election.cc:377] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [CANDIDATE]: Term 2 pre-election: Vote denied by peer 012786a8f1b64995847cf7dacbc029a4. Message: Invalid argument: T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9da22e5afdce42078fe8f94ad166ac3c for term 2 because replica is either leader or believes a valid leader to be alive. I0312 20:10:31.749081 1676 leader_election.cc:258] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. I0312 20:10:31.749964 2101 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c: Snoozing failure detection for 0.105s (election complete) I0312 20:10:31.750196 2101 raft_consensus.cc:2455] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority I0312 20:10:31.812237 2100 raft_consensus.cc:436] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e3aa58d3045c4a05b94b94eb8bf58ae1) I0312 20:10:31.812486 2100 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4: Snoozing failure detection for 0.040s (starting election) I0312 20:10:31.812712 2100 raft_consensus.cc:458] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "012786a8f1b64995847cf7dacbc029a4" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 43059 } } peers { permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 } } peers { permanent_uuid: "9da22e5afdce42078fe8f94ad166ac3c" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 39175 } } I0312 20:10:31.814689 2100 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer e3aa58d3045c4a05b94b94eb8bf58ae1 I0312 20:10:31.815045 2100 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 9da22e5afdce42078fe8f94ad166ac3c I0312 20:10:31.816578 1741 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "012786a8f1b64995847cf7dacbc029a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 64 } } ignore_live_leader: false dest_uuid: "9da22e5afdce42078fe8f94ad166ac3c" is_pre_election: true I0312 20:10:31.817641 1741 raft_consensus.cc:2190] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 012786a8f1b64995847cf7dacbc029a4 for term 2 because replica is already servicing an update from a current leader or another vote. I0312 20:10:31.817597 1991 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "012786a8f1b64995847cf7dacbc029a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 64 } } ignore_live_leader: false dest_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" is_pre_election: true I0312 20:10:31.818120 1991 raft_consensus.cc:2176] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate 012786a8f1b64995847cf7dacbc029a4 for term 2 because replica is either leader or believes a valid leader to be alive. I0312 20:10:31.818660 1797 leader_election.cc:377] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 9da22e5afdce42078fe8f94ad166ac3c. Message: Service unavailable: T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 012786a8f1b64995847cf7dacbc029a4 for term 2 because replica is already servicing an update from a current leader or another vote. I0312 20:10:31.819157 1797 leader_election.cc:377] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [CANDIDATE]: Term 2 pre-election: Vote denied by peer e3aa58d3045c4a05b94b94eb8bf58ae1. Message: Invalid argument: T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate 012786a8f1b64995847cf7dacbc029a4 for term 2 because replica is either leader or believes a valid leader to be alive. I0312 20:10:31.819370 1797 leader_election.cc:258] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. I0312 20:10:31.820147 2100 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4: Snoozing failure detection for 0.128s (election complete) I0312 20:10:31.820355 2100 raft_consensus.cc:2455] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority I0312 20:10:31.845455 2100 raft_consensus.cc:436] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e3aa58d3045c4a05b94b94eb8bf58ae1) I0312 20:10:31.845676 2100 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4: Snoozing failure detection for 0.119s (starting election) I0312 20:10:31.845902 2100 raft_consensus.cc:458] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "012786a8f1b64995847cf7dacbc029a4" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 43059 } } peers { permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 } } peers { permanent_uuid: "9da22e5afdce42078fe8f94ad166ac3c" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 39175 } } I0312 20:10:31.846377 2102 raft_consensus.cc:436] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e3aa58d3045c4a05b94b94eb8bf58ae1) I0312 20:10:31.846626 2102 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c: Snoozing failure detection for 0.123s (starting election) I0312 20:10:31.846860 2100 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer e3aa58d3045c4a05b94b94eb8bf58ae1 I0312 20:10:31.847201 2100 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 9da22e5afdce42078fe8f94ad166ac3c I0312 20:10:31.846849 2102 raft_consensus.cc:458] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "012786a8f1b64995847cf7dacbc029a4" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 43059 } } peers { permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 } } peers { permanent_uuid: "9da22e5afdce42078fe8f94ad166ac3c" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 39175 } } I0312 20:10:31.847862 2102 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 012786a8f1b64995847cf7dacbc029a4 I0312 20:10:31.848235 2102 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer e3aa58d3045c4a05b94b94eb8bf58ae1 I0312 20:10:31.848994 1740 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "012786a8f1b64995847cf7dacbc029a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 65 } } ignore_live_leader: false dest_uuid: "9da22e5afdce42078fe8f94ad166ac3c" is_pre_election: true I0312 20:10:31.849522 1740 raft_consensus.cc:2176] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 012786a8f1b64995847cf7dacbc029a4 for term 2 because replica is either leader or believes a valid leader to be alive. I0312 20:10:31.849885 1991 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "9da22e5afdce42078fe8f94ad166ac3c" candidate_term: 2 candidate_status { last_received { term: 1 index: 65 } } ignore_live_leader: false dest_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" is_pre_election: true I0312 20:10:31.850401 1991 raft_consensus.cc:2176] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate 9da22e5afdce42078fe8f94ad166ac3c for term 2 because replica is either leader or believes a valid leader to be alive. I0312 20:10:31.850365 1797 leader_election.cc:377] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 9da22e5afdce42078fe8f94ad166ac3c. Message: Invalid argument: T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 012786a8f1b64995847cf7dacbc029a4 for term 2 because replica is either leader or believes a valid leader to be alive. I0312 20:10:31.850899 1865 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "9da22e5afdce42078fe8f94ad166ac3c" candidate_term: 2 candidate_status { last_received { term: 1 index: 65 } } ignore_live_leader: false dest_uuid: "012786a8f1b64995847cf7dacbc029a4" is_pre_election: true I0312 20:10:31.851351 1663 leader_election.cc:377] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [CANDIDATE]: Term 2 pre-election: Vote denied by peer e3aa58d3045c4a05b94b94eb8bf58ae1. Message: Invalid argument: T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate 9da22e5afdce42078fe8f94ad166ac3c for term 2 because replica is either leader or believes a valid leader to be alive. I0312 20:10:31.851495 1865 raft_consensus.cc:2176] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9da22e5afdce42078fe8f94ad166ac3c for term 2 because replica is either leader or believes a valid leader to be alive. I0312 20:10:31.862702 1676 leader_election.cc:377] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [CANDIDATE]: Term 2 pre-election: Vote denied by peer 012786a8f1b64995847cf7dacbc029a4. Message: Invalid argument: T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9da22e5afdce42078fe8f94ad166ac3c for term 2 because replica is either leader or believes a valid leader to be alive. I0312 20:10:31.862936 1676 leader_election.cc:258] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. I0312 20:10:31.862862 1991 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "012786a8f1b64995847cf7dacbc029a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 65 } } ignore_live_leader: false dest_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" is_pre_election: true I0312 20:10:31.863368 1991 raft_consensus.cc:2176] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate 012786a8f1b64995847cf7dacbc029a4 for term 2 because replica is either leader or believes a valid leader to be alive. I0312 20:10:31.864226 1797 leader_election.cc:377] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [CANDIDATE]: Term 2 pre-election: Vote denied by peer e3aa58d3045c4a05b94b94eb8bf58ae1. Message: Invalid argument: T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate 012786a8f1b64995847cf7dacbc029a4 for term 2 because replica is either leader or believes a valid leader to be alive. I0312 20:10:31.864423 1797 leader_election.cc:258] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. I0312 20:10:31.864769 2100 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4: Snoozing failure detection for 0.021s (election complete) I0312 20:10:31.864987 2100 raft_consensus.cc:2455] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority I0312 20:10:31.867795 2102 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c: Snoozing failure detection for 0.095s (election complete) I0312 20:10:31.868021 2102 raft_consensus.cc:2455] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority I0312 20:10:31.913655 2100 raft_consensus.cc:436] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e3aa58d3045c4a05b94b94eb8bf58ae1) I0312 20:10:31.913882 2100 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4: Snoozing failure detection for 0.223s (starting election) I0312 20:10:31.914093 2100 raft_consensus.cc:458] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "012786a8f1b64995847cf7dacbc029a4" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 43059 } } peers { permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 } } peers { permanent_uuid: "9da22e5afdce42078fe8f94ad166ac3c" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 39175 } } I0312 20:10:31.915027 2100 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer e3aa58d3045c4a05b94b94eb8bf58ae1 I0312 20:10:31.915380 2100 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 9da22e5afdce42078fe8f94ad166ac3c I0312 20:10:31.915606 2102 raft_consensus.cc:436] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e3aa58d3045c4a05b94b94eb8bf58ae1) I0312 20:10:31.915835 2102 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c: Snoozing failure detection for 0.152s (starting election) I0312 20:10:31.916054 2102 raft_consensus.cc:458] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "012786a8f1b64995847cf7dacbc029a4" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 43059 } } peers { permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 } } peers { permanent_uuid: "9da22e5afdce42078fe8f94ad166ac3c" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 39175 } } I0312 20:10:31.918071 2102 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 012786a8f1b64995847cf7dacbc029a4 I0312 20:10:31.918426 2102 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer e3aa58d3045c4a05b94b94eb8bf58ae1 I0312 20:10:31.917886 1741 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "012786a8f1b64995847cf7dacbc029a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 67 } } ignore_live_leader: false dest_uuid: "9da22e5afdce42078fe8f94ad166ac3c" is_pre_election: true I0312 20:10:31.919231 1741 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c: Snoozing failure detection for 0.238s (vote granted) I0312 20:10:31.919427 1741 raft_consensus.cc:2215] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 012786a8f1b64995847cf7dacbc029a4 in term 1. I0312 20:10:31.920629 1797 leader_election.cc:363] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [CANDIDATE]: Term 2 pre-election: Vote granted by peer 9da22e5afdce42078fe8f94ad166ac3c I0312 20:10:31.920816 1797 leader_election.cc:258] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. I0312 20:10:31.922051 1991 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "012786a8f1b64995847cf7dacbc029a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 67 } } ignore_live_leader: false dest_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" is_pre_election: true I0312 20:10:31.922631 1991 raft_consensus.cc:2176] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate 012786a8f1b64995847cf7dacbc029a4 for term 2 because replica is either leader or believes a valid leader to be alive. I0312 20:10:31.924685 1991 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "9da22e5afdce42078fe8f94ad166ac3c" candidate_term: 2 candidate_status { last_received { term: 1 index: 67 } } ignore_live_leader: false dest_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" is_pre_election: true I0312 20:10:31.925241 1991 raft_consensus.cc:2176] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate 9da22e5afdce42078fe8f94ad166ac3c for term 2 because replica is either leader or believes a valid leader to be alive. I0312 20:10:31.925848 2100 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4: Snoozing failure detection for 0.381s (election complete) I0312 20:10:31.925909 1797 leader_election.cc:377] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [CANDIDATE]: Term 2 pre-election: Vote denied by peer e3aa58d3045c4a05b94b94eb8bf58ae1. Message: Invalid argument: T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate 012786a8f1b64995847cf7dacbc029a4 for term 2 because replica is either leader or believes a valid leader to be alive. I0312 20:10:31.926066 2100 raft_consensus.cc:2507] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 1 FOLLOWER]: Leader pre-election won for term 2 I0312 20:10:31.926278 2100 raft_consensus.cc:436] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 1 FOLLOWER]: Starting leader election (detected failure of leader e3aa58d3045c4a05b94b94eb8bf58ae1) I0312 20:10:31.926542 2100 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4: Snoozing failure detection for 0.320s (starting election) I0312 20:10:31.926775 2100 raft_consensus.cc:2752] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 1 FOLLOWER]: Advancing to term 2 I0312 20:10:31.931398 2100 raft_consensus.cc:458] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "012786a8f1b64995847cf7dacbc029a4" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 43059 } } peers { permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 } } peers { permanent_uuid: "9da22e5afdce42078fe8f94ad166ac3c" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 39175 } } I0312 20:10:31.932890 1663 leader_election.cc:377] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [CANDIDATE]: Term 2 pre-election: Vote denied by peer e3aa58d3045c4a05b94b94eb8bf58ae1. Message: Invalid argument: T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate 9da22e5afdce42078fe8f94ad166ac3c for term 2 because replica is either leader or believes a valid leader to be alive. I0312 20:10:31.933701 1865 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "9da22e5afdce42078fe8f94ad166ac3c" candidate_term: 2 candidate_status { last_received { term: 1 index: 67 } } ignore_live_leader: false dest_uuid: "012786a8f1b64995847cf7dacbc029a4" is_pre_election: true I0312 20:10:31.960160 2100 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [CANDIDATE]: Term 2 election: Requesting vote from peer e3aa58d3045c4a05b94b94eb8bf58ae1 I0312 20:10:31.960319 1865 raft_consensus.cc:2145] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9da22e5afdce42078fe8f94ad166ac3c in current term 2: Already voted for candidate 012786a8f1b64995847cf7dacbc029a4 in this term. I0312 20:10:31.961273 1866 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 2 FOLLOWER]: Rejecting Update request from peer e3aa58d3045c4a05b94b94eb8bf58ae1 for earlier term 1. Current term is 2. Ops: [] I0312 20:10:31.961710 2100 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [CANDIDATE]: Term 2 election: Requesting vote from peer 9da22e5afdce42078fe8f94ad166ac3c I0312 20:10:31.961628 1991 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "012786a8f1b64995847cf7dacbc029a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 67 } } ignore_live_leader: false dest_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" I0312 20:10:31.962165 1991 raft_consensus.cc:2176] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 1 LEADER]: Leader election vote request: Denying vote to candidate 012786a8f1b64995847cf7dacbc029a4 for term 2 because replica is either leader or believes a valid leader to be alive. I0312 20:10:31.963057 1797 leader_election.cc:377] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [CANDIDATE]: Term 2 election: Vote denied by peer e3aa58d3045c4a05b94b94eb8bf58ae1. Message: Invalid argument: T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 1 LEADER]: Leader election vote request: Denying vote to candidate 012786a8f1b64995847cf7dacbc029a4 for term 2 because replica is either leader or believes a valid leader to be alive. I0312 20:10:31.963656 1676 leader_election.cc:377] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [CANDIDATE]: Term 2 pre-election: Vote denied by peer 012786a8f1b64995847cf7dacbc029a4. Message: Invalid argument: T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9da22e5afdce42078fe8f94ad166ac3c in current term 2: Already voted for candidate 012786a8f1b64995847cf7dacbc029a4 in this term. I0312 20:10:31.963898 1676 leader_election.cc:258] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. I0312 20:10:31.963726 1740 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "012786a8f1b64995847cf7dacbc029a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 67 } } ignore_live_leader: false dest_uuid: "9da22e5afdce42078fe8f94ad166ac3c" I0312 20:10:31.964058 2102 raft_consensus.cc:436] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e3aa58d3045c4a05b94b94eb8bf58ae1) I0312 20:10:31.964301 2102 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c: Snoozing failure detection for 0.142s (starting election) I0312 20:10:31.964645 2063 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "012786a8f1b64995847cf7dacbc029a4" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 43059 }, Status: INVALID_TERM, Last received: 1.67, Next index: 68, Last known committed idx: 67, Time since last communication: 0.000s I0312 20:10:31.964512 2102 raft_consensus.cc:458] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "012786a8f1b64995847cf7dacbc029a4" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 43059 } } peers { permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 } } peers { permanent_uuid: "9da22e5afdce42078fe8f94ad166ac3c" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 39175 } } I0312 20:10:31.965469 2102 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 012786a8f1b64995847cf7dacbc029a4 I0312 20:10:31.965502 1740 raft_consensus.cc:2752] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 1 FOLLOWER]: Advancing to term 2 I0312 20:10:31.965873 2102 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer e3aa58d3045c4a05b94b94eb8bf58ae1 I0312 20:10:31.967406 1991 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "9da22e5afdce42078fe8f94ad166ac3c" candidate_term: 2 candidate_status { last_received { term: 1 index: 69 } } ignore_live_leader: false dest_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" is_pre_election: true I0312 20:10:31.967897 1991 raft_consensus.cc:2176] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate 9da22e5afdce42078fe8f94ad166ac3c for term 2 because replica is either leader or believes a valid leader to be alive. I0312 20:10:31.968086 1866 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 2 FOLLOWER]: Rejecting Update request from peer e3aa58d3045c4a05b94b94eb8bf58ae1 for earlier term 1. Current term is 2. Ops: [] I0312 20:10:31.968809 1663 leader_election.cc:377] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [CANDIDATE]: Term 2 pre-election: Vote denied by peer e3aa58d3045c4a05b94b94eb8bf58ae1. Message: Invalid argument: T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate 9da22e5afdce42078fe8f94ad166ac3c for term 2 because replica is either leader or believes a valid leader to be alive. I0312 20:10:31.969272 1740 raft_consensus.cc:2162] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 012786a8f1b64995847cf7dacbc029a4 for term 2 because replica has last-logged OpId of term: 1 index: 69, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 67. I0312 20:10:31.969099 1866 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "9da22e5afdce42078fe8f94ad166ac3c" candidate_term: 2 candidate_status { last_received { term: 1 index: 69 } } ignore_live_leader: false dest_uuid: "012786a8f1b64995847cf7dacbc029a4" is_pre_election: true I0312 20:10:31.969583 1866 raft_consensus.cc:2145] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9da22e5afdce42078fe8f94ad166ac3c in current term 2: Already voted for candidate 012786a8f1b64995847cf7dacbc029a4 in this term. I0312 20:10:31.969547 2101 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c: Snoozing failure detection for 0.144s (election complete) I0312 20:10:31.969744 2101 raft_consensus.cc:2455] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority I0312 20:10:31.970441 1741 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 2 FOLLOWER]: Rejecting Update request from peer e3aa58d3045c4a05b94b94eb8bf58ae1 for earlier term 1. Current term is 2. Ops: [] I0312 20:10:31.970610 1676 leader_election.cc:377] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [CANDIDATE]: Term 2 pre-election: Vote denied by peer 012786a8f1b64995847cf7dacbc029a4. Message: Invalid argument: T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9da22e5afdce42078fe8f94ad166ac3c in current term 2: Already voted for candidate 012786a8f1b64995847cf7dacbc029a4 in this term. I0312 20:10:31.970827 1676 leader_election.cc:258] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. I0312 20:10:31.971735 1797 leader_election.cc:377] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [CANDIDATE]: Term 2 election: Vote denied by peer 9da22e5afdce42078fe8f94ad166ac3c. Message: Invalid argument: T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 012786a8f1b64995847cf7dacbc029a4 for term 2 because replica has last-logged OpId of term: 1 index: 69, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 67. I0312 20:10:31.971969 1797 leader_election.cc:258] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. I0312 20:10:31.971846 2063 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9da22e5afdce42078fe8f94ad166ac3c" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 39175 }, Status: INVALID_TERM, Last received: 1.69, Next index: 70, Last known committed idx: 69, Time since last communication: 0.000s I0312 20:10:31.972559 2101 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c: Snoozing failure detection for 0.191s (election complete) I0312 20:10:31.972765 2101 raft_consensus.cc:2455] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority I0312 20:10:31.973745 2075 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "012786a8f1b64995847cf7dacbc029a4" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 43059 }, Status: INVALID_TERM, Last received: 1.67, Next index: 68, Last known committed idx: 67, Time since last communication: 0.000s I0312 20:10:31.974275 2100 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4: Snoozing failure detection for 0.317s (election complete) I0312 20:10:31.974480 2100 raft_consensus.cc:2455] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority I0312 20:10:31.978276 1866 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 2 FOLLOWER]: Rejecting Update request from peer e3aa58d3045c4a05b94b94eb8bf58ae1 for earlier term 1. Current term is 2. Ops: [] I0312 20:10:31.979465 2063 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "012786a8f1b64995847cf7dacbc029a4" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 43059 }, Status: INVALID_TERM, Last received: 1.67, Next index: 68, Last known committed idx: 67, Time since last communication: 0.000s I0312 20:10:31.983261 1866 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 2 FOLLOWER]: Rejecting Update request from peer e3aa58d3045c4a05b94b94eb8bf58ae1 for earlier term 1. Current term is 2. Ops: [] I0312 20:10:31.984436 2063 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "012786a8f1b64995847cf7dacbc029a4" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 43059 }, Status: INVALID_TERM, Last received: 1.67, Next index: 68, Last known committed idx: 67, Time since last communication: 0.000s I0312 20:10:31.988481 1866 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 2 FOLLOWER]: Rejecting Update request from peer e3aa58d3045c4a05b94b94eb8bf58ae1 for earlier term 1. Current term is 2. Ops: [] I0312 20:10:31.989748 2063 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "012786a8f1b64995847cf7dacbc029a4" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 43059 }, Status: INVALID_TERM, Last received: 1.67, Next index: 68, Last known committed idx: 67, Time since last communication: 0.000s I0312 20:10:31.993150 1866 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 2 FOLLOWER]: Rejecting Update request from peer e3aa58d3045c4a05b94b94eb8bf58ae1 for earlier term 1. Current term is 2. Ops: [] I0312 20:10:31.994343 2063 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "012786a8f1b64995847cf7dacbc029a4" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 43059 }, Status: INVALID_TERM, Last received: 1.67, Next index: 68, Last known committed idx: 67, Time since last communication: 0.000s I0312 20:10:32.001339 1866 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 2 FOLLOWER]: Rejecting Update request from peer e3aa58d3045c4a05b94b94eb8bf58ae1 for earlier term 1. Current term is 2. Ops: [] I0312 20:10:32.002517 2063 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "012786a8f1b64995847cf7dacbc029a4" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 43059 }, Status: INVALID_TERM, Last received: 1.67, Next index: 68, Last known committed idx: 67, Time since last communication: 0.000s I0312 20:10:32.006078 1866 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 2 FOLLOWER]: Rejecting Update request from peer e3aa58d3045c4a05b94b94eb8bf58ae1 for earlier term 1. Current term is 2. Ops: [] I0312 20:10:32.008432 2063 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "012786a8f1b64995847cf7dacbc029a4" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 43059 }, Status: INVALID_TERM, Last received: 1.67, Next index: 68, Last known committed idx: 67, Time since last communication: 0.001s I0312 20:10:32.009677 2091 raft_consensus.cc:2747] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 1 LEADER]: Stepping down as leader of term 1 I0312 20:10:32.009874 2091 raft_consensus.cc:605] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e3aa58d3045c4a05b94b94eb8bf58ae1, State: Running, Role: LEADER I0312 20:10:32.010298 2091 consensus_queue.cc:219] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 69, Committed index: 69, Last appended: 1.71, Last appended by leader: 71, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "012786a8f1b64995847cf7dacbc029a4" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 43059 } } peers { permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 } } peers { permanent_uuid: "9da22e5afdce42078fe8f94ad166ac3c" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 39175 } } I0312 20:10:32.010952 2091 consensus_peers.cc:458] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 -> Peer 9da22e5afdce42078fe8f94ad166ac3c (127.0.53.1:39175): Closing peer: 9da22e5afdce42078fe8f94ad166ac3c I0312 20:10:32.011162 2091 consensus_peers.cc:458] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 -> Peer 012786a8f1b64995847cf7dacbc029a4 (127.0.53.2:43059): Closing peer: 012786a8f1b64995847cf7dacbc029a4 I0312 20:10:32.011167 1740 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 2 FOLLOWER]: Rejecting Update request from peer e3aa58d3045c4a05b94b94eb8bf58ae1 for earlier term 1. Current term is 2. Ops: [] I0312 20:10:32.011497 2091 raft_consensus.cc:2752] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 1 FOLLOWER]: Advancing to term 2 I0312 20:10:32.021896 2064 raft_consensus.cc:436] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0312 20:10:32.022106 2064 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1: Snoozing failure detection for 0.021s (starting election) I0312 20:10:32.022310 2064 raft_consensus.cc:458] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "012786a8f1b64995847cf7dacbc029a4" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 43059 } } peers { permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 } } peers { permanent_uuid: "9da22e5afdce42078fe8f94ad166ac3c" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 39175 } } I0312 20:10:32.023201 2064 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 3 pre-election: Requesting pre-vote from peer 012786a8f1b64995847cf7dacbc029a4 I0312 20:10:32.023502 2064 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 3 pre-election: Requesting pre-vote from peer 9da22e5afdce42078fe8f94ad166ac3c I0312 20:10:32.024435 1866 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" candidate_term: 3 candidate_status { last_received { term: 1 index: 71 } } ignore_live_leader: false dest_uuid: "012786a8f1b64995847cf7dacbc029a4" is_pre_election: true I0312 20:10:32.024881 1866 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4: Snoozing failure detection for 0.330s (vote granted) I0312 20:10:32.024729 1740 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" candidate_term: 3 candidate_status { last_received { term: 1 index: 71 } } ignore_live_leader: false dest_uuid: "9da22e5afdce42078fe8f94ad166ac3c" is_pre_election: true I0312 20:10:32.025168 1740 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c: Snoozing failure detection for 0.191s (vote granted) I0312 20:10:32.025133 1866 raft_consensus.cc:2215] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3aa58d3045c4a05b94b94eb8bf58ae1 in term 2. I0312 20:10:32.025367 1740 raft_consensus.cc:2215] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3aa58d3045c4a05b94b94eb8bf58ae1 in term 2. I0312 20:10:32.026106 1926 leader_election.cc:363] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 3 pre-election: Vote granted by peer 012786a8f1b64995847cf7dacbc029a4 I0312 20:10:32.026326 1926 leader_election.cc:258] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. I0312 20:10:32.026551 1922 leader_election.cc:363] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 3 pre-election: Vote granted by peer 9da22e5afdce42078fe8f94ad166ac3c I0312 20:10:32.026758 2064 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1: Snoozing failure detection for 0.021s (election complete) I0312 20:10:32.026974 2064 raft_consensus.cc:2507] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 2 FOLLOWER]: Leader pre-election won for term 3 I0312 20:10:32.027168 2064 raft_consensus.cc:436] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0312 20:10:32.027325 2064 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1: Snoozing failure detection for 0.016s (starting election) I0312 20:10:32.027498 2064 raft_consensus.cc:2752] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 2 FOLLOWER]: Advancing to term 3 I0312 20:10:32.216840 2101 raft_consensus.cc:436] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0312 20:10:32.217123 2101 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c: Snoozing failure detection for 0.231s (starting election) I0312 20:10:32.217371 2101 raft_consensus.cc:458] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "012786a8f1b64995847cf7dacbc029a4" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 43059 } } peers { permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 } } peers { permanent_uuid: "9da22e5afdce42078fe8f94ad166ac3c" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 39175 } } I0312 20:10:32.218451 2101 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [CANDIDATE]: Term 3 pre-election: Requesting pre-vote from peer 012786a8f1b64995847cf7dacbc029a4 I0312 20:10:32.218844 2101 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [CANDIDATE]: Term 3 pre-election: Requesting pre-vote from peer e3aa58d3045c4a05b94b94eb8bf58ae1 I0312 20:10:32.219928 1866 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "9da22e5afdce42078fe8f94ad166ac3c" candidate_term: 3 candidate_status { last_received { term: 1 index: 69 } } ignore_live_leader: false dest_uuid: "012786a8f1b64995847cf7dacbc029a4" is_pre_election: true I0312 20:10:32.220173 1991 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "9da22e5afdce42078fe8f94ad166ac3c" candidate_term: 3 candidate_status { last_received { term: 1 index: 69 } } ignore_live_leader: false dest_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" is_pre_election: true I0312 20:10:32.220422 1866 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4: Snoozing failure detection for 0.055s (vote granted) I0312 20:10:32.220631 1866 raft_consensus.cc:2215] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9da22e5afdce42078fe8f94ad166ac3c in term 2. I0312 20:10:32.221876 1676 leader_election.cc:363] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [CANDIDATE]: Term 3 pre-election: Vote granted by peer 012786a8f1b64995847cf7dacbc029a4 I0312 20:10:32.222082 1676 leader_election.cc:258] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. I0312 20:10:32.222780 2101 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c: Snoozing failure detection for 0.548s (election complete) I0312 20:10:32.223009 2101 raft_consensus.cc:2507] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 2 FOLLOWER]: Leader pre-election won for term 3 I0312 20:10:32.223201 2101 raft_consensus.cc:436] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0312 20:10:32.223448 2101 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c: Snoozing failure detection for 0.041s (starting election) I0312 20:10:32.223739 2101 raft_consensus.cc:2752] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 2 FOLLOWER]: Advancing to term 3 I0312 20:10:32.276255 2100 raft_consensus.cc:436] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0312 20:10:32.276491 2100 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4: Snoozing failure detection for 0.146s (starting election) I0312 20:10:32.276727 2100 raft_consensus.cc:458] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "012786a8f1b64995847cf7dacbc029a4" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 43059 } } peers { permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 } } peers { permanent_uuid: "9da22e5afdce42078fe8f94ad166ac3c" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 39175 } } I0312 20:10:32.277757 2100 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [CANDIDATE]: Term 3 pre-election: Requesting pre-vote from peer e3aa58d3045c4a05b94b94eb8bf58ae1 I0312 20:10:32.278142 2100 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [CANDIDATE]: Term 3 pre-election: Requesting pre-vote from peer 9da22e5afdce42078fe8f94ad166ac3c I0312 20:10:32.279553 1740 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "012786a8f1b64995847cf7dacbc029a4" candidate_term: 3 candidate_status { last_received { term: 1 index: 67 } } ignore_live_leader: false dest_uuid: "9da22e5afdce42078fe8f94ad166ac3c" is_pre_election: true I0312 20:10:32.279722 1990 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "012786a8f1b64995847cf7dacbc029a4" candidate_term: 3 candidate_status { last_received { term: 1 index: 67 } } ignore_live_leader: false dest_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" is_pre_election: true I0312 20:10:32.423663 2100 raft_consensus.cc:436] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0312 20:10:32.423907 2100 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4: Snoozing failure detection for 0.570s (starting election) I0312 20:10:32.424173 2100 raft_consensus.cc:458] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "012786a8f1b64995847cf7dacbc029a4" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 43059 } } peers { permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 } } peers { permanent_uuid: "9da22e5afdce42078fe8f94ad166ac3c" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 39175 } } I0312 20:10:32.425541 2100 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [CANDIDATE]: Term 3 pre-election: Requesting pre-vote from peer e3aa58d3045c4a05b94b94eb8bf58ae1 W0312 20:10:32.425627 1797 leader_election.cc:287] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e3aa58d3045c4a05b94b94eb8bf58ae1: Timed out: RequestConsensusVote RPC to 127.0.53.3:49316 timed out after 0.146s (SENT) I0312 20:10:32.425928 2100 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [CANDIDATE]: Term 3 pre-election: Requesting pre-vote from peer 9da22e5afdce42078fe8f94ad166ac3c W0312 20:10:32.426035 1797 leader_election.cc:287] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9da22e5afdce42078fe8f94ad166ac3c: Timed out: RequestConsensusVote RPC to 127.0.53.1:39175 timed out after 0.146s (SENT) I0312 20:10:32.426265 1797 leader_election.cc:258] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. I0312 20:10:32.428120 1741 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "012786a8f1b64995847cf7dacbc029a4" candidate_term: 3 candidate_status { last_received { term: 1 index: 67 } } ignore_live_leader: false dest_uuid: "9da22e5afdce42078fe8f94ad166ac3c" is_pre_election: true I0312 20:10:32.428293 1989 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "012786a8f1b64995847cf7dacbc029a4" candidate_term: 3 candidate_status { last_received { term: 1 index: 67 } } ignore_live_leader: false dest_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" is_pre_election: true I0312 20:10:32.429138 2100 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4: Snoozing failure detection for 0.338s (election complete) I0312 20:10:32.429370 2100 raft_consensus.cc:2455] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority W0312 20:10:32.451373 1663 leader_election.cc:287] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e3aa58d3045c4a05b94b94eb8bf58ae1: Timed out: RequestConsensusVote RPC to 127.0.53.3:49316 timed out after 0.231s (SENT) I0312 20:10:32.768532 2100 raft_consensus.cc:436] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0312 20:10:32.768802 2100 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4: Snoozing failure detection for 0.318s (starting election) I0312 20:10:32.769057 2100 raft_consensus.cc:458] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "012786a8f1b64995847cf7dacbc029a4" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 43059 } } peers { permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 } } peers { permanent_uuid: "9da22e5afdce42078fe8f94ad166ac3c" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 39175 } } I0312 20:10:32.770150 2100 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [CANDIDATE]: Term 3 pre-election: Requesting pre-vote from peer e3aa58d3045c4a05b94b94eb8bf58ae1 I0312 20:10:32.770565 2100 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [CANDIDATE]: Term 3 pre-election: Requesting pre-vote from peer 9da22e5afdce42078fe8f94ad166ac3c I0312 20:10:32.771594 1988 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "012786a8f1b64995847cf7dacbc029a4" candidate_term: 3 candidate_status { last_received { term: 1 index: 67 } } ignore_live_leader: false dest_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" is_pre_election: true I0312 20:10:32.771885 1739 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "012786a8f1b64995847cf7dacbc029a4" candidate_term: 3 candidate_status { last_received { term: 1 index: 67 } } ignore_live_leader: false dest_uuid: "9da22e5afdce42078fe8f94ad166ac3c" is_pre_election: true W0312 20:10:32.775557 2064 consensus_meta.cc:216] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1: Time spent flushing consensus metadata: real 0.748s user 0.004s sys 0.000s W0312 20:10:32.775553 2101 consensus_meta.cc:216] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c: Time spent flushing consensus metadata: real 0.551s user 0.004s sys 0.000s I0312 20:10:32.775815 2101 raft_consensus.cc:458] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "012786a8f1b64995847cf7dacbc029a4" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 43059 } } peers { permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 } } peers { permanent_uuid: "9da22e5afdce42078fe8f94ad166ac3c" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 39175 } } I0312 20:10:32.775815 2064 raft_consensus.cc:458] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "012786a8f1b64995847cf7dacbc029a4" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 43059 } } peers { permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 } } peers { permanent_uuid: "9da22e5afdce42078fe8f94ad166ac3c" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 39175 } } I0312 20:10:32.776825 2101 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [CANDIDATE]: Term 3 election: Requesting vote from peer 012786a8f1b64995847cf7dacbc029a4 I0312 20:10:32.777003 1740 raft_consensus.cc:2145] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 012786a8f1b64995847cf7dacbc029a4 in current term 3: Already voted for candidate 9da22e5afdce42078fe8f94ad166ac3c in this term. I0312 20:10:32.777230 2101 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [CANDIDATE]: Term 3 election: Requesting vote from peer e3aa58d3045c4a05b94b94eb8bf58ae1 I0312 20:10:32.777240 2102 raft_consensus.cc:436] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0312 20:10:32.777519 2102 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c: Snoozing failure detection for 0.162s (starting election) W0312 20:10:32.777506 1740 rpcz_store.cc:251] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.53.2:55850 (request call id 13) took 498ms (client timeout 146). I0312 20:10:32.778595 1866 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "9da22e5afdce42078fe8f94ad166ac3c" candidate_term: 3 candidate_status { last_received { term: 1 index: 69 } } ignore_live_leader: false dest_uuid: "012786a8f1b64995847cf7dacbc029a4" W0312 20:10:32.778993 1740 rpcz_store.cc:255] Trace: 0312 20:10:32.279292 (+ 0us) service_pool.cc:163] Inserting onto call queue 0312 20:10:32.279406 (+ 114us) service_pool.cc:222] Handling call 0312 20:10:32.777345 (+497939us) spinlock_profiling.cc:261] Waited 497 ms on lock 0x7b480003c39c. stack: 00007fede5da3c74 00007fede5da3b70 00007fede4893314 00007feded8e0294 00007feded8e0229 00007fedec48aaf3 00007feded91e56b 00007fede82d1b24 00007fede82d1ab1 00007fede82d19c4 00007fede792dc29 00007fede792d6ec 00007fede792ebcd 00007fede7930946 00007fede793089a 00007fede7930823 0312 20:10:32.777447 (+ 102us) inbound_call.cc:157] Queueing success response Metrics: {"spinlock_wait_cycles":1143114496} I0312 20:10:32.779003 1866 raft_consensus.cc:2752] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 2 FOLLOWER]: Advancing to term 3 I0312 20:10:32.779170 1866 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4: Snoozing failure detection for 0.074s (vote granted) I0312 20:10:32.777761 2102 raft_consensus.cc:458] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "012786a8f1b64995847cf7dacbc029a4" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 43059 } } peers { permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 } } peers { permanent_uuid: "9da22e5afdce42078fe8f94ad166ac3c" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 39175 } } I0312 20:10:32.782313 1866 raft_consensus.cc:2215] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9da22e5afdce42078fe8f94ad166ac3c in term 3. I0312 20:10:32.782956 2064 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 3 election: Requesting vote from peer 012786a8f1b64995847cf7dacbc029a4 I0312 20:10:32.783090 1741 raft_consensus.cc:2190] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 012786a8f1b64995847cf7dacbc029a4 for term 3 because replica is already servicing an update from a current leader or another vote. I0312 20:10:32.783138 1676 leader_election.cc:363] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [CANDIDATE]: Term 3 election: Vote granted by peer 012786a8f1b64995847cf7dacbc029a4 I0312 20:10:32.783363 2064 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 3 election: Requesting vote from peer 9da22e5afdce42078fe8f94ad166ac3c I0312 20:10:32.783357 1676 leader_election.cc:258] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. I0312 20:10:32.783780 1739 raft_consensus.cc:2190] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 012786a8f1b64995847cf7dacbc029a4 for term 3 because replica is already servicing an update from a current leader or another vote. I0312 20:10:32.784152 2101 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c: Snoozing failure detection for 0.541s (election complete) I0312 20:10:32.784396 2101 raft_consensus.cc:2507] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 3 FOLLOWER]: Leader election won for term 3 I0312 20:10:32.784668 1988 raft_consensus.cc:2190] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 012786a8f1b64995847cf7dacbc029a4 for term 3 because replica is already servicing an update from a current leader or another vote. I0312 20:10:32.784726 2101 raft_consensus.cc:572] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 3 LEADER]: Becoming Leader. State: Replica: 9da22e5afdce42078fe8f94ad166ac3c, State: Running, Role: LEADER I0312 20:10:32.784958 2102 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [CANDIDATE]: Term 4 pre-election: Requesting pre-vote from peer 012786a8f1b64995847cf7dacbc029a4 I0312 20:10:32.785506 2102 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [CANDIDATE]: Term 4 pre-election: Requesting pre-vote from peer e3aa58d3045c4a05b94b94eb8bf58ae1 I0312 20:10:32.785733 1797 leader_election.cc:377] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [CANDIDATE]: Term 3 pre-election: Vote denied by peer 9da22e5afdce42078fe8f94ad166ac3c. Message: Service unavailable: T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 012786a8f1b64995847cf7dacbc029a4 for term 3 because replica is already servicing an update from a current leader or another vote. I0312 20:10:32.785856 1991 raft_consensus.cc:2145] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9da22e5afdce42078fe8f94ad166ac3c in current term 3: Already voted for candidate e3aa58d3045c4a05b94b94eb8bf58ae1 in this term. W0312 20:10:32.786315 1991 rpcz_store.cc:251] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.53.1:52135 (request call id 11) took 566ms (client timeout 231). I0312 20:10:32.786535 1865 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "9da22e5afdce42078fe8f94ad166ac3c" candidate_term: 4 candidate_status { last_received { term: 1 index: 69 } } ignore_live_leader: false dest_uuid: "012786a8f1b64995847cf7dacbc029a4" is_pre_election: true I0312 20:10:32.786957 1865 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4: Snoozing failure detection for 0.627s (vote granted) W0312 20:10:32.787081 1991 rpcz_store.cc:255] Trace: 0312 20:10:32.219916 (+ 0us) service_pool.cc:163] Inserting onto call queue 0312 20:10:32.220039 (+ 123us) service_pool.cc:222] Handling call 0312 20:10:32.786173 (+566134us) spinlock_profiling.cc:261] Waited 565 ms on lock 0x7b480003039c. stack: 00007efd9a88dc74 00007efd9a88db70 00007efd9937d314 00007efda23ca294 00007efda23ca229 00007efda0f74af3 00007efda240856b 00007efd9cdbbb24 00007efd9cdbbab1 00007efd9cdbb9c4 00007efd9c417c29 00007efd9c4176ec 00007efd9c418bcd 00007efd9c41a946 00007efd9c41a89a 00007efd9c41a823 0312 20:10:32.786260 (+ 87us) inbound_call.cc:157] Queueing success response Metrics: {"spinlock_wait_cycles":1300291968} I0312 20:10:32.787144 1865 raft_consensus.cc:2215] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9da22e5afdce42078fe8f94ad166ac3c in term 3. I0312 20:10:32.787497 1990 raft_consensus.cc:2190] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 012786a8f1b64995847cf7dacbc029a4 for term 3 because replica is already servicing an update from a current leader or another vote. I0312 20:10:32.787691 2075 raft_consensus.cc:436] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0312 20:10:32.787892 2075 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1: Snoozing failure detection for 0.022s (starting election) W0312 20:10:32.787964 1990 rpcz_store.cc:251] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.53.2:45159 (request call id 13) took 508ms (client timeout 146). I0312 20:10:32.788071 1797 leader_election.cc:377] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [CANDIDATE]: Term 3 pre-election: Vote denied by peer e3aa58d3045c4a05b94b94eb8bf58ae1. Message: Service unavailable: T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 012786a8f1b64995847cf7dacbc029a4 for term 3 because replica is already servicing an update from a current leader or another vote. I0312 20:10:32.788447 1797 leader_election.cc:377] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [CANDIDATE]: Term 3 pre-election: Vote denied by peer 9da22e5afdce42078fe8f94ad166ac3c. Message: Service unavailable: T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 012786a8f1b64995847cf7dacbc029a4 for term 3 because replica is already servicing an update from a current leader or another vote. I0312 20:10:32.788116 2075 raft_consensus.cc:458] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "012786a8f1b64995847cf7dacbc029a4" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 43059 } } peers { permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 } } peers { permanent_uuid: "9da22e5afdce42078fe8f94ad166ac3c" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 39175 } } I0312 20:10:32.788635 1797 leader_election.cc:258] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. I0312 20:10:32.788841 1866 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" candidate_term: 3 candidate_status { last_received { term: 1 index: 71 } } ignore_live_leader: false dest_uuid: "012786a8f1b64995847cf7dacbc029a4" I0312 20:10:32.789393 1866 raft_consensus.cc:2145] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate e3aa58d3045c4a05b94b94eb8bf58ae1 in current term 3: Already voted for candidate 9da22e5afdce42078fe8f94ad166ac3c in this term. I0312 20:10:32.789491 2075 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 4 pre-election: Requesting pre-vote from peer 012786a8f1b64995847cf7dacbc029a4 I0312 20:10:32.789829 2075 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 4 pre-election: Requesting pre-vote from peer 9da22e5afdce42078fe8f94ad166ac3c I0312 20:10:32.786494 2101 consensus_queue.cc:196] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 69, Committed index: 69, Last appended: 1.69, Last appended by leader: 69, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "012786a8f1b64995847cf7dacbc029a4" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 43059 } } peers { permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 } } peers { permanent_uuid: "9da22e5afdce42078fe8f94ad166ac3c" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 39175 } } I0312 20:10:32.790403 1991 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "9da22e5afdce42078fe8f94ad166ac3c" candidate_term: 3 candidate_status { last_received { term: 1 index: 69 } } ignore_live_leader: false dest_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" I0312 20:10:32.790616 1926 leader_election.cc:377] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 3 election: Vote denied by peer 012786a8f1b64995847cf7dacbc029a4. Message: Invalid argument: T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate e3aa58d3045c4a05b94b94eb8bf58ae1 in current term 3: Already voted for candidate 9da22e5afdce42078fe8f94ad166ac3c in this term. I0312 20:10:32.789424 1676 leader_election.cc:363] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [CANDIDATE]: Term 4 pre-election: Vote granted by peer 012786a8f1b64995847cf7dacbc029a4 I0312 20:10:32.790920 1991 raft_consensus.cc:2145] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 9da22e5afdce42078fe8f94ad166ac3c in current term 3: Already voted for candidate e3aa58d3045c4a05b94b94eb8bf58ae1 in this term. I0312 20:10:32.790923 1676 leader_election.cc:258] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. I0312 20:10:32.791664 1989 raft_consensus.cc:2190] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 012786a8f1b64995847cf7dacbc029a4 for term 3 because replica is already servicing an update from a current leader or another vote. W0312 20:10:32.792503 1990 rpcz_store.cc:255] Trace: 0312 20:10:32.279450 (+ 0us) service_pool.cc:163] Inserting onto call queue 0312 20:10:32.279555 (+ 105us) service_pool.cc:222] Handling call 0312 20:10:32.787828 (+508273us) spinlock_profiling.cc:261] Waited 507 ms on lock 0x7b480003039c. stack: 00007efd9a88dc74 00007efd9a88db70 00007efd9937d314 00007efda23ca294 00007efda23ca229 00007efda0f74606 00007efda240856b 00007efd9cdbbb24 00007efd9cdbbab1 00007efd9cdbb9c4 00007efd9c417c29 00007efd9c4176ec 00007efd9c418bcd 00007efd9c41a946 00007efd9c41a89a 00007efd9c41a823 0312 20:10:32.787920 (+ 92us) inbound_call.cc:157] Queueing success response Metrics: {"spinlock_wait_cycles":1166911232} I0312 20:10:32.792321 1739 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" candidate_term: 3 candidate_status { last_received { term: 1 index: 71 } } ignore_live_leader: false dest_uuid: "9da22e5afdce42078fe8f94ad166ac3c" I0312 20:10:32.793243 1739 raft_consensus.cc:2176] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 3 LEADER]: Leader election vote request: Denying vote to candidate e3aa58d3045c4a05b94b94eb8bf58ae1 for term 3 because replica is either leader or believes a valid leader to be alive. I0312 20:10:32.793265 1797 leader_election.cc:377] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [CANDIDATE]: Term 3 pre-election: Vote denied by peer e3aa58d3045c4a05b94b94eb8bf58ae1. Message: Service unavailable: T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 012786a8f1b64995847cf7dacbc029a4 for term 3 because replica is already servicing an update from a current leader or another vote. I0312 20:10:32.793480 1797 leader_election.cc:258] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. I0312 20:10:32.793803 2102 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c: Snoozing failure detection for 0.018s (election complete) I0312 20:10:32.794315 2100 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4: Snoozing failure detection for 0.741s (election complete) I0312 20:10:32.794544 2100 raft_consensus.cc:2455] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority I0312 20:10:32.794509 1866 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" candidate_term: 4 candidate_status { last_received { term: 1 index: 71 } } ignore_live_leader: false dest_uuid: "012786a8f1b64995847cf7dacbc029a4" is_pre_election: true I0312 20:10:32.795096 2100 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4: Snoozing failure detection for 1.126s (election complete) I0312 20:10:32.795317 2100 raft_consensus.cc:2455] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority I0312 20:10:32.795619 1866 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4: Snoozing failure detection for 0.961s (vote granted) I0312 20:10:32.795374 1618 catalog_manager.cc:3592] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c reported cstate change: term changed from 1 to 3, leader changed from e3aa58d3045c4a05b94b94eb8bf58ae1 (127.0.53.3) to 9da22e5afdce42078fe8f94ad166ac3c (127.0.53.1). New cstate: current_term: 3 leader_uuid: "9da22e5afdce42078fe8f94ad166ac3c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "012786a8f1b64995847cf7dacbc029a4" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 43059 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9da22e5afdce42078fe8f94ad166ac3c" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 39175 } health_report { overall_health: HEALTHY } } } I0312 20:10:32.796612 1866 raft_consensus.cc:2215] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3aa58d3045c4a05b94b94eb8bf58ae1 in term 3. I0312 20:10:32.797456 1926 leader_election.cc:363] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 4 pre-election: Vote granted by peer 012786a8f1b64995847cf7dacbc029a4 I0312 20:10:32.797643 1926 leader_election.cc:258] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. I0312 20:10:32.798138 2075 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1: Snoozing failure detection for 0.018s (election complete) I0312 20:10:32.798347 2075 raft_consensus.cc:2507] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 3 FOLLOWER]: Leader pre-election won for term 4 I0312 20:10:32.798517 2075 raft_consensus.cc:436] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0312 20:10:32.798647 1865 raft_consensus.cc:1058] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 3 FOLLOWER]: Refusing update from remote peer 9da22e5afdce42078fe8f94ad166ac3c: Log matching property violated. Preceding OpId in replica: term: 1 index: 67. Preceding OpId from leader: term: 3 index: 70. (index mismatch) I0312 20:10:32.798684 2075 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1: Snoozing failure detection for 0.015s (starting election) I0312 20:10:32.798852 2075 raft_consensus.cc:2752] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 3 FOLLOWER]: Advancing to term 4 I0312 20:10:32.800045 2101 consensus_queue.cc:1017] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "012786a8f1b64995847cf7dacbc029a4" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 43059 }, Status: LMP_MISMATCH, Last received: 1.67, Next index: 68, Last known committed idx: 67, Time since last communication: 0.000s I0312 20:10:32.803854 2075 raft_consensus.cc:458] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "012786a8f1b64995847cf7dacbc029a4" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 43059 } } peers { permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 } } peers { permanent_uuid: "9da22e5afdce42078fe8f94ad166ac3c" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 39175 } } I0312 20:10:32.805780 2075 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 4 election: Requesting vote from peer 012786a8f1b64995847cf7dacbc029a4 I0312 20:10:32.806589 2075 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 4 election: Requesting vote from peer 9da22e5afdce42078fe8f94ad166ac3c I0312 20:10:32.807848 1866 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" candidate_term: 4 candidate_status { last_received { term: 1 index: 71 } } ignore_live_leader: false dest_uuid: "012786a8f1b64995847cf7dacbc029a4" I0312 20:10:32.809482 1739 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" candidate_term: 4 candidate_status { last_received { term: 1 index: 71 } } ignore_live_leader: false dest_uuid: "9da22e5afdce42078fe8f94ad166ac3c" is_pre_election: true I0312 20:10:32.809978 1739 raft_consensus.cc:2176] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 3 LEADER]: Leader pre-election vote request: Denying vote to candidate e3aa58d3045c4a05b94b94eb8bf58ae1 for term 4 because replica is either leader or believes a valid leader to be alive. I0312 20:10:32.810751 1989 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "9da22e5afdce42078fe8f94ad166ac3c" candidate_term: 4 candidate_status { last_received { term: 1 index: 69 } } ignore_live_leader: false dest_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" is_pre_election: true I0312 20:10:32.811285 1989 raft_consensus.cc:2145] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9da22e5afdce42078fe8f94ad166ac3c in current term 4: Already voted for candidate e3aa58d3045c4a05b94b94eb8bf58ae1 in this term. I0312 20:10:32.812043 1866 raft_consensus.cc:2190] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate e3aa58d3045c4a05b94b94eb8bf58ae1 for term 4 because replica is already servicing an update from a current leader or another vote. W0312 20:10:32.812139 2102 fault_injection.cc:41] FAULT INJECTION ENABLED! W0312 20:10:32.812270 2102 fault_injection.cc:42] THIS SERVER MAY CRASH! I0312 20:10:32.814610 1926 leader_election.cc:377] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 4 election: Vote denied by peer 012786a8f1b64995847cf7dacbc029a4. Message: Service unavailable: T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate e3aa58d3045c4a05b94b94eb8bf58ae1 for term 4 because replica is already servicing an update from a current leader or another vote. I0312 20:10:32.815601 1663 leader_election.cc:377] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [CANDIDATE]: Term 3 election: Vote denied by peer e3aa58d3045c4a05b94b94eb8bf58ae1. Message: Invalid argument: T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 9da22e5afdce42078fe8f94ad166ac3c in current term 3: Already voted for candidate e3aa58d3045c4a05b94b94eb8bf58ae1 in this term. I0312 20:10:32.816308 2075 raft_consensus.cc:436] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0312 20:10:32.816520 2075 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1: Snoozing failure detection for 0.022s (starting election) W0312 20:10:32.816545 1922 leader_election.cc:287] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 9da22e5afdce42078fe8f94ad166ac3c: Timed out: RequestConsensusVote RPC to 127.0.53.1:39175 timed out after 0.016s (SENT) I0312 20:10:32.816764 1922 leader_election.cc:258] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. I0312 20:10:32.816764 2075 raft_consensus.cc:458] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "012786a8f1b64995847cf7dacbc029a4" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 43059 } } peers { permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 } } peers { permanent_uuid: "9da22e5afdce42078fe8f94ad166ac3c" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 39175 } } W0312 20:10:32.817597 1922 leader_election.cc:287] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 9da22e5afdce42078fe8f94ad166ac3c: Timed out: RequestConsensusVote RPC to 127.0.53.1:39175 timed out after 0.022s (SENT) I0312 20:10:32.817751 2075 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 5 pre-election: Requesting pre-vote from peer 012786a8f1b64995847cf7dacbc029a4 I0312 20:10:32.818080 2075 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 5 pre-election: Requesting pre-vote from peer 9da22e5afdce42078fe8f94ad166ac3c I0312 20:10:32.818795 2064 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1: Snoozing failure detection for 0.015s (election complete) I0312 20:10:32.819020 2064 raft_consensus.cc:2455] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 4 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority I0312 20:10:32.819006 1739 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" candidate_term: 4 candidate_status { last_received { term: 1 index: 71 } } ignore_live_leader: false dest_uuid: "9da22e5afdce42078fe8f94ad166ac3c" I0312 20:10:32.819500 1739 raft_consensus.cc:2176] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 3 LEADER]: Leader election vote request: Denying vote to candidate e3aa58d3045c4a05b94b94eb8bf58ae1 for term 4 because replica is either leader or believes a valid leader to be alive. I0312 20:10:32.820845 1989 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 4 FOLLOWER]: Rejecting Update request from peer 9da22e5afdce42078fe8f94ad166ac3c for earlier term 3. Current term is 4. Ops: [] I0312 20:10:32.821645 1865 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" candidate_term: 5 candidate_status { last_received { term: 1 index: 71 } } ignore_live_leader: false dest_uuid: "012786a8f1b64995847cf7dacbc029a4" is_pre_election: true I0312 20:10:32.821862 1663 leader_election.cc:377] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [CANDIDATE]: Term 4 pre-election: Vote denied by peer e3aa58d3045c4a05b94b94eb8bf58ae1. Message: Invalid argument: T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9da22e5afdce42078fe8f94ad166ac3c in current term 4: Already voted for candidate e3aa58d3045c4a05b94b94eb8bf58ae1 in this term. I0312 20:10:32.822206 1865 raft_consensus.cc:2176] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e3aa58d3045c4a05b94b94eb8bf58ae1 for term 5 because replica is either leader or believes a valid leader to be alive. I0312 20:10:32.822965 1926 leader_election.cc:377] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 5 pre-election: Vote denied by peer 012786a8f1b64995847cf7dacbc029a4. Message: Invalid argument: T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e3aa58d3045c4a05b94b94eb8bf58ae1 for term 5 because replica is either leader or believes a valid leader to be alive. I0312 20:10:32.823192 2101 consensus_queue.cc:1004] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [LEADER]: Peer e3aa58d3045c4a05b94b94eb8bf58ae1 log is divergent from this leader: its last log entry 1.71 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 69 I0312 20:10:32.823381 2101 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 }, Status: INVALID_TERM, Last received: 0.0, Next index: 70, Last known committed idx: 69, Time since last communication: 0.000s I0312 20:10:32.824264 1922 leader_election.cc:377] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 4 election: Vote denied by peer 9da22e5afdce42078fe8f94ad166ac3c. Message: Invalid argument: T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 3 LEADER]: Leader election vote request: Denying vote to candidate e3aa58d3045c4a05b94b94eb8bf58ae1 for term 4 because replica is either leader or believes a valid leader to be alive. I0312 20:10:32.824494 1922 leader_election.cc:258] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. I0312 20:10:32.824499 1739 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" candidate_term: 5 candidate_status { last_received { term: 1 index: 71 } } ignore_live_leader: false dest_uuid: "9da22e5afdce42078fe8f94ad166ac3c" is_pre_election: true I0312 20:10:32.824931 2064 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1: Snoozing failure detection for 0.032s (election complete) I0312 20:10:32.825018 1739 raft_consensus.cc:2176] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 3 LEADER]: Leader pre-election vote request: Denying vote to candidate e3aa58d3045c4a05b94b94eb8bf58ae1 for term 5 because replica is either leader or believes a valid leader to be alive. I0312 20:10:32.825147 2064 raft_consensus.cc:2455] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority I0312 20:10:32.825992 1922 leader_election.cc:377] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 5 pre-election: Vote denied by peer 9da22e5afdce42078fe8f94ad166ac3c. Message: Invalid argument: T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 3 LEADER]: Leader pre-election vote request: Denying vote to candidate e3aa58d3045c4a05b94b94eb8bf58ae1 for term 5 because replica is either leader or believes a valid leader to be alive. I0312 20:10:32.826189 1922 leader_election.cc:258] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. I0312 20:10:32.826212 1989 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 4 FOLLOWER]: Rejecting Update request from peer 9da22e5afdce42078fe8f94ad166ac3c for earlier term 3. Current term is 4. Ops: [] I0312 20:10:32.826563 2064 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1: Snoozing failure detection for 0.049s (election complete) I0312 20:10:32.826753 2064 raft_consensus.cc:2455] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority I0312 20:10:32.829157 2101 consensus_queue.cc:1004] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [LEADER]: Peer e3aa58d3045c4a05b94b94eb8bf58ae1 log is divergent from this leader: its last log entry 1.71 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 69 I0312 20:10:32.829349 2101 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 }, Status: INVALID_TERM, Last received: 0.0, Next index: 70, Last known committed idx: 69, Time since last communication: 0.001s I0312 20:10:32.836474 1989 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 4 FOLLOWER]: Rejecting Update request from peer 9da22e5afdce42078fe8f94ad166ac3c for earlier term 3. Current term is 4. Ops: [] I0312 20:10:32.837740 2101 consensus_queue.cc:1004] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [LEADER]: Peer e3aa58d3045c4a05b94b94eb8bf58ae1 log is divergent from this leader: its last log entry 1.71 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 69 I0312 20:10:32.837966 2101 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 }, Status: INVALID_TERM, Last received: 0.0, Next index: 70, Last known committed idx: 69, Time since last communication: 0.001s I0312 20:10:32.841123 1989 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 4 FOLLOWER]: Rejecting Update request from peer 9da22e5afdce42078fe8f94ad166ac3c for earlier term 3. Current term is 4. Ops: [] I0312 20:10:32.842284 2108 consensus_queue.cc:1004] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [LEADER]: Peer e3aa58d3045c4a05b94b94eb8bf58ae1 log is divergent from this leader: its last log entry 1.71 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 69 I0312 20:10:32.842454 2108 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 }, Status: INVALID_TERM, Last received: 0.0, Next index: 70, Last known committed idx: 69, Time since last communication: 0.001s I0312 20:10:32.846817 1989 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 4 FOLLOWER]: Rejecting Update request from peer 9da22e5afdce42078fe8f94ad166ac3c for earlier term 3. Current term is 4. Ops: [] I0312 20:10:32.847795 2108 consensus_queue.cc:1004] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [LEADER]: Peer e3aa58d3045c4a05b94b94eb8bf58ae1 log is divergent from this leader: its last log entry 1.71 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 69 I0312 20:10:32.848207 2108 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 }, Status: INVALID_TERM, Last received: 0.0, Next index: 70, Last known committed idx: 69, Time since last communication: 0.001s I0312 20:10:32.851457 1989 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 4 FOLLOWER]: Rejecting Update request from peer 9da22e5afdce42078fe8f94ad166ac3c for earlier term 3. Current term is 4. Ops: [] I0312 20:10:32.852824 2108 consensus_queue.cc:1004] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [LEADER]: Peer e3aa58d3045c4a05b94b94eb8bf58ae1 log is divergent from this leader: its last log entry 1.71 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 69 I0312 20:10:32.853185 2108 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 }, Status: INVALID_TERM, Last received: 0.0, Next index: 70, Last known committed idx: 69, Time since last communication: 0.001s I0312 20:10:32.856132 1989 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 4 FOLLOWER]: Rejecting Update request from peer 9da22e5afdce42078fe8f94ad166ac3c for earlier term 3. Current term is 4. Ops: [] I0312 20:10:32.857192 2108 consensus_queue.cc:1004] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [LEADER]: Peer e3aa58d3045c4a05b94b94eb8bf58ae1 log is divergent from this leader: its last log entry 1.71 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 69 I0312 20:10:32.857429 2108 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 }, Status: INVALID_TERM, Last received: 0.0, Next index: 70, Last known committed idx: 69, Time since last communication: 0.001s I0312 20:10:32.860381 2102 raft_consensus.cc:2747] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 3 LEADER]: Stepping down as leader of term 3 I0312 20:10:32.860579 2102 raft_consensus.cc:605] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 9da22e5afdce42078fe8f94ad166ac3c, State: Running, Role: LEADER I0312 20:10:32.860994 2102 consensus_queue.cc:219] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 70, Committed index: 70, Last appended: 3.70, Last appended by leader: 70, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "012786a8f1b64995847cf7dacbc029a4" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 43059 } } peers { permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 } } peers { permanent_uuid: "9da22e5afdce42078fe8f94ad166ac3c" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 39175 } } I0312 20:10:32.861621 2102 consensus_peers.cc:458] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c -> Peer e3aa58d3045c4a05b94b94eb8bf58ae1 (127.0.53.3:49316): Closing peer: e3aa58d3045c4a05b94b94eb8bf58ae1 I0312 20:10:32.861810 2102 consensus_peers.cc:458] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c -> Peer 012786a8f1b64995847cf7dacbc029a4 (127.0.53.2:43059): Closing peer: 012786a8f1b64995847cf7dacbc029a4 I0312 20:10:32.862218 2102 raft_consensus.cc:2752] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 3 FOLLOWER]: Advancing to term 4 I0312 20:10:32.874905 2109 raft_consensus.cc:436] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 9da22e5afdce42078fe8f94ad166ac3c) I0312 20:10:32.875157 2109 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4: Snoozing failure detection for 0.016s (starting election) I0312 20:10:32.875404 2109 raft_consensus.cc:458] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "012786a8f1b64995847cf7dacbc029a4" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 43059 } } peers { permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 } } peers { permanent_uuid: "9da22e5afdce42078fe8f94ad166ac3c" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 39175 } } I0312 20:10:32.876406 2109 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [CANDIDATE]: Term 4 pre-election: Requesting pre-vote from peer e3aa58d3045c4a05b94b94eb8bf58ae1 I0312 20:10:32.876503 2064 raft_consensus.cc:436] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0312 20:10:32.876713 2064 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1: Snoozing failure detection for 0.065s (starting election) I0312 20:10:32.876766 2109 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [CANDIDATE]: Term 4 pre-election: Requesting pre-vote from peer 9da22e5afdce42078fe8f94ad166ac3c I0312 20:10:32.876940 2064 raft_consensus.cc:458] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "012786a8f1b64995847cf7dacbc029a4" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 43059 } } peers { permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 } } peers { permanent_uuid: "9da22e5afdce42078fe8f94ad166ac3c" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 39175 } } I0312 20:10:32.877928 2064 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 5 pre-election: Requesting pre-vote from peer 012786a8f1b64995847cf7dacbc029a4 I0312 20:10:32.878275 2064 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 5 pre-election: Requesting pre-vote from peer 9da22e5afdce42078fe8f94ad166ac3c I0312 20:10:32.878442 1989 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "012786a8f1b64995847cf7dacbc029a4" candidate_term: 4 candidate_status { last_received { term: 3 index: 70 } } ignore_live_leader: false dest_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" is_pre_election: true I0312 20:10:32.879024 1989 raft_consensus.cc:2145] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 012786a8f1b64995847cf7dacbc029a4 in current term 4: Already voted for candidate e3aa58d3045c4a05b94b94eb8bf58ae1 in this term. I0312 20:10:32.879845 1797 leader_election.cc:377] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [CANDIDATE]: Term 4 pre-election: Vote denied by peer e3aa58d3045c4a05b94b94eb8bf58ae1. Message: Invalid argument: T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 012786a8f1b64995847cf7dacbc029a4 in current term 4: Already voted for candidate e3aa58d3045c4a05b94b94eb8bf58ae1 in this term. I0312 20:10:32.879961 1739 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" candidate_term: 5 candidate_status { last_received { term: 1 index: 71 } } ignore_live_leader: false dest_uuid: "9da22e5afdce42078fe8f94ad166ac3c" is_pre_election: true I0312 20:10:32.880594 1739 raft_consensus.cc:2162] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e3aa58d3045c4a05b94b94eb8bf58ae1 for term 5 because replica has last-logged OpId of term: 3 index: 70, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 71. I0312 20:10:32.880787 1741 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "012786a8f1b64995847cf7dacbc029a4" candidate_term: 4 candidate_status { last_received { term: 3 index: 70 } } ignore_live_leader: false dest_uuid: "9da22e5afdce42078fe8f94ad166ac3c" is_pre_election: true I0312 20:10:32.881316 1741 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c: Snoozing failure detection for 0.017s (vote granted) I0312 20:10:32.881526 1741 raft_consensus.cc:2215] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 012786a8f1b64995847cf7dacbc029a4 in term 4. I0312 20:10:32.881600 1922 leader_election.cc:377] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 5 pre-election: Vote denied by peer 9da22e5afdce42078fe8f94ad166ac3c. Message: Invalid argument: T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e3aa58d3045c4a05b94b94eb8bf58ae1 for term 5 because replica has last-logged OpId of term: 3 index: 70, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 71. I0312 20:10:32.882454 1797 leader_election.cc:363] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [CANDIDATE]: Term 4 pre-election: Vote granted by peer 9da22e5afdce42078fe8f94ad166ac3c I0312 20:10:32.882328 1865 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" candidate_term: 5 candidate_status { last_received { term: 1 index: 71 } } ignore_live_leader: false dest_uuid: "012786a8f1b64995847cf7dacbc029a4" is_pre_election: true I0312 20:10:32.882648 1797 leader_election.cc:258] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. I0312 20:10:32.882961 1865 raft_consensus.cc:2162] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e3aa58d3045c4a05b94b94eb8bf58ae1 for term 5 because replica has last-logged OpId of term: 3 index: 70, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 71. I0312 20:10:32.883348 2109 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4: Snoozing failure detection for 0.016s (election complete) I0312 20:10:32.883558 2109 raft_consensus.cc:2507] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 3 FOLLOWER]: Leader pre-election won for term 4 I0312 20:10:32.883728 2109 raft_consensus.cc:436] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 9da22e5afdce42078fe8f94ad166ac3c) I0312 20:10:32.883769 1926 leader_election.cc:377] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 5 pre-election: Vote denied by peer 012786a8f1b64995847cf7dacbc029a4. Message: Invalid argument: T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e3aa58d3045c4a05b94b94eb8bf58ae1 for term 5 because replica has last-logged OpId of term: 3 index: 70, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 71. I0312 20:10:32.883915 2109 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4: Snoozing failure detection for 0.022s (starting election) I0312 20:10:32.883985 1926 leader_election.cc:258] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. I0312 20:10:32.884080 2109 raft_consensus.cc:2752] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 3 FOLLOWER]: Advancing to term 4 I0312 20:10:32.884557 2064 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1: Snoozing failure detection for 0.074s (election complete) I0312 20:10:32.884752 2064 raft_consensus.cc:2455] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority I0312 20:10:32.885191 2101 raft_consensus.cc:436] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0312 20:10:32.885401 2101 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c: Snoozing failure detection for 0.015s (starting election) I0312 20:10:32.885622 2101 raft_consensus.cc:458] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "012786a8f1b64995847cf7dacbc029a4" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 43059 } } peers { permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 } } peers { permanent_uuid: "9da22e5afdce42078fe8f94ad166ac3c" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 39175 } } I0312 20:10:32.886575 2101 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [CANDIDATE]: Term 5 pre-election: Requesting pre-vote from peer 012786a8f1b64995847cf7dacbc029a4 I0312 20:10:32.887007 2101 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [CANDIDATE]: Term 5 pre-election: Requesting pre-vote from peer e3aa58d3045c4a05b94b94eb8bf58ae1 I0312 20:10:32.887889 1865 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "9da22e5afdce42078fe8f94ad166ac3c" candidate_term: 5 candidate_status { last_received { term: 3 index: 70 } } ignore_live_leader: false dest_uuid: "012786a8f1b64995847cf7dacbc029a4" is_pre_election: true I0312 20:10:32.888341 1989 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "9da22e5afdce42078fe8f94ad166ac3c" candidate_term: 5 candidate_status { last_received { term: 3 index: 70 } } ignore_live_leader: false dest_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" is_pre_election: true I0312 20:10:32.888751 1989 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1: Snoozing failure detection for 0.104s (vote granted) I0312 20:10:32.888947 1989 raft_consensus.cc:2215] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9da22e5afdce42078fe8f94ad166ac3c in term 4. I0312 20:10:32.889811 1663 leader_election.cc:363] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [CANDIDATE]: Term 5 pre-election: Vote granted by peer e3aa58d3045c4a05b94b94eb8bf58ae1 I0312 20:10:32.890056 1663 leader_election.cc:258] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. I0312 20:10:32.890463 2108 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c: Snoozing failure detection for 0.015s (election complete) I0312 20:10:32.890734 2108 raft_consensus.cc:2507] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 4 FOLLOWER]: Leader pre-election won for term 5 I0312 20:10:32.890954 2108 raft_consensus.cc:436] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0312 20:10:32.891142 2108 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c: Snoozing failure detection for 0.022s (starting election) I0312 20:10:32.891335 2108 raft_consensus.cc:2752] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 4 FOLLOWER]: Advancing to term 5 W0312 20:10:32.902902 1676 leader_election.cc:287] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 012786a8f1b64995847cf7dacbc029a4: Timed out: RequestConsensusVote RPC to 127.0.53.2:43059 timed out after 0.015s (SENT) I0312 20:10:32.993587 2064 raft_consensus.cc:436] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0312 20:10:32.993796 2064 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1: Snoozing failure detection for 0.086s (starting election) I0312 20:10:32.993975 2064 raft_consensus.cc:458] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "012786a8f1b64995847cf7dacbc029a4" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 43059 } } peers { permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 } } peers { permanent_uuid: "9da22e5afdce42078fe8f94ad166ac3c" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 39175 } } I0312 20:10:32.994771 2064 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 5 pre-election: Requesting pre-vote from peer 012786a8f1b64995847cf7dacbc029a4 I0312 20:10:32.995079 2064 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 5 pre-election: Requesting pre-vote from peer 9da22e5afdce42078fe8f94ad166ac3c I0312 20:10:32.995856 1866 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" candidate_term: 5 candidate_status { last_received { term: 1 index: 71 } } ignore_live_leader: false dest_uuid: "012786a8f1b64995847cf7dacbc029a4" is_pre_election: true I0312 20:10:32.996094 1741 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" candidate_term: 5 candidate_status { last_received { term: 1 index: 71 } } ignore_live_leader: false dest_uuid: "9da22e5afdce42078fe8f94ad166ac3c" is_pre_election: true I0312 20:10:33.080569 2064 raft_consensus.cc:436] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0312 20:10:33.080776 2064 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1: Snoozing failure detection for 0.047s (starting election) I0312 20:10:33.080945 2064 raft_consensus.cc:458] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "012786a8f1b64995847cf7dacbc029a4" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 43059 } } peers { permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 } } peers { permanent_uuid: "9da22e5afdce42078fe8f94ad166ac3c" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 39175 } } I0312 20:10:33.081773 2064 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 5 pre-election: Requesting pre-vote from peer 012786a8f1b64995847cf7dacbc029a4 W0312 20:10:33.081805 1922 leader_election.cc:287] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 9da22e5afdce42078fe8f94ad166ac3c: Timed out: RequestConsensusVote RPC to 127.0.53.1:39175 timed out after 0.086s (SENT) W0312 20:10:33.082059 1926 leader_election.cc:287] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 012786a8f1b64995847cf7dacbc029a4: Timed out: RequestConsensusVote RPC to 127.0.53.2:43059 timed out after 0.086s (SENT) I0312 20:10:33.082151 2064 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 5 pre-election: Requesting pre-vote from peer 9da22e5afdce42078fe8f94ad166ac3c I0312 20:10:33.082252 1926 leader_election.cc:258] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. I0312 20:10:33.082635 2064 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1: Snoozing failure detection for 0.111s (election complete) I0312 20:10:33.082832 2064 raft_consensus.cc:2455] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority I0312 20:10:33.083407 1739 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" candidate_term: 5 candidate_status { last_received { term: 1 index: 71 } } ignore_live_leader: false dest_uuid: "9da22e5afdce42078fe8f94ad166ac3c" is_pre_election: true I0312 20:10:33.083535 1864 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" candidate_term: 5 candidate_status { last_received { term: 1 index: 71 } } ignore_live_leader: false dest_uuid: "012786a8f1b64995847cf7dacbc029a4" is_pre_election: true W0312 20:10:33.130074 1922 leader_election.cc:287] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 9da22e5afdce42078fe8f94ad166ac3c: Timed out: RequestConsensusVote RPC to 127.0.53.1:39175 timed out after 0.047s (SENT) W0312 20:10:33.130475 1926 leader_election.cc:287] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 012786a8f1b64995847cf7dacbc029a4: Timed out: RequestConsensusVote RPC to 127.0.53.2:43059 timed out after 0.047s (SENT) I0312 20:10:33.130625 1926 leader_election.cc:258] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. I0312 20:10:33.130923 2064 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1: Snoozing failure detection for 0.115s (election complete) I0312 20:10:33.131080 2064 raft_consensus.cc:2455] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority I0312 20:10:33.246932 2064 raft_consensus.cc:436] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0312 20:10:33.247117 2064 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1: Snoozing failure detection for 0.163s (starting election) I0312 20:10:33.247328 2064 raft_consensus.cc:458] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "012786a8f1b64995847cf7dacbc029a4" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 43059 } } peers { permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 } } peers { permanent_uuid: "9da22e5afdce42078fe8f94ad166ac3c" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 39175 } } I0312 20:10:33.248095 2064 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 5 pre-election: Requesting pre-vote from peer 012786a8f1b64995847cf7dacbc029a4 I0312 20:10:33.248420 2064 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 5 pre-election: Requesting pre-vote from peer 9da22e5afdce42078fe8f94ad166ac3c I0312 20:10:33.249402 1863 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" candidate_term: 5 candidate_status { last_received { term: 1 index: 71 } } ignore_live_leader: false dest_uuid: "012786a8f1b64995847cf7dacbc029a4" is_pre_election: true I0312 20:10:33.249559 1740 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" candidate_term: 5 candidate_status { last_received { term: 1 index: 71 } } ignore_live_leader: false dest_uuid: "9da22e5afdce42078fe8f94ad166ac3c" is_pre_election: true I0312 20:10:33.410961 2064 raft_consensus.cc:436] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0312 20:10:33.411157 2064 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1: Snoozing failure detection for 0.213s (starting election) I0312 20:10:33.411376 2064 raft_consensus.cc:458] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "012786a8f1b64995847cf7dacbc029a4" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 43059 } } peers { permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 } } peers { permanent_uuid: "9da22e5afdce42078fe8f94ad166ac3c" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 39175 } } I0312 20:10:33.412166 2064 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 5 pre-election: Requesting pre-vote from peer 012786a8f1b64995847cf7dacbc029a4 W0312 20:10:33.412225 1922 leader_election.cc:287] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 9da22e5afdce42078fe8f94ad166ac3c: Timed out: RequestConsensusVote RPC to 127.0.53.1:39175 timed out after 0.163s (SENT) I0312 20:10:33.412497 2064 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 5 pre-election: Requesting pre-vote from peer 9da22e5afdce42078fe8f94ad166ac3c W0312 20:10:33.412621 1926 leader_election.cc:287] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 012786a8f1b64995847cf7dacbc029a4: Timed out: RequestConsensusVote RPC to 127.0.53.2:43059 timed out after 0.163s (SENT) I0312 20:10:33.412833 1926 leader_election.cc:258] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. I0312 20:10:33.413254 2064 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1: Snoozing failure detection for 0.186s (election complete) I0312 20:10:33.413408 2064 raft_consensus.cc:2455] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority I0312 20:10:33.414263 1862 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" candidate_term: 5 candidate_status { last_received { term: 1 index: 71 } } ignore_live_leader: false dest_uuid: "012786a8f1b64995847cf7dacbc029a4" is_pre_election: true I0312 20:10:33.415150 1738 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" candidate_term: 5 candidate_status { last_received { term: 1 index: 71 } } ignore_live_leader: false dest_uuid: "9da22e5afdce42078fe8f94ad166ac3c" is_pre_election: true I0312 20:10:33.599766 2064 raft_consensus.cc:436] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0312 20:10:33.600029 2064 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1: Snoozing failure detection for 0.360s (starting election) I0312 20:10:33.600237 2064 raft_consensus.cc:458] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "012786a8f1b64995847cf7dacbc029a4" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 43059 } } peers { permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 } } peers { permanent_uuid: "9da22e5afdce42078fe8f94ad166ac3c" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 39175 } } I0312 20:10:33.600996 2064 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 5 pre-election: Requesting pre-vote from peer 012786a8f1b64995847cf7dacbc029a4 I0312 20:10:33.601337 2064 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 5 pre-election: Requesting pre-vote from peer 9da22e5afdce42078fe8f94ad166ac3c I0312 20:10:33.602274 1861 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" candidate_term: 5 candidate_status { last_received { term: 1 index: 71 } } ignore_live_leader: false dest_uuid: "012786a8f1b64995847cf7dacbc029a4" is_pre_election: true I0312 20:10:33.602470 1737 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" candidate_term: 5 candidate_status { last_received { term: 1 index: 71 } } ignore_live_leader: false dest_uuid: "9da22e5afdce42078fe8f94ad166ac3c" is_pre_election: true W0312 20:10:33.627104 1926 leader_election.cc:287] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 012786a8f1b64995847cf7dacbc029a4: Timed out: RequestConsensusVote RPC to 127.0.53.2:43059 timed out after 0.213s (SENT) W0312 20:10:33.627813 1922 leader_election.cc:287] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 9da22e5afdce42078fe8f94ad166ac3c: Timed out: RequestConsensusVote RPC to 127.0.53.1:39175 timed out after 0.213s (SENT) I0312 20:10:33.627979 1922 leader_election.cc:258] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. I0312 20:10:33.628348 2064 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1: Snoozing failure detection for 0.130s (election complete) I0312 20:10:33.628502 2064 raft_consensus.cc:2455] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority I0312 20:10:33.759699 2064 raft_consensus.cc:436] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0312 20:10:33.759896 2064 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1: Snoozing failure detection for 0.369s (starting election) I0312 20:10:33.760176 2064 raft_consensus.cc:458] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "012786a8f1b64995847cf7dacbc029a4" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 43059 } } peers { permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 } } peers { permanent_uuid: "9da22e5afdce42078fe8f94ad166ac3c" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 39175 } } I0312 20:10:33.760938 2064 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 5 pre-election: Requesting pre-vote from peer 012786a8f1b64995847cf7dacbc029a4 I0312 20:10:33.761301 2064 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 5 pre-election: Requesting pre-vote from peer 9da22e5afdce42078fe8f94ad166ac3c I0312 20:10:33.762362 1860 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" candidate_term: 5 candidate_status { last_received { term: 1 index: 71 } } ignore_live_leader: false dest_uuid: "012786a8f1b64995847cf7dacbc029a4" is_pre_election: true I0312 20:10:33.762498 1736 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" candidate_term: 5 candidate_status { last_received { term: 1 index: 71 } } ignore_live_leader: false dest_uuid: "9da22e5afdce42078fe8f94ad166ac3c" is_pre_election: true W0312 20:10:33.962298 1926 leader_election.cc:287] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 012786a8f1b64995847cf7dacbc029a4: Timed out: RequestConsensusVote RPC to 127.0.53.2:43059 timed out after 0.360s (SENT) W0312 20:10:33.962529 1922 leader_election.cc:287] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 9da22e5afdce42078fe8f94ad166ac3c: Timed out: RequestConsensusVote RPC to 127.0.53.1:39175 timed out after 0.360s (SENT) I0312 20:10:33.962699 1922 leader_election.cc:258] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. I0312 20:10:33.963065 2064 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1: Snoozing failure detection for 0.060s (election complete) I0312 20:10:33.963228 2064 raft_consensus.cc:2455] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority I0312 20:10:34.023799 2064 raft_consensus.cc:436] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0312 20:10:34.024034 2064 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1: Snoozing failure detection for 0.063s (starting election) I0312 20:10:34.024211 2064 raft_consensus.cc:458] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "012786a8f1b64995847cf7dacbc029a4" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 43059 } } peers { permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 } } peers { permanent_uuid: "9da22e5afdce42078fe8f94ad166ac3c" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 39175 } } I0312 20:10:34.025029 2064 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 5 pre-election: Requesting pre-vote from peer 012786a8f1b64995847cf7dacbc029a4 I0312 20:10:34.025346 2064 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 5 pre-election: Requesting pre-vote from peer 9da22e5afdce42078fe8f94ad166ac3c I0312 20:10:34.026129 1859 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" candidate_term: 5 candidate_status { last_received { term: 1 index: 71 } } ignore_live_leader: false dest_uuid: "012786a8f1b64995847cf7dacbc029a4" is_pre_election: true I0312 20:10:34.026386 1735 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" candidate_term: 5 candidate_status { last_received { term: 1 index: 71 } } ignore_live_leader: false dest_uuid: "9da22e5afdce42078fe8f94ad166ac3c" is_pre_election: true I0312 20:10:34.087659 2064 raft_consensus.cc:436] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0312 20:10:34.087863 2064 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1: Snoozing failure detection for 0.848s (starting election) I0312 20:10:34.088034 2064 raft_consensus.cc:458] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "012786a8f1b64995847cf7dacbc029a4" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 43059 } } peers { permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 } } peers { permanent_uuid: "9da22e5afdce42078fe8f94ad166ac3c" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 39175 } } W0312 20:10:34.088881 1922 leader_election.cc:287] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 9da22e5afdce42078fe8f94ad166ac3c: Timed out: RequestConsensusVote RPC to 127.0.53.1:39175 timed out after 0.063s (SENT) I0312 20:10:34.089010 2064 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 5 pre-election: Requesting pre-vote from peer 012786a8f1b64995847cf7dacbc029a4 W0312 20:10:34.089169 1926 leader_election.cc:287] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 012786a8f1b64995847cf7dacbc029a4: Timed out: RequestConsensusVote RPC to 127.0.53.2:43059 timed out after 0.063s (SENT) I0312 20:10:34.089362 1926 leader_election.cc:258] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. I0312 20:10:34.089337 2064 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 5 pre-election: Requesting pre-vote from peer 9da22e5afdce42078fe8f94ad166ac3c I0312 20:10:34.089737 2064 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1: Snoozing failure detection for 0.094s (election complete) I0312 20:10:34.089944 2064 raft_consensus.cc:2455] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority I0312 20:10:34.090632 1734 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" candidate_term: 5 candidate_status { last_received { term: 1 index: 71 } } ignore_live_leader: false dest_uuid: "9da22e5afdce42078fe8f94ad166ac3c" is_pre_election: true I0312 20:10:34.091763 1858 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" candidate_term: 5 candidate_status { last_received { term: 1 index: 71 } } ignore_live_leader: false dest_uuid: "012786a8f1b64995847cf7dacbc029a4" is_pre_election: true W0312 20:10:34.130611 1926 leader_election.cc:287] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 012786a8f1b64995847cf7dacbc029a4: Timed out: RequestConsensusVote RPC to 127.0.53.2:43059 timed out after 0.369s (SENT) W0312 20:10:34.131140 1922 leader_election.cc:287] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 9da22e5afdce42078fe8f94ad166ac3c: Timed out: RequestConsensusVote RPC to 127.0.53.1:39175 timed out after 0.369s (SENT) I0312 20:10:34.131294 1922 leader_election.cc:258] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. I0312 20:10:34.131606 2119 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1: Snoozing failure detection for 0.652s (election complete) I0312 20:10:34.131809 2119 raft_consensus.cc:2455] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority W0312 20:10:34.317242 2109 consensus_meta.cc:216] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4: Time spent flushing consensus metadata: real 1.433s user 0.004s sys 0.000s W0312 20:10:34.317242 2108 consensus_meta.cc:216] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c: Time spent flushing consensus metadata: real 1.426s user 0.000s sys 0.004s I0312 20:10:34.317545 2108 raft_consensus.cc:458] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "012786a8f1b64995847cf7dacbc029a4" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 43059 } } peers { permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 } } peers { permanent_uuid: "9da22e5afdce42078fe8f94ad166ac3c" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 39175 } } I0312 20:10:34.317544 2109 raft_consensus.cc:458] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "012786a8f1b64995847cf7dacbc029a4" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 43059 } } peers { permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 } } peers { permanent_uuid: "9da22e5afdce42078fe8f94ad166ac3c" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 39175 } } I0312 20:10:34.318637 2108 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [CANDIDATE]: Term 5 election: Requesting vote from peer 012786a8f1b64995847cf7dacbc029a4 I0312 20:10:34.318637 2109 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [CANDIDATE]: Term 4 election: Requesting vote from peer e3aa58d3045c4a05b94b94eb8bf58ae1 I0312 20:10:34.318648 2100 raft_consensus.cc:436] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0312 20:10:34.318886 2100 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4: Snoozing failure detection for 0.021s (starting election) I0312 20:10:34.319033 2108 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [CANDIDATE]: Term 5 election: Requesting vote from peer e3aa58d3045c4a05b94b94eb8bf58ae1 I0312 20:10:34.319047 2109 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [CANDIDATE]: Term 4 election: Requesting vote from peer 9da22e5afdce42078fe8f94ad166ac3c I0312 20:10:34.319053 2100 raft_consensus.cc:458] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "012786a8f1b64995847cf7dacbc029a4" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 43059 } } peers { permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 } } peers { permanent_uuid: "9da22e5afdce42078fe8f94ad166ac3c" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 39175 } } I0312 20:10:34.319636 1741 raft_consensus.cc:2145] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e3aa58d3045c4a05b94b94eb8bf58ae1 in current term 5: Already voted for candidate 9da22e5afdce42078fe8f94ad166ac3c in this term. I0312 20:10:34.320001 1863 raft_consensus.cc:2190] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e3aa58d3045c4a05b94b94eb8bf58ae1 for term 5 because replica is already servicing an update from a current leader or another vote. I0312 20:10:34.320001 2100 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [CANDIDATE]: Term 5 pre-election: Requesting pre-vote from peer e3aa58d3045c4a05b94b94eb8bf58ae1 W0312 20:10:34.320132 1741 rpcz_store.cc:251] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.53.3:59901 (request call id 326) took 1324ms (client timeout 86). I0312 20:10:34.320396 2100 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [CANDIDATE]: Term 5 pre-election: Requesting pre-vote from peer 9da22e5afdce42078fe8f94ad166ac3c W0312 20:10:34.320538 1863 rpcz_store.cc:251] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.53.3:47255 (request call id 302) took 1071ms (client timeout 163). W0312 20:10:34.320755 1741 rpcz_store.cc:255] Trace: 0312 20:10:32.995835 (+ 0us) service_pool.cc:163] Inserting onto call queue 0312 20:10:32.995939 (+ 104us) service_pool.cc:222] Handling call 0312 20:10:34.319983 (+1324044us) spinlock_profiling.cc:261] Waited 1.32 s on lock 0x7b480003c39c. stack: 00007fede5da3c74 00007fede5da3b70 00007fede4893314 00007feded8e0294 00007feded8e0229 00007fedec48aaf3 00007feded91e56b 00007fede82d1b24 00007fede82d1ab1 00007fede82d19c4 00007fede792dc29 00007fede792d6ec 00007fede792ebcd 00007fede7930946 00007fede793089a 00007fede7930823 0312 20:10:34.320081 (+ 98us) inbound_call.cc:157] Queueing success response Metrics: {"spinlock_wait_cycles":3043754368} I0312 20:10:34.320860 1989 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "012786a8f1b64995847cf7dacbc029a4" candidate_term: 4 candidate_status { last_received { term: 3 index: 70 } } ignore_live_leader: false dest_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" I0312 20:10:34.321110 2101 raft_consensus.cc:436] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0312 20:10:34.321323 2101 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c: Snoozing failure detection for 0.018s (starting election) W0312 20:10:34.321395 1863 rpcz_store.cc:255] Trace: 0312 20:10:33.249127 (+ 0us) service_pool.cc:163] Inserting onto call queue 0312 20:10:33.249227 (+ 100us) service_pool.cc:222] Handling call 0312 20:10:34.320375 (+1071148us) spinlock_profiling.cc:261] Waited 1.07 s on lock 0x7b480003039c. stack: 00007fee78fc7c74 00007fee78fc7b70 00007fee77ab7314 00007fee80b04294 00007fee80b04229 00007fee7f6ae606 00007fee80b4256b 00007fee7b4f5b24 00007fee7b4f5ab1 00007fee7b4f59c4 00007fee7ab51c29 00007fee7ab516ec 00007fee7ab52bcd 00007fee7ab54946 00007fee7ab5489a 00007fee7ab54823 0312 20:10:34.320470 (+ 95us) inbound_call.cc:157] Queueing success response Metrics: {"spinlock_wait_cycles":2461967744} I0312 20:10:34.321475 1989 raft_consensus.cc:2145] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 012786a8f1b64995847cf7dacbc029a4 in current term 4: Already voted for candidate e3aa58d3045c4a05b94b94eb8bf58ae1 in this term. I0312 20:10:34.321828 1864 raft_consensus.cc:2190] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e3aa58d3045c4a05b94b94eb8bf58ae1 for term 5 because replica is already servicing an update from a current leader or another vote. I0312 20:10:34.321563 2101 raft_consensus.cc:458] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "012786a8f1b64995847cf7dacbc029a4" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 43059 } } peers { permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 } } peers { permanent_uuid: "9da22e5afdce42078fe8f94ad166ac3c" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 39175 } } W0312 20:10:34.322304 1864 rpcz_store.cc:251] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.53.3:47255 (request call id 301) took 1239ms (client timeout 47). I0312 20:10:34.322185 1990 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "012786a8f1b64995847cf7dacbc029a4" candidate_term: 5 candidate_status { last_received { term: 3 index: 70 } } ignore_live_leader: false dest_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" is_pre_election: true I0312 20:10:34.322526 1797 leader_election.cc:377] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [CANDIDATE]: Term 4 election: Vote denied by peer e3aa58d3045c4a05b94b94eb8bf58ae1. Message: Invalid argument: T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 012786a8f1b64995847cf7dacbc029a4 in current term 4: Already voted for candidate e3aa58d3045c4a05b94b94eb8bf58ae1 in this term. I0312 20:10:34.322646 2101 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [CANDIDATE]: Term 6 pre-election: Requesting pre-vote from peer 012786a8f1b64995847cf7dacbc029a4 I0312 20:10:34.322636 1990 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1: Snoozing failure detection for 1.004s (vote granted) I0312 20:10:34.322957 2101 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [CANDIDATE]: Term 6 pre-election: Requesting pre-vote from peer e3aa58d3045c4a05b94b94eb8bf58ae1 I0312 20:10:34.322851 1990 raft_consensus.cc:2215] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 012786a8f1b64995847cf7dacbc029a4 in term 4. W0312 20:10:34.322993 1864 rpcz_store.cc:255] Trace: 0312 20:10:33.083267 (+ 0us) service_pool.cc:163] Inserting onto call queue 0312 20:10:33.083374 (+ 107us) service_pool.cc:222] Handling call 0312 20:10:34.322168 (+1238794us) spinlock_profiling.cc:261] Waited 1.24 s on lock 0x7b480003039c. stack: 00007fee78fc7c74 00007fee78fc7b70 00007fee77ab7314 00007fee80b04294 00007fee80b04229 00007fee7f6ae606 00007fee80b4256b 00007fee7b4f5b24 00007fee7b4f5ab1 00007fee7b4f59c4 00007fee7ab51c29 00007fee7ab516ec 00007fee7ab52bcd 00007fee7ab54946 00007fee7ab5489a 00007fee7ab54823 0312 20:10:34.322257 (+ 89us) inbound_call.cc:157] Queueing success response Metrics: {"spinlock_wait_cycles":2847644032} I0312 20:10:34.323312 1865 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4: Snoozing failure detection for 0.018s (vote granted) I0312 20:10:34.323503 1865 raft_consensus.cc:2215] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9da22e5afdce42078fe8f94ad166ac3c in term 4. I0312 20:10:34.323695 1735 raft_consensus.cc:2190] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e3aa58d3045c4a05b94b94eb8bf58ae1 for term 5 because replica is already servicing an update from a current leader or another vote. I0312 20:10:34.323789 1797 leader_election.cc:363] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [CANDIDATE]: Term 5 pre-election: Vote granted by peer e3aa58d3045c4a05b94b94eb8bf58ae1 I0312 20:10:34.324019 1797 leader_election.cc:258] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. W0312 20:10:34.323997 1865 rpcz_store.cc:251] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.53.1:43855 (request call id 26) took 1436ms (client timeout 15). W0312 20:10:34.324229 1735 rpcz_store.cc:251] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.53.3:59901 (request call id 332) took 298ms (client timeout 63). I0312 20:10:34.324527 2100 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4: Snoozing failure detection for 0.016s (election complete) W0312 20:10:34.324707 1865 rpcz_store.cc:255] Trace: 0312 20:10:32.887673 (+ 0us) service_pool.cc:163] Inserting onto call queue 0312 20:10:32.887744 (+ 71us) service_pool.cc:222] Handling call 0312 20:10:34.323874 (+1436130us) spinlock_profiling.cc:261] Waited 1.44 s on lock 0x7b480003039c. stack: 00007fee78fc7c74 00007fee78fc7b70 00007fee77ab7314 00007fee80b04294 00007fee80b04229 00007fee7f6aeaf3 00007fee80b4256b 00007fee7b4f5b24 00007fee7b4f5ab1 00007fee7b4f59c4 00007fee7ab51c29 00007fee7ab516ec 00007fee7ab52bcd 00007fee7ab54946 00007fee7ab5489a 00007fee7ab54823 0312 20:10:34.323949 (+ 75us) inbound_call.cc:157] Queueing success response Metrics: {"spinlock_wait_cycles":3301404928} I0312 20:10:34.324753 2100 raft_consensus.cc:2507] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 4 FOLLOWER]: Leader pre-election won for term 5 I0312 20:10:34.324973 2100 raft_consensus.cc:436] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0312 20:10:34.325150 2100 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4: Snoozing failure detection for 0.022s (starting election) I0312 20:10:34.325173 1737 raft_consensus.cc:2190] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e3aa58d3045c4a05b94b94eb8bf58ae1 for term 5 because replica is already servicing an update from a current leader or another vote. I0312 20:10:34.325314 2100 raft_consensus.cc:2752] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 4 FOLLOWER]: Advancing to term 5 W0312 20:10:34.325669 1737 rpcz_store.cc:251] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.53.3:59901 (request call id 330) took 723ms (client timeout 360). I0312 20:10:34.325769 1741 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "012786a8f1b64995847cf7dacbc029a4" candidate_term: 4 candidate_status { last_received { term: 3 index: 70 } } ignore_live_leader: false dest_uuid: "9da22e5afdce42078fe8f94ad166ac3c" W0312 20:10:34.326519 1737 rpcz_store.cc:255] Trace: 0312 20:10:33.602196 (+ 0us) service_pool.cc:163] Inserting onto call queue 0312 20:10:33.602300 (+ 104us) service_pool.cc:222] Handling call 0312 20:10:34.325533 (+723233us) spinlock_profiling.cc:261] Waited 722 ms on lock 0x7b480003c39c. stack: 00007fede5da3c74 00007fede5da3b70 00007fede4893314 00007feded8e0294 00007feded8e0229 00007fedec48a606 00007feded91e56b 00007fede82d1b24 00007fede82d1ab1 00007fede82d19c4 00007fede792dc29 00007fede792d6ec 00007fede792ebcd 00007fede7930946 00007fede793089a 00007fede7930823 0312 20:10:34.325620 (+ 87us) inbound_call.cc:157] Queueing success response Metrics: {"spinlock_wait_cycles":1660916992} I0312 20:10:34.327064 1739 raft_consensus.cc:2190] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e3aa58d3045c4a05b94b94eb8bf58ae1 for term 5 because replica is already servicing an update from a current leader or another vote. I0312 20:10:34.327077 1864 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "9da22e5afdce42078fe8f94ad166ac3c" candidate_term: 6 candidate_status { last_received { term: 3 index: 70 } } ignore_live_leader: false dest_uuid: "012786a8f1b64995847cf7dacbc029a4" is_pre_election: true W0312 20:10:34.327533 1739 rpcz_store.cc:251] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.53.3:59901 (request call id 327) took 1244ms (client timeout 47). I0312 20:10:34.327726 1865 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "9da22e5afdce42078fe8f94ad166ac3c" candidate_term: 5 candidate_status { last_received { term: 3 index: 70 } } ignore_live_leader: false dest_uuid: "012786a8f1b64995847cf7dacbc029a4" W0312 20:10:34.328313 1739 rpcz_store.cc:255] Trace: 0312 20:10:33.083181 (+ 0us) service_pool.cc:163] Inserting onto call queue 0312 20:10:33.083277 (+ 96us) service_pool.cc:222] Handling call 0312 20:10:34.327399 (+1244122us) spinlock_profiling.cc:261] Waited 1.24 s on lock 0x7b480003c39c. stack: 00007fede5da3c74 00007fede5da3b70 00007fede4893314 00007feded8e0294 00007feded8e0229 00007fedec48a606 00007feded91e56b 00007fede82d1b24 00007fede82d1ab1 00007fede82d19c4 00007fede792dc29 00007fede792d6ec 00007fede792ebcd 00007fede7930946 00007fede793089a 00007fede7930823 0312 20:10:34.327480 (+ 81us) inbound_call.cc:157] Queueing success response Metrics: {"spinlock_wait_cycles":2858200448} I0312 20:10:34.328631 1736 raft_consensus.cc:2190] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e3aa58d3045c4a05b94b94eb8bf58ae1 for term 5 because replica is already servicing an update from a current leader or another vote. I0312 20:10:34.328960 1741 raft_consensus.cc:2120] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 012786a8f1b64995847cf7dacbc029a4 for earlier term 4. Current term is 5. W0312 20:10:34.329100 1736 rpcz_store.cc:251] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.53.3:59901 (request call id 331) took 566ms (client timeout 369). W0312 20:10:34.328546 1735 rpcz_store.cc:255] Trace: 0312 20:10:34.026133 (+ 0us) service_pool.cc:163] Inserting onto call queue 0312 20:10:34.026227 (+ 94us) service_pool.cc:222] Handling call 0312 20:10:34.324091 (+297864us) spinlock_profiling.cc:261] Waited 297 ms on lock 0x7b480003c39c. stack: 00007fede5da3c74 00007fede5da3b70 00007fede4893314 00007feded8e0294 00007feded8e0229 00007fedec48a606 00007feded91e56b 00007fede82d1b24 00007fede82d1ab1 00007fede82d19c4 00007fede792dc29 00007fede792d6ec 00007fede792ebcd 00007fede7930946 00007fede793089a 00007fede7930823 0312 20:10:34.324183 (+ 92us) inbound_call.cc:157] Queueing success response Metrics: {"spinlock_wait_cycles":682895104} I0312 20:10:34.329710 1740 raft_consensus.cc:2190] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e3aa58d3045c4a05b94b94eb8bf58ae1 for term 5 because replica is already servicing an update from a current leader or another vote. W0312 20:10:34.329802 1736 rpcz_store.cc:255] Trace: 0312 20:10:33.762225 (+ 0us) service_pool.cc:163] Inserting onto call queue 0312 20:10:33.762335 (+ 110us) service_pool.cc:222] Handling call 0312 20:10:34.328982 (+566647us) spinlock_profiling.cc:261] Waited 565 ms on lock 0x7b480003c39c. stack: 00007fede5da3c74 00007fede5da3b70 00007fede4893314 00007feded8e0294 00007feded8e0229 00007fedec48a606 00007feded91e56b 00007fede82d1b24 00007fede82d1ab1 00007fede82d19c4 00007fede792dc29 00007fede792d6ec 00007fede792ebcd 00007fede7930946 00007fede793089a 00007fede7930823 0312 20:10:34.329045 (+ 63us) inbound_call.cc:157] Queueing success response Metrics: {"spinlock_wait_cycles":1299963136} W0312 20:10:34.330163 1740 rpcz_store.cc:251] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.53.3:59901 (request call id 328) took 1080ms (client timeout 163). I0312 20:10:34.330108 1733 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "012786a8f1b64995847cf7dacbc029a4" candidate_term: 5 candidate_status { last_received { term: 3 index: 70 } } ignore_live_leader: false dest_uuid: "9da22e5afdce42078fe8f94ad166ac3c" is_pre_election: true I0312 20:10:34.330654 1733 raft_consensus.cc:2145] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 012786a8f1b64995847cf7dacbc029a4 in current term 5: Already voted for candidate 9da22e5afdce42078fe8f94ad166ac3c in this term. W0312 20:10:34.330844 1740 rpcz_store.cc:255] Trace: 0312 20:10:33.249338 (+ 0us) service_pool.cc:163] Inserting onto call queue 0312 20:10:33.249428 (+ 90us) service_pool.cc:222] Handling call 0312 20:10:34.330030 (+1080602us) spinlock_profiling.cc:261] Waited 1.08 s on lock 0x7b480003c39c. stack: 00007fede5da3c74 00007fede5da3b70 00007fede4893314 00007feded8e0294 00007feded8e0229 00007fedec48a606 00007feded91e56b 00007fede82d1b24 00007fede82d1ab1 00007fede82d19c4 00007fede792dc29 00007fede792d6ec 00007fede792ebcd 00007fede7930946 00007fede793089a 00007fede7930823 0312 20:10:34.330116 (+ 86us) inbound_call.cc:157] Queueing success response Metrics: {"spinlock_wait_cycles":2484024576} I0312 20:10:34.331254 1734 raft_consensus.cc:2190] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e3aa58d3045c4a05b94b94eb8bf58ae1 for term 5 because replica is already servicing an update from a current leader or another vote. I0312 20:10:34.331097 1990 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "9da22e5afdce42078fe8f94ad166ac3c" candidate_term: 5 candidate_status { last_received { term: 3 index: 70 } } ignore_live_leader: false dest_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" I0312 20:10:34.331478 1990 raft_consensus.cc:2752] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 4 FOLLOWER]: Advancing to term 5 I0312 20:10:34.331591 1738 raft_consensus.cc:2190] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e3aa58d3045c4a05b94b94eb8bf58ae1 for term 5 because replica is already servicing an update from a current leader or another vote. I0312 20:10:34.331681 1990 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1: Snoozing failure detection for 0.825s (vote granted) W0312 20:10:34.331991 1738 rpcz_store.cc:251] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.53.3:59901 (request call id 329) took 917ms (client timeout 213). I0312 20:10:34.332505 1922 leader_election.cc:377] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 5 pre-election: Vote denied by peer 9da22e5afdce42078fe8f94ad166ac3c. Message: Service unavailable: T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e3aa58d3045c4a05b94b94eb8bf58ae1 for term 5 because replica is already servicing an update from a current leader or another vote. W0312 20:10:34.332612 1738 rpcz_store.cc:255] Trace: 0312 20:10:33.414906 (+ 0us) service_pool.cc:163] Inserting onto call queue 0312 20:10:33.415002 (+ 96us) service_pool.cc:222] Handling call 0312 20:10:34.331860 (+916858us) spinlock_profiling.cc:261] Waited 916 ms on lock 0x7b480003c39c. stack: 00007fede5da3c74 00007fede5da3b70 00007fede4893314 00007feded8e0294 00007feded8e0229 00007fedec48a606 00007feded91e56b 00007fede82d1b24 00007fede82d1ab1 00007fede82d19c4 00007fede792dc29 00007fede792d6ec 00007fede792ebcd 00007fede7930946 00007fede793089a 00007fede7930823 0312 20:10:34.331945 (+ 85us) inbound_call.cc:157] Queueing success response Metrics: {"spinlock_wait_cycles":2107388160} W0312 20:10:34.333171 1797 leader_election.cc:347] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [CANDIDATE]: Term 4 election: Vote denied by peer 9da22e5afdce42078fe8f94ad166ac3c with higher term. Message: Invalid argument: T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 012786a8f1b64995847cf7dacbc029a4 for earlier term 4. Current term is 5. I0312 20:10:34.333356 1797 leader_election.cc:350] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [CANDIDATE]: Term 4 election: Cancelling election due to peer responding with higher term I0312 20:10:34.333747 1989 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "9da22e5afdce42078fe8f94ad166ac3c" candidate_term: 6 candidate_status { last_received { term: 3 index: 70 } } ignore_live_leader: false dest_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" is_pre_election: true I0312 20:10:34.334166 1797 leader_election.cc:377] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [CANDIDATE]: Term 5 pre-election: Vote denied by peer 9da22e5afdce42078fe8f94ad166ac3c. Message: Invalid argument: T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 012786a8f1b64995847cf7dacbc029a4 in current term 5: Already voted for candidate 9da22e5afdce42078fe8f94ad166ac3c in this term. I0312 20:10:34.336002 1990 raft_consensus.cc:2215] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9da22e5afdce42078fe8f94ad166ac3c in term 5. I0312 20:10:34.336298 1989 raft_consensus.cc:2190] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9da22e5afdce42078fe8f94ad166ac3c for term 6 because replica is already servicing an update from a current leader or another vote. I0312 20:10:34.328953 2100 raft_consensus.cc:458] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "012786a8f1b64995847cf7dacbc029a4" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 43059 } } peers { permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 } } peers { permanent_uuid: "9da22e5afdce42078fe8f94ad166ac3c" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 39175 } } I0312 20:10:34.336809 1663 leader_election.cc:363] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [CANDIDATE]: Term 5 election: Vote granted by peer e3aa58d3045c4a05b94b94eb8bf58ae1 I0312 20:10:34.336935 2100 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [CANDIDATE]: Term 5 election: Requesting vote from peer e3aa58d3045c4a05b94b94eb8bf58ae1 I0312 20:10:34.337101 1861 raft_consensus.cc:2190] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e3aa58d3045c4a05b94b94eb8bf58ae1 for term 5 because replica is already servicing an update from a current leader or another vote. I0312 20:10:34.337095 1663 leader_election.cc:258] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. I0312 20:10:34.337471 2101 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c: Snoozing failure detection for 0.018s (election complete) W0312 20:10:34.337584 1861 rpcz_store.cc:251] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.53.3:47255 (request call id 304) took 735ms (client timeout 360). I0312 20:10:34.337678 2101 raft_consensus.cc:2507] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 5 FOLLOWER]: Leader election won for term 5 I0312 20:10:34.337698 1663 leader_election.cc:377] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [CANDIDATE]: Term 6 pre-election: Vote denied by peer e3aa58d3045c4a05b94b94eb8bf58ae1. Message: Service unavailable: T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9da22e5afdce42078fe8f94ad166ac3c for term 6 because replica is already servicing an update from a current leader or another vote. I0312 20:10:34.338032 2101 raft_consensus.cc:572] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 5 LEADER]: Becoming Leader. State: Replica: 9da22e5afdce42078fe8f94ad166ac3c, State: Running, Role: LEADER I0312 20:10:34.338111 1989 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "012786a8f1b64995847cf7dacbc029a4" candidate_term: 5 candidate_status { last_received { term: 3 index: 70 } } ignore_live_leader: false dest_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" I0312 20:10:34.338502 1860 raft_consensus.cc:2190] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e3aa58d3045c4a05b94b94eb8bf58ae1 for term 5 because replica is already servicing an update from a current leader or another vote. I0312 20:10:34.338584 1989 raft_consensus.cc:2145] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 012786a8f1b64995847cf7dacbc029a4 in current term 5: Already voted for candidate 9da22e5afdce42078fe8f94ad166ac3c in this term. I0312 20:10:34.338340 2101 consensus_queue.cc:196] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 70, Committed index: 70, Last appended: 3.70, Last appended by leader: 70, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "012786a8f1b64995847cf7dacbc029a4" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 43059 } } peers { permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 } } peers { permanent_uuid: "9da22e5afdce42078fe8f94ad166ac3c" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 39175 } } W0312 20:10:34.338964 1860 rpcz_store.cc:251] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.53.3:47255 (request call id 305) took 577ms (client timeout 369). I0312 20:10:34.339838 1797 leader_election.cc:377] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [CANDIDATE]: Term 5 election: Vote denied by peer e3aa58d3045c4a05b94b94eb8bf58ae1. Message: Invalid argument: T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 012786a8f1b64995847cf7dacbc029a4 in current term 5: Already voted for candidate 9da22e5afdce42078fe8f94ad166ac3c in this term. I0312 20:10:34.340065 1862 raft_consensus.cc:2190] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e3aa58d3045c4a05b94b94eb8bf58ae1 for term 5 because replica is already servicing an update from a current leader or another vote. W0312 20:10:34.340533 1862 rpcz_store.cc:251] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.53.3:47255 (request call id 303) took 926ms (client timeout 213). W0312 20:10:34.339912 1861 rpcz_store.cc:255] Trace: 0312 20:10:33.601897 (+ 0us) service_pool.cc:163] Inserting onto call queue 0312 20:10:33.602007 (+ 110us) service_pool.cc:222] Handling call 0312 20:10:34.337456 (+735449us) spinlock_profiling.cc:261] Waited 734 ms on lock 0x7b480003039c. stack: 00007fee78fc7c74 00007fee78fc7b70 00007fee77ab7314 00007fee80b04294 00007fee80b04229 00007fee7f6ae606 00007fee80b4256b 00007fee7b4f5b24 00007fee7b4f5ab1 00007fee7b4f59c4 00007fee7ab51c29 00007fee7ab516ec 00007fee7ab52bcd 00007fee7ab54946 00007fee7ab5489a 00007fee7ab54823 0312 20:10:34.337537 (+ 81us) inbound_call.cc:157] Queueing success response Metrics: {"spinlock_wait_cycles":1689331456} W0312 20:10:34.341387 1860 rpcz_store.cc:255] Trace: 0312 20:10:33.761983 (+ 0us) service_pool.cc:163] Inserting onto call queue 0312 20:10:33.762092 (+ 109us) service_pool.cc:222] Handling call 0312 20:10:34.338842 (+576750us) spinlock_profiling.cc:261] Waited 575 ms on lock 0x7b480003039c. stack: 00007fee78fc7c74 00007fee78fc7b70 00007fee77ab7314 00007fee80b04294 00007fee80b04229 00007fee7f6ae606 00007fee80b4256b 00007fee7b4f5b24 00007fee7b4f5ab1 00007fee7b4f59c4 00007fee7ab51c29 00007fee7ab516ec 00007fee7ab52bcd 00007fee7ab54946 00007fee7ab5489a 00007fee7ab54823 0312 20:10:34.338921 (+ 79us) inbound_call.cc:157] Queueing success response Metrics: {"spinlock_wait_cycles":1323247872} I0312 20:10:34.339356 2100 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [CANDIDATE]: Term 5 election: Requesting vote from peer 9da22e5afdce42078fe8f94ad166ac3c I0312 20:10:34.341658 1858 raft_consensus.cc:2190] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e3aa58d3045c4a05b94b94eb8bf58ae1 for term 5 because replica is already servicing an update from a current leader or another vote. I0312 20:10:34.341925 1864 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4: Snoozing failure detection for 0.020s (vote granted) I0312 20:10:34.342133 1864 raft_consensus.cc:2215] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9da22e5afdce42078fe8f94ad166ac3c in term 5. W0312 20:10:34.342361 1676 leader_election.cc:287] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 012786a8f1b64995847cf7dacbc029a4: Timed out: RequestConsensusVote RPC to 127.0.53.2:43059 timed out after 0.018s (SENT) I0312 20:10:34.342576 1676 leader_election.cc:258] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. I0312 20:10:34.342928 1926 leader_election.cc:377] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 5 pre-election: Vote denied by peer 012786a8f1b64995847cf7dacbc029a4. Message: Service unavailable: T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e3aa58d3045c4a05b94b94eb8bf58ae1 for term 5 because replica is already servicing an update from a current leader or another vote. I0312 20:10:34.343173 1926 leader_election.cc:258] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. I0312 20:10:34.342589 1617 catalog_manager.cc:3592] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "9da22e5afdce42078fe8f94ad166ac3c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "012786a8f1b64995847cf7dacbc029a4" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 43059 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9da22e5afdce42078fe8f94ad166ac3c" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 39175 } health_report { overall_health: HEALTHY } } } I0312 20:10:34.343722 1738 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "012786a8f1b64995847cf7dacbc029a4" candidate_term: 5 candidate_status { last_received { term: 3 index: 70 } } ignore_live_leader: false dest_uuid: "9da22e5afdce42078fe8f94ad166ac3c" W0312 20:10:34.344092 1862 rpcz_store.cc:255] Trace: 0312 20:10:33.413975 (+ 0us) service_pool.cc:163] Inserting onto call queue 0312 20:10:33.414072 (+ 97us) service_pool.cc:222] Handling call 0312 20:10:34.340404 (+926332us) spinlock_profiling.cc:261] Waited 925 ms on lock 0x7b480003039c. stack: 00007fee78fc7c74 00007fee78fc7b70 00007fee77ab7314 00007fee80b04294 00007fee80b04229 00007fee7f6ae606 00007fee80b4256b 00007fee7b4f5b24 00007fee7b4f5ab1 00007fee7b4f59c4 00007fee7ab51c29 00007fee7ab516ec 00007fee7ab52bcd 00007fee7ab54946 00007fee7ab5489a 00007fee7ab54823 0312 20:10:34.340485 (+ 81us) inbound_call.cc:157] Queueing success response Metrics: {"spinlock_wait_cycles":2127127808} I0312 20:10:34.344199 1738 raft_consensus.cc:2176] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 5 LEADER]: Leader election vote request: Denying vote to candidate 012786a8f1b64995847cf7dacbc029a4 for term 5 because replica is either leader or believes a valid leader to be alive. I0312 20:10:34.344516 1859 raft_consensus.cc:2190] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e3aa58d3045c4a05b94b94eb8bf58ae1 for term 5 because replica is already servicing an update from a current leader or another vote. I0312 20:10:34.344473 2102 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c: Snoozing failure detection for 0.015s (election complete) I0312 20:10:34.344677 2102 raft_consensus.cc:2455] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 5 LEADER]: Leader pre-election lost for term 6. Reason: could not achieve majority W0312 20:10:34.344969 1859 rpcz_store.cc:251] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.53.3:47255 (request call id 306) took 319ms (client timeout 63). W0312 20:10:34.344097 1864 rpcz_store.cc:251] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.53.1:43855 (request call id 28) took 17ms (client timeout 18). I0312 20:10:34.345832 2119 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1: Snoozing failure detection for 1.879s (election complete) I0312 20:10:34.346063 2119 raft_consensus.cc:2455] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority I0312 20:10:34.346498 1989 raft_consensus.cc:1058] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 5 FOLLOWER]: Refusing update from remote peer 9da22e5afdce42078fe8f94ad166ac3c: Log matching property violated. Preceding OpId in replica: term: 1 index: 71. Preceding OpId from leader: term: 5 index: 71. (term mismatch) I0312 20:10:34.346715 1989 pending_rounds.cc:76] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1: Aborting all transactions after (but not including) 70 I0312 20:10:34.346850 1989 pending_rounds.cc:97] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1: Aborting uncommitted WRITE_OP operation due to leader change: 1.71 W0312 20:10:34.347179 1859 rpcz_store.cc:255] Trace: 0312 20:10:34.025875 (+ 0us) service_pool.cc:163] Inserting onto call queue 0312 20:10:34.025972 (+ 97us) service_pool.cc:222] Handling call 0312 20:10:34.344841 (+318869us) spinlock_profiling.cc:261] Waited 316 ms on lock 0x7b480003039c. stack: 00007fee78fc7c74 00007fee78fc7b70 00007fee77ab7314 00007fee80b04294 00007fee80b04229 00007fee7f6ae606 00007fee80b4256b 00007fee7b4f5b24 00007fee7b4f5ab1 00007fee7b4f59c4 00007fee7ab51c29 00007fee7ab516ec 00007fee7ab52bcd 00007fee7ab54946 00007fee7ab5489a 00007fee7ab54823 0312 20:10:34.344920 (+ 79us) inbound_call.cc:157] Queueing success response Metrics: {"spinlock_wait_cycles":726792960} I0312 20:10:34.347563 1797 leader_election.cc:377] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [CANDIDATE]: Term 5 election: Vote denied by peer 9da22e5afdce42078fe8f94ad166ac3c. Message: Invalid argument: T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 5 LEADER]: Leader election vote request: Denying vote to candidate 012786a8f1b64995847cf7dacbc029a4 for term 5 because replica is either leader or believes a valid leader to be alive. I0312 20:10:34.347784 1797 leader_election.cc:258] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. I0312 20:10:34.347527 1989 rpcz_store.cc:265] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:57130 (ReqId={client: 65000f5e93144329adc212a378e8be47, seq_no=69, attempt_no=0}) took 2373ms. Request Metrics: {"child_traces":[["txn",{"num_ops":1,"prepare.queue_time_us":385,"prepare.run_cpu_time_us":929,"prepare.run_wall_time_us":927,"raft.queue_time_us":1545,"raft.run_cpu_time_us":559,"raft.run_wall_time_us":32944,"replication_time_us":2371506,"spinlock_wait_cycles":75489920}]]} I0312 20:10:34.348495 1865 raft_consensus.cc:2190] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 9da22e5afdce42078fe8f94ad166ac3c for term 5 because replica is already servicing an update from a current leader or another vote. W0312 20:10:34.348937 1865 rpcz_store.cc:251] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.53.1:43855 (request call id 27) took 23ms (client timeout 22). I0312 20:10:34.349468 1866 raft_consensus.cc:2190] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e3aa58d3045c4a05b94b94eb8bf58ae1 for term 5 because replica is already servicing an update from a current leader or another vote. I0312 20:10:34.349500 2101 consensus_queue.cc:1004] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [LEADER]: Peer e3aa58d3045c4a05b94b94eb8bf58ae1 log is divergent from this leader: its last log entry 1.70 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 69 I0312 20:10:34.349680 2101 consensus_queue.cc:1017] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 70, Last known committed idx: 69, Time since last communication: 0.000s I0312 20:10:34.349673 2109 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4: Snoozing failure detection for 0.015s (election complete) I0312 20:10:34.350471 2109 raft_consensus.cc:2455] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 5 FOLLOWER]: Leader election lost for term 4. Reason: Vote denied by peer 9da22e5afdce42078fe8f94ad166ac3c with higher term. Message: Invalid argument: T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 012786a8f1b64995847cf7dacbc029a4 for earlier term 4. Current term is 5. I0312 20:10:34.351439 1989 pending_rounds.cc:76] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1: Aborting all transactions after (but not including) 69 I0312 20:10:34.351565 1989 pending_rounds.cc:97] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1: Aborting uncommitted WRITE_OP operation due to leader change: 1.70 W0312 20:10:34.351629 1866 rpcz_store.cc:251] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.53.3:47255 (request call id 300) took 1356ms (client timeout 86). I0312 20:10:34.352149 2117 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4: Snoozing failure detection for 0.033s (election complete) I0312 20:10:34.352318 2117 raft_consensus.cc:2455] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority I0312 20:10:34.352028 1989 rpcz_store.cc:265] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:57130 (ReqId={client: 65000f5e93144329adc212a378e8be47, seq_no=68, attempt_no=0}) took 2379ms. Request Metrics: {"child_traces":[["txn",{"num_ops":1,"prepare.queue_time_us":98,"prepare.run_cpu_time_us":1175,"prepare.run_wall_time_us":1679,"raft.queue_time_us":1129,"raft.run_cpu_time_us":583,"raft.run_wall_time_us":590,"replication_time_us":2377903,"spinlock_wait_cycles":2044544}]]} W0312 20:10:34.352895 1866 rpcz_store.cc:255] Trace: 0312 20:10:32.995615 (+ 0us) service_pool.cc:163] Inserting onto call queue 0312 20:10:32.995715 (+ 100us) service_pool.cc:222] Handling call 0312 20:10:34.351489 (+1355774us) spinlock_profiling.cc:261] Waited 1.35 s on lock 0x7b480003039c. stack: 00007fee78fc7c74 00007fee78fc7b70 00007fee77ab7314 00007fee80b04294 00007fee80b04229 00007fee7f6ae606 00007fee80b4256b 00007fee7b4f5b24 00007fee7b4f5ab1 00007fee7b4f59c4 00007fee7ab51c29 00007fee7ab516ec 00007fee7ab52bcd 00007fee7ab54946 00007fee7ab5489a 00007fee7ab54823 0312 20:10:34.351582 (+ 93us) inbound_call.cc:157] Queueing success response Metrics: {"spinlock_wait_cycles":3112940416} I0312 20:10:34.352893 1859 raft_consensus.cc:1058] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 5 FOLLOWER]: Refusing update from remote peer 9da22e5afdce42078fe8f94ad166ac3c: Log matching property violated. Preceding OpId in replica: term: 3 index: 70. Preceding OpId from leader: term: 5 index: 71. (index mismatch) W0312 20:10:34.353612 1865 rpcz_store.cc:255] Trace: 0312 20:10:34.325812 (+ 0us) service_pool.cc:163] Inserting onto call queue 0312 20:10:34.327594 (+ 1782us) service_pool.cc:222] Handling call 0312 20:10:34.348812 (+ 21218us) spinlock_profiling.cc:261] Waited 17.5 ms on lock 0x7b480003039c. stack: 00007fee78fc7c74 00007fee78fc7b70 00007fee77ab7314 00007fee80b04294 00007fee80b04229 00007fee7f6ae606 00007fee80b4256b 00007fee7b4f5b24 00007fee7b4f5ab1 00007fee7b4f59c4 00007fee7ab51c29 00007fee7ab516ec 00007fee7ab52bcd 00007fee7ab54946 00007fee7ab5489a 00007fee7ab54823 0312 20:10:34.348895 (+ 83us) inbound_call.cc:157] Queueing success response Metrics: {"spinlock_wait_cycles":40162560} W0312 20:10:34.354374 1864 rpcz_store.cc:255] Trace: 0312 20:10:34.326279 (+ 0us) service_pool.cc:163] Inserting onto call queue 0312 20:10:34.326932 (+ 653us) service_pool.cc:222] Handling call 0312 20:10:34.343986 (+ 17054us) spinlock_profiling.cc:261] Waited 14.4 ms on lock 0x7b480003039c. stack: 00007fee78fc7c74 00007fee78fc7b70 00007fee77ab7314 00007fee80b04294 00007fee80b04229 00007fee7f6aeaf3 00007fee80b4256b 00007fee7b4f5b24 00007fee7b4f5ab1 00007fee7b4f59c4 00007fee7ab51c29 00007fee7ab516ec 00007fee7ab52bcd 00007fee7ab54946 00007fee7ab5489a 00007fee7ab54823 0312 20:10:34.344057 (+ 71us) inbound_call.cc:157] Queueing success response Metrics: {"spinlock_wait_cycles":33097088} W0312 20:10:34.354986 1676 leader_election.cc:287] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 012786a8f1b64995847cf7dacbc029a4: Timed out: RequestConsensusVote RPC to 127.0.53.2:43059 timed out after 0.022s (SENT) I0312 20:10:34.356040 2101 consensus_queue.cc:1017] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "012786a8f1b64995847cf7dacbc029a4" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 43059 }, Status: LMP_MISMATCH, Last received: 3.70, Next index: 71, Last known committed idx: 70, Time since last communication: 0.000s I0312 20:10:34.378506 1949 rpcz_store.cc:265] Call kudu.tserver.TabletServerService.Scan from 127.0.0.1:57130 (request call id 77) took 2369ms. Request Metrics: {} I0312 20:10:34.882709 2135 raft_consensus.cc:436] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 9da22e5afdce42078fe8f94ad166ac3c) I0312 20:10:34.882966 2135 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4: Snoozing failure detection for 0.022s (starting election) I0312 20:10:34.883224 2135 raft_consensus.cc:458] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "012786a8f1b64995847cf7dacbc029a4" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 43059 } } peers { permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 } } peers { permanent_uuid: "9da22e5afdce42078fe8f94ad166ac3c" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 39175 } } I0312 20:10:34.884219 2135 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [CANDIDATE]: Term 6 pre-election: Requesting pre-vote from peer e3aa58d3045c4a05b94b94eb8bf58ae1 I0312 20:10:34.885551 2135 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [CANDIDATE]: Term 6 pre-election: Requesting pre-vote from peer 9da22e5afdce42078fe8f94ad166ac3c I0312 20:10:34.885485 1990 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "012786a8f1b64995847cf7dacbc029a4" candidate_term: 6 candidate_status { last_received { term: 5 index: 88 } } ignore_live_leader: false dest_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" is_pre_election: true I0312 20:10:34.886029 1990 raft_consensus.cc:2176] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 012786a8f1b64995847cf7dacbc029a4 for term 6 because replica is either leader or believes a valid leader to be alive. I0312 20:10:34.886854 1797 leader_election.cc:377] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [CANDIDATE]: Term 6 pre-election: Vote denied by peer e3aa58d3045c4a05b94b94eb8bf58ae1. Message: Invalid argument: T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 012786a8f1b64995847cf7dacbc029a4 for term 6 because replica is either leader or believes a valid leader to be alive. I0312 20:10:34.887074 1738 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "012786a8f1b64995847cf7dacbc029a4" candidate_term: 6 candidate_status { last_received { term: 5 index: 88 } } ignore_live_leader: false dest_uuid: "9da22e5afdce42078fe8f94ad166ac3c" is_pre_election: true I0312 20:10:34.887604 1738 raft_consensus.cc:2176] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 5 LEADER]: Leader pre-election vote request: Denying vote to candidate 012786a8f1b64995847cf7dacbc029a4 for term 6 because replica is either leader or believes a valid leader to be alive. I0312 20:10:34.888335 1797 leader_election.cc:377] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [CANDIDATE]: Term 6 pre-election: Vote denied by peer 9da22e5afdce42078fe8f94ad166ac3c. Message: Invalid argument: T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 5 LEADER]: Leader pre-election vote request: Denying vote to candidate 012786a8f1b64995847cf7dacbc029a4 for term 6 because replica is either leader or believes a valid leader to be alive. I0312 20:10:34.888556 1797 leader_election.cc:258] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. I0312 20:10:34.889164 2135 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4: Snoozing failure detection for 0.021s (election complete) I0312 20:10:34.889387 2135 raft_consensus.cc:2455] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority I0312 20:10:36.882143 2141 raft_consensus.cc:436] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 9da22e5afdce42078fe8f94ad166ac3c) I0312 20:10:36.882484 2141 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1: Snoozing failure detection for 0.020s (starting election) I0312 20:10:36.882728 2141 raft_consensus.cc:458] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "012786a8f1b64995847cf7dacbc029a4" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 43059 } } peers { permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 } } peers { permanent_uuid: "9da22e5afdce42078fe8f94ad166ac3c" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 39175 } } I0312 20:10:36.884624 2141 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 6 pre-election: Requesting pre-vote from peer 012786a8f1b64995847cf7dacbc029a4 I0312 20:10:36.885380 2141 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 6 pre-election: Requesting pre-vote from peer 9da22e5afdce42078fe8f94ad166ac3c I0312 20:10:36.887305 1864 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" candidate_term: 6 candidate_status { last_received { term: 5 index: 165 } } ignore_live_leader: false dest_uuid: "012786a8f1b64995847cf7dacbc029a4" is_pre_election: true I0312 20:10:36.887883 1864 raft_consensus.cc:2176] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e3aa58d3045c4a05b94b94eb8bf58ae1 for term 6 because replica is either leader or believes a valid leader to be alive. I0312 20:10:36.888820 1926 leader_election.cc:377] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 6 pre-election: Vote denied by peer 012786a8f1b64995847cf7dacbc029a4. Message: Invalid argument: T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e3aa58d3045c4a05b94b94eb8bf58ae1 for term 6 because replica is either leader or believes a valid leader to be alive. I0312 20:10:36.889883 1738 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" candidate_term: 6 candidate_status { last_received { term: 5 index: 165 } } ignore_live_leader: false dest_uuid: "9da22e5afdce42078fe8f94ad166ac3c" is_pre_election: true I0312 20:10:36.890445 1738 raft_consensus.cc:2176] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 5 LEADER]: Leader pre-election vote request: Denying vote to candidate e3aa58d3045c4a05b94b94eb8bf58ae1 for term 6 because replica is either leader or believes a valid leader to be alive. I0312 20:10:36.901666 1922 leader_election.cc:377] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 6 pre-election: Vote denied by peer 9da22e5afdce42078fe8f94ad166ac3c. Message: Invalid argument: T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 5 LEADER]: Leader pre-election vote request: Denying vote to candidate e3aa58d3045c4a05b94b94eb8bf58ae1 for term 6 because replica is either leader or believes a valid leader to be alive. I0312 20:10:36.901923 1922 leader_election.cc:258] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. I0312 20:10:36.902335 2141 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1: Snoozing failure detection for 0.016s (election complete) I0312 20:10:36.902560 2141 raft_consensus.cc:2455] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority I0312 20:10:38.464759 2146 raft_consensus.cc:436] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 9da22e5afdce42078fe8f94ad166ac3c) I0312 20:10:38.465337 2146 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4: Snoozing failure detection for 0.017s (starting election) I0312 20:10:38.465662 2146 raft_consensus.cc:458] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "012786a8f1b64995847cf7dacbc029a4" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 43059 } } peers { permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 } } peers { permanent_uuid: "9da22e5afdce42078fe8f94ad166ac3c" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 39175 } } I0312 20:10:38.466835 2146 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [CANDIDATE]: Term 6 pre-election: Requesting pre-vote from peer e3aa58d3045c4a05b94b94eb8bf58ae1 I0312 20:10:38.468225 1989 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "012786a8f1b64995847cf7dacbc029a4" candidate_term: 6 candidate_status { last_received { term: 5 index: 224 } } ignore_live_leader: false dest_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" is_pre_election: true I0312 20:10:38.468776 1989 raft_consensus.cc:2190] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 012786a8f1b64995847cf7dacbc029a4 for term 6 because replica is already servicing an update from a current leader or another vote. I0312 20:10:38.469310 2146 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [CANDIDATE]: Term 6 pre-election: Requesting pre-vote from peer 9da22e5afdce42078fe8f94ad166ac3c I0312 20:10:38.470118 1797 leader_election.cc:377] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [CANDIDATE]: Term 6 pre-election: Vote denied by peer e3aa58d3045c4a05b94b94eb8bf58ae1. Message: Service unavailable: T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 012786a8f1b64995847cf7dacbc029a4 for term 6 because replica is already servicing an update from a current leader or another vote. I0312 20:10:38.471169 1738 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "012786a8f1b64995847cf7dacbc029a4" candidate_term: 6 candidate_status { last_received { term: 5 index: 224 } } ignore_live_leader: false dest_uuid: "9da22e5afdce42078fe8f94ad166ac3c" is_pre_election: true I0312 20:10:38.471763 1738 raft_consensus.cc:2176] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 5 LEADER]: Leader pre-election vote request: Denying vote to candidate 012786a8f1b64995847cf7dacbc029a4 for term 6 because replica is either leader or believes a valid leader to be alive. I0312 20:10:38.475040 1797 leader_election.cc:377] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [CANDIDATE]: Term 6 pre-election: Vote denied by peer 9da22e5afdce42078fe8f94ad166ac3c. Message: Invalid argument: T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 5 LEADER]: Leader pre-election vote request: Denying vote to candidate 012786a8f1b64995847cf7dacbc029a4 for term 6 because replica is either leader or believes a valid leader to be alive. I0312 20:10:38.475276 1797 leader_election.cc:258] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. I0312 20:10:38.477668 2146 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4: Snoozing failure detection for 0.019s (election complete) I0312 20:10:38.477881 2146 raft_consensus.cc:2455] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority I0312 20:10:38.667842 2146 raft_consensus.cc:436] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 9da22e5afdce42078fe8f94ad166ac3c) I0312 20:10:38.668115 2146 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4: Snoozing failure detection for 0.043s (starting election) I0312 20:10:38.668349 2146 raft_consensus.cc:458] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "012786a8f1b64995847cf7dacbc029a4" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 43059 } } peers { permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 } } peers { permanent_uuid: "9da22e5afdce42078fe8f94ad166ac3c" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 39175 } } I0312 20:10:38.669420 2146 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [CANDIDATE]: Term 6 pre-election: Requesting pre-vote from peer e3aa58d3045c4a05b94b94eb8bf58ae1 I0312 20:10:38.669802 2146 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [CANDIDATE]: Term 6 pre-election: Requesting pre-vote from peer 9da22e5afdce42078fe8f94ad166ac3c I0312 20:10:38.672200 1990 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "012786a8f1b64995847cf7dacbc029a4" candidate_term: 6 candidate_status { last_received { term: 5 index: 232 } } ignore_live_leader: false dest_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" is_pre_election: true I0312 20:10:38.672685 1990 raft_consensus.cc:2176] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 012786a8f1b64995847cf7dacbc029a4 for term 6 because replica is either leader or believes a valid leader to be alive. I0312 20:10:38.673478 1797 leader_election.cc:377] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [CANDIDATE]: Term 6 pre-election: Vote denied by peer e3aa58d3045c4a05b94b94eb8bf58ae1. Message: Invalid argument: T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 012786a8f1b64995847cf7dacbc029a4 for term 6 because replica is either leader or believes a valid leader to be alive. I0312 20:10:38.674156 1738 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "012786a8f1b64995847cf7dacbc029a4" candidate_term: 6 candidate_status { last_received { term: 5 index: 232 } } ignore_live_leader: false dest_uuid: "9da22e5afdce42078fe8f94ad166ac3c" is_pre_election: true I0312 20:10:38.674616 1738 raft_consensus.cc:2176] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 5 LEADER]: Leader pre-election vote request: Denying vote to candidate 012786a8f1b64995847cf7dacbc029a4 for term 6 because replica is either leader or believes a valid leader to be alive. I0312 20:10:38.676115 1797 leader_election.cc:377] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [CANDIDATE]: Term 6 pre-election: Vote denied by peer 9da22e5afdce42078fe8f94ad166ac3c. Message: Invalid argument: T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 5 LEADER]: Leader pre-election vote request: Denying vote to candidate 012786a8f1b64995847cf7dacbc029a4 for term 6 because replica is either leader or believes a valid leader to be alive. I0312 20:10:38.676283 1797 leader_election.cc:258] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. I0312 20:10:38.676605 2146 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4: Snoozing failure detection for 0.017s (election complete) I0312 20:10:38.676791 2146 raft_consensus.cc:2455] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority I0312 20:10:39.182238 2151 raft_consensus.cc:436] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 9da22e5afdce42078fe8f94ad166ac3c) I0312 20:10:39.183105 2151 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1: Snoozing failure detection for 0.031s (starting election) I0312 20:10:39.183526 2151 raft_consensus.cc:458] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "012786a8f1b64995847cf7dacbc029a4" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 43059 } } peers { permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 } } peers { permanent_uuid: "9da22e5afdce42078fe8f94ad166ac3c" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 39175 } } I0312 20:10:39.185570 2151 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 6 pre-election: Requesting pre-vote from peer 012786a8f1b64995847cf7dacbc029a4 I0312 20:10:39.186004 2151 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 6 pre-election: Requesting pre-vote from peer 9da22e5afdce42078fe8f94ad166ac3c I0312 20:10:39.186854 1864 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" candidate_term: 6 candidate_status { last_received { term: 5 index: 255 } } ignore_live_leader: false dest_uuid: "012786a8f1b64995847cf7dacbc029a4" is_pre_election: true I0312 20:10:39.187417 1864 raft_consensus.cc:2176] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e3aa58d3045c4a05b94b94eb8bf58ae1 for term 6 because replica is either leader or believes a valid leader to be alive. I0312 20:10:39.187597 1738 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" candidate_term: 6 candidate_status { last_received { term: 5 index: 255 } } ignore_live_leader: false dest_uuid: "9da22e5afdce42078fe8f94ad166ac3c" is_pre_election: true I0312 20:10:39.188349 1738 raft_consensus.cc:2176] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 5 LEADER]: Leader pre-election vote request: Denying vote to candidate e3aa58d3045c4a05b94b94eb8bf58ae1 for term 6 because replica is either leader or believes a valid leader to be alive. I0312 20:10:39.189103 1926 leader_election.cc:377] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 6 pre-election: Vote denied by peer 012786a8f1b64995847cf7dacbc029a4. Message: Invalid argument: T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e3aa58d3045c4a05b94b94eb8bf58ae1 for term 6 because replica is either leader or believes a valid leader to be alive. I0312 20:10:39.190985 1922 leader_election.cc:377] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 6 pre-election: Vote denied by peer 9da22e5afdce42078fe8f94ad166ac3c. Message: Invalid argument: T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 5 LEADER]: Leader pre-election vote request: Denying vote to candidate e3aa58d3045c4a05b94b94eb8bf58ae1 for term 6 because replica is either leader or believes a valid leader to be alive. I0312 20:10:39.191193 1922 leader_election.cc:258] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. I0312 20:10:39.193323 2151 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1: Snoozing failure detection for 0.015s (election complete) I0312 20:10:39.193480 2151 raft_consensus.cc:2455] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority I0312 20:10:39.389911 2151 raft_consensus.cc:436] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 9da22e5afdce42078fe8f94ad166ac3c) I0312 20:10:39.390195 2151 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1: Snoozing failure detection for 0.029s (starting election) I0312 20:10:39.390434 2151 raft_consensus.cc:458] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "012786a8f1b64995847cf7dacbc029a4" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 43059 } } peers { permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 } } peers { permanent_uuid: "9da22e5afdce42078fe8f94ad166ac3c" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 39175 } } I0312 20:10:39.391341 2151 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 6 pre-election: Requesting pre-vote from peer 012786a8f1b64995847cf7dacbc029a4 I0312 20:10:39.391705 2151 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 6 pre-election: Requesting pre-vote from peer 9da22e5afdce42078fe8f94ad166ac3c I0312 20:10:39.404373 1864 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" candidate_term: 6 candidate_status { last_received { term: 5 index: 265 } } ignore_live_leader: false dest_uuid: "012786a8f1b64995847cf7dacbc029a4" is_pre_election: true I0312 20:10:39.404955 1864 raft_consensus.cc:2176] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e3aa58d3045c4a05b94b94eb8bf58ae1 for term 6 because replica is either leader or believes a valid leader to be alive. I0312 20:10:39.405110 1738 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" candidate_term: 6 candidate_status { last_received { term: 5 index: 265 } } ignore_live_leader: false dest_uuid: "9da22e5afdce42078fe8f94ad166ac3c" is_pre_election: true I0312 20:10:39.405534 1738 raft_consensus.cc:2176] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 5 LEADER]: Leader pre-election vote request: Denying vote to candidate e3aa58d3045c4a05b94b94eb8bf58ae1 for term 6 because replica is either leader or believes a valid leader to be alive. I0312 20:10:39.405830 1926 leader_election.cc:377] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 6 pre-election: Vote denied by peer 012786a8f1b64995847cf7dacbc029a4. Message: Invalid argument: T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e3aa58d3045c4a05b94b94eb8bf58ae1 for term 6 because replica is either leader or believes a valid leader to be alive. I0312 20:10:39.406566 1922 leader_election.cc:377] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 6 pre-election: Vote denied by peer 9da22e5afdce42078fe8f94ad166ac3c. Message: Invalid argument: T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 5 LEADER]: Leader pre-election vote request: Denying vote to candidate e3aa58d3045c4a05b94b94eb8bf58ae1 for term 6 because replica is either leader or believes a valid leader to be alive. I0312 20:10:39.406781 1922 leader_election.cc:258] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. I0312 20:10:39.407855 2151 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1: Snoozing failure detection for 0.031s (election complete) I0312 20:10:39.408006 2151 raft_consensus.cc:2455] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority I0312 20:10:41.994081 2162 raft_consensus.cc:436] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 9da22e5afdce42078fe8f94ad166ac3c) I0312 20:10:41.994415 2162 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1: Snoozing failure detection for 0.020s (starting election) I0312 20:10:41.994729 2162 raft_consensus.cc:458] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "012786a8f1b64995847cf7dacbc029a4" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 43059 } } peers { permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 } } peers { permanent_uuid: "9da22e5afdce42078fe8f94ad166ac3c" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 39175 } } I0312 20:10:41.995836 2162 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 6 pre-election: Requesting pre-vote from peer 012786a8f1b64995847cf7dacbc029a4 I0312 20:10:41.996230 2162 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 6 pre-election: Requesting pre-vote from peer 9da22e5afdce42078fe8f94ad166ac3c I0312 20:10:41.998594 1864 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" candidate_term: 6 candidate_status { last_received { term: 5 index: 369 } } ignore_live_leader: false dest_uuid: "012786a8f1b64995847cf7dacbc029a4" is_pre_election: true I0312 20:10:41.999004 1864 raft_consensus.cc:2176] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e3aa58d3045c4a05b94b94eb8bf58ae1 for term 6 because replica is either leader or believes a valid leader to be alive. I0312 20:10:42.000375 1926 leader_election.cc:377] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 6 pre-election: Vote denied by peer 012786a8f1b64995847cf7dacbc029a4. Message: Invalid argument: T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e3aa58d3045c4a05b94b94eb8bf58ae1 for term 6 because replica is either leader or believes a valid leader to be alive. I0312 20:10:42.001358 1738 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" candidate_term: 6 candidate_status { last_received { term: 5 index: 369 } } ignore_live_leader: false dest_uuid: "9da22e5afdce42078fe8f94ad166ac3c" is_pre_election: true I0312 20:10:42.001827 1738 raft_consensus.cc:2176] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 5 LEADER]: Leader pre-election vote request: Denying vote to candidate e3aa58d3045c4a05b94b94eb8bf58ae1 for term 6 because replica is either leader or believes a valid leader to be alive. I0312 20:10:42.002868 1922 leader_election.cc:377] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 6 pre-election: Vote denied by peer 9da22e5afdce42078fe8f94ad166ac3c. Message: Invalid argument: T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 5 LEADER]: Leader pre-election vote request: Denying vote to candidate e3aa58d3045c4a05b94b94eb8bf58ae1 for term 6 because replica is either leader or believes a valid leader to be alive. I0312 20:10:42.003111 1922 leader_election.cc:258] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. I0312 20:10:42.004515 2162 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1: Snoozing failure detection for 0.042s (election complete) I0312 20:10:42.004679 2162 raft_consensus.cc:2455] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority I0312 20:10:42.404436 2162 raft_consensus.cc:436] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 9da22e5afdce42078fe8f94ad166ac3c) I0312 20:10:42.404660 2162 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1: Snoozing failure detection for 0.105s (starting election) I0312 20:10:42.404851 2162 raft_consensus.cc:458] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "012786a8f1b64995847cf7dacbc029a4" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 43059 } } peers { permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 } } peers { permanent_uuid: "9da22e5afdce42078fe8f94ad166ac3c" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 39175 } } I0312 20:10:42.405673 2162 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 6 pre-election: Requesting pre-vote from peer 012786a8f1b64995847cf7dacbc029a4 I0312 20:10:42.405964 2162 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 6 pre-election: Requesting pre-vote from peer 9da22e5afdce42078fe8f94ad166ac3c I0312 20:10:42.406751 1865 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" candidate_term: 6 candidate_status { last_received { term: 5 index: 386 } } ignore_live_leader: false dest_uuid: "012786a8f1b64995847cf7dacbc029a4" is_pre_election: true I0312 20:10:42.407042 1738 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" candidate_term: 6 candidate_status { last_received { term: 5 index: 386 } } ignore_live_leader: false dest_uuid: "9da22e5afdce42078fe8f94ad166ac3c" is_pre_election: true I0312 20:10:42.407277 1865 raft_consensus.cc:2176] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e3aa58d3045c4a05b94b94eb8bf58ae1 for term 6 because replica is either leader or believes a valid leader to be alive. I0312 20:10:42.407531 1738 raft_consensus.cc:2176] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 5 LEADER]: Leader pre-election vote request: Denying vote to candidate e3aa58d3045c4a05b94b94eb8bf58ae1 for term 6 because replica is either leader or believes a valid leader to be alive. I0312 20:10:42.407954 1926 leader_election.cc:377] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 6 pre-election: Vote denied by peer 012786a8f1b64995847cf7dacbc029a4. Message: Invalid argument: T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e3aa58d3045c4a05b94b94eb8bf58ae1 for term 6 because replica is either leader or believes a valid leader to be alive. I0312 20:10:42.408387 1922 leader_election.cc:377] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 6 pre-election: Vote denied by peer 9da22e5afdce42078fe8f94ad166ac3c. Message: Invalid argument: T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 5 LEADER]: Leader pre-election vote request: Denying vote to candidate e3aa58d3045c4a05b94b94eb8bf58ae1 for term 6 because replica is either leader or believes a valid leader to be alive. I0312 20:10:42.408558 1922 leader_election.cc:258] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. I0312 20:10:42.408890 2162 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1: Snoozing failure detection for 0.051s (election complete) I0312 20:10:42.409107 2162 raft_consensus.cc:2455] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority I0312 20:10:42.870386 2162 raft_consensus.cc:436] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 9da22e5afdce42078fe8f94ad166ac3c) I0312 20:10:42.870640 2162 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1: Snoozing failure detection for 0.051s (starting election) I0312 20:10:42.870865 2162 raft_consensus.cc:458] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "012786a8f1b64995847cf7dacbc029a4" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 43059 } } peers { permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 } } peers { permanent_uuid: "9da22e5afdce42078fe8f94ad166ac3c" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 39175 } } I0312 20:10:42.871887 2162 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 6 pre-election: Requesting pre-vote from peer 012786a8f1b64995847cf7dacbc029a4 I0312 20:10:42.872253 2162 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 6 pre-election: Requesting pre-vote from peer 9da22e5afdce42078fe8f94ad166ac3c I0312 20:10:42.872921 1864 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" candidate_term: 6 candidate_status { last_received { term: 5 index: 404 } } ignore_live_leader: false dest_uuid: "012786a8f1b64995847cf7dacbc029a4" is_pre_election: true I0312 20:10:42.873409 1864 raft_consensus.cc:2176] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e3aa58d3045c4a05b94b94eb8bf58ae1 for term 6 because replica is either leader or believes a valid leader to be alive. I0312 20:10:42.874114 1926 leader_election.cc:377] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 6 pre-election: Vote denied by peer 012786a8f1b64995847cf7dacbc029a4. Message: Invalid argument: T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e3aa58d3045c4a05b94b94eb8bf58ae1 for term 6 because replica is either leader or believes a valid leader to be alive. I0312 20:10:42.875058 1738 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" candidate_term: 6 candidate_status { last_received { term: 5 index: 404 } } ignore_live_leader: false dest_uuid: "9da22e5afdce42078fe8f94ad166ac3c" is_pre_election: true I0312 20:10:42.875602 1738 raft_consensus.cc:2176] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 5 LEADER]: Leader pre-election vote request: Denying vote to candidate e3aa58d3045c4a05b94b94eb8bf58ae1 for term 6 because replica is either leader or believes a valid leader to be alive. I0312 20:10:42.882238 1922 leader_election.cc:377] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 6 pre-election: Vote denied by peer 9da22e5afdce42078fe8f94ad166ac3c. Message: Invalid argument: T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 5 LEADER]: Leader pre-election vote request: Denying vote to candidate e3aa58d3045c4a05b94b94eb8bf58ae1 for term 6 because replica is either leader or believes a valid leader to be alive. I0312 20:10:42.882457 1922 leader_election.cc:258] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. I0312 20:10:42.883175 2162 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1: Snoozing failure detection for 0.064s (election complete) I0312 20:10:42.883388 2162 raft_consensus.cc:2455] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority I0312 20:10:43.921607 2165 raft_consensus.cc:436] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 9da22e5afdce42078fe8f94ad166ac3c) I0312 20:10:43.922220 2165 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1: Snoozing failure detection for 0.171s (starting election) I0312 20:10:43.922430 2165 raft_consensus.cc:458] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "012786a8f1b64995847cf7dacbc029a4" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 43059 } } peers { permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 } } peers { permanent_uuid: "9da22e5afdce42078fe8f94ad166ac3c" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 39175 } } I0312 20:10:43.923166 2165 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 6 pre-election: Requesting pre-vote from peer 012786a8f1b64995847cf7dacbc029a4 I0312 20:10:43.923545 2165 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 6 pre-election: Requesting pre-vote from peer 9da22e5afdce42078fe8f94ad166ac3c I0312 20:10:43.926160 1865 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" candidate_term: 6 candidate_status { last_received { term: 5 index: 438 } } ignore_live_leader: false dest_uuid: "012786a8f1b64995847cf7dacbc029a4" is_pre_election: true I0312 20:10:43.928587 1738 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" candidate_term: 6 candidate_status { last_received { term: 5 index: 438 } } ignore_live_leader: false dest_uuid: "9da22e5afdce42078fe8f94ad166ac3c" is_pre_election: true I0312 20:10:43.929324 1738 raft_consensus.cc:2176] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 5 LEADER]: Leader pre-election vote request: Denying vote to candidate e3aa58d3045c4a05b94b94eb8bf58ae1 for term 6 because replica is either leader or believes a valid leader to be alive. I0312 20:10:43.930902 1922 leader_election.cc:377] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 6 pre-election: Vote denied by peer 9da22e5afdce42078fe8f94ad166ac3c. Message: Invalid argument: T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 5 LEADER]: Leader pre-election vote request: Denying vote to candidate e3aa58d3045c4a05b94b94eb8bf58ae1 for term 6 because replica is either leader or believes a valid leader to be alive. I0312 20:10:43.937800 1865 raft_consensus.cc:2190] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e3aa58d3045c4a05b94b94eb8bf58ae1 for term 6 because replica is already servicing an update from a current leader or another vote. I0312 20:10:43.938899 1926 leader_election.cc:377] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 6 pre-election: Vote denied by peer 012786a8f1b64995847cf7dacbc029a4. Message: Service unavailable: T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e3aa58d3045c4a05b94b94eb8bf58ae1 for term 6 because replica is already servicing an update from a current leader or another vote. I0312 20:10:43.939122 1926 leader_election.cc:258] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. I0312 20:10:43.939548 2165 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1: Snoozing failure detection for 0.021s (election complete) I0312 20:10:43.939733 2165 raft_consensus.cc:2455] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority I0312 20:10:43.940631 2168 raft_consensus.cc:436] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 9da22e5afdce42078fe8f94ad166ac3c) I0312 20:10:43.940811 2168 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4: Snoozing failure detection for 0.037s (starting election) I0312 20:10:43.940994 2168 raft_consensus.cc:458] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "012786a8f1b64995847cf7dacbc029a4" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 43059 } } peers { permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 } } peers { permanent_uuid: "9da22e5afdce42078fe8f94ad166ac3c" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 39175 } } I0312 20:10:43.941782 2168 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [CANDIDATE]: Term 6 pre-election: Requesting pre-vote from peer e3aa58d3045c4a05b94b94eb8bf58ae1 I0312 20:10:43.943078 1990 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "012786a8f1b64995847cf7dacbc029a4" candidate_term: 6 candidate_status { last_received { term: 5 index: 440 } } ignore_live_leader: false dest_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" is_pre_election: true I0312 20:10:43.943599 1990 raft_consensus.cc:2176] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 012786a8f1b64995847cf7dacbc029a4 for term 6 because replica is either leader or believes a valid leader to be alive. I0312 20:10:43.944087 2168 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [CANDIDATE]: Term 6 pre-election: Requesting pre-vote from peer 9da22e5afdce42078fe8f94ad166ac3c I0312 20:10:43.944502 1797 leader_election.cc:377] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [CANDIDATE]: Term 6 pre-election: Vote denied by peer e3aa58d3045c4a05b94b94eb8bf58ae1. Message: Invalid argument: T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 012786a8f1b64995847cf7dacbc029a4 for term 6 because replica is either leader or believes a valid leader to be alive. I0312 20:10:43.945462 1738 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "012786a8f1b64995847cf7dacbc029a4" candidate_term: 6 candidate_status { last_received { term: 5 index: 440 } } ignore_live_leader: false dest_uuid: "9da22e5afdce42078fe8f94ad166ac3c" is_pre_election: true I0312 20:10:43.945832 1738 raft_consensus.cc:2176] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 5 LEADER]: Leader pre-election vote request: Denying vote to candidate 012786a8f1b64995847cf7dacbc029a4 for term 6 because replica is either leader or believes a valid leader to be alive. I0312 20:10:43.946768 1797 leader_election.cc:377] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [CANDIDATE]: Term 6 pre-election: Vote denied by peer 9da22e5afdce42078fe8f94ad166ac3c. Message: Invalid argument: T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 5 LEADER]: Leader pre-election vote request: Denying vote to candidate 012786a8f1b64995847cf7dacbc029a4 for term 6 because replica is either leader or believes a valid leader to be alive. I0312 20:10:43.946979 1797 leader_election.cc:258] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. I0312 20:10:43.947625 2168 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4: Snoozing failure detection for 0.024s (election complete) I0312 20:10:43.947845 2168 raft_consensus.cc:2455] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority I0312 20:10:44.092483 2168 raft_consensus.cc:436] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 9da22e5afdce42078fe8f94ad166ac3c) I0312 20:10:44.092677 2168 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4: Snoozing failure detection for 0.103s (starting election) I0312 20:10:44.092852 2168 raft_consensus.cc:458] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "012786a8f1b64995847cf7dacbc029a4" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 43059 } } peers { permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 } } peers { permanent_uuid: "9da22e5afdce42078fe8f94ad166ac3c" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 39175 } } I0312 20:10:44.093688 2168 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [CANDIDATE]: Term 6 pre-election: Requesting pre-vote from peer e3aa58d3045c4a05b94b94eb8bf58ae1 I0312 20:10:44.093996 2168 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [CANDIDATE]: Term 6 pre-election: Requesting pre-vote from peer 9da22e5afdce42078fe8f94ad166ac3c I0312 20:10:44.095012 1989 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "012786a8f1b64995847cf7dacbc029a4" candidate_term: 6 candidate_status { last_received { term: 5 index: 445 } } ignore_live_leader: false dest_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" is_pre_election: true I0312 20:10:44.095057 1738 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "012786a8f1b64995847cf7dacbc029a4" candidate_term: 6 candidate_status { last_received { term: 5 index: 445 } } ignore_live_leader: false dest_uuid: "9da22e5afdce42078fe8f94ad166ac3c" is_pre_election: true I0312 20:10:44.095528 1989 raft_consensus.cc:2190] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 012786a8f1b64995847cf7dacbc029a4 for term 6 because replica is already servicing an update from a current leader or another vote. I0312 20:10:44.095561 1738 raft_consensus.cc:2176] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 5 LEADER]: Leader pre-election vote request: Denying vote to candidate 012786a8f1b64995847cf7dacbc029a4 for term 6 because replica is either leader or believes a valid leader to be alive. I0312 20:10:44.096329 1797 leader_election.cc:377] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [CANDIDATE]: Term 6 pre-election: Vote denied by peer e3aa58d3045c4a05b94b94eb8bf58ae1. Message: Service unavailable: T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 012786a8f1b64995847cf7dacbc029a4 for term 6 because replica is already servicing an update from a current leader or another vote. I0312 20:10:44.096717 1797 leader_election.cc:377] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [CANDIDATE]: Term 6 pre-election: Vote denied by peer 9da22e5afdce42078fe8f94ad166ac3c. Message: Invalid argument: T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 5 LEADER]: Leader pre-election vote request: Denying vote to candidate 012786a8f1b64995847cf7dacbc029a4 for term 6 because replica is either leader or believes a valid leader to be alive. I0312 20:10:44.096930 1797 leader_election.cc:258] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. I0312 20:10:44.097266 2168 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4: Snoozing failure detection for 0.095s (election complete) I0312 20:10:44.097462 2168 raft_consensus.cc:2455] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority I0312 20:10:44.497833 2170 raft_consensus.cc:436] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 9da22e5afdce42078fe8f94ad166ac3c) I0312 20:10:44.498142 2170 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1: Snoozing failure detection for 0.357s (starting election) I0312 20:10:44.498350 2170 raft_consensus.cc:458] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "012786a8f1b64995847cf7dacbc029a4" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 43059 } } peers { permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 } } peers { permanent_uuid: "9da22e5afdce42078fe8f94ad166ac3c" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 39175 } } I0312 20:10:44.499596 2170 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 6 pre-election: Requesting pre-vote from peer 012786a8f1b64995847cf7dacbc029a4 I0312 20:10:44.500002 2170 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 6 pre-election: Requesting pre-vote from peer 9da22e5afdce42078fe8f94ad166ac3c I0312 20:10:44.501166 1864 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" candidate_term: 6 candidate_status { last_received { term: 5 index: 465 } } ignore_live_leader: false dest_uuid: "012786a8f1b64995847cf7dacbc029a4" is_pre_election: true I0312 20:10:44.501281 1738 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" candidate_term: 6 candidate_status { last_received { term: 5 index: 465 } } ignore_live_leader: false dest_uuid: "9da22e5afdce42078fe8f94ad166ac3c" is_pre_election: true I0312 20:10:44.501596 1864 raft_consensus.cc:2176] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e3aa58d3045c4a05b94b94eb8bf58ae1 for term 6 because replica is either leader or believes a valid leader to be alive. I0312 20:10:44.501657 1738 raft_consensus.cc:2176] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 5 LEADER]: Leader pre-election vote request: Denying vote to candidate e3aa58d3045c4a05b94b94eb8bf58ae1 for term 6 because replica is either leader or believes a valid leader to be alive. I0312 20:10:44.502334 1926 leader_election.cc:377] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 6 pre-election: Vote denied by peer 012786a8f1b64995847cf7dacbc029a4. Message: Invalid argument: T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e3aa58d3045c4a05b94b94eb8bf58ae1 for term 6 because replica is either leader or believes a valid leader to be alive. I0312 20:10:44.503626 1922 leader_election.cc:377] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 6 pre-election: Vote denied by peer 9da22e5afdce42078fe8f94ad166ac3c. Message: Invalid argument: T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 5 LEADER]: Leader pre-election vote request: Denying vote to candidate e3aa58d3045c4a05b94b94eb8bf58ae1 for term 6 because replica is either leader or believes a valid leader to be alive. I0312 20:10:44.503821 1922 leader_election.cc:258] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. I0312 20:10:44.504307 2170 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1: Snoozing failure detection for 0.107s (election complete) I0312 20:10:44.504462 2170 raft_consensus.cc:2455] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority I0312 20:10:44.871317 2170 raft_consensus.cc:436] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 9da22e5afdce42078fe8f94ad166ac3c) I0312 20:10:44.871531 2170 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1: Snoozing failure detection for 0.400s (starting election) I0312 20:10:44.871700 2170 raft_consensus.cc:458] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "012786a8f1b64995847cf7dacbc029a4" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 43059 } } peers { permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 } } peers { permanent_uuid: "9da22e5afdce42078fe8f94ad166ac3c" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 39175 } } I0312 20:10:44.872495 2170 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 6 pre-election: Requesting pre-vote from peer 012786a8f1b64995847cf7dacbc029a4 I0312 20:10:44.872782 2170 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 6 pre-election: Requesting pre-vote from peer 9da22e5afdce42078fe8f94ad166ac3c I0312 20:10:44.873386 1864 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" candidate_term: 6 candidate_status { last_received { term: 5 index: 478 } } ignore_live_leader: false dest_uuid: "012786a8f1b64995847cf7dacbc029a4" is_pre_election: true I0312 20:10:44.873807 1864 raft_consensus.cc:2176] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e3aa58d3045c4a05b94b94eb8bf58ae1 for term 6 because replica is either leader or believes a valid leader to be alive. I0312 20:10:44.874429 1926 leader_election.cc:377] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 6 pre-election: Vote denied by peer 012786a8f1b64995847cf7dacbc029a4. Message: Invalid argument: T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e3aa58d3045c4a05b94b94eb8bf58ae1 for term 6 because replica is either leader or believes a valid leader to be alive. I0312 20:10:44.878527 1738 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" candidate_term: 6 candidate_status { last_received { term: 5 index: 478 } } ignore_live_leader: false dest_uuid: "9da22e5afdce42078fe8f94ad166ac3c" is_pre_election: true I0312 20:10:44.878913 1738 raft_consensus.cc:2176] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 5 LEADER]: Leader pre-election vote request: Denying vote to candidate e3aa58d3045c4a05b94b94eb8bf58ae1 for term 6 because replica is either leader or believes a valid leader to be alive. I0312 20:10:44.881587 1922 leader_election.cc:377] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 6 pre-election: Vote denied by peer 9da22e5afdce42078fe8f94ad166ac3c. Message: Invalid argument: T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 5 LEADER]: Leader pre-election vote request: Denying vote to candidate e3aa58d3045c4a05b94b94eb8bf58ae1 for term 6 because replica is either leader or believes a valid leader to be alive. I0312 20:10:44.881878 1922 leader_election.cc:258] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. I0312 20:10:44.882196 2170 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1: Snoozing failure detection for 0.208s (election complete) I0312 20:10:44.882355 2170 raft_consensus.cc:2455] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority I0312 20:10:45.005519 2170 raft_consensus.cc:436] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 9da22e5afdce42078fe8f94ad166ac3c) I0312 20:10:45.005719 2170 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1: Snoozing failure detection for 0.270s (starting election) I0312 20:10:45.005880 2170 raft_consensus.cc:458] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "012786a8f1b64995847cf7dacbc029a4" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 43059 } } peers { permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 } } peers { permanent_uuid: "9da22e5afdce42078fe8f94ad166ac3c" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 39175 } } I0312 20:10:45.006629 2170 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 6 pre-election: Requesting pre-vote from peer 012786a8f1b64995847cf7dacbc029a4 I0312 20:10:45.006904 2170 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 6 pre-election: Requesting pre-vote from peer 9da22e5afdce42078fe8f94ad166ac3c I0312 20:10:45.007535 1865 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" candidate_term: 6 candidate_status { last_received { term: 5 index: 482 } } ignore_live_leader: false dest_uuid: "012786a8f1b64995847cf7dacbc029a4" is_pre_election: true I0312 20:10:45.007949 1865 raft_consensus.cc:2176] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e3aa58d3045c4a05b94b94eb8bf58ae1 for term 6 because replica is either leader or believes a valid leader to be alive. I0312 20:10:45.008584 1926 leader_election.cc:377] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 6 pre-election: Vote denied by peer 012786a8f1b64995847cf7dacbc029a4. Message: Invalid argument: T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e3aa58d3045c4a05b94b94eb8bf58ae1 for term 6 because replica is either leader or believes a valid leader to be alive. I0312 20:10:45.008533 1738 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" candidate_term: 6 candidate_status { last_received { term: 5 index: 482 } } ignore_live_leader: false dest_uuid: "9da22e5afdce42078fe8f94ad166ac3c" is_pre_election: true I0312 20:10:45.008899 1738 raft_consensus.cc:2176] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 5 LEADER]: Leader pre-election vote request: Denying vote to candidate e3aa58d3045c4a05b94b94eb8bf58ae1 for term 6 because replica is either leader or believes a valid leader to be alive. I0312 20:10:45.009603 1922 leader_election.cc:377] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 6 pre-election: Vote denied by peer 9da22e5afdce42078fe8f94ad166ac3c. Message: Invalid argument: T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 5 LEADER]: Leader pre-election vote request: Denying vote to candidate e3aa58d3045c4a05b94b94eb8bf58ae1 for term 6 because replica is either leader or believes a valid leader to be alive. I0312 20:10:45.010226 1922 leader_election.cc:258] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. I0312 20:10:45.010586 2170 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1: Snoozing failure detection for 0.743s (election complete) I0312 20:10:45.010740 2170 raft_consensus.cc:2455] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority I0312 20:10:47.115236 2173 raft_consensus.cc:436] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 9da22e5afdce42078fe8f94ad166ac3c) I0312 20:10:47.115516 2173 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1: Snoozing failure detection for 0.853s (starting election) I0312 20:10:47.115710 2173 raft_consensus.cc:458] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "012786a8f1b64995847cf7dacbc029a4" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 43059 } } peers { permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 } } peers { permanent_uuid: "9da22e5afdce42078fe8f94ad166ac3c" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 39175 } } I0312 20:10:47.116860 2173 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 6 pre-election: Requesting pre-vote from peer 012786a8f1b64995847cf7dacbc029a4 I0312 20:10:47.117601 2173 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 6 pre-election: Requesting pre-vote from peer 9da22e5afdce42078fe8f94ad166ac3c I0312 20:10:47.118543 1738 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" candidate_term: 6 candidate_status { last_received { term: 5 index: 575 } } ignore_live_leader: false dest_uuid: "9da22e5afdce42078fe8f94ad166ac3c" is_pre_election: true I0312 20:10:47.118940 1738 raft_consensus.cc:2176] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 5 LEADER]: Leader pre-election vote request: Denying vote to candidate e3aa58d3045c4a05b94b94eb8bf58ae1 for term 6 because replica is either leader or believes a valid leader to be alive. I0312 20:10:47.121022 1864 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" candidate_term: 6 candidate_status { last_received { term: 5 index: 575 } } ignore_live_leader: false dest_uuid: "012786a8f1b64995847cf7dacbc029a4" is_pre_election: true I0312 20:10:47.121464 1864 raft_consensus.cc:2176] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e3aa58d3045c4a05b94b94eb8bf58ae1 for term 6 because replica is either leader or believes a valid leader to be alive. I0312 20:10:47.121903 1922 leader_election.cc:377] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 6 pre-election: Vote denied by peer 9da22e5afdce42078fe8f94ad166ac3c. Message: Invalid argument: T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 5 LEADER]: Leader pre-election vote request: Denying vote to candidate e3aa58d3045c4a05b94b94eb8bf58ae1 for term 6 because replica is either leader or believes a valid leader to be alive. I0312 20:10:47.122447 1926 leader_election.cc:377] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 6 pre-election: Vote denied by peer 012786a8f1b64995847cf7dacbc029a4. Message: Invalid argument: T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e3aa58d3045c4a05b94b94eb8bf58ae1 for term 6 because replica is either leader or believes a valid leader to be alive. I0312 20:10:47.122638 1926 leader_election.cc:258] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. I0312 20:10:47.123201 2173 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1: Snoozing failure detection for 0.186s (election complete) I0312 20:10:47.123741 2173 raft_consensus.cc:2455] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority I0312 20:10:47.501345 2173 raft_consensus.cc:436] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 9da22e5afdce42078fe8f94ad166ac3c) I0312 20:10:47.501536 2173 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1: Snoozing failure detection for 0.495s (starting election) I0312 20:10:47.501687 2173 raft_consensus.cc:458] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "012786a8f1b64995847cf7dacbc029a4" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 43059 } } peers { permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 } } peers { permanent_uuid: "9da22e5afdce42078fe8f94ad166ac3c" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 39175 } } I0312 20:10:47.502449 2173 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 6 pre-election: Requesting pre-vote from peer 012786a8f1b64995847cf7dacbc029a4 I0312 20:10:47.502712 2173 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 6 pre-election: Requesting pre-vote from peer 9da22e5afdce42078fe8f94ad166ac3c I0312 20:10:47.504348 1738 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" candidate_term: 6 candidate_status { last_received { term: 5 index: 591 } } ignore_live_leader: false dest_uuid: "9da22e5afdce42078fe8f94ad166ac3c" is_pre_election: true I0312 20:10:47.504747 1738 raft_consensus.cc:2176] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 5 LEADER]: Leader pre-election vote request: Denying vote to candidate e3aa58d3045c4a05b94b94eb8bf58ae1 for term 6 because replica is either leader or believes a valid leader to be alive. I0312 20:10:47.507158 1864 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" candidate_term: 6 candidate_status { last_received { term: 5 index: 591 } } ignore_live_leader: false dest_uuid: "012786a8f1b64995847cf7dacbc029a4" is_pre_election: true I0312 20:10:47.506120 1922 leader_election.cc:377] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 6 pre-election: Vote denied by peer 9da22e5afdce42078fe8f94ad166ac3c. Message: Invalid argument: T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 5 LEADER]: Leader pre-election vote request: Denying vote to candidate e3aa58d3045c4a05b94b94eb8bf58ae1 for term 6 because replica is either leader or believes a valid leader to be alive. I0312 20:10:47.508028 1864 raft_consensus.cc:2176] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e3aa58d3045c4a05b94b94eb8bf58ae1 for term 6 because replica is either leader or believes a valid leader to be alive. I0312 20:10:47.508711 1926 leader_election.cc:377] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 6 pre-election: Vote denied by peer 012786a8f1b64995847cf7dacbc029a4. Message: Invalid argument: T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e3aa58d3045c4a05b94b94eb8bf58ae1 for term 6 because replica is either leader or believes a valid leader to be alive. I0312 20:10:47.508883 1926 leader_election.cc:258] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. I0312 20:10:47.509219 2173 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1: Snoozing failure detection for 1.205s (election complete) I0312 20:10:47.509372 2173 raft_consensus.cc:2455] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority I0312 20:10:47.825783 2173 raft_consensus.cc:436] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 9da22e5afdce42078fe8f94ad166ac3c) I0312 20:10:47.825978 2173 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1: Snoozing failure detection for 0.358s (starting election) I0312 20:10:47.826144 2173 raft_consensus.cc:458] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "012786a8f1b64995847cf7dacbc029a4" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 43059 } } peers { permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 } } peers { permanent_uuid: "9da22e5afdce42078fe8f94ad166ac3c" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 39175 } } I0312 20:10:47.826885 2173 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 6 pre-election: Requesting pre-vote from peer 012786a8f1b64995847cf7dacbc029a4 I0312 20:10:47.827167 2173 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 6 pre-election: Requesting pre-vote from peer 9da22e5afdce42078fe8f94ad166ac3c I0312 20:10:47.833843 1864 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" candidate_term: 6 candidate_status { last_received { term: 5 index: 603 } } ignore_live_leader: false dest_uuid: "012786a8f1b64995847cf7dacbc029a4" is_pre_election: true I0312 20:10:47.834379 1864 raft_consensus.cc:2176] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e3aa58d3045c4a05b94b94eb8bf58ae1 for term 6 because replica is either leader or believes a valid leader to be alive. I0312 20:10:47.835038 1926 leader_election.cc:377] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 6 pre-election: Vote denied by peer 012786a8f1b64995847cf7dacbc029a4. Message: Invalid argument: T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e3aa58d3045c4a05b94b94eb8bf58ae1 for term 6 because replica is either leader or believes a valid leader to be alive. I0312 20:10:47.842252 1738 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" candidate_term: 6 candidate_status { last_received { term: 5 index: 603 } } ignore_live_leader: false dest_uuid: "9da22e5afdce42078fe8f94ad166ac3c" is_pre_election: true I0312 20:10:47.842917 1738 raft_consensus.cc:2176] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 5 LEADER]: Leader pre-election vote request: Denying vote to candidate e3aa58d3045c4a05b94b94eb8bf58ae1 for term 6 because replica is either leader or believes a valid leader to be alive. I0312 20:10:47.844228 1922 leader_election.cc:377] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 6 pre-election: Vote denied by peer 9da22e5afdce42078fe8f94ad166ac3c. Message: Invalid argument: T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 5 LEADER]: Leader pre-election vote request: Denying vote to candidate e3aa58d3045c4a05b94b94eb8bf58ae1 for term 6 because replica is either leader or believes a valid leader to be alive. I0312 20:10:47.844413 1922 leader_election.cc:258] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. I0312 20:10:47.844710 2173 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1: Snoozing failure detection for 0.575s (election complete) I0312 20:10:47.844847 2173 raft_consensus.cc:2455] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority I0312 20:10:48.571705 2177 raft_consensus.cc:436] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 9da22e5afdce42078fe8f94ad166ac3c) I0312 20:10:48.572203 2177 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4: Snoozing failure detection for 0.099s (starting election) I0312 20:10:48.572448 2177 raft_consensus.cc:458] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "012786a8f1b64995847cf7dacbc029a4" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 43059 } } peers { permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 } } peers { permanent_uuid: "9da22e5afdce42078fe8f94ad166ac3c" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 39175 } } I0312 20:10:48.573493 2177 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [CANDIDATE]: Term 6 pre-election: Requesting pre-vote from peer e3aa58d3045c4a05b94b94eb8bf58ae1 I0312 20:10:48.573806 2177 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [CANDIDATE]: Term 6 pre-election: Requesting pre-vote from peer 9da22e5afdce42078fe8f94ad166ac3c I0312 20:10:48.574892 1738 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "012786a8f1b64995847cf7dacbc029a4" candidate_term: 6 candidate_status { last_received { term: 5 index: 629 } } ignore_live_leader: false dest_uuid: "9da22e5afdce42078fe8f94ad166ac3c" is_pre_election: true I0312 20:10:48.575282 1738 raft_consensus.cc:2176] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 5 LEADER]: Leader pre-election vote request: Denying vote to candidate 012786a8f1b64995847cf7dacbc029a4 for term 6 because replica is either leader or believes a valid leader to be alive. I0312 20:10:48.574554 1990 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "012786a8f1b64995847cf7dacbc029a4" candidate_term: 6 candidate_status { last_received { term: 5 index: 629 } } ignore_live_leader: false dest_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" is_pre_election: true I0312 20:10:48.575901 1990 raft_consensus.cc:2176] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 012786a8f1b64995847cf7dacbc029a4 for term 6 because replica is either leader or believes a valid leader to be alive. I0312 20:10:48.576139 1797 leader_election.cc:377] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [CANDIDATE]: Term 6 pre-election: Vote denied by peer 9da22e5afdce42078fe8f94ad166ac3c. Message: Invalid argument: T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 5 LEADER]: Leader pre-election vote request: Denying vote to candidate 012786a8f1b64995847cf7dacbc029a4 for term 6 because replica is either leader or believes a valid leader to be alive. I0312 20:10:48.576545 1797 leader_election.cc:377] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [CANDIDATE]: Term 6 pre-election: Vote denied by peer e3aa58d3045c4a05b94b94eb8bf58ae1. Message: Invalid argument: T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 012786a8f1b64995847cf7dacbc029a4 for term 6 because replica is either leader or believes a valid leader to be alive. I0312 20:10:48.576700 1797 leader_election.cc:258] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. I0312 20:10:48.577056 2177 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4: Snoozing failure detection for 0.027s (election complete) I0312 20:10:48.577244 2177 raft_consensus.cc:2455] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority I0312 20:10:48.788239 2178 raft_consensus.cc:436] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 9da22e5afdce42078fe8f94ad166ac3c) I0312 20:10:48.788491 2178 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1: Snoozing failure detection for 3.056s (starting election) I0312 20:10:48.788688 2178 raft_consensus.cc:458] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "012786a8f1b64995847cf7dacbc029a4" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 43059 } } peers { permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 } } peers { permanent_uuid: "9da22e5afdce42078fe8f94ad166ac3c" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 39175 } } I0312 20:10:48.789921 2178 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 6 pre-election: Requesting pre-vote from peer 012786a8f1b64995847cf7dacbc029a4 I0312 20:10:48.790191 2178 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 6 pre-election: Requesting pre-vote from peer 9da22e5afdce42078fe8f94ad166ac3c I0312 20:10:48.791304 1738 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" candidate_term: 6 candidate_status { last_received { term: 5 index: 636 } } ignore_live_leader: false dest_uuid: "9da22e5afdce42078fe8f94ad166ac3c" is_pre_election: true I0312 20:10:48.791352 1865 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" candidate_term: 6 candidate_status { last_received { term: 5 index: 636 } } ignore_live_leader: false dest_uuid: "012786a8f1b64995847cf7dacbc029a4" is_pre_election: true I0312 20:10:48.791682 1738 raft_consensus.cc:2176] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 5 LEADER]: Leader pre-election vote request: Denying vote to candidate e3aa58d3045c4a05b94b94eb8bf58ae1 for term 6 because replica is either leader or believes a valid leader to be alive. I0312 20:10:48.791826 1865 raft_consensus.cc:2176] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e3aa58d3045c4a05b94b94eb8bf58ae1 for term 6 because replica is either leader or believes a valid leader to be alive. I0312 20:10:48.793501 1926 leader_election.cc:377] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 6 pre-election: Vote denied by peer 012786a8f1b64995847cf7dacbc029a4. Message: Invalid argument: T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e3aa58d3045c4a05b94b94eb8bf58ae1 for term 6 because replica is either leader or believes a valid leader to be alive. I0312 20:10:48.794085 1922 leader_election.cc:377] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 6 pre-election: Vote denied by peer 9da22e5afdce42078fe8f94ad166ac3c. Message: Invalid argument: T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 5 LEADER]: Leader pre-election vote request: Denying vote to candidate e3aa58d3045c4a05b94b94eb8bf58ae1 for term 6 because replica is either leader or believes a valid leader to be alive. I0312 20:10:48.794270 1922 leader_election.cc:258] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. I0312 20:10:48.795004 2178 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1: Snoozing failure detection for 0.639s (election complete) I0312 20:10:48.795217 2178 raft_consensus.cc:2455] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority I0312 20:10:49.494832 2180 raft_consensus.cc:436] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 9da22e5afdce42078fe8f94ad166ac3c) I0312 20:10:49.495039 2180 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4: Snoozing failure detection for 0.128s (starting election) I0312 20:10:49.495208 2180 raft_consensus.cc:458] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "012786a8f1b64995847cf7dacbc029a4" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 43059 } } peers { permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 } } peers { permanent_uuid: "9da22e5afdce42078fe8f94ad166ac3c" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 39175 } } I0312 20:10:49.495929 2180 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [CANDIDATE]: Term 6 pre-election: Requesting pre-vote from peer e3aa58d3045c4a05b94b94eb8bf58ae1 I0312 20:10:49.496201 2180 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [CANDIDATE]: Term 6 pre-election: Requesting pre-vote from peer 9da22e5afdce42078fe8f94ad166ac3c I0312 20:10:49.497819 1990 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "012786a8f1b64995847cf7dacbc029a4" candidate_term: 6 candidate_status { last_received { term: 5 index: 663 } } ignore_live_leader: false dest_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" is_pre_election: true I0312 20:10:49.498384 1990 raft_consensus.cc:2176] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 012786a8f1b64995847cf7dacbc029a4 for term 6 because replica is either leader or believes a valid leader to be alive. I0312 20:10:49.499104 1797 leader_election.cc:377] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [CANDIDATE]: Term 6 pre-election: Vote denied by peer e3aa58d3045c4a05b94b94eb8bf58ae1. Message: Invalid argument: T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 012786a8f1b64995847cf7dacbc029a4 for term 6 because replica is either leader or believes a valid leader to be alive. I0312 20:10:49.501955 1738 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "012786a8f1b64995847cf7dacbc029a4" candidate_term: 6 candidate_status { last_received { term: 5 index: 663 } } ignore_live_leader: false dest_uuid: "9da22e5afdce42078fe8f94ad166ac3c" is_pre_election: true I0312 20:10:49.502336 1738 raft_consensus.cc:2176] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 5 LEADER]: Leader pre-election vote request: Denying vote to candidate 012786a8f1b64995847cf7dacbc029a4 for term 6 because replica is either leader or believes a valid leader to be alive. I0312 20:10:49.502996 1797 leader_election.cc:377] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [CANDIDATE]: Term 6 pre-election: Vote denied by peer 9da22e5afdce42078fe8f94ad166ac3c. Message: Invalid argument: T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 5 LEADER]: Leader pre-election vote request: Denying vote to candidate 012786a8f1b64995847cf7dacbc029a4 for term 6 because replica is either leader or believes a valid leader to be alive. I0312 20:10:49.503155 1797 leader_election.cc:258] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. I0312 20:10:49.503878 2180 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4: Snoozing failure detection for 0.245s (election complete) I0312 20:10:49.504065 2180 raft_consensus.cc:2455] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority I0312 20:10:49.514932 2181 raft_consensus.cc:436] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 9da22e5afdce42078fe8f94ad166ac3c) I0312 20:10:49.515192 2181 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1: Snoozing failure detection for 2.170s (starting election) I0312 20:10:49.515395 2181 raft_consensus.cc:458] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "012786a8f1b64995847cf7dacbc029a4" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 43059 } } peers { permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 } } peers { permanent_uuid: "9da22e5afdce42078fe8f94ad166ac3c" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 39175 } } I0312 20:10:49.518013 2181 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 6 pre-election: Requesting pre-vote from peer 012786a8f1b64995847cf7dacbc029a4 I0312 20:10:49.519249 1864 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" candidate_term: 6 candidate_status { last_received { term: 5 index: 663 } } ignore_live_leader: false dest_uuid: "012786a8f1b64995847cf7dacbc029a4" is_pre_election: true I0312 20:10:49.519632 1864 raft_consensus.cc:2176] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e3aa58d3045c4a05b94b94eb8bf58ae1 for term 6 because replica is either leader or believes a valid leader to be alive. I0312 20:10:49.520515 1926 leader_election.cc:377] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 6 pre-election: Vote denied by peer 012786a8f1b64995847cf7dacbc029a4. Message: Invalid argument: T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e3aa58d3045c4a05b94b94eb8bf58ae1 for term 6 because replica is either leader or believes a valid leader to be alive. I0312 20:10:49.522390 2181 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 6 pre-election: Requesting pre-vote from peer 9da22e5afdce42078fe8f94ad166ac3c I0312 20:10:49.524569 1738 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" candidate_term: 6 candidate_status { last_received { term: 5 index: 663 } } ignore_live_leader: false dest_uuid: "9da22e5afdce42078fe8f94ad166ac3c" is_pre_election: true I0312 20:10:49.525315 1738 raft_consensus.cc:2190] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 5 LEADER]: Leader pre-election vote request: Denying vote to candidate e3aa58d3045c4a05b94b94eb8bf58ae1 for term 6 because replica is already servicing an update from a current leader or another vote. I0312 20:10:49.526260 1922 leader_election.cc:377] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 6 pre-election: Vote denied by peer 9da22e5afdce42078fe8f94ad166ac3c. Message: Service unavailable: T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 5 LEADER]: Leader pre-election vote request: Denying vote to candidate e3aa58d3045c4a05b94b94eb8bf58ae1 for term 6 because replica is already servicing an update from a current leader or another vote. I0312 20:10:49.526521 1922 leader_election.cc:258] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. I0312 20:10:49.527094 2181 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1: Snoozing failure detection for 3.666s (election complete) I0312 20:10:49.527285 2181 raft_consensus.cc:2455] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority I0312 20:10:49.871469 2180 raft_consensus.cc:436] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 9da22e5afdce42078fe8f94ad166ac3c) I0312 20:10:49.871680 2180 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4: Snoozing failure detection for 0.017s (starting election) I0312 20:10:49.871837 2180 raft_consensus.cc:458] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "012786a8f1b64995847cf7dacbc029a4" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 43059 } } peers { permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 } } peers { permanent_uuid: "9da22e5afdce42078fe8f94ad166ac3c" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 39175 } } I0312 20:10:49.872591 2180 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [CANDIDATE]: Term 6 pre-election: Requesting pre-vote from peer e3aa58d3045c4a05b94b94eb8bf58ae1 I0312 20:10:49.872859 2180 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [CANDIDATE]: Term 6 pre-election: Requesting pre-vote from peer 9da22e5afdce42078fe8f94ad166ac3c I0312 20:10:49.873790 1990 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "012786a8f1b64995847cf7dacbc029a4" candidate_term: 6 candidate_status { last_received { term: 5 index: 675 } } ignore_live_leader: false dest_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" is_pre_election: true I0312 20:10:49.875095 1738 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "012786a8f1b64995847cf7dacbc029a4" candidate_term: 6 candidate_status { last_received { term: 5 index: 675 } } ignore_live_leader: false dest_uuid: "9da22e5afdce42078fe8f94ad166ac3c" is_pre_election: true I0312 20:10:49.875353 1990 raft_consensus.cc:2190] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 012786a8f1b64995847cf7dacbc029a4 for term 6 because replica is already servicing an update from a current leader or another vote. I0312 20:10:49.875481 1738 raft_consensus.cc:2176] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 5 LEADER]: Leader pre-election vote request: Denying vote to candidate 012786a8f1b64995847cf7dacbc029a4 for term 6 because replica is either leader or believes a valid leader to be alive. I0312 20:10:49.876257 1797 leader_election.cc:377] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [CANDIDATE]: Term 6 pre-election: Vote denied by peer e3aa58d3045c4a05b94b94eb8bf58ae1. Message: Service unavailable: T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 012786a8f1b64995847cf7dacbc029a4 for term 6 because replica is already servicing an update from a current leader or another vote. I0312 20:10:49.876876 1797 leader_election.cc:377] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [CANDIDATE]: Term 6 pre-election: Vote denied by peer 9da22e5afdce42078fe8f94ad166ac3c. Message: Invalid argument: T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 5 LEADER]: Leader pre-election vote request: Denying vote to candidate 012786a8f1b64995847cf7dacbc029a4 for term 6 because replica is either leader or believes a valid leader to be alive. I0312 20:10:49.877120 1797 leader_election.cc:258] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. I0312 20:10:49.877791 2180 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4: Snoozing failure detection for 0.188s (election complete) I0312 20:10:49.878002 2180 raft_consensus.cc:2455] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority I0312 20:10:49.906801 2180 raft_consensus.cc:436] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 9da22e5afdce42078fe8f94ad166ac3c) I0312 20:10:49.906967 2180 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4: Snoozing failure detection for 0.562s (starting election) I0312 20:10:49.907117 2180 raft_consensus.cc:458] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "012786a8f1b64995847cf7dacbc029a4" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 43059 } } peers { permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 } } peers { permanent_uuid: "9da22e5afdce42078fe8f94ad166ac3c" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 39175 } } I0312 20:10:49.907801 2180 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [CANDIDATE]: Term 6 pre-election: Requesting pre-vote from peer e3aa58d3045c4a05b94b94eb8bf58ae1 I0312 20:10:49.908064 2180 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [CANDIDATE]: Term 6 pre-election: Requesting pre-vote from peer 9da22e5afdce42078fe8f94ad166ac3c I0312 20:10:49.909394 1989 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "012786a8f1b64995847cf7dacbc029a4" candidate_term: 6 candidate_status { last_received { term: 5 index: 676 } } ignore_live_leader: false dest_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" is_pre_election: true I0312 20:10:49.909730 1989 raft_consensus.cc:2176] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 012786a8f1b64995847cf7dacbc029a4 for term 6 because replica is either leader or believes a valid leader to be alive. I0312 20:10:49.910416 1797 leader_election.cc:377] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [CANDIDATE]: Term 6 pre-election: Vote denied by peer e3aa58d3045c4a05b94b94eb8bf58ae1. Message: Invalid argument: T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 012786a8f1b64995847cf7dacbc029a4 for term 6 because replica is either leader or believes a valid leader to be alive. I0312 20:10:49.913808 1738 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "012786a8f1b64995847cf7dacbc029a4" candidate_term: 6 candidate_status { last_received { term: 5 index: 676 } } ignore_live_leader: false dest_uuid: "9da22e5afdce42078fe8f94ad166ac3c" is_pre_election: true I0312 20:10:49.914212 1738 raft_consensus.cc:2176] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 5 LEADER]: Leader pre-election vote request: Denying vote to candidate 012786a8f1b64995847cf7dacbc029a4 for term 6 because replica is either leader or believes a valid leader to be alive. I0312 20:10:49.915343 1797 leader_election.cc:377] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [CANDIDATE]: Term 6 pre-election: Vote denied by peer 9da22e5afdce42078fe8f94ad166ac3c. Message: Invalid argument: T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 5 LEADER]: Leader pre-election vote request: Denying vote to candidate 012786a8f1b64995847cf7dacbc029a4 for term 6 because replica is either leader or believes a valid leader to be alive. I0312 20:10:49.915499 1797 leader_election.cc:258] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. I0312 20:10:49.916797 2180 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4: Snoozing failure detection for 0.091s (election complete) I0312 20:10:49.916954 2180 raft_consensus.cc:2455] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority I0312 20:10:50.598835 2182 raft_consensus.cc:436] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 9da22e5afdce42078fe8f94ad166ac3c) I0312 20:10:50.599031 2182 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1: Snoozing failure detection for 8.479s (starting election) I0312 20:10:50.599222 2182 raft_consensus.cc:458] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "012786a8f1b64995847cf7dacbc029a4" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 43059 } } peers { permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 } } peers { permanent_uuid: "9da22e5afdce42078fe8f94ad166ac3c" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 39175 } } I0312 20:10:50.600030 2182 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 6 pre-election: Requesting pre-vote from peer 012786a8f1b64995847cf7dacbc029a4 I0312 20:10:50.601222 1865 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" candidate_term: 6 candidate_status { last_received { term: 5 index: 703 } } ignore_live_leader: false dest_uuid: "012786a8f1b64995847cf7dacbc029a4" is_pre_election: true I0312 20:10:50.601632 1865 raft_consensus.cc:2176] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e3aa58d3045c4a05b94b94eb8bf58ae1 for term 6 because replica is either leader or believes a valid leader to be alive. I0312 20:10:50.601135 2182 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 6 pre-election: Requesting pre-vote from peer 9da22e5afdce42078fe8f94ad166ac3c I0312 20:10:50.602727 1926 leader_election.cc:377] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 6 pre-election: Vote denied by peer 012786a8f1b64995847cf7dacbc029a4. Message: Invalid argument: T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e3aa58d3045c4a05b94b94eb8bf58ae1 for term 6 because replica is either leader or believes a valid leader to be alive. I0312 20:10:50.603781 1738 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" candidate_term: 6 candidate_status { last_received { term: 5 index: 703 } } ignore_live_leader: false dest_uuid: "9da22e5afdce42078fe8f94ad166ac3c" is_pre_election: true I0312 20:10:50.604210 1738 raft_consensus.cc:2176] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 5 LEADER]: Leader pre-election vote request: Denying vote to candidate e3aa58d3045c4a05b94b94eb8bf58ae1 for term 6 because replica is either leader or believes a valid leader to be alive. I0312 20:10:50.606101 1922 leader_election.cc:377] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 6 pre-election: Vote denied by peer 9da22e5afdce42078fe8f94ad166ac3c. Message: Invalid argument: T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 5 LEADER]: Leader pre-election vote request: Denying vote to candidate e3aa58d3045c4a05b94b94eb8bf58ae1 for term 6 because replica is either leader or believes a valid leader to be alive. I0312 20:10:50.606289 1922 leader_election.cc:258] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. I0312 20:10:50.606895 2182 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1: Snoozing failure detection for 7.182s (election complete) I0312 20:10:50.607064 2182 raft_consensus.cc:2455] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority W0312 20:10:51.202178 1927 debug-util.cc:396] Leaking SignalData structure 0x7b080002b060 after lost signal to thread 1914 W0312 20:10:51.202899 1927 debug-util.cc:396] Leaking SignalData structure 0x7b080000cb40 after lost signal to thread 2032 I0312 20:10:51.225708 2183 raft_consensus.cc:436] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 9da22e5afdce42078fe8f94ad166ac3c) I0312 20:10:51.226020 2183 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4: Snoozing failure detection for 0.621s (starting election) I0312 20:10:51.226213 2183 raft_consensus.cc:458] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "012786a8f1b64995847cf7dacbc029a4" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 43059 } } peers { permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 } } peers { permanent_uuid: "9da22e5afdce42078fe8f94ad166ac3c" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 39175 } } I0312 20:10:51.227176 2183 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [CANDIDATE]: Term 6 pre-election: Requesting pre-vote from peer e3aa58d3045c4a05b94b94eb8bf58ae1 I0312 20:10:51.228278 2183 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [CANDIDATE]: Term 6 pre-election: Requesting pre-vote from peer 9da22e5afdce42078fe8f94ad166ac3c I0312 20:10:51.228924 1989 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "012786a8f1b64995847cf7dacbc029a4" candidate_term: 6 candidate_status { last_received { term: 5 index: 724 } } ignore_live_leader: false dest_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" is_pre_election: true I0312 20:10:51.229326 1989 raft_consensus.cc:2176] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 012786a8f1b64995847cf7dacbc029a4 for term 6 because replica is either leader or believes a valid leader to be alive. I0312 20:10:51.229317 1738 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "012786a8f1b64995847cf7dacbc029a4" candidate_term: 6 candidate_status { last_received { term: 5 index: 724 } } ignore_live_leader: false dest_uuid: "9da22e5afdce42078fe8f94ad166ac3c" is_pre_election: true I0312 20:10:51.229734 1738 raft_consensus.cc:2176] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 5 LEADER]: Leader pre-election vote request: Denying vote to candidate 012786a8f1b64995847cf7dacbc029a4 for term 6 because replica is either leader or believes a valid leader to be alive. I0312 20:10:51.230542 1797 leader_election.cc:377] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [CANDIDATE]: Term 6 pre-election: Vote denied by peer e3aa58d3045c4a05b94b94eb8bf58ae1. Message: Invalid argument: T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 012786a8f1b64995847cf7dacbc029a4 for term 6 because replica is either leader or believes a valid leader to be alive. I0312 20:10:51.230867 1797 leader_election.cc:377] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [CANDIDATE]: Term 6 pre-election: Vote denied by peer 9da22e5afdce42078fe8f94ad166ac3c. Message: Invalid argument: T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 5 LEADER]: Leader pre-election vote request: Denying vote to candidate 012786a8f1b64995847cf7dacbc029a4 for term 6 because replica is either leader or believes a valid leader to be alive. I0312 20:10:51.231016 1797 leader_election.cc:258] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. I0312 20:10:51.231355 2183 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4: Snoozing failure detection for 0.648s (election complete) I0312 20:10:51.231536 2183 raft_consensus.cc:2455] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority I0312 20:10:51.342484 2183 raft_consensus.cc:436] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 9da22e5afdce42078fe8f94ad166ac3c) I0312 20:10:51.342667 2183 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4: Snoozing failure detection for 1.207s (starting election) I0312 20:10:51.342818 2183 raft_consensus.cc:458] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "012786a8f1b64995847cf7dacbc029a4" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 43059 } } peers { permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 } } peers { permanent_uuid: "9da22e5afdce42078fe8f94ad166ac3c" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 39175 } } I0312 20:10:51.343523 2183 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [CANDIDATE]: Term 6 pre-election: Requesting pre-vote from peer e3aa58d3045c4a05b94b94eb8bf58ae1 I0312 20:10:51.343773 2183 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [CANDIDATE]: Term 6 pre-election: Requesting pre-vote from peer 9da22e5afdce42078fe8f94ad166ac3c I0312 20:10:51.344923 1989 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "012786a8f1b64995847cf7dacbc029a4" candidate_term: 6 candidate_status { last_received { term: 5 index: 726 } } ignore_live_leader: false dest_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" is_pre_election: true I0312 20:10:51.345319 1989 raft_consensus.cc:2176] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 012786a8f1b64995847cf7dacbc029a4 for term 6 because replica is either leader or believes a valid leader to be alive. I0312 20:10:51.345654 1738 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "012786a8f1b64995847cf7dacbc029a4" candidate_term: 6 candidate_status { last_received { term: 5 index: 726 } } ignore_live_leader: false dest_uuid: "9da22e5afdce42078fe8f94ad166ac3c" is_pre_election: true I0312 20:10:51.346071 1738 raft_consensus.cc:2176] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 5 LEADER]: Leader pre-election vote request: Denying vote to candidate 012786a8f1b64995847cf7dacbc029a4 for term 6 because replica is either leader or believes a valid leader to be alive. I0312 20:10:51.346040 1797 leader_election.cc:377] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [CANDIDATE]: Term 6 pre-election: Vote denied by peer e3aa58d3045c4a05b94b94eb8bf58ae1. Message: Invalid argument: T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 012786a8f1b64995847cf7dacbc029a4 for term 6 because replica is either leader or believes a valid leader to be alive. I0312 20:10:51.346657 1797 leader_election.cc:377] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [CANDIDATE]: Term 6 pre-election: Vote denied by peer 9da22e5afdce42078fe8f94ad166ac3c. Message: Invalid argument: T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 5 LEADER]: Leader pre-election vote request: Denying vote to candidate 012786a8f1b64995847cf7dacbc029a4 for term 6 because replica is either leader or believes a valid leader to be alive. I0312 20:10:51.346806 1797 leader_election.cc:258] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. I0312 20:10:51.347291 2183 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4: Snoozing failure detection for 0.845s (election complete) I0312 20:10:51.347429 2183 raft_consensus.cc:2455] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority I0312 20:10:51.353493 2184 raft_consensus.cc:436] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 9da22e5afdce42078fe8f94ad166ac3c) I0312 20:10:51.353673 2184 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1: Snoozing failure detection for 8.780s (starting election) I0312 20:10:51.353849 2184 raft_consensus.cc:458] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "012786a8f1b64995847cf7dacbc029a4" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 43059 } } peers { permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 } } peers { permanent_uuid: "9da22e5afdce42078fe8f94ad166ac3c" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 39175 } } I0312 20:10:51.354596 2184 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 6 pre-election: Requesting pre-vote from peer 012786a8f1b64995847cf7dacbc029a4 I0312 20:10:51.354866 2184 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 6 pre-election: Requesting pre-vote from peer 9da22e5afdce42078fe8f94ad166ac3c I0312 20:10:51.356277 1738 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" candidate_term: 6 candidate_status { last_received { term: 5 index: 726 } } ignore_live_leader: false dest_uuid: "9da22e5afdce42078fe8f94ad166ac3c" is_pre_election: true I0312 20:10:51.356024 1865 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" candidate_term: 6 candidate_status { last_received { term: 5 index: 726 } } ignore_live_leader: false dest_uuid: "012786a8f1b64995847cf7dacbc029a4" is_pre_election: true I0312 20:10:51.356639 1738 raft_consensus.cc:2176] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 5 LEADER]: Leader pre-election vote request: Denying vote to candidate e3aa58d3045c4a05b94b94eb8bf58ae1 for term 6 because replica is either leader or believes a valid leader to be alive. I0312 20:10:51.356771 1865 raft_consensus.cc:2176] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e3aa58d3045c4a05b94b94eb8bf58ae1 for term 6 because replica is either leader or believes a valid leader to be alive. I0312 20:10:51.357456 1926 leader_election.cc:377] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 6 pre-election: Vote denied by peer 012786a8f1b64995847cf7dacbc029a4. Message: Invalid argument: T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e3aa58d3045c4a05b94b94eb8bf58ae1 for term 6 because replica is either leader or believes a valid leader to be alive. I0312 20:10:51.361198 1922 leader_election.cc:377] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 6 pre-election: Vote denied by peer 9da22e5afdce42078fe8f94ad166ac3c. Message: Invalid argument: T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 5 LEADER]: Leader pre-election vote request: Denying vote to candidate e3aa58d3045c4a05b94b94eb8bf58ae1 for term 6 because replica is either leader or believes a valid leader to be alive. I0312 20:10:51.361385 1922 leader_election.cc:258] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. I0312 20:10:51.362977 2184 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1: Snoozing failure detection for 2.788s (election complete) I0312 20:10:51.363135 2184 raft_consensus.cc:2455] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority I0312 20:10:51.383484 2183 raft_consensus.cc:436] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 9da22e5afdce42078fe8f94ad166ac3c) I0312 20:10:51.383667 2183 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4: Snoozing failure detection for 1.156s (starting election) I0312 20:10:51.383828 2183 raft_consensus.cc:458] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "012786a8f1b64995847cf7dacbc029a4" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 43059 } } peers { permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 } } peers { permanent_uuid: "9da22e5afdce42078fe8f94ad166ac3c" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 39175 } } I0312 20:10:51.384573 2183 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [CANDIDATE]: Term 6 pre-election: Requesting pre-vote from peer e3aa58d3045c4a05b94b94eb8bf58ae1 I0312 20:10:51.384841 2183 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [CANDIDATE]: Term 6 pre-election: Requesting pre-vote from peer 9da22e5afdce42078fe8f94ad166ac3c I0312 20:10:51.386221 1989 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "012786a8f1b64995847cf7dacbc029a4" candidate_term: 6 candidate_status { last_received { term: 5 index: 728 } } ignore_live_leader: false dest_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" is_pre_election: true I0312 20:10:51.386551 1989 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1: Snoozing failure detection for 14.768s (vote granted) I0312 20:10:51.386679 1989 raft_consensus.cc:2215] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 012786a8f1b64995847cf7dacbc029a4 in term 5. I0312 20:10:51.387403 1797 leader_election.cc:363] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [CANDIDATE]: Term 6 pre-election: Vote granted by peer e3aa58d3045c4a05b94b94eb8bf58ae1 I0312 20:10:51.387547 1797 leader_election.cc:258] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. I0312 20:10:51.387965 2183 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4: Snoozing failure detection for 1.789s (election complete) I0312 20:10:51.388131 2183 raft_consensus.cc:2507] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 5 FOLLOWER]: Leader pre-election won for term 6 I0312 20:10:51.388249 2183 raft_consensus.cc:436] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 9da22e5afdce42078fe8f94ad166ac3c) I0312 20:10:51.388375 2183 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4: Snoozing failure detection for 0.447s (starting election) I0312 20:10:51.388484 2183 raft_consensus.cc:2752] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 5 FOLLOWER]: Advancing to term 6 I0312 20:10:51.392325 2183 raft_consensus.cc:458] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "012786a8f1b64995847cf7dacbc029a4" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 43059 } } peers { permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 } } peers { permanent_uuid: "9da22e5afdce42078fe8f94ad166ac3c" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 39175 } } I0312 20:10:51.393030 2183 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [CANDIDATE]: Term 6 election: Requesting vote from peer e3aa58d3045c4a05b94b94eb8bf58ae1 I0312 20:10:51.394201 1989 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "012786a8f1b64995847cf7dacbc029a4" candidate_term: 6 candidate_status { last_received { term: 5 index: 728 } } ignore_live_leader: false dest_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" I0312 20:10:51.394462 1989 raft_consensus.cc:2752] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 5 FOLLOWER]: Advancing to term 6 I0312 20:10:51.394598 1989 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1: Snoozing failure detection for 14.697s (vote granted) I0312 20:10:51.397909 1738 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "012786a8f1b64995847cf7dacbc029a4" candidate_term: 6 candidate_status { last_received { term: 5 index: 728 } } ignore_live_leader: false dest_uuid: "9da22e5afdce42078fe8f94ad166ac3c" is_pre_election: true I0312 20:10:51.398314 1738 raft_consensus.cc:2176] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 5 LEADER]: Leader pre-election vote request: Denying vote to candidate 012786a8f1b64995847cf7dacbc029a4 for term 6 because replica is either leader or believes a valid leader to be alive. I0312 20:10:51.398368 1865 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 6 FOLLOWER]: Rejecting Update request from peer 9da22e5afdce42078fe8f94ad166ac3c for earlier term 5. Current term is 6. Ops: [] I0312 20:10:51.398442 1989 raft_consensus.cc:2215] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 012786a8f1b64995847cf7dacbc029a4 in term 6. I0312 20:10:51.399112 1797 leader_election.cc:363] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [CANDIDATE]: Term 6 election: Vote granted by peer e3aa58d3045c4a05b94b94eb8bf58ae1 I0312 20:10:51.399262 1797 leader_election.cc:258] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. I0312 20:10:51.400015 2183 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [CANDIDATE]: Term 6 election: Requesting vote from peer 9da22e5afdce42078fe8f94ad166ac3c I0312 20:10:51.400233 2187 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4: Snoozing failure detection for 0.147s (election complete) I0312 20:10:51.400133 2171 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "012786a8f1b64995847cf7dacbc029a4" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 43059 }, Status: INVALID_TERM, Last received: 5.728, Next index: 729, Last known committed idx: 728, Time since last communication: 0.000s I0312 20:10:51.400717 1797 leader_election.cc:377] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [CANDIDATE]: Term 6 pre-election: Vote denied by peer 9da22e5afdce42078fe8f94ad166ac3c. Message: Invalid argument: T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 5 LEADER]: Leader pre-election vote request: Denying vote to candidate 012786a8f1b64995847cf7dacbc029a4 for term 6 because replica is either leader or believes a valid leader to be alive. I0312 20:10:51.400909 2187 raft_consensus.cc:2507] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 6 FOLLOWER]: Leader election won for term 6 I0312 20:10:51.401151 2187 raft_consensus.cc:572] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 6 LEADER]: Becoming Leader. State: Replica: 012786a8f1b64995847cf7dacbc029a4, State: Running, Role: LEADER I0312 20:10:51.401693 1989 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 6 FOLLOWER]: Rejecting Update request from peer 9da22e5afdce42078fe8f94ad166ac3c for earlier term 5. Current term is 6. Ops: [] I0312 20:10:51.401513 2187 consensus_queue.cc:196] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 728, Committed index: 728, Last appended: 5.728, Last appended by leader: 728, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "012786a8f1b64995847cf7dacbc029a4" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 43059 } } peers { permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 } } peers { permanent_uuid: "9da22e5afdce42078fe8f94ad166ac3c" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 39175 } } I0312 20:10:51.401885 1738 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "012786a8f1b64995847cf7dacbc029a4" candidate_term: 6 candidate_status { last_received { term: 5 index: 728 } } ignore_live_leader: false dest_uuid: "9da22e5afdce42078fe8f94ad166ac3c" I0312 20:10:51.402215 1738 raft_consensus.cc:2176] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 5 LEADER]: Leader election vote request: Denying vote to candidate 012786a8f1b64995847cf7dacbc029a4 for term 6 because replica is either leader or believes a valid leader to be alive. I0312 20:10:51.402715 2171 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 }, Status: INVALID_TERM, Last received: 5.727, Next index: 728, Last known committed idx: 726, Time since last communication: 0.000s I0312 20:10:51.402935 1797 leader_election.cc:377] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [CANDIDATE]: Term 6 election: Vote denied by peer 9da22e5afdce42078fe8f94ad166ac3c. Message: Invalid argument: T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 5 LEADER]: Leader election vote request: Denying vote to candidate 012786a8f1b64995847cf7dacbc029a4 for term 6 because replica is either leader or believes a valid leader to be alive. I0312 20:10:51.403509 1865 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 6 LEADER]: Rejecting Update request from peer 9da22e5afdce42078fe8f94ad166ac3c for earlier term 5. Current term is 6. Ops: [] I0312 20:10:51.404661 2171 consensus_queue.cc:1004] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [LEADER]: Peer 012786a8f1b64995847cf7dacbc029a4 log is divergent from this leader: its last log entry 6.729 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 728 I0312 20:10:51.404798 2171 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "012786a8f1b64995847cf7dacbc029a4" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 43059 }, Status: INVALID_TERM, Last received: 5.728, Next index: 729, Last known committed idx: 728, Time since last communication: 0.000s I0312 20:10:51.405485 1617 catalog_manager.cc:3592] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 reported cstate change: term changed from 5 to 6, leader changed from 9da22e5afdce42078fe8f94ad166ac3c (127.0.53.1) to 012786a8f1b64995847cf7dacbc029a4 (127.0.53.2). New cstate: current_term: 6 leader_uuid: "012786a8f1b64995847cf7dacbc029a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "012786a8f1b64995847cf7dacbc029a4" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 43059 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9da22e5afdce42078fe8f94ad166ac3c" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 39175 } health_report { overall_health: UNKNOWN } } } I0312 20:10:51.407742 1989 raft_consensus.cc:1058] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 6 FOLLOWER]: Refusing update from remote peer 012786a8f1b64995847cf7dacbc029a4: Log matching property violated. Preceding OpId in replica: term: 5 index: 727. Preceding OpId from leader: term: 6 index: 729. (index mismatch) I0312 20:10:51.408093 1990 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 6 FOLLOWER]: Rejecting Update request from peer 9da22e5afdce42078fe8f94ad166ac3c for earlier term 5. Current term is 6. Ops: [] I0312 20:10:51.408996 2187 consensus_queue.cc:1017] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 }, Status: LMP_MISMATCH, Last received: 5.727, Next index: 728, Last known committed idx: 726, Time since last communication: 0.000s I0312 20:10:51.409768 2171 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 }, Status: INVALID_TERM, Last received: 5.727, Next index: 728, Last known committed idx: 726, Time since last communication: 0.000s I0312 20:10:51.410588 1865 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 6 LEADER]: Rejecting Update request from peer 9da22e5afdce42078fe8f94ad166ac3c for earlier term 5. Current term is 6. Ops: [] I0312 20:10:51.413861 2171 consensus_queue.cc:1004] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [LEADER]: Peer 012786a8f1b64995847cf7dacbc029a4 log is divergent from this leader: its last log entry 6.729 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 728 I0312 20:10:51.414002 2171 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "012786a8f1b64995847cf7dacbc029a4" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 43059 }, Status: INVALID_TERM, Last received: 5.728, Next index: 729, Last known committed idx: 728, Time since last communication: 0.000s I0312 20:10:51.414659 1738 raft_consensus.cc:2747] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 5 LEADER]: Stepping down as leader of term 5 I0312 20:10:51.414813 1738 raft_consensus.cc:605] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 9da22e5afdce42078fe8f94ad166ac3c, State: Running, Role: LEADER I0312 20:10:51.415107 1738 consensus_queue.cc:219] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 728, Committed index: 728, Last appended: 5.728, Last appended by leader: 728, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "012786a8f1b64995847cf7dacbc029a4" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 43059 } } peers { permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 } } peers { permanent_uuid: "9da22e5afdce42078fe8f94ad166ac3c" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 39175 } } I0312 20:10:51.415524 1738 consensus_peers.cc:458] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c -> Peer e3aa58d3045c4a05b94b94eb8bf58ae1 (127.0.53.3:49316): Closing peer: e3aa58d3045c4a05b94b94eb8bf58ae1 I0312 20:10:51.415645 1738 consensus_peers.cc:458] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c -> Peer 012786a8f1b64995847cf7dacbc029a4 (127.0.53.2:43059): Closing peer: 012786a8f1b64995847cf7dacbc029a4 I0312 20:10:51.415765 1738 raft_consensus.cc:2752] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 5 FOLLOWER]: Advancing to term 6 I0312 20:10:51.416100 1990 raft_consensus.cc:1003] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 6 FOLLOWER]: Deduplicated request from leader. Original: 5.727->[5.728-5.728] Dedup: 5.728->[] I0312 20:10:51.416311 1990 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 6 FOLLOWER]: Rejecting Update request from peer 9da22e5afdce42078fe8f94ad166ac3c for earlier term 5. Current term is 6. Ops: [5.728-5.728] W0312 20:10:51.419492 2187 fault_injection.cc:41] FAULT INJECTION ENABLED! W0312 20:10:51.419615 2187 fault_injection.cc:42] THIS SERVER MAY CRASH! I0312 20:10:51.419286 1738 raft_consensus.cc:1058] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 6 FOLLOWER]: Refusing update from remote peer 012786a8f1b64995847cf7dacbc029a4: Log matching property violated. Preceding OpId in replica: term: 5 index: 728. Preceding OpId from leader: term: 6 index: 729. (index mismatch) I0312 20:10:51.422734 2183 consensus_queue.cc:1017] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9da22e5afdce42078fe8f94ad166ac3c" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 39175 }, Status: LMP_MISMATCH, Last received: 5.728, Next index: 729, Last known committed idx: 728, Time since last communication: 0.000s I0312 20:10:51.439654 2185 raft_consensus.cc:436] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 012786a8f1b64995847cf7dacbc029a4) I0312 20:10:51.439842 2185 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c: Snoozing failure detection for 0.021s (starting election) I0312 20:10:51.440007 2185 raft_consensus.cc:458] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "012786a8f1b64995847cf7dacbc029a4" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 43059 } } peers { permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 } } peers { permanent_uuid: "9da22e5afdce42078fe8f94ad166ac3c" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 39175 } } I0312 20:10:51.440768 2185 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [CANDIDATE]: Term 7 pre-election: Requesting pre-vote from peer 012786a8f1b64995847cf7dacbc029a4 I0312 20:10:51.441051 2185 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [CANDIDATE]: Term 7 pre-election: Requesting pre-vote from peer e3aa58d3045c4a05b94b94eb8bf58ae1 I0312 20:10:51.442217 1990 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "9da22e5afdce42078fe8f94ad166ac3c" candidate_term: 7 candidate_status { last_received { term: 6 index: 729 } } ignore_live_leader: false dest_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" is_pre_election: true I0312 20:10:51.442566 1990 raft_consensus.cc:2176] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9da22e5afdce42078fe8f94ad166ac3c for term 7 because replica is either leader or believes a valid leader to be alive. I0312 20:10:51.443616 1663 leader_election.cc:377] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [CANDIDATE]: Term 7 pre-election: Vote denied by peer e3aa58d3045c4a05b94b94eb8bf58ae1. Message: Invalid argument: T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9da22e5afdce42078fe8f94ad166ac3c for term 7 because replica is either leader or believes a valid leader to be alive. I0312 20:10:51.448746 1865 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "9da22e5afdce42078fe8f94ad166ac3c" candidate_term: 7 candidate_status { last_received { term: 6 index: 729 } } ignore_live_leader: false dest_uuid: "012786a8f1b64995847cf7dacbc029a4" is_pre_election: true I0312 20:10:51.449092 1865 raft_consensus.cc:2190] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 6 LEADER]: Leader pre-election vote request: Denying vote to candidate 9da22e5afdce42078fe8f94ad166ac3c for term 7 because replica is already servicing an update from a current leader or another vote. I0312 20:10:51.452514 2184 raft_consensus.cc:436] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 012786a8f1b64995847cf7dacbc029a4) I0312 20:10:51.452678 2184 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1: Snoozing failure detection for 0.018s (starting election) I0312 20:10:51.452857 2184 raft_consensus.cc:458] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "012786a8f1b64995847cf7dacbc029a4" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 43059 } } peers { permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 } } peers { permanent_uuid: "9da22e5afdce42078fe8f94ad166ac3c" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 39175 } } I0312 20:10:51.453814 2184 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 7 pre-election: Requesting pre-vote from peer 012786a8f1b64995847cf7dacbc029a4 I0312 20:10:51.454084 2184 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 7 pre-election: Requesting pre-vote from peer 9da22e5afdce42078fe8f94ad166ac3c I0312 20:10:51.454962 1738 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" candidate_term: 7 candidate_status { last_received { term: 6 index: 730 } } ignore_live_leader: false dest_uuid: "9da22e5afdce42078fe8f94ad166ac3c" is_pre_election: true I0312 20:10:51.455338 1738 raft_consensus.cc:2176] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e3aa58d3045c4a05b94b94eb8bf58ae1 for term 7 because replica is either leader or believes a valid leader to be alive. I0312 20:10:51.455960 1922 leader_election.cc:377] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 7 pre-election: Vote denied by peer 9da22e5afdce42078fe8f94ad166ac3c. Message: Invalid argument: T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e3aa58d3045c4a05b94b94eb8bf58ae1 for term 7 because replica is either leader or believes a valid leader to be alive. I0312 20:10:51.460081 2185 raft_consensus.cc:436] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 012786a8f1b64995847cf7dacbc029a4) I0312 20:10:51.460243 2185 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c: Snoozing failure detection for 0.019s (starting election) I0312 20:10:51.460392 2185 raft_consensus.cc:458] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "012786a8f1b64995847cf7dacbc029a4" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 43059 } } peers { permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 } } peers { permanent_uuid: "9da22e5afdce42078fe8f94ad166ac3c" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 39175 } } I0312 20:10:51.461053 2185 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [CANDIDATE]: Term 7 pre-election: Requesting pre-vote from peer 012786a8f1b64995847cf7dacbc029a4 I0312 20:10:51.461333 2185 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [CANDIDATE]: Term 7 pre-election: Requesting pre-vote from peer e3aa58d3045c4a05b94b94eb8bf58ae1 I0312 20:10:51.462206 1990 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "9da22e5afdce42078fe8f94ad166ac3c" candidate_term: 7 candidate_status { last_received { term: 6 index: 730 } } ignore_live_leader: false dest_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" is_pre_election: true I0312 20:10:51.462568 1990 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1: Snoozing failure detection for 0.016s (vote granted) I0312 20:10:51.462854 1990 raft_consensus.cc:2215] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9da22e5afdce42078fe8f94ad166ac3c in term 6. I0312 20:10:51.463493 1663 leader_election.cc:363] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [CANDIDATE]: Term 7 pre-election: Vote granted by peer e3aa58d3045c4a05b94b94eb8bf58ae1 I0312 20:10:51.463624 1663 leader_election.cc:258] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. I0312 20:10:51.464155 2185 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c: Snoozing failure detection for 0.016s (election complete) I0312 20:10:51.464303 2185 raft_consensus.cc:2507] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 6 FOLLOWER]: Leader pre-election won for term 7 I0312 20:10:51.464431 2185 raft_consensus.cc:436] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 6 FOLLOWER]: Starting leader election (detected failure of leader 012786a8f1b64995847cf7dacbc029a4) I0312 20:10:51.464557 2185 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c: Snoozing failure detection for 0.019s (starting election) I0312 20:10:51.464674 2185 raft_consensus.cc:2752] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 6 FOLLOWER]: Advancing to term 7 I0312 20:10:51.468156 2185 raft_consensus.cc:458] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "012786a8f1b64995847cf7dacbc029a4" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 43059 } } peers { permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 } } peers { permanent_uuid: "9da22e5afdce42078fe8f94ad166ac3c" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 39175 } } I0312 20:10:51.468643 1865 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" candidate_term: 7 candidate_status { last_received { term: 6 index: 730 } } ignore_live_leader: false dest_uuid: "012786a8f1b64995847cf7dacbc029a4" is_pre_election: true I0312 20:10:51.469024 1865 raft_consensus.cc:2176] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 6 LEADER]: Leader pre-election vote request: Denying vote to candidate e3aa58d3045c4a05b94b94eb8bf58ae1 for term 7 because replica is either leader or believes a valid leader to be alive. I0312 20:10:51.469790 1926 leader_election.cc:377] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 7 pre-election: Vote denied by peer 012786a8f1b64995847cf7dacbc029a4. Message: Invalid argument: T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 6 LEADER]: Leader pre-election vote request: Denying vote to candidate e3aa58d3045c4a05b94b94eb8bf58ae1 for term 7 because replica is either leader or believes a valid leader to be alive. I0312 20:10:51.469952 1926 leader_election.cc:258] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. I0312 20:10:51.470842 2184 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1: Snoozing failure detection for 0.019s (election complete) I0312 20:10:51.470993 2184 raft_consensus.cc:2455] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority I0312 20:10:51.472370 2185 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [CANDIDATE]: Term 7 election: Requesting vote from peer 012786a8f1b64995847cf7dacbc029a4 I0312 20:10:51.472668 2185 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [CANDIDATE]: Term 7 election: Requesting vote from peer e3aa58d3045c4a05b94b94eb8bf58ae1 I0312 20:10:51.473700 1990 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "9da22e5afdce42078fe8f94ad166ac3c" candidate_term: 7 candidate_status { last_received { term: 6 index: 730 } } ignore_live_leader: false dest_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" I0312 20:10:51.474063 1990 raft_consensus.cc:2176] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 9da22e5afdce42078fe8f94ad166ac3c for term 7 because replica is either leader or believes a valid leader to be alive. I0312 20:10:51.474675 1663 leader_election.cc:377] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [CANDIDATE]: Term 7 election: Vote denied by peer e3aa58d3045c4a05b94b94eb8bf58ae1. Message: Invalid argument: T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 9da22e5afdce42078fe8f94ad166ac3c for term 7 because replica is either leader or believes a valid leader to be alive. W0312 20:10:51.475468 1676 leader_election.cc:287] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 012786a8f1b64995847cf7dacbc029a4: Timed out: RequestConsensusVote RPC to 127.0.53.2:43059 timed out after 0.021s (SENT) I0312 20:10:51.475637 1676 leader_election.cc:258] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. I0312 20:10:51.476572 2185 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c: Snoozing failure detection for 0.017s (election complete) I0312 20:10:51.476642 1865 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "9da22e5afdce42078fe8f94ad166ac3c" candidate_term: 7 candidate_status { last_received { term: 6 index: 730 } } ignore_live_leader: false dest_uuid: "012786a8f1b64995847cf7dacbc029a4" is_pre_election: true I0312 20:10:51.476779 2185 raft_consensus.cc:2455] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority I0312 20:10:51.476979 1865 raft_consensus.cc:2176] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 6 LEADER]: Leader pre-election vote request: Denying vote to candidate 9da22e5afdce42078fe8f94ad166ac3c for term 7 because replica is either leader or believes a valid leader to be alive. I0312 20:10:51.477421 1865 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "9da22e5afdce42078fe8f94ad166ac3c" candidate_term: 7 candidate_status { last_received { term: 6 index: 730 } } ignore_live_leader: false dest_uuid: "012786a8f1b64995847cf7dacbc029a4" I0312 20:10:51.477725 1865 raft_consensus.cc:2176] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 6 LEADER]: Leader election vote request: Denying vote to candidate 9da22e5afdce42078fe8f94ad166ac3c for term 7 because replica is either leader or believes a valid leader to be alive. I0312 20:10:51.477885 1676 leader_election.cc:377] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [CANDIDATE]: Term 7 pre-election: Vote denied by peer 012786a8f1b64995847cf7dacbc029a4. Message: Invalid argument: T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 6 LEADER]: Leader pre-election vote request: Denying vote to candidate 9da22e5afdce42078fe8f94ad166ac3c for term 7 because replica is either leader or believes a valid leader to be alive. I0312 20:10:51.478346 1738 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 7 FOLLOWER]: Rejecting Update request from peer 012786a8f1b64995847cf7dacbc029a4 for earlier term 6. Current term is 7. Ops: [] I0312 20:10:51.478832 1676 leader_election.cc:377] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [CANDIDATE]: Term 7 election: Vote denied by peer 012786a8f1b64995847cf7dacbc029a4. Message: Invalid argument: T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 6 LEADER]: Leader election vote request: Denying vote to candidate 9da22e5afdce42078fe8f94ad166ac3c for term 7 because replica is either leader or believes a valid leader to be alive. I0312 20:10:51.478987 1676 leader_election.cc:258] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. I0312 20:10:51.479256 2185 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c: Snoozing failure detection for 0.024s (election complete) I0312 20:10:51.479429 2185 raft_consensus.cc:2455] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority I0312 20:10:51.479868 2191 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9da22e5afdce42078fe8f94ad166ac3c" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 39175 }, Status: INVALID_TERM, Last received: 6.730, Next index: 731, Last known committed idx: 730, Time since last communication: 0.000s I0312 20:10:51.481914 1738 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 7 FOLLOWER]: Rejecting Update request from peer 012786a8f1b64995847cf7dacbc029a4 for earlier term 6. Current term is 7. Ops: [] I0312 20:10:51.482843 2191 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9da22e5afdce42078fe8f94ad166ac3c" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 39175 }, Status: INVALID_TERM, Last received: 6.730, Next index: 731, Last known committed idx: 730, Time since last communication: 0.000s I0312 20:10:51.487226 1738 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 7 FOLLOWER]: Rejecting Update request from peer 012786a8f1b64995847cf7dacbc029a4 for earlier term 6. Current term is 7. Ops: [] I0312 20:10:51.488183 2193 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9da22e5afdce42078fe8f94ad166ac3c" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 39175 }, Status: INVALID_TERM, Last received: 6.730, Next index: 731, Last known committed idx: 730, Time since last communication: 0.000s I0312 20:10:51.491663 1738 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 7 FOLLOWER]: Rejecting Update request from peer 012786a8f1b64995847cf7dacbc029a4 for earlier term 6. Current term is 7. Ops: [] I0312 20:10:51.492496 2191 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9da22e5afdce42078fe8f94ad166ac3c" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 39175 }, Status: INVALID_TERM, Last received: 6.730, Next index: 731, Last known committed idx: 730, Time since last communication: 0.000s I0312 20:10:51.496239 1738 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 7 FOLLOWER]: Rejecting Update request from peer 012786a8f1b64995847cf7dacbc029a4 for earlier term 6. Current term is 7. Ops: [] I0312 20:10:51.497123 2183 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9da22e5afdce42078fe8f94ad166ac3c" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 39175 }, Status: INVALID_TERM, Last received: 6.730, Next index: 731, Last known committed idx: 730, Time since last communication: 0.000s I0312 20:10:51.500711 1738 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 7 FOLLOWER]: Rejecting Update request from peer 012786a8f1b64995847cf7dacbc029a4 for earlier term 6. Current term is 7. Ops: [] I0312 20:10:51.502130 2193 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9da22e5afdce42078fe8f94ad166ac3c" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 39175 }, Status: INVALID_TERM, Last received: 6.730, Next index: 731, Last known committed idx: 730, Time since last communication: 0.000s I0312 20:10:51.504797 2185 raft_consensus.cc:436] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0312 20:10:51.504974 2185 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c: Snoozing failure detection for 0.042s (starting election) I0312 20:10:51.505161 2185 raft_consensus.cc:458] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "012786a8f1b64995847cf7dacbc029a4" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 43059 } } peers { permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 } } peers { permanent_uuid: "9da22e5afdce42078fe8f94ad166ac3c" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 39175 } } I0312 20:10:51.506018 2185 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [CANDIDATE]: Term 8 pre-election: Requesting pre-vote from peer 012786a8f1b64995847cf7dacbc029a4 I0312 20:10:51.508178 2185 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [CANDIDATE]: Term 8 pre-election: Requesting pre-vote from peer e3aa58d3045c4a05b94b94eb8bf58ae1 I0312 20:10:51.508847 1865 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "9da22e5afdce42078fe8f94ad166ac3c" candidate_term: 8 candidate_status { last_received { term: 6 index: 730 } } ignore_live_leader: false dest_uuid: "012786a8f1b64995847cf7dacbc029a4" is_pre_election: true I0312 20:10:51.509004 1990 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "9da22e5afdce42078fe8f94ad166ac3c" candidate_term: 8 candidate_status { last_received { term: 6 index: 730 } } ignore_live_leader: false dest_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" is_pre_election: true I0312 20:10:51.509194 1865 raft_consensus.cc:2176] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 6 LEADER]: Leader pre-election vote request: Denying vote to candidate 9da22e5afdce42078fe8f94ad166ac3c for term 8 because replica is either leader or believes a valid leader to be alive. I0312 20:10:51.509371 1990 raft_consensus.cc:2176] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9da22e5afdce42078fe8f94ad166ac3c for term 8 because replica is either leader or believes a valid leader to be alive. I0312 20:10:51.509933 1663 leader_election.cc:377] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [CANDIDATE]: Term 8 pre-election: Vote denied by peer e3aa58d3045c4a05b94b94eb8bf58ae1. Message: Invalid argument: T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9da22e5afdce42078fe8f94ad166ac3c for term 8 because replica is either leader or believes a valid leader to be alive. I0312 20:10:51.511801 1676 leader_election.cc:377] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [CANDIDATE]: Term 8 pre-election: Vote denied by peer 012786a8f1b64995847cf7dacbc029a4. Message: Invalid argument: T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 6 LEADER]: Leader pre-election vote request: Denying vote to candidate 9da22e5afdce42078fe8f94ad166ac3c for term 8 because replica is either leader or believes a valid leader to be alive. I0312 20:10:51.512063 1676 leader_election.cc:258] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. I0312 20:10:51.512339 2185 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c: Snoozing failure detection for 0.029s (election complete) I0312 20:10:51.512493 2185 raft_consensus.cc:2455] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority I0312 20:10:51.512887 1738 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 7 FOLLOWER]: Rejecting Update request from peer 012786a8f1b64995847cf7dacbc029a4 for earlier term 6. Current term is 7. Ops: [] I0312 20:10:51.513989 2193 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9da22e5afdce42078fe8f94ad166ac3c" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 39175 }, Status: INVALID_TERM, Last received: 6.730, Next index: 731, Last known committed idx: 730, Time since last communication: 0.000s I0312 20:10:51.517503 1738 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 7 FOLLOWER]: Rejecting Update request from peer 012786a8f1b64995847cf7dacbc029a4 for earlier term 6. Current term is 7. Ops: [] I0312 20:10:51.519125 2183 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9da22e5afdce42078fe8f94ad166ac3c" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 39175 }, Status: INVALID_TERM, Last received: 6.730, Next index: 731, Last known committed idx: 730, Time since last communication: 0.000s I0312 20:10:51.521273 1738 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 7 FOLLOWER]: Rejecting Update request from peer 012786a8f1b64995847cf7dacbc029a4 for earlier term 6. Current term is 7. Ops: [] I0312 20:10:51.522095 2191 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9da22e5afdce42078fe8f94ad166ac3c" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 39175 }, Status: INVALID_TERM, Last received: 6.730, Next index: 731, Last known committed idx: 730, Time since last communication: 0.000s I0312 20:10:51.524660 1738 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 7 FOLLOWER]: Rejecting Update request from peer 012786a8f1b64995847cf7dacbc029a4 for earlier term 6. Current term is 7. Ops: [] I0312 20:10:51.525540 2191 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9da22e5afdce42078fe8f94ad166ac3c" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 39175 }, Status: INVALID_TERM, Last received: 6.730, Next index: 731, Last known committed idx: 730, Time since last communication: 0.000s I0312 20:10:51.529115 1738 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 7 FOLLOWER]: Rejecting Update request from peer 012786a8f1b64995847cf7dacbc029a4 for earlier term 6. Current term is 7. Ops: [] I0312 20:10:51.530122 2191 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9da22e5afdce42078fe8f94ad166ac3c" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 39175 }, Status: INVALID_TERM, Last received: 6.730, Next index: 731, Last known committed idx: 730, Time since last communication: 0.000s I0312 20:10:51.534770 1738 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 7 FOLLOWER]: Rejecting Update request from peer 012786a8f1b64995847cf7dacbc029a4 for earlier term 6. Current term is 7. Ops: [] I0312 20:10:51.535665 2191 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9da22e5afdce42078fe8f94ad166ac3c" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 39175 }, Status: INVALID_TERM, Last received: 6.730, Next index: 731, Last known committed idx: 730, Time since last communication: 0.000s I0312 20:10:51.539252 1738 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 7 FOLLOWER]: Rejecting Update request from peer 012786a8f1b64995847cf7dacbc029a4 for earlier term 6. Current term is 7. Ops: [] I0312 20:10:51.540127 2191 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9da22e5afdce42078fe8f94ad166ac3c" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 39175 }, Status: INVALID_TERM, Last received: 6.730, Next index: 731, Last known committed idx: 730, Time since last communication: 0.000s I0312 20:10:51.542047 2185 raft_consensus.cc:436] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0312 20:10:51.542203 2185 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c: Snoozing failure detection for 0.046s (starting election) I0312 20:10:51.542356 2185 raft_consensus.cc:458] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "012786a8f1b64995847cf7dacbc029a4" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 43059 } } peers { permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 } } peers { permanent_uuid: "9da22e5afdce42078fe8f94ad166ac3c" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 39175 } } I0312 20:10:51.543005 2185 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [CANDIDATE]: Term 8 pre-election: Requesting pre-vote from peer 012786a8f1b64995847cf7dacbc029a4 I0312 20:10:51.543283 2185 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [CANDIDATE]: Term 8 pre-election: Requesting pre-vote from peer e3aa58d3045c4a05b94b94eb8bf58ae1 I0312 20:10:51.543586 1738 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 7 FOLLOWER]: Rejecting Update request from peer 012786a8f1b64995847cf7dacbc029a4 for earlier term 6. Current term is 7. Ops: [] I0312 20:10:51.544028 1865 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "9da22e5afdce42078fe8f94ad166ac3c" candidate_term: 8 candidate_status { last_received { term: 6 index: 730 } } ignore_live_leader: false dest_uuid: "012786a8f1b64995847cf7dacbc029a4" is_pre_election: true I0312 20:10:51.544239 1990 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "9da22e5afdce42078fe8f94ad166ac3c" candidate_term: 8 candidate_status { last_received { term: 6 index: 730 } } ignore_live_leader: false dest_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" is_pre_election: true I0312 20:10:51.544389 1865 raft_consensus.cc:2176] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 6 LEADER]: Leader pre-election vote request: Denying vote to candidate 9da22e5afdce42078fe8f94ad166ac3c for term 8 because replica is either leader or believes a valid leader to be alive. I0312 20:10:51.544567 1990 raft_consensus.cc:2176] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9da22e5afdce42078fe8f94ad166ac3c for term 8 because replica is either leader or believes a valid leader to be alive. I0312 20:10:51.545104 1676 leader_election.cc:377] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [CANDIDATE]: Term 8 pre-election: Vote denied by peer 012786a8f1b64995847cf7dacbc029a4. Message: Invalid argument: T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 6 LEADER]: Leader pre-election vote request: Denying vote to candidate 9da22e5afdce42078fe8f94ad166ac3c for term 8 because replica is either leader or believes a valid leader to be alive. I0312 20:10:51.545353 1663 leader_election.cc:377] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [CANDIDATE]: Term 8 pre-election: Vote denied by peer e3aa58d3045c4a05b94b94eb8bf58ae1. Message: Invalid argument: T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9da22e5afdce42078fe8f94ad166ac3c for term 8 because replica is either leader or believes a valid leader to be alive. I0312 20:10:51.545511 1663 leader_election.cc:258] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. I0312 20:10:51.545471 2191 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9da22e5afdce42078fe8f94ad166ac3c" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 39175 }, Status: INVALID_TERM, Last received: 6.730, Next index: 731, Last known committed idx: 730, Time since last communication: 0.000s I0312 20:10:51.545965 2185 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c: Snoozing failure detection for 0.048s (election complete) I0312 20:10:51.546118 2185 raft_consensus.cc:2455] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority I0312 20:10:51.550806 1738 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 7 FOLLOWER]: Rejecting Update request from peer 012786a8f1b64995847cf7dacbc029a4 for earlier term 6. Current term is 7. Ops: [] I0312 20:10:51.551748 2191 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9da22e5afdce42078fe8f94ad166ac3c" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 39175 }, Status: INVALID_TERM, Last received: 6.730, Next index: 731, Last known committed idx: 730, Time since last communication: 0.000s I0312 20:10:51.556686 1738 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 7 FOLLOWER]: Rejecting Update request from peer 012786a8f1b64995847cf7dacbc029a4 for earlier term 6. Current term is 7. Ops: [] I0312 20:10:51.559103 2187 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9da22e5afdce42078fe8f94ad166ac3c" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 39175 }, Status: INVALID_TERM, Last received: 6.730, Next index: 731, Last known committed idx: 730, Time since last communication: 0.000s I0312 20:10:51.562155 1738 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 7 FOLLOWER]: Rejecting Update request from peer 012786a8f1b64995847cf7dacbc029a4 for earlier term 6. Current term is 7. Ops: [] I0312 20:10:51.563508 2187 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9da22e5afdce42078fe8f94ad166ac3c" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 39175 }, Status: INVALID_TERM, Last received: 6.730, Next index: 731, Last known committed idx: 730, Time since last communication: 0.000s I0312 20:10:51.567085 1738 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 7 FOLLOWER]: Rejecting Update request from peer 012786a8f1b64995847cf7dacbc029a4 for earlier term 6. Current term is 7. Ops: [] I0312 20:10:51.568042 2187 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9da22e5afdce42078fe8f94ad166ac3c" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 39175 }, Status: INVALID_TERM, Last received: 6.730, Next index: 731, Last known committed idx: 730, Time since last communication: 0.000s I0312 20:10:51.571590 1738 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 7 FOLLOWER]: Rejecting Update request from peer 012786a8f1b64995847cf7dacbc029a4 for earlier term 6. Current term is 7. Ops: [] I0312 20:10:51.572487 2187 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9da22e5afdce42078fe8f94ad166ac3c" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 39175 }, Status: INVALID_TERM, Last received: 6.730, Next index: 731, Last known committed idx: 730, Time since last communication: 0.000s I0312 20:10:51.575238 1738 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 7 FOLLOWER]: Rejecting Update request from peer 012786a8f1b64995847cf7dacbc029a4 for earlier term 6. Current term is 7. Ops: [] I0312 20:10:51.576126 2187 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9da22e5afdce42078fe8f94ad166ac3c" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 39175 }, Status: INVALID_TERM, Last received: 6.730, Next index: 731, Last known committed idx: 730, Time since last communication: 0.000s I0312 20:10:51.580483 1738 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 7 FOLLOWER]: Rejecting Update request from peer 012786a8f1b64995847cf7dacbc029a4 for earlier term 6. Current term is 7. Ops: [] I0312 20:10:51.581461 2187 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9da22e5afdce42078fe8f94ad166ac3c" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 39175 }, Status: INVALID_TERM, Last received: 6.730, Next index: 731, Last known committed idx: 730, Time since last communication: 0.000s I0312 20:10:51.584903 1738 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 7 FOLLOWER]: Rejecting Update request from peer 012786a8f1b64995847cf7dacbc029a4 for earlier term 6. Current term is 7. Ops: [] I0312 20:10:51.586138 2187 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9da22e5afdce42078fe8f94ad166ac3c" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 39175 }, Status: INVALID_TERM, Last received: 6.730, Next index: 731, Last known committed idx: 730, Time since last communication: 0.000s I0312 20:10:51.589342 1738 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 7 FOLLOWER]: Rejecting Update request from peer 012786a8f1b64995847cf7dacbc029a4 for earlier term 6. Current term is 7. Ops: [] I0312 20:10:51.590298 2187 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9da22e5afdce42078fe8f94ad166ac3c" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 39175 }, Status: INVALID_TERM, Last received: 6.730, Next index: 731, Last known committed idx: 730, Time since last communication: 0.000s I0312 20:10:51.592635 1738 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 7 FOLLOWER]: Rejecting Update request from peer 012786a8f1b64995847cf7dacbc029a4 for earlier term 6. Current term is 7. Ops: [] I0312 20:10:51.593551 2187 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9da22e5afdce42078fe8f94ad166ac3c" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 39175 }, Status: INVALID_TERM, Last received: 6.730, Next index: 731, Last known committed idx: 730, Time since last communication: 0.000s I0312 20:10:51.594318 2185 raft_consensus.cc:436] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0312 20:10:51.594485 2185 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c: Snoozing failure detection for 0.047s (starting election) I0312 20:10:51.594653 2185 raft_consensus.cc:458] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "012786a8f1b64995847cf7dacbc029a4" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 43059 } } peers { permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 } } peers { permanent_uuid: "9da22e5afdce42078fe8f94ad166ac3c" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 39175 } } I0312 20:10:51.595320 2185 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [CANDIDATE]: Term 8 pre-election: Requesting pre-vote from peer 012786a8f1b64995847cf7dacbc029a4 I0312 20:10:51.595624 2185 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [CANDIDATE]: Term 8 pre-election: Requesting pre-vote from peer e3aa58d3045c4a05b94b94eb8bf58ae1 I0312 20:10:51.596673 1865 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "9da22e5afdce42078fe8f94ad166ac3c" candidate_term: 8 candidate_status { last_received { term: 6 index: 730 } } ignore_live_leader: false dest_uuid: "012786a8f1b64995847cf7dacbc029a4" is_pre_election: true I0312 20:10:51.596657 1990 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "9da22e5afdce42078fe8f94ad166ac3c" candidate_term: 8 candidate_status { last_received { term: 6 index: 730 } } ignore_live_leader: false dest_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" is_pre_election: true I0312 20:10:51.597015 1865 raft_consensus.cc:2176] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 6 LEADER]: Leader pre-election vote request: Denying vote to candidate 9da22e5afdce42078fe8f94ad166ac3c for term 8 because replica is either leader or believes a valid leader to be alive. I0312 20:10:51.597041 1990 raft_consensus.cc:2176] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9da22e5afdce42078fe8f94ad166ac3c for term 8 because replica is either leader or believes a valid leader to be alive. I0312 20:10:51.597805 1676 leader_election.cc:377] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [CANDIDATE]: Term 8 pre-election: Vote denied by peer 012786a8f1b64995847cf7dacbc029a4. Message: Invalid argument: T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 6 LEADER]: Leader pre-election vote request: Denying vote to candidate 9da22e5afdce42078fe8f94ad166ac3c for term 8 because replica is either leader or believes a valid leader to be alive. I0312 20:10:51.598021 1663 leader_election.cc:377] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [CANDIDATE]: Term 8 pre-election: Vote denied by peer e3aa58d3045c4a05b94b94eb8bf58ae1. Message: Invalid argument: T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9da22e5afdce42078fe8f94ad166ac3c for term 8 because replica is either leader or believes a valid leader to be alive. I0312 20:10:51.598033 2191 raft_consensus.cc:2747] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 6 LEADER]: Stepping down as leader of term 6 I0312 20:10:51.598147 1663 leader_election.cc:258] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. I0312 20:10:51.598177 2191 raft_consensus.cc:605] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 012786a8f1b64995847cf7dacbc029a4, State: Running, Role: LEADER I0312 20:10:51.598518 2185 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c: Snoozing failure detection for 0.042s (election complete) I0312 20:10:51.598515 2191 consensus_queue.cc:219] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 733, Committed index: 733, Last appended: 6.733, Last appended by leader: 733, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "012786a8f1b64995847cf7dacbc029a4" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 43059 } } peers { permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 } } peers { permanent_uuid: "9da22e5afdce42078fe8f94ad166ac3c" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 39175 } } I0312 20:10:51.598887 2185 raft_consensus.cc:2455] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority I0312 20:10:51.598990 2191 consensus_peers.cc:458] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 -> Peer 9da22e5afdce42078fe8f94ad166ac3c (127.0.53.1:39175): Closing peer: 9da22e5afdce42078fe8f94ad166ac3c I0312 20:10:51.599239 2187 consensus_peers.cc:230] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 -> Peer e3aa58d3045c4a05b94b94eb8bf58ae1 (127.0.53.3:49316): Could not obtain request from queue for peer: e3aa58d3045c4a05b94b94eb8bf58ae1. Status: Not found: Peer not tracked or queue not in leader mode. I0312 20:10:51.599432 2191 consensus_peers.cc:458] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 -> Peer e3aa58d3045c4a05b94b94eb8bf58ae1 (127.0.53.3:49316): Closing peer: e3aa58d3045c4a05b94b94eb8bf58ae1 I0312 20:10:51.599750 2191 raft_consensus.cc:2752] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 6 FOLLOWER]: Advancing to term 7 I0312 20:10:51.599930 1738 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 7 FOLLOWER]: Rejecting Update request from peer 012786a8f1b64995847cf7dacbc029a4 for earlier term 6. Current term is 7. Ops: [] I0312 20:10:51.610280 2187 raft_consensus.cc:436] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0312 20:10:51.610446 2187 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4: Snoozing failure detection for 0.020s (starting election) I0312 20:10:51.610673 2187 raft_consensus.cc:458] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "012786a8f1b64995847cf7dacbc029a4" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 43059 } } peers { permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 } } peers { permanent_uuid: "9da22e5afdce42078fe8f94ad166ac3c" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 39175 } } I0312 20:10:51.611506 2187 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [CANDIDATE]: Term 8 pre-election: Requesting pre-vote from peer e3aa58d3045c4a05b94b94eb8bf58ae1 I0312 20:10:51.611781 2187 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [CANDIDATE]: Term 8 pre-election: Requesting pre-vote from peer 9da22e5afdce42078fe8f94ad166ac3c I0312 20:10:51.612449 2184 raft_consensus.cc:436] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 012786a8f1b64995847cf7dacbc029a4) I0312 20:10:51.612419 1990 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "012786a8f1b64995847cf7dacbc029a4" candidate_term: 8 candidate_status { last_received { term: 6 index: 733 } } ignore_live_leader: false dest_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" is_pre_election: true I0312 20:10:51.612643 2184 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1: Snoozing failure detection for 0.021s (starting election) I0312 20:10:51.612800 2184 raft_consensus.cc:458] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "012786a8f1b64995847cf7dacbc029a4" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 43059 } } peers { permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 } } peers { permanent_uuid: "9da22e5afdce42078fe8f94ad166ac3c" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 39175 } } I0312 20:10:51.613445 1738 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "012786a8f1b64995847cf7dacbc029a4" candidate_term: 8 candidate_status { last_received { term: 6 index: 733 } } ignore_live_leader: false dest_uuid: "9da22e5afdce42078fe8f94ad166ac3c" is_pre_election: true I0312 20:10:51.613601 2184 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 7 pre-election: Requesting pre-vote from peer 012786a8f1b64995847cf7dacbc029a4 I0312 20:10:51.613770 1738 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c: Snoozing failure detection for 0.110s (vote granted) I0312 20:10:51.613883 2184 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 7 pre-election: Requesting pre-vote from peer 9da22e5afdce42078fe8f94ad166ac3c I0312 20:10:51.613987 1738 raft_consensus.cc:2215] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 012786a8f1b64995847cf7dacbc029a4 in term 7. I0312 20:10:51.614524 1990 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1: Snoozing failure detection for 0.032s (vote granted) I0312 20:10:51.614475 1865 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" candidate_term: 7 candidate_status { last_received { term: 6 index: 733 } } ignore_live_leader: false dest_uuid: "012786a8f1b64995847cf7dacbc029a4" is_pre_election: true I0312 20:10:51.614675 1797 leader_election.cc:363] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [CANDIDATE]: Term 8 pre-election: Vote granted by peer 9da22e5afdce42078fe8f94ad166ac3c I0312 20:10:51.614701 1990 raft_consensus.cc:2215] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 012786a8f1b64995847cf7dacbc029a4 in term 6. I0312 20:10:51.614739 1865 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4: Snoozing failure detection for 0.020s (vote granted) I0312 20:10:51.614838 1797 leader_election.cc:258] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. I0312 20:10:51.614888 1865 raft_consensus.cc:2215] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3aa58d3045c4a05b94b94eb8bf58ae1 in term 7. I0312 20:10:51.615209 2187 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4: Snoozing failure detection for 0.022s (election complete) I0312 20:10:51.615360 2187 raft_consensus.cc:2507] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 7 FOLLOWER]: Leader pre-election won for term 8 I0312 20:10:51.615530 2187 raft_consensus.cc:436] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0312 20:10:51.615550 1926 leader_election.cc:363] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 7 pre-election: Vote granted by peer 012786a8f1b64995847cf7dacbc029a4 I0312 20:10:51.615679 1926 leader_election.cc:258] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. I0312 20:10:51.615731 2187 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4: Snoozing failure detection for 0.017s (starting election) I0312 20:10:51.615851 2187 raft_consensus.cc:2752] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 7 FOLLOWER]: Advancing to term 8 I0312 20:10:51.615960 1797 leader_election.cc:363] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [CANDIDATE]: Term 8 pre-election: Vote granted by peer e3aa58d3045c4a05b94b94eb8bf58ae1 I0312 20:10:51.616070 2184 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1: Snoozing failure detection for 0.024s (election complete) I0312 20:10:51.616219 2184 raft_consensus.cc:2507] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 6 FOLLOWER]: Leader pre-election won for term 7 I0312 20:10:51.616333 2184 raft_consensus.cc:436] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 012786a8f1b64995847cf7dacbc029a4) I0312 20:10:51.616453 2184 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1: Snoozing failure detection for 0.030s (starting election) I0312 20:10:51.616560 2184 raft_consensus.cc:2752] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 6 FOLLOWER]: Advancing to term 7 I0312 20:10:51.617239 1738 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" candidate_term: 7 candidate_status { last_received { term: 6 index: 733 } } ignore_live_leader: false dest_uuid: "9da22e5afdce42078fe8f94ad166ac3c" is_pre_election: true I0312 20:10:51.617588 1738 raft_consensus.cc:2145] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e3aa58d3045c4a05b94b94eb8bf58ae1 in current term 7: Already voted for candidate 9da22e5afdce42078fe8f94ad166ac3c in this term. I0312 20:10:51.618238 1922 leader_election.cc:377] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 7 pre-election: Vote denied by peer 9da22e5afdce42078fe8f94ad166ac3c. Message: Invalid argument: T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e3aa58d3045c4a05b94b94eb8bf58ae1 in current term 7: Already voted for candidate 9da22e5afdce42078fe8f94ad166ac3c in this term. I0312 20:10:51.619724 2184 raft_consensus.cc:458] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "012786a8f1b64995847cf7dacbc029a4" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 43059 } } peers { permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 } } peers { permanent_uuid: "9da22e5afdce42078fe8f94ad166ac3c" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 39175 } } I0312 20:10:51.620506 2184 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 7 election: Requesting vote from peer 012786a8f1b64995847cf7dacbc029a4 I0312 20:10:51.620703 2187 raft_consensus.cc:458] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "012786a8f1b64995847cf7dacbc029a4" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 43059 } } peers { permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 } } peers { permanent_uuid: "9da22e5afdce42078fe8f94ad166ac3c" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 39175 } } I0312 20:10:51.621491 2187 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [CANDIDATE]: Term 8 election: Requesting vote from peer e3aa58d3045c4a05b94b94eb8bf58ae1 I0312 20:10:51.621418 1865 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" candidate_term: 7 candidate_status { last_received { term: 6 index: 733 } } ignore_live_leader: false dest_uuid: "012786a8f1b64995847cf7dacbc029a4" I0312 20:10:51.621647 2184 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 7 election: Requesting vote from peer 9da22e5afdce42078fe8f94ad166ac3c I0312 20:10:51.621783 1865 raft_consensus.cc:2120] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate e3aa58d3045c4a05b94b94eb8bf58ae1 for earlier term 7. Current term is 8. W0312 20:10:51.622453 1926 leader_election.cc:347] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 7 election: Vote denied by peer 012786a8f1b64995847cf7dacbc029a4 with higher term. Message: Invalid argument: T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate e3aa58d3045c4a05b94b94eb8bf58ae1 for earlier term 7. Current term is 8. I0312 20:10:51.622573 1926 leader_election.cc:350] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 7 election: Cancelling election due to peer responding with higher term I0312 20:10:51.622531 1738 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" candidate_term: 7 candidate_status { last_received { term: 6 index: 733 } } ignore_live_leader: false dest_uuid: "9da22e5afdce42078fe8f94ad166ac3c" I0312 20:10:51.622910 2184 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1: Snoozing failure detection for 0.033s (election complete) I0312 20:10:51.623054 2184 raft_consensus.cc:2752] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 7 FOLLOWER]: Advancing to term 8 I0312 20:10:51.623226 1738 raft_consensus.cc:2145] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate e3aa58d3045c4a05b94b94eb8bf58ae1 in current term 7: Already voted for candidate 9da22e5afdce42078fe8f94ad166ac3c in this term. I0312 20:10:51.624114 1922 leader_election.cc:377] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 7 election: Vote denied by peer 9da22e5afdce42078fe8f94ad166ac3c. Message: Invalid argument: T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate e3aa58d3045c4a05b94b94eb8bf58ae1 in current term 7: Already voted for candidate 9da22e5afdce42078fe8f94ad166ac3c in this term. I0312 20:10:51.624270 1990 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "012786a8f1b64995847cf7dacbc029a4" candidate_term: 8 candidate_status { last_received { term: 6 index: 733 } } ignore_live_leader: false dest_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" I0312 20:10:51.622325 2187 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [CANDIDATE]: Term 8 election: Requesting vote from peer 9da22e5afdce42078fe8f94ad166ac3c I0312 20:10:51.625632 2184 raft_consensus.cc:2455] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 8 FOLLOWER]: Leader election lost for term 7. Reason: Vote denied by peer 012786a8f1b64995847cf7dacbc029a4 with higher term. Message: Invalid argument: T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate e3aa58d3045c4a05b94b94eb8bf58ae1 for earlier term 7. Current term is 8. I0312 20:10:51.625571 1738 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "012786a8f1b64995847cf7dacbc029a4" candidate_term: 8 candidate_status { last_received { term: 6 index: 733 } } ignore_live_leader: false dest_uuid: "9da22e5afdce42078fe8f94ad166ac3c" I0312 20:10:51.625854 1990 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1: Snoozing failure detection for 0.027s (vote granted) I0312 20:10:51.625910 1738 raft_consensus.cc:2752] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 7 FOLLOWER]: Advancing to term 8 I0312 20:10:51.626118 1738 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c: Snoozing failure detection for 0.048s (vote granted) I0312 20:10:51.628386 1990 raft_consensus.cc:2215] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 012786a8f1b64995847cf7dacbc029a4 in term 8. I0312 20:10:51.629484 1738 raft_consensus.cc:2215] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 012786a8f1b64995847cf7dacbc029a4 in term 8. I0312 20:10:51.630044 1797 leader_election.cc:363] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [CANDIDATE]: Term 8 election: Vote granted by peer e3aa58d3045c4a05b94b94eb8bf58ae1 I0312 20:10:51.630187 1797 leader_election.cc:258] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. I0312 20:10:51.630534 2187 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4: Snoozing failure detection for 0.016s (election complete) I0312 20:10:51.630638 1797 leader_election.cc:363] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [CANDIDATE]: Term 8 election: Vote granted by peer 9da22e5afdce42078fe8f94ad166ac3c I0312 20:10:51.630736 2187 raft_consensus.cc:2507] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 8 FOLLOWER]: Leader election won for term 8 I0312 20:10:51.631033 2187 raft_consensus.cc:572] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 8 LEADER]: Becoming Leader. State: Replica: 012786a8f1b64995847cf7dacbc029a4, State: Running, Role: LEADER I0312 20:10:51.631310 2187 consensus_queue.cc:196] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 731, Committed index: 731, Last appended: 6.733, Last appended by leader: 733, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "012786a8f1b64995847cf7dacbc029a4" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 43059 } } peers { permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 } } peers { permanent_uuid: "9da22e5afdce42078fe8f94ad166ac3c" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 39175 } } I0312 20:10:51.634268 1619 catalog_manager.cc:3592] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 reported cstate change: term changed from 6 to 8. New cstate: current_term: 8 leader_uuid: "012786a8f1b64995847cf7dacbc029a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "012786a8f1b64995847cf7dacbc029a4" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 43059 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9da22e5afdce42078fe8f94ad166ac3c" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 39175 } health_report { overall_health: UNKNOWN } } } I0312 20:10:51.638314 1738 raft_consensus.cc:1058] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 8 FOLLOWER]: Refusing update from remote peer 012786a8f1b64995847cf7dacbc029a4: Log matching property violated. Preceding OpId in replica: term: 6 index: 730. Preceding OpId from leader: term: 8 index: 734. (index mismatch) I0312 20:10:51.639509 2187 consensus_queue.cc:1017] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9da22e5afdce42078fe8f94ad166ac3c" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 39175 }, Status: LMP_MISMATCH, Last received: 6.730, Next index: 731, Last known committed idx: 730, Time since last communication: 0.000s I0312 20:10:51.639740 1990 raft_consensus.cc:1058] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 8 FOLLOWER]: Refusing update from remote peer 012786a8f1b64995847cf7dacbc029a4: Log matching property violated. Preceding OpId in replica: term: 6 index: 733. Preceding OpId from leader: term: 8 index: 734. (index mismatch) I0312 20:10:51.640846 2187 consensus_queue.cc:1017] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 }, Status: LMP_MISMATCH, Last received: 6.733, Next index: 734, Last known committed idx: 733, Time since last communication: 0.000s W0312 20:10:51.663357 2191 raft_consensus.cc:747] Couldn't advance consensus term.: Illegal state: Can't advance term to: 7 current term: 8 is higher. W0312 20:10:51.667675 2191 raft_consensus.cc:747] Couldn't advance consensus term.: Illegal state: Can't advance term to: 7 current term: 8 is higher. W0312 20:10:51.714970 2191 raft_consensus.cc:747] Couldn't advance consensus term.: Illegal state: Can't advance term to: 7 current term: 8 is higher. W0312 20:10:51.723323 2191 raft_consensus.cc:747] Couldn't advance consensus term.: Illegal state: Can't advance term to: 7 current term: 8 is higher. W0312 20:10:51.769642 2191 raft_consensus.cc:747] Couldn't advance consensus term.: Illegal state: Can't advance term to: 7 current term: 8 is higher. I0312 20:10:51.831302 2136 raft_consensus.cc:436] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 012786a8f1b64995847cf7dacbc029a4) I0312 20:10:51.831495 2136 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c: Snoozing failure detection for 0.018s (starting election) I0312 20:10:51.831638 2136 raft_consensus.cc:458] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "012786a8f1b64995847cf7dacbc029a4" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 43059 } } peers { permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 } } peers { permanent_uuid: "9da22e5afdce42078fe8f94ad166ac3c" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 39175 } } I0312 20:10:51.832361 2136 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [CANDIDATE]: Term 9 pre-election: Requesting pre-vote from peer 012786a8f1b64995847cf7dacbc029a4 I0312 20:10:51.832608 2136 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [CANDIDATE]: Term 9 pre-election: Requesting pre-vote from peer e3aa58d3045c4a05b94b94eb8bf58ae1 I0312 20:10:51.833142 2184 raft_consensus.cc:436] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 012786a8f1b64995847cf7dacbc029a4) I0312 20:10:51.833312 2184 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1: Snoozing failure detection for 0.019s (starting election) I0312 20:10:51.833461 2184 raft_consensus.cc:458] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "012786a8f1b64995847cf7dacbc029a4" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 43059 } } peers { permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 } } peers { permanent_uuid: "9da22e5afdce42078fe8f94ad166ac3c" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 39175 } } I0312 20:10:51.834213 2184 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 9 pre-election: Requesting pre-vote from peer 012786a8f1b64995847cf7dacbc029a4 I0312 20:10:51.834482 2184 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 9 pre-election: Requesting pre-vote from peer 9da22e5afdce42078fe8f94ad166ac3c I0312 20:10:51.834805 1989 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "9da22e5afdce42078fe8f94ad166ac3c" candidate_term: 9 candidate_status { last_received { term: 8 index: 734 } } ignore_live_leader: false dest_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" is_pre_election: true I0312 20:10:51.835170 1989 raft_consensus.cc:2190] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9da22e5afdce42078fe8f94ad166ac3c for term 9 because replica is already servicing an update from a current leader or another vote. I0312 20:10:51.835399 1738 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" candidate_term: 9 candidate_status { last_received { term: 8 index: 734 } } ignore_live_leader: false dest_uuid: "9da22e5afdce42078fe8f94ad166ac3c" is_pre_election: true I0312 20:10:51.835722 1738 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c: Snoozing failure detection for 0.018s (vote granted) I0312 20:10:51.835861 1738 raft_consensus.cc:2215] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3aa58d3045c4a05b94b94eb8bf58ae1 in term 8. I0312 20:10:51.835922 1663 leader_election.cc:377] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [CANDIDATE]: Term 9 pre-election: Vote denied by peer e3aa58d3045c4a05b94b94eb8bf58ae1. Message: Service unavailable: T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9da22e5afdce42078fe8f94ad166ac3c for term 9 because replica is already servicing an update from a current leader or another vote. I0312 20:10:51.836308 1865 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" candidate_term: 9 candidate_status { last_received { term: 8 index: 734 } } ignore_live_leader: false dest_uuid: "012786a8f1b64995847cf7dacbc029a4" is_pre_election: true I0312 20:10:51.836513 1922 leader_election.cc:363] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 9 pre-election: Vote granted by peer 9da22e5afdce42078fe8f94ad166ac3c I0312 20:10:51.836666 1865 raft_consensus.cc:2176] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 8 LEADER]: Leader pre-election vote request: Denying vote to candidate e3aa58d3045c4a05b94b94eb8bf58ae1 for term 9 because replica is either leader or believes a valid leader to be alive. I0312 20:10:51.836665 1922 leader_election.cc:258] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. I0312 20:10:51.837005 2184 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1: Snoozing failure detection for 0.018s (election complete) I0312 20:10:51.837178 2184 raft_consensus.cc:2507] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 8 FOLLOWER]: Leader pre-election won for term 9 I0312 20:10:51.837278 1926 leader_election.cc:377] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 9 pre-election: Vote denied by peer 012786a8f1b64995847cf7dacbc029a4. Message: Invalid argument: T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 8 LEADER]: Leader pre-election vote request: Denying vote to candidate e3aa58d3045c4a05b94b94eb8bf58ae1 for term 9 because replica is either leader or believes a valid leader to be alive. I0312 20:10:51.837294 2184 raft_consensus.cc:436] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 012786a8f1b64995847cf7dacbc029a4) I0312 20:10:51.837501 2184 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1: Snoozing failure detection for 0.017s (starting election) I0312 20:10:51.837492 1865 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "9da22e5afdce42078fe8f94ad166ac3c" candidate_term: 9 candidate_status { last_received { term: 8 index: 734 } } ignore_live_leader: false dest_uuid: "012786a8f1b64995847cf7dacbc029a4" is_pre_election: true I0312 20:10:51.837633 2184 raft_consensus.cc:2752] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 8 FOLLOWER]: Advancing to term 9 I0312 20:10:51.837826 1865 raft_consensus.cc:2176] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 8 LEADER]: Leader pre-election vote request: Denying vote to candidate 9da22e5afdce42078fe8f94ad166ac3c for term 9 because replica is either leader or believes a valid leader to be alive. I0312 20:10:51.840436 1676 leader_election.cc:377] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [CANDIDATE]: Term 9 pre-election: Vote denied by peer 012786a8f1b64995847cf7dacbc029a4. Message: Invalid argument: T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 8 LEADER]: Leader pre-election vote request: Denying vote to candidate 9da22e5afdce42078fe8f94ad166ac3c for term 9 because replica is either leader or believes a valid leader to be alive. I0312 20:10:51.840600 1676 leader_election.cc:258] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. I0312 20:10:51.840464 2184 raft_consensus.cc:458] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "012786a8f1b64995847cf7dacbc029a4" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 43059 } } peers { permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 } } peers { permanent_uuid: "9da22e5afdce42078fe8f94ad166ac3c" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 39175 } } I0312 20:10:51.840885 2136 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c: Snoozing failure detection for 0.019s (election complete) I0312 20:10:51.841051 2136 raft_consensus.cc:2455] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority I0312 20:10:51.841261 2184 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 9 election: Requesting vote from peer 012786a8f1b64995847cf7dacbc029a4 I0312 20:10:51.841325 1990 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 9 FOLLOWER]: Rejecting Update request from peer 012786a8f1b64995847cf7dacbc029a4 for earlier term 8. Current term is 9. Ops: [] I0312 20:10:51.841526 2184 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 9 election: Requesting vote from peer 9da22e5afdce42078fe8f94ad166ac3c I0312 20:10:51.842358 1865 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" candidate_term: 9 candidate_status { last_received { term: 8 index: 735 } } ignore_live_leader: false dest_uuid: "012786a8f1b64995847cf7dacbc029a4" I0312 20:10:51.842667 1865 raft_consensus.cc:2176] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 8 LEADER]: Leader election vote request: Denying vote to candidate e3aa58d3045c4a05b94b94eb8bf58ae1 for term 9 because replica is either leader or believes a valid leader to be alive. I0312 20:10:51.843292 1926 leader_election.cc:377] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 9 election: Vote denied by peer 012786a8f1b64995847cf7dacbc029a4. Message: Invalid argument: T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 8 LEADER]: Leader election vote request: Denying vote to candidate e3aa58d3045c4a05b94b94eb8bf58ae1 for term 9 because replica is either leader or believes a valid leader to be alive. I0312 20:10:51.843341 1738 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" candidate_term: 9 candidate_status { last_received { term: 8 index: 735 } } ignore_live_leader: false dest_uuid: "9da22e5afdce42078fe8f94ad166ac3c" I0312 20:10:51.843683 1738 raft_consensus.cc:2176] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate e3aa58d3045c4a05b94b94eb8bf58ae1 for term 9 because replica is either leader or believes a valid leader to be alive. I0312 20:10:51.844425 1922 leader_election.cc:377] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 9 election: Vote denied by peer 9da22e5afdce42078fe8f94ad166ac3c. Message: Invalid argument: T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate e3aa58d3045c4a05b94b94eb8bf58ae1 for term 9 because replica is either leader or believes a valid leader to be alive. I0312 20:10:51.844580 1922 leader_election.cc:258] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. I0312 20:10:51.844867 2184 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1: Snoozing failure detection for 0.019s (election complete) I0312 20:10:51.845021 2184 raft_consensus.cc:2455] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority I0312 20:10:51.846246 2187 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 }, Status: INVALID_TERM, Last received: 8.735, Next index: 736, Last known committed idx: 734, Time since last communication: 0.000s I0312 20:10:51.849241 1990 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 9 FOLLOWER]: Rejecting Update request from peer 012786a8f1b64995847cf7dacbc029a4 for earlier term 8. Current term is 9. Ops: [] I0312 20:10:51.850082 2187 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 }, Status: INVALID_TERM, Last received: 8.735, Next index: 736, Last known committed idx: 734, Time since last communication: 0.000s I0312 20:10:51.852463 1990 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 9 FOLLOWER]: Rejecting Update request from peer 012786a8f1b64995847cf7dacbc029a4 for earlier term 8. Current term is 9. Ops: [] I0312 20:10:51.853329 2187 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 }, Status: INVALID_TERM, Last received: 8.735, Next index: 736, Last known committed idx: 734, Time since last communication: 0.000s I0312 20:10:51.857256 1990 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 9 FOLLOWER]: Rejecting Update request from peer 012786a8f1b64995847cf7dacbc029a4 for earlier term 8. Current term is 9. Ops: [] W0312 20:10:51.857647 2191 raft_consensus.cc:747] Couldn't advance consensus term.: Illegal state: Can't advance term to: 7 current term: 8 is higher. I0312 20:10:51.861810 2194 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 }, Status: INVALID_TERM, Last received: 8.735, Next index: 736, Last known committed idx: 734, Time since last communication: 0.000s I0312 20:10:51.864748 2184 raft_consensus.cc:436] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0312 20:10:51.864924 2184 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1: Snoozing failure detection for 0.022s (starting election) I0312 20:10:51.865111 2184 raft_consensus.cc:458] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "012786a8f1b64995847cf7dacbc029a4" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 43059 } } peers { permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 } } peers { permanent_uuid: "9da22e5afdce42078fe8f94ad166ac3c" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 39175 } } I0312 20:10:51.865901 2184 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 10 pre-election: Requesting pre-vote from peer 012786a8f1b64995847cf7dacbc029a4 I0312 20:10:51.866127 1990 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 9 FOLLOWER]: Rejecting Update request from peer 012786a8f1b64995847cf7dacbc029a4 for earlier term 8. Current term is 9. Ops: [] I0312 20:10:51.866168 2184 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 10 pre-election: Requesting pre-vote from peer 9da22e5afdce42078fe8f94ad166ac3c I0312 20:10:51.866794 1865 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" candidate_term: 10 candidate_status { last_received { term: 8 index: 735 } } ignore_live_leader: false dest_uuid: "012786a8f1b64995847cf7dacbc029a4" is_pre_election: true I0312 20:10:51.867132 1865 raft_consensus.cc:2176] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 8 LEADER]: Leader pre-election vote request: Denying vote to candidate e3aa58d3045c4a05b94b94eb8bf58ae1 for term 10 because replica is either leader or believes a valid leader to be alive. I0312 20:10:51.867166 1738 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" candidate_term: 10 candidate_status { last_received { term: 8 index: 735 } } ignore_live_leader: false dest_uuid: "9da22e5afdce42078fe8f94ad166ac3c" is_pre_election: true I0312 20:10:51.867552 1738 raft_consensus.cc:2176] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e3aa58d3045c4a05b94b94eb8bf58ae1 for term 10 because replica is either leader or believes a valid leader to be alive. I0312 20:10:51.867707 1926 leader_election.cc:377] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 10 pre-election: Vote denied by peer 012786a8f1b64995847cf7dacbc029a4. Message: Invalid argument: T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 8 LEADER]: Leader pre-election vote request: Denying vote to candidate e3aa58d3045c4a05b94b94eb8bf58ae1 for term 10 because replica is either leader or believes a valid leader to be alive. I0312 20:10:51.868135 1922 leader_election.cc:377] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 10 pre-election: Vote denied by peer 9da22e5afdce42078fe8f94ad166ac3c. Message: Invalid argument: T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e3aa58d3045c4a05b94b94eb8bf58ae1 for term 10 because replica is either leader or believes a valid leader to be alive. I0312 20:10:51.868305 1922 leader_election.cc:258] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. I0312 20:10:51.868796 2184 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1: Snoozing failure detection for 0.025s (election complete) I0312 20:10:51.868949 2184 raft_consensus.cc:2455] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority I0312 20:10:51.870038 2194 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 }, Status: INVALID_TERM, Last received: 8.735, Next index: 736, Last known committed idx: 734, Time since last communication: 0.000s I0312 20:10:51.876698 1990 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 9 FOLLOWER]: Rejecting Update request from peer 012786a8f1b64995847cf7dacbc029a4 for earlier term 8. Current term is 9. Ops: [] I0312 20:10:51.877806 2194 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 }, Status: INVALID_TERM, Last received: 8.735, Next index: 736, Last known committed idx: 734, Time since last communication: 0.000s I0312 20:10:51.880808 1990 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 9 FOLLOWER]: Rejecting Update request from peer 012786a8f1b64995847cf7dacbc029a4 for earlier term 8. Current term is 9. Ops: [] I0312 20:10:51.883311 2187 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 }, Status: INVALID_TERM, Last received: 8.735, Next index: 736, Last known committed idx: 734, Time since last communication: 0.000s I0312 20:10:51.887318 1990 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 9 FOLLOWER]: Rejecting Update request from peer 012786a8f1b64995847cf7dacbc029a4 for earlier term 8. Current term is 9. Ops: [] I0312 20:10:51.888430 2187 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 }, Status: INVALID_TERM, Last received: 8.735, Next index: 736, Last known committed idx: 734, Time since last communication: 0.000s I0312 20:10:51.892678 1990 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 9 FOLLOWER]: Rejecting Update request from peer 012786a8f1b64995847cf7dacbc029a4 for earlier term 8. Current term is 9. Ops: [] I0312 20:10:51.893880 2194 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 }, Status: INVALID_TERM, Last received: 8.735, Next index: 736, Last known committed idx: 734, Time since last communication: 0.000s I0312 20:10:51.894495 2184 raft_consensus.cc:436] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0312 20:10:51.894664 2184 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1: Snoozing failure detection for 0.039s (starting election) I0312 20:10:51.894822 2184 raft_consensus.cc:458] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "012786a8f1b64995847cf7dacbc029a4" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 43059 } } peers { permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 } } peers { permanent_uuid: "9da22e5afdce42078fe8f94ad166ac3c" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 39175 } } I0312 20:10:51.895704 2184 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 10 pre-election: Requesting pre-vote from peer 012786a8f1b64995847cf7dacbc029a4 I0312 20:10:51.895969 2184 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 10 pre-election: Requesting pre-vote from peer 9da22e5afdce42078fe8f94ad166ac3c I0312 20:10:51.896584 1990 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 9 FOLLOWER]: Rejecting Update request from peer 012786a8f1b64995847cf7dacbc029a4 for earlier term 8. Current term is 9. Ops: [] I0312 20:10:51.896553 1865 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" candidate_term: 10 candidate_status { last_received { term: 8 index: 735 } } ignore_live_leader: false dest_uuid: "012786a8f1b64995847cf7dacbc029a4" is_pre_election: true I0312 20:10:51.896895 1865 raft_consensus.cc:2176] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 8 LEADER]: Leader pre-election vote request: Denying vote to candidate e3aa58d3045c4a05b94b94eb8bf58ae1 for term 10 because replica is either leader or believes a valid leader to be alive. I0312 20:10:51.896805 1738 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" candidate_term: 10 candidate_status { last_received { term: 8 index: 735 } } ignore_live_leader: false dest_uuid: "9da22e5afdce42078fe8f94ad166ac3c" is_pre_election: true I0312 20:10:51.897169 1738 raft_consensus.cc:2176] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e3aa58d3045c4a05b94b94eb8bf58ae1 for term 10 because replica is either leader or believes a valid leader to be alive. I0312 20:10:51.897469 1926 leader_election.cc:377] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 10 pre-election: Vote denied by peer 012786a8f1b64995847cf7dacbc029a4. Message: Invalid argument: T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 8 LEADER]: Leader pre-election vote request: Denying vote to candidate e3aa58d3045c4a05b94b94eb8bf58ae1 for term 10 because replica is either leader or believes a valid leader to be alive. I0312 20:10:51.897832 1922 leader_election.cc:377] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 10 pre-election: Vote denied by peer 9da22e5afdce42078fe8f94ad166ac3c. Message: Invalid argument: T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e3aa58d3045c4a05b94b94eb8bf58ae1 for term 10 because replica is either leader or believes a valid leader to be alive. I0312 20:10:51.897992 1922 leader_election.cc:258] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. I0312 20:10:51.898397 2184 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1: Snoozing failure detection for 0.028s (election complete) I0312 20:10:51.898552 2184 raft_consensus.cc:2455] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority W0312 20:10:51.898953 2191 raft_consensus.cc:747] Couldn't advance consensus term.: Illegal state: Can't advance term to: 7 current term: 8 is higher. I0312 20:10:51.899292 2194 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 }, Status: INVALID_TERM, Last received: 8.735, Next index: 736, Last known committed idx: 734, Time since last communication: 0.000s I0312 20:10:51.902470 1990 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 9 FOLLOWER]: Rejecting Update request from peer 012786a8f1b64995847cf7dacbc029a4 for earlier term 8. Current term is 9. Ops: [] I0312 20:10:51.903810 2194 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 }, Status: INVALID_TERM, Last received: 8.735, Next index: 736, Last known committed idx: 734, Time since last communication: 0.000s W0312 20:10:51.906298 2191 raft_consensus.cc:747] Couldn't advance consensus term.: Illegal state: Can't advance term to: 7 current term: 8 is higher. I0312 20:10:51.908107 1990 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 9 FOLLOWER]: Rejecting Update request from peer 012786a8f1b64995847cf7dacbc029a4 for earlier term 8. Current term is 9. Ops: [] I0312 20:10:51.908851 2194 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 }, Status: INVALID_TERM, Last received: 8.735, Next index: 736, Last known committed idx: 734, Time since last communication: 0.000s I0312 20:10:51.912554 1990 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 9 FOLLOWER]: Rejecting Update request from peer 012786a8f1b64995847cf7dacbc029a4 for earlier term 8. Current term is 9. Ops: [] I0312 20:10:51.913426 2187 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 }, Status: INVALID_TERM, Last received: 8.735, Next index: 736, Last known committed idx: 734, Time since last communication: 0.000s I0312 20:10:51.916940 1990 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 9 FOLLOWER]: Rejecting Update request from peer 012786a8f1b64995847cf7dacbc029a4 for earlier term 8. Current term is 9. Ops: [] I0312 20:10:51.917685 2187 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 }, Status: INVALID_TERM, Last received: 8.735, Next index: 736, Last known committed idx: 734, Time since last communication: 0.000s I0312 20:10:51.922473 1990 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 9 FOLLOWER]: Rejecting Update request from peer 012786a8f1b64995847cf7dacbc029a4 for earlier term 8. Current term is 9. Ops: [] I0312 20:10:51.923377 2187 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 }, Status: INVALID_TERM, Last received: 8.735, Next index: 736, Last known committed idx: 734, Time since last communication: 0.000s I0312 20:10:51.926973 1990 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 9 FOLLOWER]: Rejecting Update request from peer 012786a8f1b64995847cf7dacbc029a4 for earlier term 8. Current term is 9. Ops: [] I0312 20:10:51.927402 2184 raft_consensus.cc:436] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0312 20:10:51.927582 2184 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1: Snoozing failure detection for 0.062s (starting election) I0312 20:10:51.927743 2184 raft_consensus.cc:458] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "012786a8f1b64995847cf7dacbc029a4" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 43059 } } peers { permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 } } peers { permanent_uuid: "9da22e5afdce42078fe8f94ad166ac3c" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 39175 } } I0312 20:10:51.927887 2187 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 }, Status: INVALID_TERM, Last received: 8.735, Next index: 736, Last known committed idx: 734, Time since last communication: 0.000s I0312 20:10:51.928478 2184 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 10 pre-election: Requesting pre-vote from peer 012786a8f1b64995847cf7dacbc029a4 I0312 20:10:51.928758 2184 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 10 pre-election: Requesting pre-vote from peer 9da22e5afdce42078fe8f94ad166ac3c I0312 20:10:51.929529 1865 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" candidate_term: 10 candidate_status { last_received { term: 8 index: 735 } } ignore_live_leader: false dest_uuid: "012786a8f1b64995847cf7dacbc029a4" is_pre_election: true I0312 20:10:51.929662 1738 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" candidate_term: 10 candidate_status { last_received { term: 8 index: 735 } } ignore_live_leader: false dest_uuid: "9da22e5afdce42078fe8f94ad166ac3c" is_pre_election: true I0312 20:10:51.929884 1865 raft_consensus.cc:2176] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 8 LEADER]: Leader pre-election vote request: Denying vote to candidate e3aa58d3045c4a05b94b94eb8bf58ae1 for term 10 because replica is either leader or believes a valid leader to be alive. I0312 20:10:51.930027 1738 raft_consensus.cc:2176] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e3aa58d3045c4a05b94b94eb8bf58ae1 for term 10 because replica is either leader or believes a valid leader to be alive. I0312 20:10:51.930493 1926 leader_election.cc:377] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 10 pre-election: Vote denied by peer 012786a8f1b64995847cf7dacbc029a4. Message: Invalid argument: T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 8 LEADER]: Leader pre-election vote request: Denying vote to candidate e3aa58d3045c4a05b94b94eb8bf58ae1 for term 10 because replica is either leader or believes a valid leader to be alive. I0312 20:10:51.930711 1922 leader_election.cc:377] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 10 pre-election: Vote denied by peer 9da22e5afdce42078fe8f94ad166ac3c. Message: Invalid argument: T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e3aa58d3045c4a05b94b94eb8bf58ae1 for term 10 because replica is either leader or believes a valid leader to be alive. I0312 20:10:51.930881 1922 leader_election.cc:258] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. I0312 20:10:51.931310 2184 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1: Snoozing failure detection for 0.060s (election complete) I0312 20:10:51.931460 2184 raft_consensus.cc:2455] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority I0312 20:10:51.931674 1990 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 9 FOLLOWER]: Rejecting Update request from peer 012786a8f1b64995847cf7dacbc029a4 for earlier term 8. Current term is 9. Ops: [] I0312 20:10:51.932533 2187 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 }, Status: INVALID_TERM, Last received: 8.735, Next index: 736, Last known committed idx: 734, Time since last communication: 0.000s I0312 20:10:51.936051 1990 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 9 FOLLOWER]: Rejecting Update request from peer 012786a8f1b64995847cf7dacbc029a4 for earlier term 8. Current term is 9. Ops: [] I0312 20:10:51.936842 2187 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 }, Status: INVALID_TERM, Last received: 8.735, Next index: 736, Last known committed idx: 734, Time since last communication: 0.000s I0312 20:10:51.939765 1990 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 9 FOLLOWER]: Rejecting Update request from peer 012786a8f1b64995847cf7dacbc029a4 for earlier term 8. Current term is 9. Ops: [] I0312 20:10:51.940579 2187 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 }, Status: INVALID_TERM, Last received: 8.735, Next index: 736, Last known committed idx: 734, Time since last communication: 0.000s I0312 20:10:51.944010 1990 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 9 FOLLOWER]: Rejecting Update request from peer 012786a8f1b64995847cf7dacbc029a4 for earlier term 8. Current term is 9. Ops: [] I0312 20:10:51.944803 2187 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 }, Status: INVALID_TERM, Last received: 8.735, Next index: 736, Last known committed idx: 734, Time since last communication: 0.000s I0312 20:10:51.948370 1990 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 9 FOLLOWER]: Rejecting Update request from peer 012786a8f1b64995847cf7dacbc029a4 for earlier term 8. Current term is 9. Ops: [] I0312 20:10:51.949205 2194 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 }, Status: INVALID_TERM, Last received: 8.735, Next index: 736, Last known committed idx: 734, Time since last communication: 0.000s W0312 20:10:51.949620 2191 raft_consensus.cc:747] Couldn't advance consensus term.: Illegal state: Can't advance term to: 7 current term: 8 is higher. I0312 20:10:51.951704 1990 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 9 FOLLOWER]: Rejecting Update request from peer 012786a8f1b64995847cf7dacbc029a4 for earlier term 8. Current term is 9. Ops: [] I0312 20:10:51.952801 2187 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 }, Status: INVALID_TERM, Last received: 8.735, Next index: 736, Last known committed idx: 734, Time since last communication: 0.000s I0312 20:10:51.954929 1990 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 9 FOLLOWER]: Rejecting Update request from peer 012786a8f1b64995847cf7dacbc029a4 for earlier term 8. Current term is 9. Ops: [] I0312 20:10:51.955777 2194 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 }, Status: INVALID_TERM, Last received: 8.735, Next index: 736, Last known committed idx: 734, Time since last communication: 0.000s W0312 20:10:51.956228 2191 raft_consensus.cc:747] Couldn't advance consensus term.: Illegal state: Can't advance term to: 7 current term: 8 is higher. I0312 20:10:51.958307 1990 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 9 FOLLOWER]: Rejecting Update request from peer 012786a8f1b64995847cf7dacbc029a4 for earlier term 8. Current term is 9. Ops: [] I0312 20:10:51.959283 2187 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 }, Status: INVALID_TERM, Last received: 8.735, Next index: 736, Last known committed idx: 734, Time since last communication: 0.000s I0312 20:10:51.961849 1990 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 9 FOLLOWER]: Rejecting Update request from peer 012786a8f1b64995847cf7dacbc029a4 for earlier term 8. Current term is 9. Ops: [] I0312 20:10:51.962689 2194 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 }, Status: INVALID_TERM, Last received: 8.735, Next index: 736, Last known committed idx: 734, Time since last communication: 0.000s I0312 20:10:51.966125 1990 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 9 FOLLOWER]: Rejecting Update request from peer 012786a8f1b64995847cf7dacbc029a4 for earlier term 8. Current term is 9. Ops: [] I0312 20:10:51.966924 2194 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 }, Status: INVALID_TERM, Last received: 8.735, Next index: 736, Last known committed idx: 734, Time since last communication: 0.000s I0312 20:10:51.971494 1990 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 9 FOLLOWER]: Rejecting Update request from peer 012786a8f1b64995847cf7dacbc029a4 for earlier term 8. Current term is 9. Ops: [] I0312 20:10:51.972290 2187 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 }, Status: INVALID_TERM, Last received: 8.735, Next index: 736, Last known committed idx: 734, Time since last communication: 0.000s I0312 20:10:51.975869 1990 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 9 FOLLOWER]: Rejecting Update request from peer 012786a8f1b64995847cf7dacbc029a4 for earlier term 8. Current term is 9. Ops: [] I0312 20:10:51.976732 2187 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 }, Status: INVALID_TERM, Last received: 8.735, Next index: 736, Last known committed idx: 734, Time since last communication: 0.000s I0312 20:10:51.979102 1990 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 9 FOLLOWER]: Rejecting Update request from peer 012786a8f1b64995847cf7dacbc029a4 for earlier term 8. Current term is 9. Ops: [] I0312 20:10:51.979856 2187 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 }, Status: INVALID_TERM, Last received: 8.735, Next index: 736, Last known committed idx: 734, Time since last communication: 0.000s I0312 20:10:51.982336 1990 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 9 FOLLOWER]: Rejecting Update request from peer 012786a8f1b64995847cf7dacbc029a4 for earlier term 8. Current term is 9. Ops: [] I0312 20:10:51.983028 2187 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 }, Status: INVALID_TERM, Last received: 8.735, Next index: 736, Last known committed idx: 734, Time since last communication: 0.000s I0312 20:10:51.985780 1990 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 9 FOLLOWER]: Rejecting Update request from peer 012786a8f1b64995847cf7dacbc029a4 for earlier term 8. Current term is 9. Ops: [] I0312 20:10:51.986558 2187 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 }, Status: INVALID_TERM, Last received: 8.735, Next index: 736, Last known committed idx: 734, Time since last communication: 0.000s W0312 20:10:51.989512 2191 raft_consensus.cc:747] Couldn't advance consensus term.: Illegal state: Can't advance term to: 7 current term: 8 is higher. I0312 20:10:51.990288 1990 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 9 FOLLOWER]: Rejecting Update request from peer 012786a8f1b64995847cf7dacbc029a4 for earlier term 8. Current term is 9. Ops: [] I0312 20:10:51.991113 2194 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 }, Status: INVALID_TERM, Last received: 8.735, Next index: 736, Last known committed idx: 734, Time since last communication: 0.000s I0312 20:10:51.992636 2184 raft_consensus.cc:436] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0312 20:10:51.992799 2184 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1: Snoozing failure detection for 0.095s (starting election) I0312 20:10:51.992955 2184 raft_consensus.cc:458] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "012786a8f1b64995847cf7dacbc029a4" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 43059 } } peers { permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 } } peers { permanent_uuid: "9da22e5afdce42078fe8f94ad166ac3c" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 39175 } } I0312 20:10:51.993700 2184 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 10 pre-election: Requesting pre-vote from peer 012786a8f1b64995847cf7dacbc029a4 I0312 20:10:51.993752 1990 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 9 FOLLOWER]: Rejecting Update request from peer 012786a8f1b64995847cf7dacbc029a4 for earlier term 8. Current term is 9. Ops: [] I0312 20:10:51.994009 2184 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 10 pre-election: Requesting pre-vote from peer 9da22e5afdce42078fe8f94ad166ac3c I0312 20:10:51.994659 1865 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" candidate_term: 10 candidate_status { last_received { term: 8 index: 735 } } ignore_live_leader: false dest_uuid: "012786a8f1b64995847cf7dacbc029a4" is_pre_election: true I0312 20:10:51.995013 1865 raft_consensus.cc:2176] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 8 LEADER]: Leader pre-election vote request: Denying vote to candidate e3aa58d3045c4a05b94b94eb8bf58ae1 for term 10 because replica is either leader or believes a valid leader to be alive. I0312 20:10:51.994921 1738 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" candidate_term: 10 candidate_status { last_received { term: 8 index: 735 } } ignore_live_leader: false dest_uuid: "9da22e5afdce42078fe8f94ad166ac3c" is_pre_election: true I0312 20:10:51.995280 1738 raft_consensus.cc:2176] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e3aa58d3045c4a05b94b94eb8bf58ae1 for term 10 because replica is either leader or believes a valid leader to be alive. I0312 20:10:51.995409 2187 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 }, Status: INVALID_TERM, Last received: 8.735, Next index: 736, Last known committed idx: 734, Time since last communication: 0.000s I0312 20:10:51.995635 1926 leader_election.cc:377] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 10 pre-election: Vote denied by peer 012786a8f1b64995847cf7dacbc029a4. Message: Invalid argument: T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 8 LEADER]: Leader pre-election vote request: Denying vote to candidate e3aa58d3045c4a05b94b94eb8bf58ae1 for term 10 because replica is either leader or believes a valid leader to be alive. I0312 20:10:51.995970 1922 leader_election.cc:377] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 10 pre-election: Vote denied by peer 9da22e5afdce42078fe8f94ad166ac3c. Message: Invalid argument: T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e3aa58d3045c4a05b94b94eb8bf58ae1 for term 10 because replica is either leader or believes a valid leader to be alive. I0312 20:10:51.996129 1922 leader_election.cc:258] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. I0312 20:10:51.996408 2184 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1: Snoozing failure detection for 0.089s (election complete) I0312 20:10:51.996552 2184 raft_consensus.cc:2455] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority I0312 20:10:51.998034 1990 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 9 FOLLOWER]: Rejecting Update request from peer 012786a8f1b64995847cf7dacbc029a4 for earlier term 8. Current term is 9. Ops: [] I0312 20:10:51.998795 2187 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 }, Status: INVALID_TERM, Last received: 8.735, Next index: 736, Last known committed idx: 734, Time since last communication: 0.000s I0312 20:10:52.001467 1990 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 9 FOLLOWER]: Rejecting Update request from peer 012786a8f1b64995847cf7dacbc029a4 for earlier term 8. Current term is 9. Ops: [] I0312 20:10:52.002290 2187 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 }, Status: INVALID_TERM, Last received: 8.735, Next index: 736, Last known committed idx: 734, Time since last communication: 0.000s I0312 20:10:52.005815 1990 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 9 FOLLOWER]: Rejecting Update request from peer 012786a8f1b64995847cf7dacbc029a4 for earlier term 8. Current term is 9. Ops: [] I0312 20:10:52.006639 2194 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 }, Status: INVALID_TERM, Last received: 8.735, Next index: 736, Last known committed idx: 734, Time since last communication: 0.000s I0312 20:10:52.008996 1990 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 9 FOLLOWER]: Rejecting Update request from peer 012786a8f1b64995847cf7dacbc029a4 for earlier term 8. Current term is 9. Ops: [] I0312 20:10:52.009698 2194 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 }, Status: INVALID_TERM, Last received: 8.735, Next index: 736, Last known committed idx: 734, Time since last communication: 0.000s I0312 20:10:52.014577 1990 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 9 FOLLOWER]: Rejecting Update request from peer 012786a8f1b64995847cf7dacbc029a4 for earlier term 8. Current term is 9. Ops: [] W0312 20:10:52.014850 2191 raft_consensus.cc:747] Couldn't advance consensus term.: Illegal state: Can't advance term to: 7 current term: 8 is higher. I0312 20:10:52.015280 2194 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 }, Status: INVALID_TERM, Last received: 8.735, Next index: 736, Last known committed idx: 734, Time since last communication: 0.000s I0312 20:10:52.017992 1990 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 9 FOLLOWER]: Rejecting Update request from peer 012786a8f1b64995847cf7dacbc029a4 for earlier term 8. Current term is 9. Ops: [] I0312 20:10:52.019114 2187 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 }, Status: INVALID_TERM, Last received: 8.735, Next index: 736, Last known committed idx: 734, Time since last communication: 0.000s I0312 20:10:52.021265 1990 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 9 FOLLOWER]: Rejecting Update request from peer 012786a8f1b64995847cf7dacbc029a4 for earlier term 8. Current term is 9. Ops: [] I0312 20:10:52.022017 2187 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 }, Status: INVALID_TERM, Last received: 8.735, Next index: 736, Last known committed idx: 734, Time since last communication: 0.000s I0312 20:10:52.025657 1990 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 9 FOLLOWER]: Rejecting Update request from peer 012786a8f1b64995847cf7dacbc029a4 for earlier term 8. Current term is 9. Ops: [] I0312 20:10:52.026510 2194 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 }, Status: INVALID_TERM, Last received: 8.735, Next index: 736, Last known committed idx: 734, Time since last communication: 0.000s I0312 20:10:52.029055 1990 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 9 FOLLOWER]: Rejecting Update request from peer 012786a8f1b64995847cf7dacbc029a4 for earlier term 8. Current term is 9. Ops: [] I0312 20:10:52.029811 2194 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 }, Status: INVALID_TERM, Last received: 8.735, Next index: 736, Last known committed idx: 734, Time since last communication: 0.000s I0312 20:10:52.033386 1990 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 9 FOLLOWER]: Rejecting Update request from peer 012786a8f1b64995847cf7dacbc029a4 for earlier term 8. Current term is 9. Ops: [] I0312 20:10:52.034260 2187 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 }, Status: INVALID_TERM, Last received: 8.735, Next index: 736, Last known committed idx: 734, Time since last communication: 0.000s I0312 20:10:52.039072 1990 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 9 FOLLOWER]: Rejecting Update request from peer 012786a8f1b64995847cf7dacbc029a4 for earlier term 8. Current term is 9. Ops: [] I0312 20:10:52.039814 2194 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 }, Status: INVALID_TERM, Last received: 8.735, Next index: 736, Last known committed idx: 734, Time since last communication: 0.000s I0312 20:10:52.043421 1990 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 9 FOLLOWER]: Rejecting Update request from peer 012786a8f1b64995847cf7dacbc029a4 for earlier term 8. Current term is 9. Ops: [] I0312 20:10:52.044260 2187 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 }, Status: INVALID_TERM, Last received: 8.735, Next index: 736, Last known committed idx: 734, Time since last communication: 0.000s I0312 20:10:52.048000 1990 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 9 FOLLOWER]: Rejecting Update request from peer 012786a8f1b64995847cf7dacbc029a4 for earlier term 8. Current term is 9. Ops: [] I0312 20:10:52.048776 2187 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 }, Status: INVALID_TERM, Last received: 8.735, Next index: 736, Last known committed idx: 734, Time since last communication: 0.000s W0312 20:10:52.049170 2191 raft_consensus.cc:747] Couldn't advance consensus term.: Illegal state: Can't advance term to: 7 current term: 8 is higher. I0312 20:10:52.052384 1990 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 9 FOLLOWER]: Rejecting Update request from peer 012786a8f1b64995847cf7dacbc029a4 for earlier term 8. Current term is 9. Ops: [] I0312 20:10:52.053192 2194 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 }, Status: INVALID_TERM, Last received: 8.735, Next index: 736, Last known committed idx: 734, Time since last communication: 0.000s I0312 20:10:52.056891 1990 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 9 FOLLOWER]: Rejecting Update request from peer 012786a8f1b64995847cf7dacbc029a4 for earlier term 8. Current term is 9. Ops: [] I0312 20:10:52.057744 2194 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 }, Status: INVALID_TERM, Last received: 8.735, Next index: 736, Last known committed idx: 734, Time since last communication: 0.000s I0312 20:10:52.061403 1990 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 9 FOLLOWER]: Rejecting Update request from peer 012786a8f1b64995847cf7dacbc029a4 for earlier term 8. Current term is 9. Ops: [] I0312 20:10:52.062094 2194 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 }, Status: INVALID_TERM, Last received: 8.735, Next index: 736, Last known committed idx: 734, Time since last communication: 0.000s I0312 20:10:52.066870 1990 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 9 FOLLOWER]: Rejecting Update request from peer 012786a8f1b64995847cf7dacbc029a4 for earlier term 8. Current term is 9. Ops: [] I0312 20:10:52.067554 2194 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 }, Status: INVALID_TERM, Last received: 8.735, Next index: 736, Last known committed idx: 734, Time since last communication: 0.000s I0312 20:10:52.071260 1990 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 9 FOLLOWER]: Rejecting Update request from peer 012786a8f1b64995847cf7dacbc029a4 for earlier term 8. Current term is 9. Ops: [] I0312 20:10:52.072007 2194 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 }, Status: INVALID_TERM, Last received: 8.735, Next index: 736, Last known committed idx: 734, Time since last communication: 0.000s W0312 20:10:52.072465 2191 raft_consensus.cc:747] Couldn't advance consensus term.: Illegal state: Can't advance term to: 7 current term: 8 is higher. I0312 20:10:52.075754 1990 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 9 FOLLOWER]: Rejecting Update request from peer 012786a8f1b64995847cf7dacbc029a4 for earlier term 8. Current term is 9. Ops: [] I0312 20:10:52.076485 2194 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 }, Status: INVALID_TERM, Last received: 8.735, Next index: 736, Last known committed idx: 734, Time since last communication: 0.000s I0312 20:10:52.079066 1990 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 9 FOLLOWER]: Rejecting Update request from peer 012786a8f1b64995847cf7dacbc029a4 for earlier term 8. Current term is 9. Ops: [] I0312 20:10:52.079836 2194 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 }, Status: INVALID_TERM, Last received: 8.735, Next index: 736, Last known committed idx: 734, Time since last communication: 0.000s I0312 20:10:52.083672 1990 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 9 FOLLOWER]: Rejecting Update request from peer 012786a8f1b64995847cf7dacbc029a4 for earlier term 8. Current term is 9. Ops: [] I0312 20:10:52.084589 2194 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 }, Status: INVALID_TERM, Last received: 8.735, Next index: 736, Last known committed idx: 734, Time since last communication: 0.000s I0312 20:10:52.085925 2184 raft_consensus.cc:436] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0312 20:10:52.086088 2184 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1: Snoozing failure detection for 0.144s (starting election) I0312 20:10:52.086246 2184 raft_consensus.cc:458] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "012786a8f1b64995847cf7dacbc029a4" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 43059 } } peers { permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 } } peers { permanent_uuid: "9da22e5afdce42078fe8f94ad166ac3c" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 39175 } } I0312 20:10:52.086947 2184 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 10 pre-election: Requesting pre-vote from peer 012786a8f1b64995847cf7dacbc029a4 I0312 20:10:52.087214 2184 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 10 pre-election: Requesting pre-vote from peer 9da22e5afdce42078fe8f94ad166ac3c I0312 20:10:52.087877 1865 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" candidate_term: 10 candidate_status { last_received { term: 8 index: 735 } } ignore_live_leader: false dest_uuid: "012786a8f1b64995847cf7dacbc029a4" is_pre_election: true I0312 20:10:52.088059 1738 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" candidate_term: 10 candidate_status { last_received { term: 8 index: 735 } } ignore_live_leader: false dest_uuid: "9da22e5afdce42078fe8f94ad166ac3c" is_pre_election: true I0312 20:10:52.088220 1865 raft_consensus.cc:2176] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 8 LEADER]: Leader pre-election vote request: Denying vote to candidate e3aa58d3045c4a05b94b94eb8bf58ae1 for term 10 because replica is either leader or believes a valid leader to be alive. I0312 20:10:52.088412 1738 raft_consensus.cc:2176] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e3aa58d3045c4a05b94b94eb8bf58ae1 for term 10 because replica is either leader or believes a valid leader to be alive. I0312 20:10:52.088906 1926 leader_election.cc:377] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 10 pre-election: Vote denied by peer 012786a8f1b64995847cf7dacbc029a4. Message: Invalid argument: T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 8 LEADER]: Leader pre-election vote request: Denying vote to candidate e3aa58d3045c4a05b94b94eb8bf58ae1 for term 10 because replica is either leader or believes a valid leader to be alive. I0312 20:10:52.088958 1990 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 9 FOLLOWER]: Rejecting Update request from peer 012786a8f1b64995847cf7dacbc029a4 for earlier term 8. Current term is 9. Ops: [] I0312 20:10:52.089171 1922 leader_election.cc:377] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 10 pre-election: Vote denied by peer 9da22e5afdce42078fe8f94ad166ac3c. Message: Invalid argument: T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e3aa58d3045c4a05b94b94eb8bf58ae1 for term 10 because replica is either leader or believes a valid leader to be alive. I0312 20:10:52.089335 1922 leader_election.cc:258] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. I0312 20:10:52.089668 2184 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1: Snoozing failure detection for 0.144s (election complete) I0312 20:10:52.089840 2184 raft_consensus.cc:2455] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority I0312 20:10:52.089746 2194 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 }, Status: INVALID_TERM, Last received: 8.735, Next index: 736, Last known committed idx: 734, Time since last communication: 0.000s I0312 20:10:52.092450 1990 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 9 FOLLOWER]: Rejecting Update request from peer 012786a8f1b64995847cf7dacbc029a4 for earlier term 8. Current term is 9. Ops: [] I0312 20:10:52.093184 2194 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 }, Status: INVALID_TERM, Last received: 8.735, Next index: 736, Last known committed idx: 734, Time since last communication: 0.000s I0312 20:10:52.095707 1990 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 9 FOLLOWER]: Rejecting Update request from peer 012786a8f1b64995847cf7dacbc029a4 for earlier term 8. Current term is 9. Ops: [] I0312 20:10:52.096501 2194 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 }, Status: INVALID_TERM, Last received: 8.735, Next index: 736, Last known committed idx: 734, Time since last communication: 0.000s I0312 20:10:52.100235 1990 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 9 FOLLOWER]: Rejecting Update request from peer 012786a8f1b64995847cf7dacbc029a4 for earlier term 8. Current term is 9. Ops: [] I0312 20:10:52.101083 2187 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 }, Status: INVALID_TERM, Last received: 8.735, Next index: 736, Last known committed idx: 734, Time since last communication: 0.000s I0312 20:10:52.103536 1990 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 9 FOLLOWER]: Rejecting Update request from peer 012786a8f1b64995847cf7dacbc029a4 for earlier term 8. Current term is 9. Ops: [] I0312 20:10:52.104323 2187 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 }, Status: INVALID_TERM, Last received: 8.735, Next index: 736, Last known committed idx: 734, Time since last communication: 0.000s I0312 20:10:52.108018 1990 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 9 FOLLOWER]: Rejecting Update request from peer 012786a8f1b64995847cf7dacbc029a4 for earlier term 8. Current term is 9. Ops: [] I0312 20:10:52.108703 2187 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 }, Status: INVALID_TERM, Last received: 8.735, Next index: 736, Last known committed idx: 734, Time since last communication: 0.000s I0312 20:10:52.111281 1990 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 9 FOLLOWER]: Rejecting Update request from peer 012786a8f1b64995847cf7dacbc029a4 for earlier term 8. Current term is 9. Ops: [] I0312 20:10:52.111966 2187 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 }, Status: INVALID_TERM, Last received: 8.735, Next index: 736, Last known committed idx: 734, Time since last communication: 0.000s I0312 20:10:52.115783 1990 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 9 FOLLOWER]: Rejecting Update request from peer 012786a8f1b64995847cf7dacbc029a4 for earlier term 8. Current term is 9. Ops: [] I0312 20:10:52.116560 2187 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 }, Status: INVALID_TERM, Last received: 8.735, Next index: 736, Last known committed idx: 734, Time since last communication: 0.000s W0312 20:10:52.116936 2191 raft_consensus.cc:747] Couldn't advance consensus term.: Illegal state: Can't advance term to: 7 current term: 8 is higher. I0312 20:10:52.119057 1990 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 9 FOLLOWER]: Rejecting Update request from peer 012786a8f1b64995847cf7dacbc029a4 for earlier term 8. Current term is 9. Ops: [] I0312 20:10:52.119772 2187 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 }, Status: INVALID_TERM, Last received: 8.735, Next index: 736, Last known committed idx: 734, Time since last communication: 0.000s I0312 20:10:52.123212 1990 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 9 FOLLOWER]: Rejecting Update request from peer 012786a8f1b64995847cf7dacbc029a4 for earlier term 8. Current term is 9. Ops: [] I0312 20:10:52.123953 2187 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 }, Status: INVALID_TERM, Last received: 8.735, Next index: 736, Last known committed idx: 734, Time since last communication: 0.000s I0312 20:10:52.125888 1990 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 9 FOLLOWER]: Rejecting Update request from peer 012786a8f1b64995847cf7dacbc029a4 for earlier term 8. Current term is 9. Ops: [] I0312 20:10:52.126770 2187 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 }, Status: INVALID_TERM, Last received: 8.735, Next index: 736, Last known committed idx: 734, Time since last communication: 0.000s I0312 20:10:52.129235 1990 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 9 FOLLOWER]: Rejecting Update request from peer 012786a8f1b64995847cf7dacbc029a4 for earlier term 8. Current term is 9. Ops: [] I0312 20:10:52.130038 2187 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 }, Status: INVALID_TERM, Last received: 8.735, Next index: 736, Last known committed idx: 734, Time since last communication: 0.000s I0312 20:10:52.132660 1990 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 9 FOLLOWER]: Rejecting Update request from peer 012786a8f1b64995847cf7dacbc029a4 for earlier term 8. Current term is 9. Ops: [] I0312 20:10:52.133438 2194 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 }, Status: INVALID_TERM, Last received: 8.735, Next index: 736, Last known committed idx: 734, Time since last communication: 0.000s I0312 20:10:52.135797 1990 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 9 FOLLOWER]: Rejecting Update request from peer 012786a8f1b64995847cf7dacbc029a4 for earlier term 8. Current term is 9. Ops: [] I0312 20:10:52.136584 2194 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 }, Status: INVALID_TERM, Last received: 8.735, Next index: 736, Last known committed idx: 734, Time since last communication: 0.000s I0312 20:10:52.139966 1990 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 9 FOLLOWER]: Rejecting Update request from peer 012786a8f1b64995847cf7dacbc029a4 for earlier term 8. Current term is 9. Ops: [] I0312 20:10:52.140728 2194 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 }, Status: INVALID_TERM, Last received: 8.735, Next index: 736, Last known committed idx: 734, Time since last communication: 0.000s I0312 20:10:52.144734 1990 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 9 FOLLOWER]: Rejecting Update request from peer 012786a8f1b64995847cf7dacbc029a4 for earlier term 8. Current term is 9. Ops: [] I0312 20:10:52.145606 2191 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 }, Status: INVALID_TERM, Last received: 8.735, Next index: 736, Last known committed idx: 734, Time since last communication: 0.000s I0312 20:10:52.147966 1990 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 9 FOLLOWER]: Rejecting Update request from peer 012786a8f1b64995847cf7dacbc029a4 for earlier term 8. Current term is 9. Ops: [] I0312 20:10:52.148795 2194 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 }, Status: INVALID_TERM, Last received: 8.735, Next index: 736, Last known committed idx: 734, Time since last communication: 0.000s W0312 20:10:52.150806 2187 raft_consensus.cc:747] Couldn't advance consensus term.: Illegal state: Can't advance term to: 7 current term: 8 is higher. I0312 20:10:52.151259 1990 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 9 FOLLOWER]: Rejecting Update request from peer 012786a8f1b64995847cf7dacbc029a4 for earlier term 8. Current term is 9. Ops: [] I0312 20:10:52.152053 2194 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 }, Status: INVALID_TERM, Last received: 8.735, Next index: 736, Last known committed idx: 734, Time since last communication: 0.000s I0312 20:10:52.156879 1990 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 9 FOLLOWER]: Rejecting Update request from peer 012786a8f1b64995847cf7dacbc029a4 for earlier term 8. Current term is 9. Ops: [] I0312 20:10:52.157625 2191 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 }, Status: INVALID_TERM, Last received: 8.735, Next index: 736, Last known committed idx: 734, Time since last communication: 0.000s I0312 20:10:52.162412 1990 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 9 FOLLOWER]: Rejecting Update request from peer 012786a8f1b64995847cf7dacbc029a4 for earlier term 8. Current term is 9. Ops: [] I0312 20:10:52.163244 2191 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 }, Status: INVALID_TERM, Last received: 8.735, Next index: 736, Last known committed idx: 734, Time since last communication: 0.000s I0312 20:10:52.166949 1990 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 9 FOLLOWER]: Rejecting Update request from peer 012786a8f1b64995847cf7dacbc029a4 for earlier term 8. Current term is 9. Ops: [] I0312 20:10:52.167798 2194 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 }, Status: INVALID_TERM, Last received: 8.735, Next index: 736, Last known committed idx: 734, Time since last communication: 0.000s I0312 20:10:52.171262 1990 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 9 FOLLOWER]: Rejecting Update request from peer 012786a8f1b64995847cf7dacbc029a4 for earlier term 8. Current term is 9. Ops: [] I0312 20:10:52.171980 2191 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 }, Status: INVALID_TERM, Last received: 8.735, Next index: 736, Last known committed idx: 734, Time since last communication: 0.000s I0312 20:10:52.176856 1990 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 9 FOLLOWER]: Rejecting Update request from peer 012786a8f1b64995847cf7dacbc029a4 for earlier term 8. Current term is 9. Ops: [] I0312 20:10:52.177611 2191 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 }, Status: INVALID_TERM, Last received: 8.735, Next index: 736, Last known committed idx: 734, Time since last communication: 0.000s I0312 20:10:52.180111 1990 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 9 FOLLOWER]: Rejecting Update request from peer 012786a8f1b64995847cf7dacbc029a4 for earlier term 8. Current term is 9. Ops: [] I0312 20:10:52.180836 2191 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 }, Status: INVALID_TERM, Last received: 8.735, Next index: 736, Last known committed idx: 734, Time since last communication: 0.000s I0312 20:10:52.185945 1990 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 9 FOLLOWER]: Rejecting Update request from peer 012786a8f1b64995847cf7dacbc029a4 for earlier term 8. Current term is 9. Ops: [] I0312 20:10:52.186846 2191 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 }, Status: INVALID_TERM, Last received: 8.735, Next index: 736, Last known committed idx: 734, Time since last communication: 0.000s I0312 20:10:52.189900 1990 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 9 FOLLOWER]: Rejecting Update request from peer 012786a8f1b64995847cf7dacbc029a4 for earlier term 8. Current term is 9. Ops: [] I0312 20:10:52.190840 2194 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 }, Status: INVALID_TERM, Last received: 8.735, Next index: 736, Last known committed idx: 734, Time since last communication: 0.000s I0312 20:10:52.193155 1990 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 9 FOLLOWER]: Rejecting Update request from peer 012786a8f1b64995847cf7dacbc029a4 for earlier term 8. Current term is 9. Ops: [] I0312 20:10:52.194032 2194 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 }, Status: INVALID_TERM, Last received: 8.735, Next index: 736, Last known committed idx: 734, Time since last communication: 0.000s I0312 20:10:52.196455 1990 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 9 FOLLOWER]: Rejecting Update request from peer 012786a8f1b64995847cf7dacbc029a4 for earlier term 8. Current term is 9. Ops: [] W0312 20:10:52.197124 2187 raft_consensus.cc:747] Couldn't advance consensus term.: Illegal state: Can't advance term to: 7 current term: 8 is higher. I0312 20:10:52.197438 2183 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 }, Status: INVALID_TERM, Last received: 8.735, Next index: 736, Last known committed idx: 734, Time since last communication: 0.000s I0312 20:10:52.201262 1990 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 9 FOLLOWER]: Rejecting Update request from peer 012786a8f1b64995847cf7dacbc029a4 for earlier term 8. Current term is 9. Ops: [] I0312 20:10:52.202020 2183 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 }, Status: INVALID_TERM, Last received: 8.735, Next index: 736, Last known committed idx: 734, Time since last communication: 0.000s W0312 20:10:52.205454 2187 raft_consensus.cc:747] Couldn't advance consensus term.: Illegal state: Can't advance term to: 7 current term: 8 is higher. I0312 20:10:52.206627 1990 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 9 FOLLOWER]: Rejecting Update request from peer 012786a8f1b64995847cf7dacbc029a4 for earlier term 8. Current term is 9. Ops: [] I0312 20:10:52.207499 2187 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 }, Status: INVALID_TERM, Last received: 8.735, Next index: 736, Last known committed idx: 734, Time since last communication: 0.000s I0312 20:10:52.211231 1990 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 9 FOLLOWER]: Rejecting Update request from peer 012786a8f1b64995847cf7dacbc029a4 for earlier term 8. Current term is 9. Ops: [] I0312 20:10:52.212021 2194 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 }, Status: INVALID_TERM, Last received: 8.735, Next index: 736, Last known committed idx: 734, Time since last communication: 0.000s I0312 20:10:52.214412 1990 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 9 FOLLOWER]: Rejecting Update request from peer 012786a8f1b64995847cf7dacbc029a4 for earlier term 8. Current term is 9. Ops: [] I0312 20:10:52.215184 2187 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 }, Status: INVALID_TERM, Last received: 8.735, Next index: 736, Last known committed idx: 734, Time since last communication: 0.000s I0312 20:10:52.217655 1990 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 9 FOLLOWER]: Rejecting Update request from peer 012786a8f1b64995847cf7dacbc029a4 for earlier term 8. Current term is 9. Ops: [] I0312 20:10:52.218365 2194 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 }, Status: INVALID_TERM, Last received: 8.735, Next index: 736, Last known committed idx: 734, Time since last communication: 0.000s I0312 20:10:52.222051 1990 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 9 FOLLOWER]: Rejecting Update request from peer 012786a8f1b64995847cf7dacbc029a4 for earlier term 8. Current term is 9. Ops: [] I0312 20:10:52.222864 2194 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 }, Status: INVALID_TERM, Last received: 8.735, Next index: 736, Last known committed idx: 734, Time since last communication: 0.000s I0312 20:10:52.226501 1990 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 9 FOLLOWER]: Rejecting Update request from peer 012786a8f1b64995847cf7dacbc029a4 for earlier term 8. Current term is 9. Ops: [] I0312 20:10:52.227340 2194 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 }, Status: INVALID_TERM, Last received: 8.735, Next index: 736, Last known committed idx: 734, Time since last communication: 0.000s I0312 20:10:52.230013 1990 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 9 FOLLOWER]: Rejecting Update request from peer 012786a8f1b64995847cf7dacbc029a4 for earlier term 8. Current term is 9. Ops: [] I0312 20:10:52.230733 2194 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 }, Status: INVALID_TERM, Last received: 8.735, Next index: 736, Last known committed idx: 734, Time since last communication: 0.000s I0312 20:10:52.234314 1990 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 9 FOLLOWER]: Rejecting Update request from peer 012786a8f1b64995847cf7dacbc029a4 for earlier term 8. Current term is 9. Ops: [] I0312 20:10:52.234628 2184 raft_consensus.cc:436] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0312 20:10:52.234810 2184 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1: Snoozing failure detection for 0.156s (starting election) I0312 20:10:52.235091 2194 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 }, Status: INVALID_TERM, Last received: 8.735, Next index: 736, Last known committed idx: 734, Time since last communication: 0.000s I0312 20:10:52.234982 2184 raft_consensus.cc:458] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "012786a8f1b64995847cf7dacbc029a4" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 43059 } } peers { permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 } } peers { permanent_uuid: "9da22e5afdce42078fe8f94ad166ac3c" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 39175 } } I0312 20:10:52.235775 2184 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 10 pre-election: Requesting pre-vote from peer 012786a8f1b64995847cf7dacbc029a4 I0312 20:10:52.236073 2184 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 10 pre-election: Requesting pre-vote from peer 9da22e5afdce42078fe8f94ad166ac3c I0312 20:10:52.236742 1865 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" candidate_term: 10 candidate_status { last_received { term: 8 index: 735 } } ignore_live_leader: false dest_uuid: "012786a8f1b64995847cf7dacbc029a4" is_pre_election: true I0312 20:10:52.237140 1865 raft_consensus.cc:2176] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 8 LEADER]: Leader pre-election vote request: Denying vote to candidate e3aa58d3045c4a05b94b94eb8bf58ae1 for term 10 because replica is either leader or believes a valid leader to be alive. I0312 20:10:52.237053 1738 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" candidate_term: 10 candidate_status { last_received { term: 8 index: 735 } } ignore_live_leader: false dest_uuid: "9da22e5afdce42078fe8f94ad166ac3c" is_pre_election: true I0312 20:10:52.237426 1738 raft_consensus.cc:2176] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e3aa58d3045c4a05b94b94eb8bf58ae1 for term 10 because replica is either leader or believes a valid leader to be alive. I0312 20:10:52.237753 1926 leader_election.cc:377] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 10 pre-election: Vote denied by peer 012786a8f1b64995847cf7dacbc029a4. Message: Invalid argument: T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 8 LEADER]: Leader pre-election vote request: Denying vote to candidate e3aa58d3045c4a05b94b94eb8bf58ae1 for term 10 because replica is either leader or believes a valid leader to be alive. I0312 20:10:52.238162 1922 leader_election.cc:377] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 10 pre-election: Vote denied by peer 9da22e5afdce42078fe8f94ad166ac3c. Message: Invalid argument: T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e3aa58d3045c4a05b94b94eb8bf58ae1 for term 10 because replica is either leader or believes a valid leader to be alive. I0312 20:10:52.238349 1922 leader_election.cc:258] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. I0312 20:10:52.238739 2184 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1: Snoozing failure detection for 0.195s (election complete) I0312 20:10:52.238906 2184 raft_consensus.cc:2455] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority I0312 20:10:52.239116 1990 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 9 FOLLOWER]: Rejecting Update request from peer 012786a8f1b64995847cf7dacbc029a4 for earlier term 8. Current term is 9. Ops: [] I0312 20:10:52.239943 2194 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 }, Status: INVALID_TERM, Last received: 8.735, Next index: 736, Last known committed idx: 734, Time since last communication: 0.000s W0312 20:10:52.240842 2183 raft_consensus.cc:747] Couldn't advance consensus term.: Illegal state: Can't advance term to: 7 current term: 8 is higher. I0312 20:10:52.244211 1990 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 9 FOLLOWER]: Rejecting Update request from peer 012786a8f1b64995847cf7dacbc029a4 for earlier term 8. Current term is 9. Ops: [] I0312 20:10:52.244947 2187 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 }, Status: INVALID_TERM, Last received: 8.735, Next index: 736, Last known committed idx: 734, Time since last communication: 0.000s I0312 20:10:52.248400 1990 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 9 FOLLOWER]: Rejecting Update request from peer 012786a8f1b64995847cf7dacbc029a4 for earlier term 8. Current term is 9. Ops: [] I0312 20:10:52.249112 2187 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 }, Status: INVALID_TERM, Last received: 8.735, Next index: 736, Last known committed idx: 734, Time since last communication: 0.000s I0312 20:10:52.252929 1990 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 9 FOLLOWER]: Rejecting Update request from peer 012786a8f1b64995847cf7dacbc029a4 for earlier term 8. Current term is 9. Ops: [] I0312 20:10:52.253648 2187 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 }, Status: INVALID_TERM, Last received: 8.735, Next index: 736, Last known committed idx: 734, Time since last communication: 0.000s I0312 20:10:52.257354 1990 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 9 FOLLOWER]: Rejecting Update request from peer 012786a8f1b64995847cf7dacbc029a4 for earlier term 8. Current term is 9. Ops: [] I0312 20:10:52.258085 2187 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 }, Status: INVALID_TERM, Last received: 8.735, Next index: 736, Last known committed idx: 734, Time since last communication: 0.000s I0312 20:10:52.260686 1990 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 9 FOLLOWER]: Rejecting Update request from peer 012786a8f1b64995847cf7dacbc029a4 for earlier term 8. Current term is 9. Ops: [] I0312 20:10:52.261420 2187 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 }, Status: INVALID_TERM, Last received: 8.735, Next index: 736, Last known committed idx: 734, Time since last communication: 0.000s I0312 20:10:52.264089 1990 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 9 FOLLOWER]: Rejecting Update request from peer 012786a8f1b64995847cf7dacbc029a4 for earlier term 8. Current term is 9. Ops: [] I0312 20:10:52.264853 2187 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 }, Status: INVALID_TERM, Last received: 8.735, Next index: 736, Last known committed idx: 734, Time since last communication: 0.000s I0312 20:10:52.267470 1990 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 9 FOLLOWER]: Rejecting Update request from peer 012786a8f1b64995847cf7dacbc029a4 for earlier term 8. Current term is 9. Ops: [] I0312 20:10:52.268214 2187 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 }, Status: INVALID_TERM, Last received: 8.735, Next index: 736, Last known committed idx: 734, Time since last communication: 0.000s I0312 20:10:52.271726 1990 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 9 FOLLOWER]: Rejecting Update request from peer 012786a8f1b64995847cf7dacbc029a4 for earlier term 8. Current term is 9. Ops: [] I0312 20:10:52.272591 2187 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 }, Status: INVALID_TERM, Last received: 8.735, Next index: 736, Last known committed idx: 734, Time since last communication: 0.000s W0312 20:10:52.275131 2183 raft_consensus.cc:747] Couldn't advance consensus term.: Illegal state: Can't advance term to: 7 current term: 8 is higher. I0312 20:10:52.276238 1990 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 9 FOLLOWER]: Rejecting Update request from peer 012786a8f1b64995847cf7dacbc029a4 for earlier term 8. Current term is 9. Ops: [] I0312 20:10:52.276993 2187 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 }, Status: INVALID_TERM, Last received: 8.735, Next index: 736, Last known committed idx: 734, Time since last communication: 0.000s I0312 20:10:52.280670 1990 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 9 FOLLOWER]: Rejecting Update request from peer 012786a8f1b64995847cf7dacbc029a4 for earlier term 8. Current term is 9. Ops: [] I0312 20:10:52.281476 2187 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 }, Status: INVALID_TERM, Last received: 8.735, Next index: 736, Last known committed idx: 734, Time since last communication: 0.000s I0312 20:10:52.283988 1990 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 9 FOLLOWER]: Rejecting Update request from peer 012786a8f1b64995847cf7dacbc029a4 for earlier term 8. Current term is 9. Ops: [] I0312 20:10:52.284759 2187 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 }, Status: INVALID_TERM, Last received: 8.735, Next index: 736, Last known committed idx: 734, Time since last communication: 0.000s I0312 20:10:52.288421 1990 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 9 FOLLOWER]: Rejecting Update request from peer 012786a8f1b64995847cf7dacbc029a4 for earlier term 8. Current term is 9. Ops: [] I0312 20:10:52.289189 2187 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 }, Status: INVALID_TERM, Last received: 8.735, Next index: 736, Last known committed idx: 734, Time since last communication: 0.000s I0312 20:10:52.293958 1990 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 9 FOLLOWER]: Rejecting Update request from peer 012786a8f1b64995847cf7dacbc029a4 for earlier term 8. Current term is 9. Ops: [] I0312 20:10:52.294805 2194 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 }, Status: INVALID_TERM, Last received: 8.735, Next index: 736, Last known committed idx: 734, Time since last communication: 0.000s I0312 20:10:52.297193 1990 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 9 FOLLOWER]: Rejecting Update request from peer 012786a8f1b64995847cf7dacbc029a4 for earlier term 8. Current term is 9. Ops: [] I0312 20:10:52.297927 2194 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 }, Status: INVALID_TERM, Last received: 8.735, Next index: 736, Last known committed idx: 734, Time since last communication: 0.000s I0312 20:10:52.300552 1990 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 9 FOLLOWER]: Rejecting Update request from peer 012786a8f1b64995847cf7dacbc029a4 for earlier term 8. Current term is 9. Ops: [] I0312 20:10:52.301259 2194 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 }, Status: INVALID_TERM, Last received: 8.735, Next index: 736, Last known committed idx: 734, Time since last communication: 0.000s I0312 20:10:52.305029 1990 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 9 FOLLOWER]: Rejecting Update request from peer 012786a8f1b64995847cf7dacbc029a4 for earlier term 8. Current term is 9. Ops: [] I0312 20:10:52.305832 2194 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 }, Status: INVALID_TERM, Last received: 8.735, Next index: 736, Last known committed idx: 734, Time since last communication: 0.000s I0312 20:10:52.310614 1990 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 9 FOLLOWER]: Rejecting Update request from peer 012786a8f1b64995847cf7dacbc029a4 for earlier term 8. Current term is 9. Ops: [] I0312 20:10:52.311408 2187 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 }, Status: INVALID_TERM, Last received: 8.735, Next index: 736, Last known committed idx: 734, Time since last communication: 0.000s I0312 20:10:52.315035 1990 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 9 FOLLOWER]: Rejecting Update request from peer 012786a8f1b64995847cf7dacbc029a4 for earlier term 8. Current term is 9. Ops: [] I0312 20:10:52.315800 2194 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 }, Status: INVALID_TERM, Last received: 8.735, Next index: 736, Last known committed idx: 734, Time since last communication: 0.000s I0312 20:10:52.319345 1990 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 9 FOLLOWER]: Rejecting Update request from peer 012786a8f1b64995847cf7dacbc029a4 for earlier term 8. Current term is 9. Ops: [] W0312 20:10:52.319420 2183 raft_consensus.cc:747] Couldn't advance consensus term.: Illegal state: Can't advance term to: 7 current term: 8 is higher. I0312 20:10:52.320107 2194 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 }, Status: INVALID_TERM, Last received: 8.735, Next index: 736, Last known committed idx: 734, Time since last communication: 0.000s I0312 20:10:52.324889 1990 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 9 FOLLOWER]: Rejecting Update request from peer 012786a8f1b64995847cf7dacbc029a4 for earlier term 8. Current term is 9. Ops: [] I0312 20:10:52.325747 2194 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 }, Status: INVALID_TERM, Last received: 8.735, Next index: 736, Last known committed idx: 734, Time since last communication: 0.000s I0312 20:10:52.328359 1990 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 9 FOLLOWER]: Rejecting Update request from peer 012786a8f1b64995847cf7dacbc029a4 for earlier term 8. Current term is 9. Ops: [] I0312 20:10:52.329257 2194 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 }, Status: INVALID_TERM, Last received: 8.735, Next index: 736, Last known committed idx: 734, Time since last communication: 0.000s I0312 20:10:52.332815 1990 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 9 FOLLOWER]: Rejecting Update request from peer 012786a8f1b64995847cf7dacbc029a4 for earlier term 8. Current term is 9. Ops: [] I0312 20:10:52.333678 2194 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 }, Status: INVALID_TERM, Last received: 8.735, Next index: 736, Last known committed idx: 734, Time since last communication: 0.000s I0312 20:10:52.336124 1990 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 9 FOLLOWER]: Rejecting Update request from peer 012786a8f1b64995847cf7dacbc029a4 for earlier term 8. Current term is 9. Ops: [] I0312 20:10:52.336961 2187 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 }, Status: INVALID_TERM, Last received: 8.735, Next index: 736, Last known committed idx: 734, Time since last communication: 0.000s I0312 20:10:52.340709 1990 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 9 FOLLOWER]: Rejecting Update request from peer 012786a8f1b64995847cf7dacbc029a4 for earlier term 8. Current term is 9. Ops: [] I0312 20:10:52.341475 2194 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 }, Status: INVALID_TERM, Last received: 8.735, Next index: 736, Last known committed idx: 734, Time since last communication: 0.000s I0312 20:10:52.345039 1990 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 9 FOLLOWER]: Rejecting Update request from peer 012786a8f1b64995847cf7dacbc029a4 for earlier term 8. Current term is 9. Ops: [] I0312 20:10:52.345793 2194 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 }, Status: INVALID_TERM, Last received: 8.735, Next index: 736, Last known committed idx: 734, Time since last communication: 0.000s I0312 20:10:52.350497 1990 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 9 FOLLOWER]: Rejecting Update request from peer 012786a8f1b64995847cf7dacbc029a4 for earlier term 8. Current term is 9. Ops: [] I0312 20:10:52.351333 2187 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 }, Status: INVALID_TERM, Last received: 8.735, Next index: 736, Last known committed idx: 734, Time since last communication: 0.000s I0312 20:10:52.355074 1990 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 9 FOLLOWER]: Rejecting Update request from peer 012786a8f1b64995847cf7dacbc029a4 for earlier term 8. Current term is 9. Ops: [] I0312 20:10:52.355916 2194 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 }, Status: INVALID_TERM, Last received: 8.735, Next index: 736, Last known committed idx: 734, Time since last communication: 0.000s W0312 20:10:52.358719 2183 raft_consensus.cc:747] Couldn't advance consensus term.: Illegal state: Can't advance term to: 7 current term: 8 is higher. I0312 20:10:52.360654 1990 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 9 FOLLOWER]: Rejecting Update request from peer 012786a8f1b64995847cf7dacbc029a4 for earlier term 8. Current term is 9. Ops: [] I0312 20:10:52.361671 2194 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 }, Status: INVALID_TERM, Last received: 8.735, Next index: 736, Last known committed idx: 734, Time since last communication: 0.000s I0312 20:10:52.366036 1990 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 9 FOLLOWER]: Rejecting Update request from peer 012786a8f1b64995847cf7dacbc029a4 for earlier term 8. Current term is 9. Ops: [] I0312 20:10:52.366752 2194 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 }, Status: INVALID_TERM, Last received: 8.735, Next index: 736, Last known committed idx: 734, Time since last communication: 0.000s W0312 20:10:52.369179 1677 debug-util.cc:396] Leaking SignalData structure 0x7b080002b080 after lost signal to thread 1655 I0312 20:10:52.369429 1990 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 9 FOLLOWER]: Rejecting Update request from peer 012786a8f1b64995847cf7dacbc029a4 for earlier term 8. Current term is 9. Ops: [] W0312 20:10:52.369873 1677 debug-util.cc:396] Leaking SignalData structure 0x7b080009b700 after lost signal to thread 1782 I0312 20:10:52.370287 2194 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 }, Status: INVALID_TERM, Last received: 8.735, Next index: 736, Last known committed idx: 734, Time since last communication: 0.000s I0312 20:10:52.372750 1990 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 9 FOLLOWER]: Rejecting Update request from peer 012786a8f1b64995847cf7dacbc029a4 for earlier term 8. Current term is 9. Ops: [] I0312 20:10:52.373694 2194 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 }, Status: INVALID_TERM, Last received: 8.735, Next index: 736, Last known committed idx: 734, Time since last communication: 0.000s I0312 20:10:52.375942 1990 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 9 FOLLOWER]: Rejecting Update request from peer 012786a8f1b64995847cf7dacbc029a4 for earlier term 8. Current term is 9. Ops: [] I0312 20:10:52.376802 2191 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 }, Status: INVALID_TERM, Last received: 8.735, Next index: 736, Last known committed idx: 734, Time since last communication: 0.000s I0312 20:10:52.380475 1990 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 9 FOLLOWER]: Rejecting Update request from peer 012786a8f1b64995847cf7dacbc029a4 for earlier term 8. Current term is 9. Ops: [] I0312 20:10:52.381270 2191 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 }, Status: INVALID_TERM, Last received: 8.735, Next index: 736, Last known committed idx: 734, Time since last communication: 0.000s W0312 20:10:52.385006 2183 raft_consensus.cc:747] Couldn't advance consensus term.: Illegal state: Can't advance term to: 7 current term: 8 is higher. I0312 20:10:52.385169 1990 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 9 FOLLOWER]: Rejecting Update request from peer 012786a8f1b64995847cf7dacbc029a4 for earlier term 8. Current term is 9. Ops: [] I0312 20:10:52.386092 2191 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 }, Status: INVALID_TERM, Last received: 8.735, Next index: 736, Last known committed idx: 734, Time since last communication: 0.000s I0312 20:10:52.390810 1990 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 9 FOLLOWER]: Rejecting Update request from peer 012786a8f1b64995847cf7dacbc029a4 for earlier term 8. Current term is 9. Ops: [] I0312 20:10:52.391635 2187 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 }, Status: INVALID_TERM, Last received: 8.735, Next index: 736, Last known committed idx: 734, Time since last communication: 0.000s I0312 20:10:52.395777 1990 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 9 FOLLOWER]: Rejecting Update request from peer 012786a8f1b64995847cf7dacbc029a4 for earlier term 8. Current term is 9. Ops: [] I0312 20:10:52.397908 2187 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 }, Status: INVALID_TERM, Last received: 8.735, Next index: 736, Last known committed idx: 734, Time since last communication: 0.000s I0312 20:10:52.401876 1990 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 9 FOLLOWER]: Rejecting Update request from peer 012786a8f1b64995847cf7dacbc029a4 for earlier term 8. Current term is 9. Ops: [] I0312 20:10:52.402707 2194 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 }, Status: INVALID_TERM, Last received: 8.735, Next index: 736, Last known committed idx: 734, Time since last communication: 0.000s I0312 20:10:52.406394 1990 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 9 FOLLOWER]: Rejecting Update request from peer 012786a8f1b64995847cf7dacbc029a4 for earlier term 8. Current term is 9. Ops: [] I0312 20:10:52.407176 2187 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 }, Status: INVALID_TERM, Last received: 8.735, Next index: 736, Last known committed idx: 734, Time since last communication: 0.000s I0312 20:10:52.409574 1990 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 9 FOLLOWER]: Rejecting Update request from peer 012786a8f1b64995847cf7dacbc029a4 for earlier term 8. Current term is 9. Ops: [] I0312 20:10:52.410785 2187 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 }, Status: INVALID_TERM, Last received: 8.735, Next index: 736, Last known committed idx: 734, Time since last communication: 0.000s I0312 20:10:52.413965 1990 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 9 FOLLOWER]: Rejecting Update request from peer 012786a8f1b64995847cf7dacbc029a4 for earlier term 8. Current term is 9. Ops: [] I0312 20:10:52.415166 2187 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 }, Status: INVALID_TERM, Last received: 8.735, Next index: 736, Last known committed idx: 734, Time since last communication: 0.000s I0312 20:10:52.417400 1990 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 9 FOLLOWER]: Rejecting Update request from peer 012786a8f1b64995847cf7dacbc029a4 for earlier term 8. Current term is 9. Ops: [] I0312 20:10:52.418274 2187 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 }, Status: INVALID_TERM, Last received: 8.735, Next index: 736, Last known committed idx: 734, Time since last communication: 0.000s I0312 20:10:52.420114 1990 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 9 FOLLOWER]: Rejecting Update request from peer 012786a8f1b64995847cf7dacbc029a4 for earlier term 8. Current term is 9. Ops: [] I0312 20:10:52.421138 2194 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 }, Status: INVALID_TERM, Last received: 8.735, Next index: 736, Last known committed idx: 734, Time since last communication: 0.000s I0312 20:10:52.423954 1990 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 9 FOLLOWER]: Rejecting Update request from peer 012786a8f1b64995847cf7dacbc029a4 for earlier term 8. Current term is 9. Ops: [] I0312 20:10:52.424919 2187 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 }, Status: INVALID_TERM, Last received: 8.735, Next index: 736, Last known committed idx: 734, Time since last communication: 0.000s I0312 20:10:52.429585 1990 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 9 FOLLOWER]: Rejecting Update request from peer 012786a8f1b64995847cf7dacbc029a4 for earlier term 8. Current term is 9. Ops: [] I0312 20:10:52.430560 2194 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 }, Status: INVALID_TERM, Last received: 8.735, Next index: 736, Last known committed idx: 734, Time since last communication: 0.000s I0312 20:10:52.432832 1990 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 9 FOLLOWER]: Rejecting Update request from peer 012786a8f1b64995847cf7dacbc029a4 for earlier term 8. Current term is 9. Ops: [] I0312 20:10:52.433697 2194 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 }, Status: INVALID_TERM, Last received: 8.735, Next index: 736, Last known committed idx: 734, Time since last communication: 0.000s I0312 20:10:52.434538 2184 raft_consensus.cc:436] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0312 20:10:52.434720 2184 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1: Snoozing failure detection for 0.223s (starting election) I0312 20:10:52.434890 2184 raft_consensus.cc:458] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "012786a8f1b64995847cf7dacbc029a4" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 43059 } } peers { permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 } } peers { permanent_uuid: "9da22e5afdce42078fe8f94ad166ac3c" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 39175 } } I0312 20:10:52.435602 2184 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 10 pre-election: Requesting pre-vote from peer 012786a8f1b64995847cf7dacbc029a4 I0312 20:10:52.435878 2184 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 10 pre-election: Requesting pre-vote from peer 9da22e5afdce42078fe8f94ad166ac3c I0312 20:10:52.436784 1738 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" candidate_term: 10 candidate_status { last_received { term: 8 index: 735 } } ignore_live_leader: false dest_uuid: "9da22e5afdce42078fe8f94ad166ac3c" is_pre_election: true I0312 20:10:52.436801 1865 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" candidate_term: 10 candidate_status { last_received { term: 8 index: 735 } } ignore_live_leader: false dest_uuid: "012786a8f1b64995847cf7dacbc029a4" is_pre_election: true I0312 20:10:52.437178 1738 raft_consensus.cc:2176] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e3aa58d3045c4a05b94b94eb8bf58ae1 for term 10 because replica is either leader or believes a valid leader to be alive. I0312 20:10:52.437299 1865 raft_consensus.cc:2176] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 8 LEADER]: Leader pre-election vote request: Denying vote to candidate e3aa58d3045c4a05b94b94eb8bf58ae1 for term 10 because replica is either leader or believes a valid leader to be alive. I0312 20:10:52.437814 1922 leader_election.cc:377] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 10 pre-election: Vote denied by peer 9da22e5afdce42078fe8f94ad166ac3c. Message: Invalid argument: T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e3aa58d3045c4a05b94b94eb8bf58ae1 for term 10 because replica is either leader or believes a valid leader to be alive. I0312 20:10:52.438053 1926 leader_election.cc:377] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 10 pre-election: Vote denied by peer 012786a8f1b64995847cf7dacbc029a4. Message: Invalid argument: T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 8 LEADER]: Leader pre-election vote request: Denying vote to candidate e3aa58d3045c4a05b94b94eb8bf58ae1 for term 10 because replica is either leader or believes a valid leader to be alive. I0312 20:10:52.438241 1926 leader_election.cc:258] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. I0312 20:10:52.438542 1990 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 9 FOLLOWER]: Rejecting Update request from peer 012786a8f1b64995847cf7dacbc029a4 for earlier term 8. Current term is 9. Ops: [] I0312 20:10:52.438789 2184 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1: Snoozing failure detection for 0.331s (election complete) I0312 20:10:52.438944 2184 raft_consensus.cc:2455] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority I0312 20:10:52.440448 2194 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 }, Status: INVALID_TERM, Last received: 8.735, Next index: 736, Last known committed idx: 734, Time since last communication: 0.000s I0312 20:10:52.442291 1990 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 9 FOLLOWER]: Rejecting Update request from peer 012786a8f1b64995847cf7dacbc029a4 for earlier term 8. Current term is 9. Ops: [] I0312 20:10:52.443060 2187 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 }, Status: INVALID_TERM, Last received: 8.735, Next index: 736, Last known committed idx: 734, Time since last communication: 0.000s I0312 20:10:52.446162 1990 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 9 FOLLOWER]: Rejecting Update request from peer 012786a8f1b64995847cf7dacbc029a4 for earlier term 8. Current term is 9. Ops: [] I0312 20:10:52.446877 2194 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 }, Status: INVALID_TERM, Last received: 8.735, Next index: 736, Last known committed idx: 734, Time since last communication: 0.000s I0312 20:10:52.449039 1990 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 9 FOLLOWER]: Rejecting Update request from peer 012786a8f1b64995847cf7dacbc029a4 for earlier term 8. Current term is 9. Ops: [] I0312 20:10:52.450233 2194 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 }, Status: INVALID_TERM, Last received: 8.735, Next index: 736, Last known committed idx: 734, Time since last communication: 0.000s I0312 20:10:52.454597 1990 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 9 FOLLOWER]: Rejecting Update request from peer 012786a8f1b64995847cf7dacbc029a4 for earlier term 8. Current term is 9. Ops: [] I0312 20:10:52.455335 2194 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 }, Status: INVALID_TERM, Last received: 8.735, Next index: 736, Last known committed idx: 734, Time since last communication: 0.000s I0312 20:10:52.458748 1990 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 9 FOLLOWER]: Rejecting Update request from peer 012786a8f1b64995847cf7dacbc029a4 for earlier term 8. Current term is 9. Ops: [] I0312 20:10:52.459509 2194 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 }, Status: INVALID_TERM, Last received: 8.735, Next index: 736, Last known committed idx: 734, Time since last communication: 0.000s I0312 20:10:52.463743 1990 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 9 FOLLOWER]: Rejecting Update request from peer 012786a8f1b64995847cf7dacbc029a4 for earlier term 8. Current term is 9. Ops: [] I0312 20:10:52.467548 2187 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 }, Status: INVALID_TERM, Last received: 8.735, Next index: 736, Last known committed idx: 734, Time since last communication: 0.000s I0312 20:10:52.471415 1990 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 9 FOLLOWER]: Rejecting Update request from peer 012786a8f1b64995847cf7dacbc029a4 for earlier term 8. Current term is 9. Ops: [] I0312 20:10:52.472296 2187 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 }, Status: INVALID_TERM, Last received: 8.735, Next index: 736, Last known committed idx: 734, Time since last communication: 0.000s I0312 20:10:52.476634 1990 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 9 FOLLOWER]: Rejecting Update request from peer 012786a8f1b64995847cf7dacbc029a4 for earlier term 8. Current term is 9. Ops: [] I0312 20:10:52.477598 2187 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 }, Status: INVALID_TERM, Last received: 8.735, Next index: 736, Last known committed idx: 734, Time since last communication: 0.000s I0312 20:10:52.481137 1990 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 9 FOLLOWER]: Rejecting Update request from peer 012786a8f1b64995847cf7dacbc029a4 for earlier term 8. Current term is 9. Ops: [] I0312 20:10:52.481909 2187 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 }, Status: INVALID_TERM, Last received: 8.735, Next index: 736, Last known committed idx: 734, Time since last communication: 0.000s I0312 20:10:52.484565 1990 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 9 FOLLOWER]: Rejecting Update request from peer 012786a8f1b64995847cf7dacbc029a4 for earlier term 8. Current term is 9. Ops: [] I0312 20:10:52.485333 2187 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 }, Status: INVALID_TERM, Last received: 8.735, Next index: 736, Last known committed idx: 734, Time since last communication: 0.000s I0312 20:10:52.489943 1990 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 9 FOLLOWER]: Rejecting Update request from peer 012786a8f1b64995847cf7dacbc029a4 for earlier term 8. Current term is 9. Ops: [] I0312 20:10:52.490864 2187 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 }, Status: INVALID_TERM, Last received: 8.735, Next index: 736, Last known committed idx: 734, Time since last communication: 0.000s I0312 20:10:52.491252 2191 raft_consensus.cc:2747] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 8 LEADER]: Stepping down as leader of term 8 I0312 20:10:52.491410 2191 raft_consensus.cc:605] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 012786a8f1b64995847cf7dacbc029a4, State: Running, Role: LEADER I0312 20:10:52.491714 2191 consensus_queue.cc:219] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 737, Committed index: 737, Last appended: 8.737, Last appended by leader: 737, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "012786a8f1b64995847cf7dacbc029a4" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 43059 } } peers { permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 } } peers { permanent_uuid: "9da22e5afdce42078fe8f94ad166ac3c" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 39175 } } I0312 20:10:52.492116 2191 consensus_peers.cc:458] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 -> Peer 9da22e5afdce42078fe8f94ad166ac3c (127.0.53.1:39175): Closing peer: 9da22e5afdce42078fe8f94ad166ac3c I0312 20:10:52.492235 2191 consensus_peers.cc:458] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 -> Peer e3aa58d3045c4a05b94b94eb8bf58ae1 (127.0.53.3:49316): Closing peer: e3aa58d3045c4a05b94b94eb8bf58ae1 I0312 20:10:52.492460 2191 raft_consensus.cc:2752] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 8 FOLLOWER]: Advancing to term 9 I0312 20:10:52.508816 2196 raft_consensus.cc:436] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 012786a8f1b64995847cf7dacbc029a4) I0312 20:10:52.509002 2196 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c: Snoozing failure detection for 0.026s (starting election) I0312 20:10:52.509186 2196 raft_consensus.cc:458] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "012786a8f1b64995847cf7dacbc029a4" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 43059 } } peers { permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 } } peers { permanent_uuid: "9da22e5afdce42078fe8f94ad166ac3c" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 39175 } } I0312 20:10:52.509935 2196 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [CANDIDATE]: Term 9 pre-election: Requesting pre-vote from peer 012786a8f1b64995847cf7dacbc029a4 I0312 20:10:52.510200 2196 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [CANDIDATE]: Term 9 pre-election: Requesting pre-vote from peer e3aa58d3045c4a05b94b94eb8bf58ae1 I0312 20:10:52.510398 2183 raft_consensus.cc:436] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0312 20:10:52.510557 2183 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4: Snoozing failure detection for 0.018s (starting election) I0312 20:10:52.510711 2183 raft_consensus.cc:458] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "012786a8f1b64995847cf7dacbc029a4" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 43059 } } peers { permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 } } peers { permanent_uuid: "9da22e5afdce42078fe8f94ad166ac3c" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 39175 } } I0312 20:10:52.510846 1865 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "9da22e5afdce42078fe8f94ad166ac3c" candidate_term: 9 candidate_status { last_received { term: 8 index: 737 } } ignore_live_leader: false dest_uuid: "012786a8f1b64995847cf7dacbc029a4" is_pre_election: true I0312 20:10:52.511397 2183 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [CANDIDATE]: Term 10 pre-election: Requesting pre-vote from peer e3aa58d3045c4a05b94b94eb8bf58ae1 I0312 20:10:52.511447 1865 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4: Snoozing failure detection for 0.018s (vote granted) I0312 20:10:52.511615 1865 raft_consensus.cc:2215] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9da22e5afdce42078fe8f94ad166ac3c in term 9. I0312 20:10:52.511680 2183 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [CANDIDATE]: Term 10 pre-election: Requesting pre-vote from peer 9da22e5afdce42078fe8f94ad166ac3c I0312 20:10:52.512261 1676 leader_election.cc:363] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [CANDIDATE]: Term 9 pre-election: Vote granted by peer 012786a8f1b64995847cf7dacbc029a4 I0312 20:10:52.512431 1676 leader_election.cc:258] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. I0312 20:10:52.512643 1990 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "9da22e5afdce42078fe8f94ad166ac3c" candidate_term: 9 candidate_status { last_received { term: 8 index: 737 } } ignore_live_leader: false dest_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" is_pre_election: true I0312 20:10:52.512985 1990 raft_consensus.cc:2145] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9da22e5afdce42078fe8f94ad166ac3c in current term 9: Already voted for candidate e3aa58d3045c4a05b94b94eb8bf58ae1 in this term. I0312 20:10:52.513041 2196 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c: Snoozing failure detection for 0.026s (election complete) I0312 20:10:52.513213 2196 raft_consensus.cc:2507] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 8 FOLLOWER]: Leader pre-election won for term 9 I0312 20:10:52.513120 1989 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "012786a8f1b64995847cf7dacbc029a4" candidate_term: 10 candidate_status { last_received { term: 8 index: 737 } } ignore_live_leader: false dest_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" is_pre_election: true I0312 20:10:52.513329 2196 raft_consensus.cc:436] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 8 FOLLOWER]: Starting leader election (detected failure of leader 012786a8f1b64995847cf7dacbc029a4) I0312 20:10:52.513420 1989 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1: Snoozing failure detection for 0.333s (vote granted) I0312 20:10:52.513456 2196 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c: Snoozing failure detection for 0.023s (starting election) I0312 20:10:52.513546 1989 raft_consensus.cc:2215] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 012786a8f1b64995847cf7dacbc029a4 in term 9. I0312 20:10:52.513566 2196 raft_consensus.cc:2752] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 8 FOLLOWER]: Advancing to term 9 I0312 20:10:52.513794 1663 leader_election.cc:377] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [CANDIDATE]: Term 9 pre-election: Vote denied by peer e3aa58d3045c4a05b94b94eb8bf58ae1. Message: Invalid argument: T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9da22e5afdce42078fe8f94ad166ac3c in current term 9: Already voted for candidate e3aa58d3045c4a05b94b94eb8bf58ae1 in this term. I0312 20:10:52.516582 2196 raft_consensus.cc:458] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "012786a8f1b64995847cf7dacbc029a4" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 43059 } } peers { permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 } } peers { permanent_uuid: "9da22e5afdce42078fe8f94ad166ac3c" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 39175 } } I0312 20:10:52.517350 2196 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [CANDIDATE]: Term 9 election: Requesting vote from peer 012786a8f1b64995847cf7dacbc029a4 I0312 20:10:52.517571 1797 leader_election.cc:363] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [CANDIDATE]: Term 10 pre-election: Vote granted by peer e3aa58d3045c4a05b94b94eb8bf58ae1 I0312 20:10:52.517647 2196 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [CANDIDATE]: Term 9 election: Requesting vote from peer e3aa58d3045c4a05b94b94eb8bf58ae1 I0312 20:10:52.517711 1797 leader_election.cc:258] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. I0312 20:10:52.518020 2183 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4: Snoozing failure detection for 0.020s (election complete) I0312 20:10:52.518177 2183 raft_consensus.cc:2507] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 9 FOLLOWER]: Leader pre-election won for term 10 I0312 20:10:52.518070 1738 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "012786a8f1b64995847cf7dacbc029a4" candidate_term: 10 candidate_status { last_received { term: 8 index: 737 } } ignore_live_leader: false dest_uuid: "9da22e5afdce42078fe8f94ad166ac3c" is_pre_election: true I0312 20:10:52.518296 2183 raft_consensus.cc:436] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0312 20:10:52.518395 1738 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c: Snoozing failure detection for 0.016s (vote granted) I0312 20:10:52.518414 2183 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4: Snoozing failure detection for 0.019s (starting election) I0312 20:10:52.518523 2183 raft_consensus.cc:2752] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 9 FOLLOWER]: Advancing to term 10 I0312 20:10:52.518522 1738 raft_consensus.cc:2215] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 012786a8f1b64995847cf7dacbc029a4 in term 9. I0312 20:10:52.518487 1989 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "9da22e5afdce42078fe8f94ad166ac3c" candidate_term: 9 candidate_status { last_received { term: 8 index: 737 } } ignore_live_leader: false dest_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" I0312 20:10:52.518796 1989 raft_consensus.cc:2145] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 9da22e5afdce42078fe8f94ad166ac3c in current term 9: Already voted for candidate e3aa58d3045c4a05b94b94eb8bf58ae1 in this term. I0312 20:10:52.519222 1797 leader_election.cc:363] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [CANDIDATE]: Term 10 pre-election: Vote granted by peer 9da22e5afdce42078fe8f94ad166ac3c I0312 20:10:52.519445 1663 leader_election.cc:377] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [CANDIDATE]: Term 9 election: Vote denied by peer e3aa58d3045c4a05b94b94eb8bf58ae1. Message: Invalid argument: T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 9da22e5afdce42078fe8f94ad166ac3c in current term 9: Already voted for candidate e3aa58d3045c4a05b94b94eb8bf58ae1 in this term. I0312 20:10:52.519361 1865 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "9da22e5afdce42078fe8f94ad166ac3c" candidate_term: 9 candidate_status { last_received { term: 8 index: 737 } } ignore_live_leader: false dest_uuid: "012786a8f1b64995847cf7dacbc029a4" I0312 20:10:52.521235 2183 raft_consensus.cc:458] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "012786a8f1b64995847cf7dacbc029a4" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 43059 } } peers { permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 } } peers { permanent_uuid: "9da22e5afdce42078fe8f94ad166ac3c" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 39175 } } I0312 20:10:52.521926 2183 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [CANDIDATE]: Term 10 election: Requesting vote from peer e3aa58d3045c4a05b94b94eb8bf58ae1 I0312 20:10:52.522032 1865 raft_consensus.cc:2120] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 9da22e5afdce42078fe8f94ad166ac3c for earlier term 9. Current term is 10. I0312 20:10:52.522186 2183 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [CANDIDATE]: Term 10 election: Requesting vote from peer 9da22e5afdce42078fe8f94ad166ac3c I0312 20:10:52.523000 1989 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "012786a8f1b64995847cf7dacbc029a4" candidate_term: 10 candidate_status { last_received { term: 8 index: 737 } } ignore_live_leader: false dest_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" W0312 20:10:52.523172 1676 leader_election.cc:347] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [CANDIDATE]: Term 9 election: Vote denied by peer 012786a8f1b64995847cf7dacbc029a4 with higher term. Message: Invalid argument: T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 9da22e5afdce42078fe8f94ad166ac3c for earlier term 9. Current term is 10. I0312 20:10:52.523262 1989 raft_consensus.cc:2752] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 9 FOLLOWER]: Advancing to term 10 I0312 20:10:52.523334 1676 leader_election.cc:350] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [CANDIDATE]: Term 9 election: Cancelling election due to peer responding with higher term I0312 20:10:52.523171 1738 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "012786a8f1b64995847cf7dacbc029a4" candidate_term: 10 candidate_status { last_received { term: 8 index: 737 } } ignore_live_leader: false dest_uuid: "9da22e5afdce42078fe8f94ad166ac3c" I0312 20:10:52.523393 1989 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1: Snoozing failure detection for 0.493s (vote granted) I0312 20:10:52.523485 1738 raft_consensus.cc:2752] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 9 FOLLOWER]: Advancing to term 10 I0312 20:10:52.523627 1738 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c: Snoozing failure detection for 0.023s (vote granted) I0312 20:10:52.526163 1989 raft_consensus.cc:2215] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 012786a8f1b64995847cf7dacbc029a4 in term 10. I0312 20:10:52.526903 1738 raft_consensus.cc:2215] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 012786a8f1b64995847cf7dacbc029a4 in term 10. I0312 20:10:52.526973 1797 leader_election.cc:363] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [CANDIDATE]: Term 10 election: Vote granted by peer e3aa58d3045c4a05b94b94eb8bf58ae1 I0312 20:10:52.527104 1797 leader_election.cc:258] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. I0312 20:10:52.527248 2197 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c: Snoozing failure detection for 0.024s (election complete) I0312 20:10:52.527415 2197 raft_consensus.cc:2455] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 10 FOLLOWER]: Leader election lost for term 9. Reason: Vote denied by peer 012786a8f1b64995847cf7dacbc029a4 with higher term. Message: Invalid argument: T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 9da22e5afdce42078fe8f94ad166ac3c for earlier term 9. Current term is 10. I0312 20:10:52.527655 2183 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4: Snoozing failure detection for 0.017s (election complete) I0312 20:10:52.527829 2183 raft_consensus.cc:2507] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 10 FOLLOWER]: Leader election won for term 10 I0312 20:10:52.528034 2183 raft_consensus.cc:572] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 10 LEADER]: Becoming Leader. State: Replica: 012786a8f1b64995847cf7dacbc029a4, State: Running, Role: LEADER I0312 20:10:52.528103 1797 leader_election.cc:363] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [CANDIDATE]: Term 10 election: Vote granted by peer 9da22e5afdce42078fe8f94ad166ac3c I0312 20:10:52.528337 2183 consensus_queue.cc:196] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 735, Committed index: 735, Last appended: 8.737, Last appended by leader: 737, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "012786a8f1b64995847cf7dacbc029a4" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 43059 } } peers { permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 } } peers { permanent_uuid: "9da22e5afdce42078fe8f94ad166ac3c" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 39175 } } I0312 20:10:52.531129 1618 catalog_manager.cc:3592] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 reported cstate change: term changed from 8 to 10. New cstate: current_term: 10 leader_uuid: "012786a8f1b64995847cf7dacbc029a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "012786a8f1b64995847cf7dacbc029a4" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 43059 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9da22e5afdce42078fe8f94ad166ac3c" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 39175 } health_report { overall_health: UNKNOWN } } } I0312 20:10:52.534857 1738 raft_consensus.cc:1058] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 10 FOLLOWER]: Refusing update from remote peer 012786a8f1b64995847cf7dacbc029a4: Log matching property violated. Preceding OpId in replica: term: 8 index: 737. Preceding OpId from leader: term: 10 index: 738. (index mismatch) I0312 20:10:52.535006 1989 raft_consensus.cc:1058] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 10 FOLLOWER]: Refusing update from remote peer 012786a8f1b64995847cf7dacbc029a4: Log matching property violated. Preceding OpId in replica: term: 8 index: 735. Preceding OpId from leader: term: 10 index: 738. (index mismatch) I0312 20:10:52.535763 2183 consensus_queue.cc:1017] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9da22e5afdce42078fe8f94ad166ac3c" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 39175 }, Status: LMP_MISMATCH, Last received: 8.737, Next index: 738, Last known committed idx: 737, Time since last communication: 0.000s I0312 20:10:52.536115 2194 consensus_queue.cc:1017] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 }, Status: LMP_MISMATCH, Last received: 8.735, Next index: 736, Last known committed idx: 734, Time since last communication: 0.000s W0312 20:10:52.536826 2191 raft_consensus.cc:747] Couldn't advance consensus term.: Illegal state: Can't advance term to: 9 current term: 10 is higher. W0312 20:10:52.578121 2191 raft_consensus.cc:747] Couldn't advance consensus term.: Illegal state: Can't advance term to: 9 current term: 10 is higher. W0312 20:10:52.596417 2191 raft_consensus.cc:747] Couldn't advance consensus term.: Illegal state: Can't advance term to: 9 current term: 10 is higher. W0312 20:10:52.610687 2191 raft_consensus.cc:747] Couldn't advance consensus term.: Illegal state: Can't advance term to: 9 current term: 10 is higher. W0312 20:10:52.647967 2191 raft_consensus.cc:747] Couldn't advance consensus term.: Illegal state: Can't advance term to: 9 current term: 10 is higher. W0312 20:10:52.686251 2191 raft_consensus.cc:747] Couldn't advance consensus term.: Illegal state: Can't advance term to: 9 current term: 10 is higher. W0312 20:10:52.711535 2191 raft_consensus.cc:747] Couldn't advance consensus term.: Illegal state: Can't advance term to: 9 current term: 10 is higher. W0312 20:10:52.749805 2191 raft_consensus.cc:747] Couldn't advance consensus term.: Illegal state: Can't advance term to: 9 current term: 10 is higher. W0312 20:10:52.791079 2191 raft_consensus.cc:747] Couldn't advance consensus term.: Illegal state: Can't advance term to: 9 current term: 10 is higher. W0312 20:10:52.801345 2191 raft_consensus.cc:747] Couldn't advance consensus term.: Illegal state: Can't advance term to: 9 current term: 10 is higher. W0312 20:10:52.808614 2191 raft_consensus.cc:747] Couldn't advance consensus term.: Illegal state: Can't advance term to: 9 current term: 10 is higher. W0312 20:10:52.853893 2191 raft_consensus.cc:747] Couldn't advance consensus term.: Illegal state: Can't advance term to: 9 current term: 10 is higher. W0312 20:10:52.855165 2191 raft_consensus.cc:747] Couldn't advance consensus term.: Illegal state: Can't advance term to: 9 current term: 10 is higher. W0312 20:10:52.873404 2191 raft_consensus.cc:747] Couldn't advance consensus term.: Illegal state: Can't advance term to: 9 current term: 10 is higher. W0312 20:10:52.908660 2191 raft_consensus.cc:747] Couldn't advance consensus term.: Illegal state: Can't advance term to: 9 current term: 10 is higher. W0312 20:10:52.929134 2194 raft_consensus.cc:747] Couldn't advance consensus term.: Illegal state: Can't advance term to: 9 current term: 10 is higher. W0312 20:10:52.971437 2194 raft_consensus.cc:747] Couldn't advance consensus term.: Illegal state: Can't advance term to: 9 current term: 10 is higher. W0312 20:10:53.012728 2194 raft_consensus.cc:747] Couldn't advance consensus term.: Illegal state: Can't advance term to: 9 current term: 10 is higher. W0312 20:10:53.036036 2194 raft_consensus.cc:747] Couldn't advance consensus term.: Illegal state: Can't advance term to: 9 current term: 10 is higher. W0312 20:10:53.038516 2194 raft_consensus.cc:747] Couldn't advance consensus term.: Illegal state: Can't advance term to: 9 current term: 10 is higher. W0312 20:10:53.057831 2194 raft_consensus.cc:747] Couldn't advance consensus term.: Illegal state: Can't advance term to: 9 current term: 10 is higher. W0312 20:10:53.106125 2194 raft_consensus.cc:747] Couldn't advance consensus term.: Illegal state: Can't advance term to: 9 current term: 10 is higher. W0312 20:10:53.152535 2194 raft_consensus.cc:747] Couldn't advance consensus term.: Illegal state: Can't advance term to: 9 current term: 10 is higher. W0312 20:10:53.192811 2194 raft_consensus.cc:747] Couldn't advance consensus term.: Illegal state: Can't advance term to: 9 current term: 10 is higher. W0312 20:10:53.203136 2194 raft_consensus.cc:747] Couldn't advance consensus term.: Illegal state: Can't advance term to: 9 current term: 10 is higher. W0312 20:10:53.226486 2194 raft_consensus.cc:747] Couldn't advance consensus term.: Illegal state: Can't advance term to: 9 current term: 10 is higher. W0312 20:10:53.236783 2194 raft_consensus.cc:747] Couldn't advance consensus term.: Illegal state: Can't advance term to: 9 current term: 10 is higher. W0312 20:10:53.250155 2183 raft_consensus.cc:747] Couldn't advance consensus term.: Illegal state: Can't advance term to: 9 current term: 10 is higher. W0312 20:10:53.291460 2183 raft_consensus.cc:747] Couldn't advance consensus term.: Illegal state: Can't advance term to: 9 current term: 10 is higher. W0312 20:10:53.338894 2183 raft_consensus.cc:747] Couldn't advance consensus term.: Illegal state: Can't advance term to: 9 current term: 10 is higher. W0312 20:10:53.382308 2183 raft_consensus.cc:747] Couldn't advance consensus term.: Illegal state: Can't advance term to: 9 current term: 10 is higher. W0312 20:10:53.387639 2183 raft_consensus.cc:747] Couldn't advance consensus term.: Illegal state: Can't advance term to: 9 current term: 10 is higher. W0312 20:10:53.429916 2183 raft_consensus.cc:747] Couldn't advance consensus term.: Illegal state: Can't advance term to: 9 current term: 10 is higher. W0312 20:10:53.444238 2183 raft_consensus.cc:747] Couldn't advance consensus term.: Illegal state: Can't advance term to: 9 current term: 10 is higher. W0312 20:10:53.477584 2183 raft_consensus.cc:747] Couldn't advance consensus term.: Illegal state: Can't advance term to: 9 current term: 10 is higher. W0312 20:10:53.503883 2183 raft_consensus.cc:747] Couldn't advance consensus term.: Illegal state: Can't advance term to: 9 current term: 10 is higher. W0312 20:10:53.514269 2183 raft_consensus.cc:747] Couldn't advance consensus term.: Illegal state: Can't advance term to: 9 current term: 10 is higher. W0312 20:10:53.532570 2183 raft_consensus.cc:747] Couldn't advance consensus term.: Illegal state: Can't advance term to: 9 current term: 10 is higher. W0312 20:10:53.564883 2183 raft_consensus.cc:747] Couldn't advance consensus term.: Illegal state: Can't advance term to: 9 current term: 10 is higher. W0312 20:10:53.603222 2183 raft_consensus.cc:747] Couldn't advance consensus term.: Illegal state: Can't advance term to: 9 current term: 10 is higher. W0312 20:10:53.608515 2183 raft_consensus.cc:747] Couldn't advance consensus term.: Illegal state: Can't advance term to: 9 current term: 10 is higher. W0312 20:10:53.628839 2183 raft_consensus.cc:747] Couldn't advance consensus term.: Illegal state: Can't advance term to: 9 current term: 10 is higher. W0312 20:10:53.664155 2183 raft_consensus.cc:747] Couldn't advance consensus term.: Illegal state: Can't advance term to: 9 current term: 10 is higher. W0312 20:10:53.694453 2183 raft_consensus.cc:747] Couldn't advance consensus term.: Illegal state: Can't advance term to: 9 current term: 10 is higher. W0312 20:10:53.735747 2183 raft_consensus.cc:747] Couldn't advance consensus term.: Illegal state: Can't advance term to: 9 current term: 10 is higher. W0312 20:10:53.774062 2183 raft_consensus.cc:747] Couldn't advance consensus term.: Illegal state: Can't advance term to: 9 current term: 10 is higher. W0312 20:10:53.780378 2183 raft_consensus.cc:747] Couldn't advance consensus term.: Illegal state: Can't advance term to: 9 current term: 10 is higher. W0312 20:10:53.796634 2183 raft_consensus.cc:747] Couldn't advance consensus term.: Illegal state: Can't advance term to: 9 current term: 10 is higher. W0312 20:10:53.811933 2183 raft_consensus.cc:747] Couldn't advance consensus term.: Illegal state: Can't advance term to: 9 current term: 10 is higher. W0312 20:10:53.829252 2183 raft_consensus.cc:747] Couldn't advance consensus term.: Illegal state: Can't advance term to: 9 current term: 10 is higher. W0312 20:10:53.863576 2183 raft_consensus.cc:747] Couldn't advance consensus term.: Illegal state: Can't advance term to: 9 current term: 10 is higher. W0312 20:10:53.863925 2183 raft_consensus.cc:747] Couldn't advance consensus term.: Illegal state: Can't advance term to: 9 current term: 10 is higher. W0312 20:10:53.900223 2183 raft_consensus.cc:747] Couldn't advance consensus term.: Illegal state: Can't advance term to: 9 current term: 10 is higher. W0312 20:10:53.928511 2183 raft_consensus.cc:747] Couldn't advance consensus term.: Illegal state: Can't advance term to: 9 current term: 10 is higher. W0312 20:10:53.935807 2183 raft_consensus.cc:747] Couldn't advance consensus term.: Illegal state: Can't advance term to: 9 current term: 10 is higher. W0312 20:10:53.965132 2183 raft_consensus.cc:747] Couldn't advance consensus term.: Illegal state: Can't advance term to: 9 current term: 10 is higher. W0312 20:10:53.967417 2183 raft_consensus.cc:747] Couldn't advance consensus term.: Illegal state: Can't advance term to: 9 current term: 10 is higher. W0312 20:10:53.975720 2183 raft_consensus.cc:747] Couldn't advance consensus term.: Illegal state: Can't advance term to: 9 current term: 10 is higher. W0312 20:10:53.985044 2183 raft_consensus.cc:747] Couldn't advance consensus term.: Illegal state: Can't advance term to: 9 current term: 10 is higher. W0312 20:10:53.997355 2183 raft_consensus.cc:747] Couldn't advance consensus term.: Illegal state: Can't advance term to: 9 current term: 10 is higher. W0312 20:10:54.042635 2183 raft_consensus.cc:747] Couldn't advance consensus term.: Illegal state: Can't advance term to: 9 current term: 10 is higher. W0312 20:10:54.070940 2183 raft_consensus.cc:747] Couldn't advance consensus term.: Illegal state: Can't advance term to: 9 current term: 10 is higher. W0312 20:10:54.093245 2183 raft_consensus.cc:747] Couldn't advance consensus term.: Illegal state: Can't advance term to: 9 current term: 10 is higher. W0312 20:10:54.103550 2183 raft_consensus.cc:747] Couldn't advance consensus term.: Illegal state: Can't advance term to: 9 current term: 10 is higher. W0312 20:10:54.129849 2183 raft_consensus.cc:747] Couldn't advance consensus term.: Illegal state: Can't advance term to: 9 current term: 10 is higher. W0312 20:10:54.162134 2183 raft_consensus.cc:747] Couldn't advance consensus term.: Illegal state: Can't advance term to: 9 current term: 10 is higher. W0312 20:10:54.200422 2183 raft_consensus.cc:747] Couldn't advance consensus term.: Illegal state: Can't advance term to: 9 current term: 10 is higher. W0312 20:10:54.248725 2183 raft_consensus.cc:747] Couldn't advance consensus term.: Illegal state: Can't advance term to: 9 current term: 10 is higher. I0312 20:10:54.261957 2204 rpcz_store.cc:265] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:42809 (ReqId={client: 65000f5e93144329adc212a378e8be47, seq_no=730, attempt_no=0}) took 2124ms. Request Metrics: {"child_traces":[["txn",{"apply.queue_time_us":862,"num_ops":1,"prepare.queue_time_us":69,"prepare.run_cpu_time_us":793,"prepare.run_wall_time_us":792,"raft.queue_time_us":98,"raft.run_cpu_time_us":300,"raft.run_wall_time_us":300,"replication_time_us":2120522,"thread_start_us":1062,"threads_started":2,"wal-append.queue_time_us":545}]]} I0312 20:10:54.280081 1826 rpcz_store.cc:265] Call kudu.tserver.TabletServerService.Scan from 127.0.0.1:42809 (request call id 90) took 1940ms. Request Metrics: {} W0312 20:10:54.282150 2183 raft_consensus.cc:747] Couldn't advance consensus term.: Illegal state: Can't advance term to: 9 current term: 10 is higher. W0312 20:10:54.307472 2183 raft_consensus.cc:747] Couldn't advance consensus term.: Illegal state: Can't advance term to: 9 current term: 10 is higher. W0312 20:10:54.354806 2183 raft_consensus.cc:747] Couldn't advance consensus term.: Illegal state: Can't advance term to: 9 current term: 10 is higher. W0312 20:10:54.388154 2183 raft_consensus.cc:747] Couldn't advance consensus term.: Illegal state: Can't advance term to: 9 current term: 10 is higher. W0312 20:10:54.406523 2183 raft_consensus.cc:747] Couldn't advance consensus term.: Illegal state: Can't advance term to: 9 current term: 10 is higher. W0312 20:10:54.453883 2183 raft_consensus.cc:747] Couldn't advance consensus term.: Illegal state: Can't advance term to: 9 current term: 10 is higher. W0312 20:10:54.457212 2183 raft_consensus.cc:747] Couldn't advance consensus term.: Illegal state: Can't advance term to: 9 current term: 10 is higher. W0312 20:10:54.490559 2183 raft_consensus.cc:747] Couldn't advance consensus term.: Illegal state: Can't advance term to: 9 current term: 10 is higher. W0312 20:10:54.503861 2183 raft_consensus.cc:747] Couldn't advance consensus term.: Illegal state: Can't advance term to: 9 current term: 10 is higher. W0312 20:10:54.523149 2183 raft_consensus.cc:747] Couldn't advance consensus term.: Illegal state: Can't advance term to: 9 current term: 10 is higher. W0312 20:10:54.532472 2183 raft_consensus.cc:747] Couldn't advance consensus term.: Illegal state: Can't advance term to: 9 current term: 10 is higher. W0312 20:10:54.581778 2183 raft_consensus.cc:747] Couldn't advance consensus term.: Illegal state: Can't advance term to: 9 current term: 10 is higher. W0312 20:10:54.624083 2183 raft_consensus.cc:747] Couldn't advance consensus term.: Illegal state: Can't advance term to: 9 current term: 10 is higher. W0312 20:10:54.632397 2183 raft_consensus.cc:747] Couldn't advance consensus term.: Illegal state: Can't advance term to: 9 current term: 10 is higher. W0312 20:10:54.679682 2183 raft_consensus.cc:747] Couldn't advance consensus term.: Illegal state: Can't advance term to: 9 current term: 10 is higher. W0312 20:10:54.680968 2183 raft_consensus.cc:747] Couldn't advance consensus term.: Illegal state: Can't advance term to: 9 current term: 10 is higher. W0312 20:10:54.690254 2183 raft_consensus.cc:747] Couldn't advance consensus term.: Illegal state: Can't advance term to: 9 current term: 10 is higher. W0312 20:10:54.716562 2183 raft_consensus.cc:747] Couldn't advance consensus term.: Illegal state: Can't advance term to: 9 current term: 10 is higher. W0312 20:10:54.740844 2183 raft_consensus.cc:747] Couldn't advance consensus term.: Illegal state: Can't advance term to: 9 current term: 10 is higher. W0312 20:10:54.751127 2183 raft_consensus.cc:747] Couldn't advance consensus term.: Illegal state: Can't advance term to: 9 current term: 10 is higher. W0312 20:10:54.759414 2183 raft_consensus.cc:747] Couldn't advance consensus term.: Illegal state: Can't advance term to: 9 current term: 10 is higher. W0312 20:10:54.763715 2183 raft_consensus.cc:747] Couldn't advance consensus term.: Illegal state: Can't advance term to: 9 current term: 10 is higher. W0312 20:10:54.772011 2183 raft_consensus.cc:747] Couldn't advance consensus term.: Illegal state: Can't advance term to: 9 current term: 10 is higher. W0312 20:10:54.806321 2183 raft_consensus.cc:747] Couldn't advance consensus term.: Illegal state: Can't advance term to: 9 current term: 10 is higher. W0312 20:10:54.813621 2183 raft_consensus.cc:747] Couldn't advance consensus term.: Illegal state: Can't advance term to: 9 current term: 10 is higher. W0312 20:10:54.833914 2183 raft_consensus.cc:747] Couldn't advance consensus term.: Illegal state: Can't advance term to: 9 current term: 10 is higher. W0312 20:10:54.878202 2183 raft_consensus.cc:747] Couldn't advance consensus term.: Illegal state: Can't advance term to: 9 current term: 10 is higher. W0312 20:10:54.904502 2183 raft_consensus.cc:747] Couldn't advance consensus term.: Illegal state: Can't advance term to: 9 current term: 10 is higher. W0312 20:10:54.916792 2183 raft_consensus.cc:747] Couldn't advance consensus term.: Illegal state: Can't advance term to: 9 current term: 10 is higher. W0312 20:10:54.951088 2183 raft_consensus.cc:747] Couldn't advance consensus term.: Illegal state: Can't advance term to: 9 current term: 10 is higher. W0312 20:10:54.984380 2183 raft_consensus.cc:747] Couldn't advance consensus term.: Illegal state: Can't advance term to: 9 current term: 10 is higher. W0312 20:10:55.016705 2183 raft_consensus.cc:747] Couldn't advance consensus term.: Illegal state: Can't advance term to: 9 current term: 10 is higher. W0312 20:10:55.042037 2183 raft_consensus.cc:747] Couldn't advance consensus term.: Illegal state: Can't advance term to: 9 current term: 10 is higher. W0312 20:10:55.063325 2183 raft_consensus.cc:747] Couldn't advance consensus term.: Illegal state: Can't advance term to: 9 current term: 10 is higher. W0312 20:10:55.096650 2183 raft_consensus.cc:747] Couldn't advance consensus term.: Illegal state: Can't advance term to: 9 current term: 10 is higher. W0312 20:10:55.108985 2183 raft_consensus.cc:747] Couldn't advance consensus term.: Illegal state: Can't advance term to: 9 current term: 10 is higher. W0312 20:10:55.133291 2183 raft_consensus.cc:747] Couldn't advance consensus term.: Illegal state: Can't advance term to: 9 current term: 10 is higher. W0312 20:10:55.148603 2183 raft_consensus.cc:747] Couldn't advance consensus term.: Illegal state: Can't advance term to: 9 current term: 10 is higher. W0312 20:10:55.188874 2183 raft_consensus.cc:747] Couldn't advance consensus term.: Illegal state: Can't advance term to: 9 current term: 10 is higher. W0312 20:10:55.200242 2183 raft_consensus.cc:747] Couldn't advance consensus term.: Illegal state: Can't advance term to: 9 current term: 10 is higher. W0312 20:10:55.205595 2183 raft_consensus.cc:747] Couldn't advance consensus term.: Illegal state: Can't advance term to: 9 current term: 10 is higher. W0312 20:10:55.214942 2183 raft_consensus.cc:747] Couldn't advance consensus term.: Illegal state: Can't advance term to: 9 current term: 10 is higher. W0312 20:10:55.219240 2183 raft_consensus.cc:747] Couldn't advance consensus term.: Illegal state: Can't advance term to: 9 current term: 10 is higher. W0312 20:10:55.232558 2183 raft_consensus.cc:747] Couldn't advance consensus term.: Illegal state: Can't advance term to: 9 current term: 10 is higher. W0312 20:10:55.239825 2183 raft_consensus.cc:747] Couldn't advance consensus term.: Illegal state: Can't advance term to: 9 current term: 10 is higher. W0312 20:10:55.288125 2183 raft_consensus.cc:747] Couldn't advance consensus term.: Illegal state: Can't advance term to: 9 current term: 10 is higher. W0312 20:10:55.297441 2183 raft_consensus.cc:747] Couldn't advance consensus term.: Illegal state: Can't advance term to: 9 current term: 10 is higher. W0312 20:10:55.328764 2183 raft_consensus.cc:747] Couldn't advance consensus term.: Illegal state: Can't advance term to: 9 current term: 10 is higher. W0312 20:10:55.377131 2183 raft_consensus.cc:747] Couldn't advance consensus term.: Illegal state: Can't advance term to: 9 current term: 10 is higher. W0312 20:10:55.389519 2183 raft_consensus.cc:747] Couldn't advance consensus term.: Illegal state: Can't advance term to: 9 current term: 10 is higher. W0312 20:10:55.418800 2183 raft_consensus.cc:747] Couldn't advance consensus term.: Illegal state: Can't advance term to: 9 current term: 10 is higher. W0312 20:10:55.461091 2183 raft_consensus.cc:747] Couldn't advance consensus term.: Illegal state: Can't advance term to: 9 current term: 10 is higher. W0312 20:10:55.501402 2183 raft_consensus.cc:747] Couldn't advance consensus term.: Illegal state: Can't advance term to: 9 current term: 10 is higher. W0312 20:10:55.548699 2183 raft_consensus.cc:747] Couldn't advance consensus term.: Illegal state: Can't advance term to: 9 current term: 10 is higher. W0312 20:10:55.549989 2183 raft_consensus.cc:747] Couldn't advance consensus term.: Illegal state: Can't advance term to: 9 current term: 10 is higher. W0312 20:10:55.583299 2183 raft_consensus.cc:747] Couldn't advance consensus term.: Illegal state: Can't advance term to: 9 current term: 10 is higher. W0312 20:10:55.590606 2183 raft_consensus.cc:747] Couldn't advance consensus term.: Illegal state: Can't advance term to: 9 current term: 10 is higher. W0312 20:10:55.604904 2183 raft_consensus.cc:747] Couldn't advance consensus term.: Illegal state: Can't advance term to: 9 current term: 10 is higher. W0312 20:10:55.625205 2183 raft_consensus.cc:747] Couldn't advance consensus term.: Illegal state: Can't advance term to: 9 current term: 10 is higher. W0312 20:10:55.665524 2183 raft_consensus.cc:747] Couldn't advance consensus term.: Illegal state: Can't advance term to: 9 current term: 10 is higher. W0312 20:10:55.686825 2183 raft_consensus.cc:747] Couldn't advance consensus term.: Illegal state: Can't advance term to: 9 current term: 10 is higher. W0312 20:10:55.709113 2183 raft_consensus.cc:747] Couldn't advance consensus term.: Illegal state: Can't advance term to: 9 current term: 10 is higher. W0312 20:10:55.737419 2183 raft_consensus.cc:747] Couldn't advance consensus term.: Illegal state: Can't advance term to: 9 current term: 10 is higher. W0312 20:10:55.780702 2183 raft_consensus.cc:747] Couldn't advance consensus term.: Illegal state: Can't advance term to: 9 current term: 10 is higher. W0312 20:10:55.830008 2183 raft_consensus.cc:747] Couldn't advance consensus term.: Illegal state: Can't advance term to: 9 current term: 10 is higher. W0312 20:10:55.854359 2183 raft_consensus.cc:747] Couldn't advance consensus term.: Illegal state: Can't advance term to: 9 current term: 10 is higher. W0312 20:10:55.896638 2183 raft_consensus.cc:747] Couldn't advance consensus term.: Illegal state: Can't advance term to: 9 current term: 10 is higher. W0312 20:10:55.921923 2183 raft_consensus.cc:747] Couldn't advance consensus term.: Illegal state: Can't advance term to: 9 current term: 10 is higher. W0312 20:10:55.955236 2183 raft_consensus.cc:747] Couldn't advance consensus term.: Illegal state: Can't advance term to: 9 current term: 10 is higher. W0312 20:10:55.966539 2183 raft_consensus.cc:747] Couldn't advance consensus term.: Illegal state: Can't advance term to: 9 current term: 10 is higher. W0312 20:10:55.999861 2183 raft_consensus.cc:747] Couldn't advance consensus term.: Illegal state: Can't advance term to: 9 current term: 10 is higher. W0312 20:10:56.029209 2183 raft_consensus.cc:747] Couldn't advance consensus term.: Illegal state: Can't advance term to: 9 current term: 10 is higher. W0312 20:10:56.039549 2183 raft_consensus.cc:747] Couldn't advance consensus term.: Illegal state: Can't advance term to: 9 current term: 10 is higher. W0312 20:10:56.069876 2183 raft_consensus.cc:747] Couldn't advance consensus term.: Illegal state: Can't advance term to: 9 current term: 10 is higher. W0312 20:10:56.099184 2183 raft_consensus.cc:747] Couldn't advance consensus term.: Illegal state: Can't advance term to: 9 current term: 10 is higher. I0312 20:10:56.126562 2213 rpcz_store.cc:265] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:42809 (ReqId={client: 65000f5e93144329adc212a378e8be47, seq_no=731, attempt_no=0}) took 3664ms. Request Metrics: {"child_traces":[["txn",{"apply.queue_time_us":522,"num_ops":1,"prepare.queue_time_us":90,"prepare.run_cpu_time_us":756,"prepare.run_wall_time_us":756,"replication_time_us":3661074,"thread_start_us":851,"threads_started":2,"wal-append.queue_time_us":669}]]} W0312 20:10:56.130481 2194 raft_consensus.cc:747] Couldn't advance consensus term.: Illegal state: Can't advance term to: 9 current term: 10 is higher. I0312 20:10:56.137517 2216 raft_consensus.cc:436] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 012786a8f1b64995847cf7dacbc029a4) I0312 20:10:56.137866 2216 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c: Snoozing failure detection for 0.017s (starting election) I0312 20:10:56.138161 2216 raft_consensus.cc:458] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "012786a8f1b64995847cf7dacbc029a4" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 43059 } } peers { permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 } } peers { permanent_uuid: "9da22e5afdce42078fe8f94ad166ac3c" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 39175 } } I0312 20:10:56.139426 2216 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [CANDIDATE]: Term 11 pre-election: Requesting pre-vote from peer 012786a8f1b64995847cf7dacbc029a4 I0312 20:10:56.141149 1865 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "9da22e5afdce42078fe8f94ad166ac3c" candidate_term: 11 candidate_status { last_received { term: 10 index: 739 } } ignore_live_leader: false dest_uuid: "012786a8f1b64995847cf7dacbc029a4" is_pre_election: true I0312 20:10:56.141360 2216 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [CANDIDATE]: Term 11 pre-election: Requesting pre-vote from peer e3aa58d3045c4a05b94b94eb8bf58ae1 I0312 20:10:56.141640 1865 raft_consensus.cc:2176] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 10 LEADER]: Leader pre-election vote request: Denying vote to candidate 9da22e5afdce42078fe8f94ad166ac3c for term 11 because replica is either leader or believes a valid leader to be alive. I0312 20:10:56.142545 1990 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "9da22e5afdce42078fe8f94ad166ac3c" candidate_term: 11 candidate_status { last_received { term: 10 index: 739 } } ignore_live_leader: false dest_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" is_pre_election: true I0312 20:10:56.143147 1676 leader_election.cc:377] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [CANDIDATE]: Term 11 pre-election: Vote denied by peer 012786a8f1b64995847cf7dacbc029a4. Message: Invalid argument: T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 10 LEADER]: Leader pre-election vote request: Denying vote to candidate 9da22e5afdce42078fe8f94ad166ac3c for term 11 because replica is either leader or believes a valid leader to be alive. I0312 20:10:56.145202 1990 raft_consensus.cc:2190] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9da22e5afdce42078fe8f94ad166ac3c for term 11 because replica is already servicing an update from a current leader or another vote. I0312 20:10:56.146353 1663 leader_election.cc:377] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [CANDIDATE]: Term 11 pre-election: Vote denied by peer e3aa58d3045c4a05b94b94eb8bf58ae1. Message: Service unavailable: T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9da22e5afdce42078fe8f94ad166ac3c for term 11 because replica is already servicing an update from a current leader or another vote. I0312 20:10:56.146553 1663 leader_election.cc:258] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. I0312 20:10:56.147917 2216 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c: Snoozing failure detection for 0.020s (election complete) I0312 20:10:56.148169 2216 raft_consensus.cc:2455] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority I0312 20:10:56.148104 1825 rpcz_store.cc:265] Call kudu.tserver.TabletServerService.Scan from 127.0.0.1:42809 (request call id 92) took 3350ms. Request Metrics: {} W0312 20:10:56.171859 2194 raft_consensus.cc:747] Couldn't advance consensus term.: Illegal state: Can't advance term to: 9 current term: 10 is higher. W0312 20:10:56.189188 2194 raft_consensus.cc:747] Couldn't advance consensus term.: Illegal state: Can't advance term to: 9 current term: 10 is higher. W0312 20:10:56.196483 2194 raft_consensus.cc:747] Couldn't advance consensus term.: Illegal state: Can't advance term to: 9 current term: 10 is higher. W0312 20:10:56.207772 2194 raft_consensus.cc:747] Couldn't advance consensus term.: Illegal state: Can't advance term to: 9 current term: 10 is higher. I0312 20:10:56.279625 2213 rpcz_store.cc:265] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:42809 (ReqId={client: 65000f5e93144329adc212a378e8be47, seq_no=732, attempt_no=0}) took 2015ms. Request Metrics: {"child_traces":[["txn",{"apply.queue_time_us":62,"num_ops":1,"prepare.queue_time_us":714,"prepare.run_cpu_time_us":1220,"prepare.run_wall_time_us":1773,"raft.queue_time_us":132,"raft.run_cpu_time_us":420,"raft.run_wall_time_us":427,"replication_time_us":2011383,"spinlock_wait_cycles":244736,"thread_start_us":542,"threads_started":1}]]} I0312 20:10:56.299154 1826 rpcz_store.cc:265] Call kudu.tserver.TabletServerService.Scan from 127.0.0.1:42809 (request call id 94) took 1866ms. Request Metrics: {} I0312 20:10:56.479058 2225 raft_consensus.cc:436] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 012786a8f1b64995847cf7dacbc029a4) I0312 20:10:56.479316 2225 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1: Snoozing failure detection for 0.015s (starting election) I0312 20:10:56.479557 2225 raft_consensus.cc:458] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "012786a8f1b64995847cf7dacbc029a4" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 43059 } } peers { permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 } } peers { permanent_uuid: "9da22e5afdce42078fe8f94ad166ac3c" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 39175 } } I0312 20:10:56.480548 2225 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 11 pre-election: Requesting pre-vote from peer 012786a8f1b64995847cf7dacbc029a4 I0312 20:10:56.480896 2225 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 11 pre-election: Requesting pre-vote from peer 9da22e5afdce42078fe8f94ad166ac3c I0312 20:10:56.481685 1865 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" candidate_term: 11 candidate_status { last_received { term: 10 index: 748 } } ignore_live_leader: false dest_uuid: "012786a8f1b64995847cf7dacbc029a4" is_pre_election: true I0312 20:10:56.482213 1865 raft_consensus.cc:2176] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 10 LEADER]: Leader pre-election vote request: Denying vote to candidate e3aa58d3045c4a05b94b94eb8bf58ae1 for term 11 because replica is either leader or believes a valid leader to be alive. I0312 20:10:56.482203 1734 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" candidate_term: 11 candidate_status { last_received { term: 10 index: 748 } } ignore_live_leader: false dest_uuid: "9da22e5afdce42078fe8f94ad166ac3c" is_pre_election: true I0312 20:10:56.482717 1734 raft_consensus.cc:2176] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e3aa58d3045c4a05b94b94eb8bf58ae1 for term 11 because replica is either leader or believes a valid leader to be alive. I0312 20:10:56.483469 1926 leader_election.cc:377] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 11 pre-election: Vote denied by peer 012786a8f1b64995847cf7dacbc029a4. Message: Invalid argument: T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 10 LEADER]: Leader pre-election vote request: Denying vote to candidate e3aa58d3045c4a05b94b94eb8bf58ae1 for term 11 because replica is either leader or believes a valid leader to be alive. I0312 20:10:56.483708 1922 leader_election.cc:377] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 11 pre-election: Vote denied by peer 9da22e5afdce42078fe8f94ad166ac3c. Message: Invalid argument: T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e3aa58d3045c4a05b94b94eb8bf58ae1 for term 11 because replica is either leader or believes a valid leader to be alive. I0312 20:10:56.483914 1922 leader_election.cc:258] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. I0312 20:10:56.484326 2225 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1: Snoozing failure detection for 0.018s (election complete) I0312 20:10:56.484525 2225 raft_consensus.cc:2455] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority I0312 20:10:57.369547 2229 raft_consensus.cc:436] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 012786a8f1b64995847cf7dacbc029a4) I0312 20:10:57.369871 2229 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c: Snoozing failure detection for 0.017s (starting election) I0312 20:10:57.370137 2229 raft_consensus.cc:458] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "012786a8f1b64995847cf7dacbc029a4" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 43059 } } peers { permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 } } peers { permanent_uuid: "9da22e5afdce42078fe8f94ad166ac3c" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 39175 } } I0312 20:10:57.371810 2229 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [CANDIDATE]: Term 11 pre-election: Requesting pre-vote from peer 012786a8f1b64995847cf7dacbc029a4 I0312 20:10:57.373003 1865 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "9da22e5afdce42078fe8f94ad166ac3c" candidate_term: 11 candidate_status { last_received { term: 10 index: 783 } } ignore_live_leader: false dest_uuid: "012786a8f1b64995847cf7dacbc029a4" is_pre_election: true I0312 20:10:57.373543 1865 raft_consensus.cc:2176] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 10 LEADER]: Leader pre-election vote request: Denying vote to candidate 9da22e5afdce42078fe8f94ad166ac3c for term 11 because replica is either leader or believes a valid leader to be alive. I0312 20:10:57.374539 1676 leader_election.cc:377] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [CANDIDATE]: Term 11 pre-election: Vote denied by peer 012786a8f1b64995847cf7dacbc029a4. Message: Invalid argument: T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 10 LEADER]: Leader pre-election vote request: Denying vote to candidate 9da22e5afdce42078fe8f94ad166ac3c for term 11 because replica is either leader or believes a valid leader to be alive. I0312 20:10:57.374852 2229 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [CANDIDATE]: Term 11 pre-election: Requesting pre-vote from peer e3aa58d3045c4a05b94b94eb8bf58ae1 I0312 20:10:57.377598 1989 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "9da22e5afdce42078fe8f94ad166ac3c" candidate_term: 11 candidate_status { last_received { term: 10 index: 783 } } ignore_live_leader: false dest_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" is_pre_election: true I0312 20:10:57.377303 2228 raft_consensus.cc:436] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 012786a8f1b64995847cf7dacbc029a4) I0312 20:10:57.379096 2228 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1: Snoozing failure detection for 0.028s (starting election) I0312 20:10:57.380182 2228 raft_consensus.cc:458] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "012786a8f1b64995847cf7dacbc029a4" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 43059 } } peers { permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 } } peers { permanent_uuid: "9da22e5afdce42078fe8f94ad166ac3c" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 39175 } } I0312 20:10:57.381338 2228 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 11 pre-election: Requesting pre-vote from peer 012786a8f1b64995847cf7dacbc029a4 I0312 20:10:57.381597 1989 raft_consensus.cc:2176] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9da22e5afdce42078fe8f94ad166ac3c for term 11 because replica is either leader or believes a valid leader to be alive. I0312 20:10:57.381613 2228 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 11 pre-election: Requesting pre-vote from peer 9da22e5afdce42078fe8f94ad166ac3c I0312 20:10:57.382522 1865 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" candidate_term: 11 candidate_status { last_received { term: 10 index: 784 } } ignore_live_leader: false dest_uuid: "012786a8f1b64995847cf7dacbc029a4" is_pre_election: true I0312 20:10:57.382912 1663 leader_election.cc:377] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [CANDIDATE]: Term 11 pre-election: Vote denied by peer e3aa58d3045c4a05b94b94eb8bf58ae1. Message: Invalid argument: T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9da22e5afdce42078fe8f94ad166ac3c for term 11 because replica is either leader or believes a valid leader to be alive. I0312 20:10:57.383033 1865 raft_consensus.cc:2176] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 10 LEADER]: Leader pre-election vote request: Denying vote to candidate e3aa58d3045c4a05b94b94eb8bf58ae1 for term 11 because replica is either leader or believes a valid leader to be alive. I0312 20:10:57.383116 1663 leader_election.cc:258] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. I0312 20:10:57.383913 1926 leader_election.cc:377] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 11 pre-election: Vote denied by peer 012786a8f1b64995847cf7dacbc029a4. Message: Invalid argument: T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 10 LEADER]: Leader pre-election vote request: Denying vote to candidate e3aa58d3045c4a05b94b94eb8bf58ae1 for term 11 because replica is either leader or believes a valid leader to be alive. I0312 20:10:57.384054 1738 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" candidate_term: 11 candidate_status { last_received { term: 10 index: 784 } } ignore_live_leader: false dest_uuid: "9da22e5afdce42078fe8f94ad166ac3c" is_pre_election: true I0312 20:10:57.384538 1738 raft_consensus.cc:2176] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e3aa58d3045c4a05b94b94eb8bf58ae1 for term 11 because replica is either leader or believes a valid leader to be alive. I0312 20:10:57.385805 1922 leader_election.cc:377] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 11 pre-election: Vote denied by peer 9da22e5afdce42078fe8f94ad166ac3c. Message: Invalid argument: T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e3aa58d3045c4a05b94b94eb8bf58ae1 for term 11 because replica is either leader or believes a valid leader to be alive. I0312 20:10:57.386050 1922 leader_election.cc:258] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. I0312 20:10:57.386781 2228 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1: Snoozing failure detection for 0.028s (election complete) I0312 20:10:57.389137 2229 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c: Snoozing failure detection for 0.023s (election complete) I0312 20:10:57.389916 2229 raft_consensus.cc:2455] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority I0312 20:10:57.390071 2228 raft_consensus.cc:2455] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority I0312 20:10:57.598637 2229 raft_consensus.cc:436] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 012786a8f1b64995847cf7dacbc029a4) I0312 20:10:57.598963 2229 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c: Snoozing failure detection for 0.020s (starting election) I0312 20:10:57.599205 2229 raft_consensus.cc:458] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "012786a8f1b64995847cf7dacbc029a4" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 43059 } } peers { permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 } } peers { permanent_uuid: "9da22e5afdce42078fe8f94ad166ac3c" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 39175 } } I0312 20:10:57.600301 2229 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [CANDIDATE]: Term 11 pre-election: Requesting pre-vote from peer 012786a8f1b64995847cf7dacbc029a4 I0312 20:10:57.600615 2229 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [CANDIDATE]: Term 11 pre-election: Requesting pre-vote from peer e3aa58d3045c4a05b94b94eb8bf58ae1 I0312 20:10:57.601541 1865 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "9da22e5afdce42078fe8f94ad166ac3c" candidate_term: 11 candidate_status { last_received { term: 10 index: 795 } } ignore_live_leader: false dest_uuid: "012786a8f1b64995847cf7dacbc029a4" is_pre_election: true I0312 20:10:57.602058 1865 raft_consensus.cc:2176] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 10 LEADER]: Leader pre-election vote request: Denying vote to candidate 9da22e5afdce42078fe8f94ad166ac3c for term 11 because replica is either leader or believes a valid leader to be alive. I0312 20:10:57.602277 1990 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "9da22e5afdce42078fe8f94ad166ac3c" candidate_term: 11 candidate_status { last_received { term: 10 index: 795 } } ignore_live_leader: false dest_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" is_pre_election: true I0312 20:10:57.602653 1990 raft_consensus.cc:2176] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9da22e5afdce42078fe8f94ad166ac3c for term 11 because replica is either leader or believes a valid leader to be alive. I0312 20:10:57.603189 1676 leader_election.cc:377] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [CANDIDATE]: Term 11 pre-election: Vote denied by peer 012786a8f1b64995847cf7dacbc029a4. Message: Invalid argument: T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 10 LEADER]: Leader pre-election vote request: Denying vote to candidate 9da22e5afdce42078fe8f94ad166ac3c for term 11 because replica is either leader or believes a valid leader to be alive. I0312 20:10:57.603693 1663 leader_election.cc:377] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [CANDIDATE]: Term 11 pre-election: Vote denied by peer e3aa58d3045c4a05b94b94eb8bf58ae1. Message: Invalid argument: T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9da22e5afdce42078fe8f94ad166ac3c for term 11 because replica is either leader or believes a valid leader to be alive. I0312 20:10:57.603850 1663 leader_election.cc:258] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. I0312 20:10:57.604282 2229 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c: Snoozing failure detection for 0.022s (election complete) I0312 20:10:57.604435 2229 raft_consensus.cc:2455] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority I0312 20:10:58.521337 2233 raft_consensus.cc:436] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 012786a8f1b64995847cf7dacbc029a4) I0312 20:10:58.521586 2233 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1: Snoozing failure detection for 0.015s (starting election) I0312 20:10:58.521786 2233 raft_consensus.cc:458] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "012786a8f1b64995847cf7dacbc029a4" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 43059 } } peers { permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 } } peers { permanent_uuid: "9da22e5afdce42078fe8f94ad166ac3c" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 39175 } } I0312 20:10:58.522704 2233 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 11 pre-election: Requesting pre-vote from peer 012786a8f1b64995847cf7dacbc029a4 I0312 20:10:58.523172 2233 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 11 pre-election: Requesting pre-vote from peer 9da22e5afdce42078fe8f94ad166ac3c I0312 20:10:58.524291 1738 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" candidate_term: 11 candidate_status { last_received { term: 10 index: 830 } } ignore_live_leader: false dest_uuid: "9da22e5afdce42078fe8f94ad166ac3c" is_pre_election: true I0312 20:10:58.524708 1738 raft_consensus.cc:2176] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e3aa58d3045c4a05b94b94eb8bf58ae1 for term 11 because replica is either leader or believes a valid leader to be alive. I0312 20:10:58.524575 1865 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" candidate_term: 11 candidate_status { last_received { term: 10 index: 830 } } ignore_live_leader: false dest_uuid: "012786a8f1b64995847cf7dacbc029a4" is_pre_election: true I0312 20:10:58.525452 1865 raft_consensus.cc:2176] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 10 LEADER]: Leader pre-election vote request: Denying vote to candidate e3aa58d3045c4a05b94b94eb8bf58ae1 for term 11 because replica is either leader or believes a valid leader to be alive. I0312 20:10:58.525446 1922 leader_election.cc:377] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 11 pre-election: Vote denied by peer 9da22e5afdce42078fe8f94ad166ac3c. Message: Invalid argument: T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e3aa58d3045c4a05b94b94eb8bf58ae1 for term 11 because replica is either leader or believes a valid leader to be alive. I0312 20:10:58.526351 1926 leader_election.cc:377] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 11 pre-election: Vote denied by peer 012786a8f1b64995847cf7dacbc029a4. Message: Invalid argument: T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 10 LEADER]: Leader pre-election vote request: Denying vote to candidate e3aa58d3045c4a05b94b94eb8bf58ae1 for term 11 because replica is either leader or believes a valid leader to be alive. I0312 20:10:58.526553 1926 leader_election.cc:258] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. I0312 20:10:58.526886 2233 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1: Snoozing failure detection for 0.050s (election complete) I0312 20:10:58.527071 2233 raft_consensus.cc:2455] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority I0312 20:10:58.925850 2233 raft_consensus.cc:436] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 012786a8f1b64995847cf7dacbc029a4) I0312 20:10:58.926105 2233 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1: Snoozing failure detection for 0.035s (starting election) I0312 20:10:58.926290 2233 raft_consensus.cc:458] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "012786a8f1b64995847cf7dacbc029a4" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 43059 } } peers { permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 } } peers { permanent_uuid: "9da22e5afdce42078fe8f94ad166ac3c" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 39175 } } I0312 20:10:58.927011 2233 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 11 pre-election: Requesting pre-vote from peer 012786a8f1b64995847cf7dacbc029a4 I0312 20:10:58.927289 2233 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 11 pre-election: Requesting pre-vote from peer 9da22e5afdce42078fe8f94ad166ac3c I0312 20:10:58.928269 1865 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" candidate_term: 11 candidate_status { last_received { term: 10 index: 847 } } ignore_live_leader: false dest_uuid: "012786a8f1b64995847cf7dacbc029a4" is_pre_election: true I0312 20:10:58.928746 1865 raft_consensus.cc:2176] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 10 LEADER]: Leader pre-election vote request: Denying vote to candidate e3aa58d3045c4a05b94b94eb8bf58ae1 for term 11 because replica is either leader or believes a valid leader to be alive. I0312 20:10:58.929716 1926 leader_election.cc:377] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 11 pre-election: Vote denied by peer 012786a8f1b64995847cf7dacbc029a4. Message: Invalid argument: T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 10 LEADER]: Leader pre-election vote request: Denying vote to candidate e3aa58d3045c4a05b94b94eb8bf58ae1 for term 11 because replica is either leader or believes a valid leader to be alive. I0312 20:10:58.939620 2235 raft_consensus.cc:436] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 012786a8f1b64995847cf7dacbc029a4) I0312 20:10:58.939818 2235 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c: Snoozing failure detection for 0.017s (starting election) I0312 20:10:58.940088 1738 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" candidate_term: 11 candidate_status { last_received { term: 10 index: 847 } } ignore_live_leader: false dest_uuid: "9da22e5afdce42078fe8f94ad166ac3c" is_pre_election: true I0312 20:10:58.940030 2235 raft_consensus.cc:458] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "012786a8f1b64995847cf7dacbc029a4" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 43059 } } peers { permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 } } peers { permanent_uuid: "9da22e5afdce42078fe8f94ad166ac3c" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 39175 } } I0312 20:10:58.941076 2235 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [CANDIDATE]: Term 11 pre-election: Requesting pre-vote from peer 012786a8f1b64995847cf7dacbc029a4 I0312 20:10:58.941220 1738 raft_consensus.cc:2176] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e3aa58d3045c4a05b94b94eb8bf58ae1 for term 11 because replica is either leader or believes a valid leader to be alive. I0312 20:10:58.941465 2235 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [CANDIDATE]: Term 11 pre-election: Requesting pre-vote from peer e3aa58d3045c4a05b94b94eb8bf58ae1 I0312 20:10:58.942356 1865 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "9da22e5afdce42078fe8f94ad166ac3c" candidate_term: 11 candidate_status { last_received { term: 10 index: 849 } } ignore_live_leader: false dest_uuid: "012786a8f1b64995847cf7dacbc029a4" is_pre_election: true I0312 20:10:58.942718 1865 raft_consensus.cc:2176] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 10 LEADER]: Leader pre-election vote request: Denying vote to candidate 9da22e5afdce42078fe8f94ad166ac3c for term 11 because replica is either leader or believes a valid leader to be alive. I0312 20:10:58.943323 1676 leader_election.cc:377] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [CANDIDATE]: Term 11 pre-election: Vote denied by peer 012786a8f1b64995847cf7dacbc029a4. Message: Invalid argument: T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 10 LEADER]: Leader pre-election vote request: Denying vote to candidate 9da22e5afdce42078fe8f94ad166ac3c for term 11 because replica is either leader or believes a valid leader to be alive. I0312 20:10:58.943539 1922 leader_election.cc:377] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 11 pre-election: Vote denied by peer 9da22e5afdce42078fe8f94ad166ac3c. Message: Invalid argument: T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e3aa58d3045c4a05b94b94eb8bf58ae1 for term 11 because replica is either leader or believes a valid leader to be alive. I0312 20:10:58.943564 1990 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "9da22e5afdce42078fe8f94ad166ac3c" candidate_term: 11 candidate_status { last_received { term: 10 index: 849 } } ignore_live_leader: false dest_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" is_pre_election: true I0312 20:10:58.943792 1922 leader_election.cc:258] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. I0312 20:10:58.944033 1990 raft_consensus.cc:2176] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9da22e5afdce42078fe8f94ad166ac3c for term 11 because replica is either leader or believes a valid leader to be alive. I0312 20:10:58.944399 2233 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1: Snoozing failure detection for 0.021s (election complete) I0312 20:10:58.944598 2233 raft_consensus.cc:2455] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority I0312 20:10:58.945780 1663 leader_election.cc:377] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [CANDIDATE]: Term 11 pre-election: Vote denied by peer e3aa58d3045c4a05b94b94eb8bf58ae1. Message: Invalid argument: T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9da22e5afdce42078fe8f94ad166ac3c for term 11 because replica is either leader or believes a valid leader to be alive. I0312 20:10:58.946025 1663 leader_election.cc:258] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. I0312 20:10:58.946444 2235 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c: Snoozing failure detection for 0.027s (election complete) I0312 20:10:58.946660 2235 raft_consensus.cc:2455] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority I0312 20:10:59.405944 2235 raft_consensus.cc:436] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 012786a8f1b64995847cf7dacbc029a4) I0312 20:10:59.406234 2235 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c: Snoozing failure detection for 0.094s (starting election) I0312 20:10:59.406457 2235 raft_consensus.cc:458] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "012786a8f1b64995847cf7dacbc029a4" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 43059 } } peers { permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 } } peers { permanent_uuid: "9da22e5afdce42078fe8f94ad166ac3c" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 39175 } } I0312 20:10:59.407213 2235 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [CANDIDATE]: Term 11 pre-election: Requesting pre-vote from peer 012786a8f1b64995847cf7dacbc029a4 I0312 20:10:59.407510 2235 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [CANDIDATE]: Term 11 pre-election: Requesting pre-vote from peer e3aa58d3045c4a05b94b94eb8bf58ae1 I0312 20:10:59.408736 1989 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "9da22e5afdce42078fe8f94ad166ac3c" candidate_term: 11 candidate_status { last_received { term: 10 index: 869 } } ignore_live_leader: false dest_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" is_pre_election: true I0312 20:10:59.409317 1989 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1: Snoozing failure detection for 0.107s (vote granted) I0312 20:10:59.409497 1989 raft_consensus.cc:2215] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9da22e5afdce42078fe8f94ad166ac3c in term 10. I0312 20:10:59.410411 1663 leader_election.cc:363] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [CANDIDATE]: Term 11 pre-election: Vote granted by peer e3aa58d3045c4a05b94b94eb8bf58ae1 I0312 20:10:59.410605 1663 leader_election.cc:258] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. I0312 20:10:59.412045 2235 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c: Snoozing failure detection for 0.052s (election complete) I0312 20:10:59.412246 2235 raft_consensus.cc:2507] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 10 FOLLOWER]: Leader pre-election won for term 11 I0312 20:10:59.412379 2235 raft_consensus.cc:436] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 10 FOLLOWER]: Starting leader election (detected failure of leader 012786a8f1b64995847cf7dacbc029a4) I0312 20:10:59.412514 2235 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c: Snoozing failure detection for 0.093s (starting election) I0312 20:10:59.412684 2235 raft_consensus.cc:2752] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 10 FOLLOWER]: Advancing to term 11 I0312 20:10:59.417338 2235 raft_consensus.cc:458] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "012786a8f1b64995847cf7dacbc029a4" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 43059 } } peers { permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 } } peers { permanent_uuid: "9da22e5afdce42078fe8f94ad166ac3c" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 39175 } } I0312 20:10:59.418093 2235 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [CANDIDATE]: Term 11 election: Requesting vote from peer 012786a8f1b64995847cf7dacbc029a4 I0312 20:10:59.418424 2235 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [CANDIDATE]: Term 11 election: Requesting vote from peer e3aa58d3045c4a05b94b94eb8bf58ae1 I0312 20:10:59.419421 1989 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "9da22e5afdce42078fe8f94ad166ac3c" candidate_term: 11 candidate_status { last_received { term: 10 index: 869 } } ignore_live_leader: false dest_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" I0312 20:10:59.419603 1865 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "9da22e5afdce42078fe8f94ad166ac3c" candidate_term: 11 candidate_status { last_received { term: 10 index: 869 } } ignore_live_leader: false dest_uuid: "012786a8f1b64995847cf7dacbc029a4" is_pre_election: true I0312 20:10:59.419914 1989 raft_consensus.cc:2176] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 9da22e5afdce42078fe8f94ad166ac3c for term 11 because replica is either leader or believes a valid leader to be alive. I0312 20:10:59.419968 1738 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 11 FOLLOWER]: Rejecting Update request from peer 012786a8f1b64995847cf7dacbc029a4 for earlier term 10. Current term is 11. Ops: [] I0312 20:10:59.420131 1865 raft_consensus.cc:2176] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 10 LEADER]: Leader pre-election vote request: Denying vote to candidate 9da22e5afdce42078fe8f94ad166ac3c for term 11 because replica is either leader or believes a valid leader to be alive. I0312 20:10:59.420754 1663 leader_election.cc:377] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [CANDIDATE]: Term 11 election: Vote denied by peer e3aa58d3045c4a05b94b94eb8bf58ae1. Message: Invalid argument: T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 9da22e5afdce42078fe8f94ad166ac3c for term 11 because replica is either leader or believes a valid leader to be alive. I0312 20:10:59.420902 1865 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "9da22e5afdce42078fe8f94ad166ac3c" candidate_term: 11 candidate_status { last_received { term: 10 index: 869 } } ignore_live_leader: false dest_uuid: "012786a8f1b64995847cf7dacbc029a4" I0312 20:10:59.421391 1865 raft_consensus.cc:2176] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 10 LEADER]: Leader election vote request: Denying vote to candidate 9da22e5afdce42078fe8f94ad166ac3c for term 11 because replica is either leader or believes a valid leader to be alive. I0312 20:10:59.421378 1676 leader_election.cc:377] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [CANDIDATE]: Term 11 pre-election: Vote denied by peer 012786a8f1b64995847cf7dacbc029a4. Message: Invalid argument: T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 10 LEADER]: Leader pre-election vote request: Denying vote to candidate 9da22e5afdce42078fe8f94ad166ac3c for term 11 because replica is either leader or believes a valid leader to be alive. I0312 20:10:59.421584 2224 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9da22e5afdce42078fe8f94ad166ac3c" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 39175 }, Status: INVALID_TERM, Last received: 10.869, Next index: 870, Last known committed idx: 869, Time since last communication: 0.000s I0312 20:10:59.422482 1676 leader_election.cc:377] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [CANDIDATE]: Term 11 election: Vote denied by peer 012786a8f1b64995847cf7dacbc029a4. Message: Invalid argument: T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 10 LEADER]: Leader election vote request: Denying vote to candidate 9da22e5afdce42078fe8f94ad166ac3c for term 11 because replica is either leader or believes a valid leader to be alive. I0312 20:10:59.422636 1676 leader_election.cc:258] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. I0312 20:10:59.422932 2235 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c: Snoozing failure detection for 0.019s (election complete) I0312 20:10:59.423147 2235 raft_consensus.cc:2455] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority I0312 20:10:59.425892 1738 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 11 FOLLOWER]: Rejecting Update request from peer 012786a8f1b64995847cf7dacbc029a4 for earlier term 10. Current term is 11. Ops: [] I0312 20:10:59.426661 2224 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9da22e5afdce42078fe8f94ad166ac3c" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 39175 }, Status: INVALID_TERM, Last received: 10.869, Next index: 870, Last known committed idx: 869, Time since last communication: 0.000s I0312 20:10:59.429296 1738 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 11 FOLLOWER]: Rejecting Update request from peer 012786a8f1b64995847cf7dacbc029a4 for earlier term 10. Current term is 11. Ops: [] I0312 20:10:59.430037 2224 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9da22e5afdce42078fe8f94ad166ac3c" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 39175 }, Status: INVALID_TERM, Last received: 10.869, Next index: 870, Last known committed idx: 869, Time since last communication: 0.000s I0312 20:10:59.432798 1738 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 11 FOLLOWER]: Rejecting Update request from peer 012786a8f1b64995847cf7dacbc029a4 for earlier term 10. Current term is 11. Ops: [] I0312 20:10:59.433606 2224 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9da22e5afdce42078fe8f94ad166ac3c" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 39175 }, Status: INVALID_TERM, Last received: 10.869, Next index: 870, Last known committed idx: 869, Time since last communication: 0.000s I0312 20:10:59.436285 1738 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 11 FOLLOWER]: Rejecting Update request from peer 012786a8f1b64995847cf7dacbc029a4 for earlier term 10. Current term is 11. Ops: [] I0312 20:10:59.437144 2224 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9da22e5afdce42078fe8f94ad166ac3c" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 39175 }, Status: INVALID_TERM, Last received: 10.869, Next index: 870, Last known committed idx: 869, Time since last communication: 0.000s I0312 20:10:59.440328 1738 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 11 FOLLOWER]: Rejecting Update request from peer 012786a8f1b64995847cf7dacbc029a4 for earlier term 10. Current term is 11. Ops: [] I0312 20:10:59.441397 2224 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9da22e5afdce42078fe8f94ad166ac3c" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 39175 }, Status: INVALID_TERM, Last received: 10.869, Next index: 870, Last known committed idx: 869, Time since last communication: 0.000s I0312 20:10:59.442752 2235 raft_consensus.cc:436] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0312 20:10:59.442905 2235 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c: Snoozing failure detection for 0.067s (starting election) I0312 20:10:59.443112 2235 raft_consensus.cc:458] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "012786a8f1b64995847cf7dacbc029a4" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 43059 } } peers { permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 } } peers { permanent_uuid: "9da22e5afdce42078fe8f94ad166ac3c" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 39175 } } I0312 20:10:59.443816 2235 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [CANDIDATE]: Term 12 pre-election: Requesting pre-vote from peer 012786a8f1b64995847cf7dacbc029a4 I0312 20:10:59.444075 2235 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [CANDIDATE]: Term 12 pre-election: Requesting pre-vote from peer e3aa58d3045c4a05b94b94eb8bf58ae1 I0312 20:10:59.444836 1865 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "9da22e5afdce42078fe8f94ad166ac3c" candidate_term: 12 candidate_status { last_received { term: 10 index: 869 } } ignore_live_leader: false dest_uuid: "012786a8f1b64995847cf7dacbc029a4" is_pre_election: true I0312 20:10:59.445235 1865 raft_consensus.cc:2176] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 10 LEADER]: Leader pre-election vote request: Denying vote to candidate 9da22e5afdce42078fe8f94ad166ac3c for term 12 because replica is either leader or believes a valid leader to be alive. I0312 20:10:59.444972 1989 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "9da22e5afdce42078fe8f94ad166ac3c" candidate_term: 12 candidate_status { last_received { term: 10 index: 869 } } ignore_live_leader: false dest_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" is_pre_election: true I0312 20:10:59.445477 1989 raft_consensus.cc:2176] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9da22e5afdce42078fe8f94ad166ac3c for term 12 because replica is either leader or believes a valid leader to be alive. I0312 20:10:59.446113 1676 leader_election.cc:377] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [CANDIDATE]: Term 12 pre-election: Vote denied by peer 012786a8f1b64995847cf7dacbc029a4. Message: Invalid argument: T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 10 LEADER]: Leader pre-election vote request: Denying vote to candidate 9da22e5afdce42078fe8f94ad166ac3c for term 12 because replica is either leader or believes a valid leader to be alive. I0312 20:10:59.446350 1663 leader_election.cc:377] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [CANDIDATE]: Term 12 pre-election: Vote denied by peer e3aa58d3045c4a05b94b94eb8bf58ae1. Message: Invalid argument: T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9da22e5afdce42078fe8f94ad166ac3c for term 12 because replica is either leader or believes a valid leader to be alive. I0312 20:10:59.446566 1663 leader_election.cc:258] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. I0312 20:10:59.446945 2235 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c: Snoozing failure detection for 0.127s (election complete) I0312 20:10:59.447132 2235 raft_consensus.cc:2455] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority I0312 20:10:59.447495 1738 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 11 FOLLOWER]: Rejecting Update request from peer 012786a8f1b64995847cf7dacbc029a4 for earlier term 10. Current term is 11. Ops: [] I0312 20:10:59.448274 2224 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9da22e5afdce42078fe8f94ad166ac3c" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 39175 }, Status: INVALID_TERM, Last received: 10.869, Next index: 870, Last known committed idx: 869, Time since last communication: 0.000s I0312 20:10:59.451267 1738 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 11 FOLLOWER]: Rejecting Update request from peer 012786a8f1b64995847cf7dacbc029a4 for earlier term 10. Current term is 11. Ops: [] I0312 20:10:59.452476 2231 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9da22e5afdce42078fe8f94ad166ac3c" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 39175 }, Status: INVALID_TERM, Last received: 10.869, Next index: 870, Last known committed idx: 869, Time since last communication: 0.000s I0312 20:10:59.454303 1738 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 11 FOLLOWER]: Rejecting Update request from peer 012786a8f1b64995847cf7dacbc029a4 for earlier term 10. Current term is 11. Ops: [] I0312 20:10:59.455271 2191 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9da22e5afdce42078fe8f94ad166ac3c" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 39175 }, Status: INVALID_TERM, Last received: 10.869, Next index: 870, Last known committed idx: 869, Time since last communication: 0.000s I0312 20:10:59.458581 1738 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 11 FOLLOWER]: Rejecting Update request from peer 012786a8f1b64995847cf7dacbc029a4 for earlier term 10. Current term is 11. Ops: [] I0312 20:10:59.459332 2191 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9da22e5afdce42078fe8f94ad166ac3c" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 39175 }, Status: INVALID_TERM, Last received: 10.869, Next index: 870, Last known committed idx: 869, Time since last communication: 0.000s I0312 20:10:59.462858 1738 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 11 FOLLOWER]: Rejecting Update request from peer 012786a8f1b64995847cf7dacbc029a4 for earlier term 10. Current term is 11. Ops: [] I0312 20:10:59.463608 2191 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9da22e5afdce42078fe8f94ad166ac3c" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 39175 }, Status: INVALID_TERM, Last received: 10.869, Next index: 870, Last known committed idx: 869, Time since last communication: 0.000s I0312 20:10:59.467562 1738 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 11 FOLLOWER]: Rejecting Update request from peer 012786a8f1b64995847cf7dacbc029a4 for earlier term 10. Current term is 11. Ops: [] I0312 20:10:59.468348 2191 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9da22e5afdce42078fe8f94ad166ac3c" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 39175 }, Status: INVALID_TERM, Last received: 10.869, Next index: 870, Last known committed idx: 869, Time since last communication: 0.000s I0312 20:10:59.473098 1738 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 11 FOLLOWER]: Rejecting Update request from peer 012786a8f1b64995847cf7dacbc029a4 for earlier term 10. Current term is 11. Ops: [] I0312 20:10:59.473843 2191 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9da22e5afdce42078fe8f94ad166ac3c" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 39175 }, Status: INVALID_TERM, Last received: 10.869, Next index: 870, Last known committed idx: 869, Time since last communication: 0.000s I0312 20:10:59.476388 1738 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 11 FOLLOWER]: Rejecting Update request from peer 012786a8f1b64995847cf7dacbc029a4 for earlier term 10. Current term is 11. Ops: [] I0312 20:10:59.477242 2191 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9da22e5afdce42078fe8f94ad166ac3c" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 39175 }, Status: INVALID_TERM, Last received: 10.869, Next index: 870, Last known committed idx: 869, Time since last communication: 0.000s I0312 20:10:59.480881 1738 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 11 FOLLOWER]: Rejecting Update request from peer 012786a8f1b64995847cf7dacbc029a4 for earlier term 10. Current term is 11. Ops: [] I0312 20:10:59.481925 2191 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9da22e5afdce42078fe8f94ad166ac3c" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 39175 }, Status: INVALID_TERM, Last received: 10.869, Next index: 870, Last known committed idx: 869, Time since last communication: 0.000s I0312 20:10:59.484019 1738 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 11 FOLLOWER]: Rejecting Update request from peer 012786a8f1b64995847cf7dacbc029a4 for earlier term 10. Current term is 11. Ops: [] I0312 20:10:59.485288 2191 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9da22e5afdce42078fe8f94ad166ac3c" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 39175 }, Status: INVALID_TERM, Last received: 10.869, Next index: 870, Last known committed idx: 869, Time since last communication: 0.000s I0312 20:10:59.490046 1738 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 11 FOLLOWER]: Rejecting Update request from peer 012786a8f1b64995847cf7dacbc029a4 for earlier term 10. Current term is 11. Ops: [] I0312 20:10:59.490878 2183 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9da22e5afdce42078fe8f94ad166ac3c" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 39175 }, Status: INVALID_TERM, Last received: 10.869, Next index: 870, Last known committed idx: 869, Time since last communication: 0.000s I0312 20:10:59.494722 1738 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 11 FOLLOWER]: Rejecting Update request from peer 012786a8f1b64995847cf7dacbc029a4 for earlier term 10. Current term is 11. Ops: [] I0312 20:10:59.495550 2183 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9da22e5afdce42078fe8f94ad166ac3c" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 39175 }, Status: INVALID_TERM, Last received: 10.869, Next index: 870, Last known committed idx: 869, Time since last communication: 0.000s I0312 20:10:59.497812 1738 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 11 FOLLOWER]: Rejecting Update request from peer 012786a8f1b64995847cf7dacbc029a4 for earlier term 10. Current term is 11. Ops: [] I0312 20:10:59.498585 2183 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9da22e5afdce42078fe8f94ad166ac3c" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 39175 }, Status: INVALID_TERM, Last received: 10.869, Next index: 870, Last known committed idx: 869, Time since last communication: 0.000s I0312 20:10:59.502670 1738 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 11 FOLLOWER]: Rejecting Update request from peer 012786a8f1b64995847cf7dacbc029a4 for earlier term 10. Current term is 11. Ops: [] I0312 20:10:59.503566 2191 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9da22e5afdce42078fe8f94ad166ac3c" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 39175 }, Status: INVALID_TERM, Last received: 10.869, Next index: 870, Last known committed idx: 869, Time since last communication: 0.000s I0312 20:10:59.508244 1738 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 11 FOLLOWER]: Rejecting Update request from peer 012786a8f1b64995847cf7dacbc029a4 for earlier term 10. Current term is 11. Ops: [] I0312 20:10:59.509052 2183 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9da22e5afdce42078fe8f94ad166ac3c" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 39175 }, Status: INVALID_TERM, Last received: 10.869, Next index: 870, Last known committed idx: 869, Time since last communication: 0.000s I0312 20:10:59.513597 1738 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 11 FOLLOWER]: Rejecting Update request from peer 012786a8f1b64995847cf7dacbc029a4 for earlier term 10. Current term is 11. Ops: [] I0312 20:10:59.514407 2191 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9da22e5afdce42078fe8f94ad166ac3c" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 39175 }, Status: INVALID_TERM, Last received: 10.869, Next index: 870, Last known committed idx: 869, Time since last communication: 0.000s I0312 20:10:59.517251 1738 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 11 FOLLOWER]: Rejecting Update request from peer 012786a8f1b64995847cf7dacbc029a4 for earlier term 10. Current term is 11. Ops: [] I0312 20:10:59.518084 2183 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9da22e5afdce42078fe8f94ad166ac3c" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 39175 }, Status: INVALID_TERM, Last received: 10.869, Next index: 870, Last known committed idx: 869, Time since last communication: 0.000s I0312 20:10:59.518730 2224 raft_consensus.cc:2747] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 10 LEADER]: Stepping down as leader of term 10 I0312 20:10:59.518860 2224 raft_consensus.cc:605] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 012786a8f1b64995847cf7dacbc029a4, State: Running, Role: LEADER I0312 20:10:59.519213 2224 consensus_queue.cc:219] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 872, Committed index: 872, Last appended: 10.872, Last appended by leader: 872, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "012786a8f1b64995847cf7dacbc029a4" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 43059 } } peers { permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 } } peers { permanent_uuid: "9da22e5afdce42078fe8f94ad166ac3c" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 39175 } } I0312 20:10:59.519771 2183 consensus_peers.cc:230] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 -> Peer 9da22e5afdce42078fe8f94ad166ac3c (127.0.53.1:39175): Could not obtain request from queue for peer: 9da22e5afdce42078fe8f94ad166ac3c. Status: Not found: Peer not tracked or queue not in leader mode. I0312 20:10:59.519872 2191 consensus_peers.cc:230] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 -> Peer e3aa58d3045c4a05b94b94eb8bf58ae1 (127.0.53.3:49316): Could not obtain request from queue for peer: e3aa58d3045c4a05b94b94eb8bf58ae1. Status: Not found: Peer not tracked or queue not in leader mode. I0312 20:10:59.519940 2224 consensus_peers.cc:458] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 -> Peer 9da22e5afdce42078fe8f94ad166ac3c (127.0.53.1:39175): Closing peer: 9da22e5afdce42078fe8f94ad166ac3c I0312 20:10:59.520131 2224 consensus_peers.cc:458] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 -> Peer e3aa58d3045c4a05b94b94eb8bf58ae1 (127.0.53.3:49316): Closing peer: e3aa58d3045c4a05b94b94eb8bf58ae1 I0312 20:10:59.520443 2224 raft_consensus.cc:2752] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 10 FOLLOWER]: Advancing to term 11 I0312 20:10:59.531697 2238 raft_consensus.cc:436] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 012786a8f1b64995847cf7dacbc029a4) I0312 20:10:59.531903 2238 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1: Snoozing failure detection for 0.031s (starting election) I0312 20:10:59.532137 2238 raft_consensus.cc:458] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "012786a8f1b64995847cf7dacbc029a4" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 43059 } } peers { permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 } } peers { permanent_uuid: "9da22e5afdce42078fe8f94ad166ac3c" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 39175 } } I0312 20:10:59.533025 2238 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 11 pre-election: Requesting pre-vote from peer 012786a8f1b64995847cf7dacbc029a4 I0312 20:10:59.533324 2238 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 11 pre-election: Requesting pre-vote from peer 9da22e5afdce42078fe8f94ad166ac3c I0312 20:10:59.534289 1865 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" candidate_term: 11 candidate_status { last_received { term: 10 index: 872 } } ignore_live_leader: false dest_uuid: "012786a8f1b64995847cf7dacbc029a4" is_pre_election: true I0312 20:10:59.534597 1865 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4: Snoozing failure detection for 0.015s (vote granted) I0312 20:10:59.534757 1865 raft_consensus.cc:2215] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3aa58d3045c4a05b94b94eb8bf58ae1 in term 11. I0312 20:10:59.535034 1738 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" candidate_term: 11 candidate_status { last_received { term: 10 index: 872 } } ignore_live_leader: false dest_uuid: "9da22e5afdce42078fe8f94ad166ac3c" is_pre_election: true I0312 20:10:59.535444 1738 raft_consensus.cc:2145] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e3aa58d3045c4a05b94b94eb8bf58ae1 in current term 11: Already voted for candidate 9da22e5afdce42078fe8f94ad166ac3c in this term. I0312 20:10:59.535502 1926 leader_election.cc:363] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 11 pre-election: Vote granted by peer 012786a8f1b64995847cf7dacbc029a4 I0312 20:10:59.535704 1926 leader_election.cc:258] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. I0312 20:10:59.536135 2238 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1: Snoozing failure detection for 0.045s (election complete) I0312 20:10:59.536201 1922 leader_election.cc:377] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 11 pre-election: Vote denied by peer 9da22e5afdce42078fe8f94ad166ac3c. Message: Invalid argument: T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e3aa58d3045c4a05b94b94eb8bf58ae1 in current term 11: Already voted for candidate 9da22e5afdce42078fe8f94ad166ac3c in this term. I0312 20:10:59.536306 2238 raft_consensus.cc:2507] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 10 FOLLOWER]: Leader pre-election won for term 11 I0312 20:10:59.536471 2238 raft_consensus.cc:436] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 012786a8f1b64995847cf7dacbc029a4) I0312 20:10:59.536610 2238 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1: Snoozing failure detection for 0.111s (starting election) I0312 20:10:59.536744 2238 raft_consensus.cc:2752] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 10 FOLLOWER]: Advancing to term 11 I0312 20:10:59.540041 2238 raft_consensus.cc:458] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "012786a8f1b64995847cf7dacbc029a4" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 43059 } } peers { permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 } } peers { permanent_uuid: "9da22e5afdce42078fe8f94ad166ac3c" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 39175 } } I0312 20:10:59.540701 2238 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 11 election: Requesting vote from peer 012786a8f1b64995847cf7dacbc029a4 I0312 20:10:59.540973 2238 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 11 election: Requesting vote from peer 9da22e5afdce42078fe8f94ad166ac3c I0312 20:10:59.541802 1738 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" candidate_term: 11 candidate_status { last_received { term: 10 index: 872 } } ignore_live_leader: false dest_uuid: "9da22e5afdce42078fe8f94ad166ac3c" I0312 20:10:59.541815 1865 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" candidate_term: 11 candidate_status { last_received { term: 10 index: 872 } } ignore_live_leader: false dest_uuid: "012786a8f1b64995847cf7dacbc029a4" I0312 20:10:59.542075 1865 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4: Snoozing failure detection for 0.016s (vote granted) I0312 20:10:59.542160 1738 raft_consensus.cc:2145] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate e3aa58d3045c4a05b94b94eb8bf58ae1 in current term 11: Already voted for candidate 9da22e5afdce42078fe8f94ad166ac3c in this term. I0312 20:10:59.542886 1922 leader_election.cc:377] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 11 election: Vote denied by peer 9da22e5afdce42078fe8f94ad166ac3c. Message: Invalid argument: T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate e3aa58d3045c4a05b94b94eb8bf58ae1 in current term 11: Already voted for candidate 9da22e5afdce42078fe8f94ad166ac3c in this term. I0312 20:10:59.544947 1865 raft_consensus.cc:2215] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3aa58d3045c4a05b94b94eb8bf58ae1 in term 11. I0312 20:10:59.545569 1926 leader_election.cc:363] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 11 election: Vote granted by peer 012786a8f1b64995847cf7dacbc029a4 I0312 20:10:59.545704 1926 leader_election.cc:258] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. I0312 20:10:59.546008 2238 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1: Snoozing failure detection for 0.055s (election complete) I0312 20:10:59.546170 2238 raft_consensus.cc:2507] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 11 FOLLOWER]: Leader election won for term 11 I0312 20:10:59.546723 2238 raft_consensus.cc:572] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 11 LEADER]: Becoming Leader. State: Replica: e3aa58d3045c4a05b94b94eb8bf58ae1, State: Running, Role: LEADER I0312 20:10:59.547127 2238 consensus_queue.cc:196] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 872, Committed index: 872, Last appended: 10.872, Last appended by leader: 872, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "012786a8f1b64995847cf7dacbc029a4" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 43059 } } peers { permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 } } peers { permanent_uuid: "9da22e5afdce42078fe8f94ad166ac3c" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 39175 } } I0312 20:10:59.552659 1738 raft_consensus.cc:1058] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 11 FOLLOWER]: Refusing update from remote peer e3aa58d3045c4a05b94b94eb8bf58ae1: Log matching property violated. Preceding OpId in replica: term: 10 index: 869. Preceding OpId from leader: term: 11 index: 873. (index mismatch) I0312 20:10:59.562260 2191 raft_consensus.cc:436] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0312 20:10:59.562526 2191 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4: Snoozing failure detection for 0.017s (starting election) I0312 20:10:59.562767 2191 raft_consensus.cc:458] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "012786a8f1b64995847cf7dacbc029a4" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 43059 } } peers { permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 } } peers { permanent_uuid: "9da22e5afdce42078fe8f94ad166ac3c" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 39175 } } I0312 20:10:59.562670 1619 catalog_manager.cc:3592] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 reported cstate change: term changed from 10 to 11, leader changed from 012786a8f1b64995847cf7dacbc029a4 (127.0.53.2) to e3aa58d3045c4a05b94b94eb8bf58ae1 (127.0.53.3). New cstate: current_term: 11 leader_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "012786a8f1b64995847cf7dacbc029a4" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 43059 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9da22e5afdce42078fe8f94ad166ac3c" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 39175 } health_report { overall_health: UNKNOWN } } } I0312 20:10:59.563738 2191 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [CANDIDATE]: Term 12 pre-election: Requesting pre-vote from peer e3aa58d3045c4a05b94b94eb8bf58ae1 I0312 20:10:59.564685 2238 consensus_queue.cc:1017] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9da22e5afdce42078fe8f94ad166ac3c" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 39175 }, Status: LMP_MISMATCH, Last received: 10.869, Next index: 870, Last known committed idx: 869, Time since last communication: 0.011s I0312 20:10:59.564887 1989 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "012786a8f1b64995847cf7dacbc029a4" candidate_term: 12 candidate_status { last_received { term: 10 index: 872 } } ignore_live_leader: false dest_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" is_pre_election: true I0312 20:10:59.565428 1989 raft_consensus.cc:2176] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 11 LEADER]: Leader pre-election vote request: Denying vote to candidate 012786a8f1b64995847cf7dacbc029a4 for term 12 because replica is either leader or believes a valid leader to be alive. I0312 20:10:59.567111 1797 leader_election.cc:377] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [CANDIDATE]: Term 12 pre-election: Vote denied by peer e3aa58d3045c4a05b94b94eb8bf58ae1. Message: Invalid argument: T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 11 LEADER]: Leader pre-election vote request: Denying vote to candidate 012786a8f1b64995847cf7dacbc029a4 for term 12 because replica is either leader or believes a valid leader to be alive. I0312 20:10:59.567157 1865 raft_consensus.cc:1058] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 11 FOLLOWER]: Refusing update from remote peer e3aa58d3045c4a05b94b94eb8bf58ae1: Log matching property violated. Preceding OpId in replica: term: 10 index: 872. Preceding OpId from leader: term: 11 index: 873. (index mismatch) I0312 20:10:59.567412 2191 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [CANDIDATE]: Term 12 pre-election: Requesting pre-vote from peer 9da22e5afdce42078fe8f94ad166ac3c I0312 20:10:59.568397 2239 consensus_queue.cc:1017] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "012786a8f1b64995847cf7dacbc029a4" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 43059 }, Status: LMP_MISMATCH, Last received: 10.872, Next index: 873, Last known committed idx: 872, Time since last communication: 0.000s I0312 20:10:59.569476 1734 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "012786a8f1b64995847cf7dacbc029a4" candidate_term: 12 candidate_status { last_received { term: 10 index: 872 } } ignore_live_leader: false dest_uuid: "9da22e5afdce42078fe8f94ad166ac3c" is_pre_election: true I0312 20:10:59.576690 1734 raft_consensus.cc:2190] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 012786a8f1b64995847cf7dacbc029a4 for term 12 because replica is already servicing an update from a current leader or another vote. I0312 20:10:59.578266 1797 leader_election.cc:377] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [CANDIDATE]: Term 12 pre-election: Vote denied by peer 9da22e5afdce42078fe8f94ad166ac3c. Message: Service unavailable: T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 012786a8f1b64995847cf7dacbc029a4 for term 12 because replica is already servicing an update from a current leader or another vote. I0312 20:10:59.578487 1797 leader_election.cc:258] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. I0312 20:10:59.578902 2191 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4: Snoozing failure detection for 0.017s (election complete) I0312 20:10:59.579107 2191 raft_consensus.cc:2455] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority I0312 20:10:59.869699 2235 raft_consensus.cc:436] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 11 FOLLOWER]: Starting pre-election (detected failure of leader e3aa58d3045c4a05b94b94eb8bf58ae1) I0312 20:10:59.869952 2235 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c: Snoozing failure detection for 0.017s (starting election) I0312 20:10:59.870115 2235 raft_consensus.cc:458] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "012786a8f1b64995847cf7dacbc029a4" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 43059 } } peers { permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 } } peers { permanent_uuid: "9da22e5afdce42078fe8f94ad166ac3c" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 39175 } } I0312 20:10:59.870960 2235 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [CANDIDATE]: Term 12 pre-election: Requesting pre-vote from peer 012786a8f1b64995847cf7dacbc029a4 I0312 20:10:59.871240 2235 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [CANDIDATE]: Term 12 pre-election: Requesting pre-vote from peer e3aa58d3045c4a05b94b94eb8bf58ae1 I0312 20:10:59.872031 1865 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "9da22e5afdce42078fe8f94ad166ac3c" candidate_term: 12 candidate_status { last_received { term: 11 index: 880 } } ignore_live_leader: false dest_uuid: "012786a8f1b64995847cf7dacbc029a4" is_pre_election: true I0312 20:10:59.872547 1865 raft_consensus.cc:2176] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9da22e5afdce42078fe8f94ad166ac3c for term 12 because replica is either leader or believes a valid leader to be alive. I0312 20:10:59.873349 1676 leader_election.cc:377] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [CANDIDATE]: Term 12 pre-election: Vote denied by peer 012786a8f1b64995847cf7dacbc029a4. Message: Invalid argument: T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9da22e5afdce42078fe8f94ad166ac3c for term 12 because replica is either leader or believes a valid leader to be alive. I0312 20:10:59.873440 1989 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "9da22e5afdce42078fe8f94ad166ac3c" candidate_term: 12 candidate_status { last_received { term: 11 index: 880 } } ignore_live_leader: false dest_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" is_pre_election: true I0312 20:10:59.873971 1989 raft_consensus.cc:2176] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 11 LEADER]: Leader pre-election vote request: Denying vote to candidate 9da22e5afdce42078fe8f94ad166ac3c for term 12 because replica is either leader or believes a valid leader to be alive. I0312 20:10:59.874346 2191 raft_consensus.cc:436] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader e3aa58d3045c4a05b94b94eb8bf58ae1) I0312 20:10:59.874562 2191 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4: Snoozing failure detection for 0.025s (starting election) I0312 20:10:59.874883 1663 leader_election.cc:377] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [CANDIDATE]: Term 12 pre-election: Vote denied by peer e3aa58d3045c4a05b94b94eb8bf58ae1. Message: Invalid argument: T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 11 LEADER]: Leader pre-election vote request: Denying vote to candidate 9da22e5afdce42078fe8f94ad166ac3c for term 12 because replica is either leader or believes a valid leader to be alive. I0312 20:10:59.875078 1663 leader_election.cc:258] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. I0312 20:10:59.874761 2191 raft_consensus.cc:458] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "012786a8f1b64995847cf7dacbc029a4" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 43059 } } peers { permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 } } peers { permanent_uuid: "9da22e5afdce42078fe8f94ad166ac3c" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 39175 } } I0312 20:10:59.875368 2235 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c: Snoozing failure detection for 0.016s (election complete) I0312 20:10:59.875553 2235 raft_consensus.cc:2455] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority I0312 20:10:59.875695 2191 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [CANDIDATE]: Term 12 pre-election: Requesting pre-vote from peer e3aa58d3045c4a05b94b94eb8bf58ae1 I0312 20:10:59.876024 2191 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [CANDIDATE]: Term 12 pre-election: Requesting pre-vote from peer 9da22e5afdce42078fe8f94ad166ac3c I0312 20:10:59.877336 1989 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "012786a8f1b64995847cf7dacbc029a4" candidate_term: 12 candidate_status { last_received { term: 11 index: 880 } } ignore_live_leader: false dest_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" is_pre_election: true I0312 20:10:59.877813 1989 raft_consensus.cc:2176] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 11 LEADER]: Leader pre-election vote request: Denying vote to candidate 012786a8f1b64995847cf7dacbc029a4 for term 12 because replica is either leader or believes a valid leader to be alive. I0312 20:10:59.877677 1738 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "012786a8f1b64995847cf7dacbc029a4" candidate_term: 12 candidate_status { last_received { term: 11 index: 880 } } ignore_live_leader: false dest_uuid: "9da22e5afdce42078fe8f94ad166ac3c" is_pre_election: true I0312 20:10:59.878165 1738 raft_consensus.cc:2176] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 012786a8f1b64995847cf7dacbc029a4 for term 12 because replica is either leader or believes a valid leader to be alive. I0312 20:10:59.878584 1797 leader_election.cc:377] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [CANDIDATE]: Term 12 pre-election: Vote denied by peer e3aa58d3045c4a05b94b94eb8bf58ae1. Message: Invalid argument: T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 11 LEADER]: Leader pre-election vote request: Denying vote to candidate 012786a8f1b64995847cf7dacbc029a4 for term 12 because replica is either leader or believes a valid leader to be alive. I0312 20:10:59.879055 1797 leader_election.cc:377] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [CANDIDATE]: Term 12 pre-election: Vote denied by peer 9da22e5afdce42078fe8f94ad166ac3c. Message: Invalid argument: T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 012786a8f1b64995847cf7dacbc029a4 for term 12 because replica is either leader or believes a valid leader to be alive. I0312 20:10:59.879242 1797 leader_election.cc:258] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. I0312 20:10:59.879622 2191 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4: Snoozing failure detection for 0.022s (election complete) I0312 20:10:59.879761 2191 raft_consensus.cc:2455] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority I0312 20:10:59.951759 2191 raft_consensus.cc:436] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader e3aa58d3045c4a05b94b94eb8bf58ae1) I0312 20:10:59.951980 2191 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4: Snoozing failure detection for 0.047s (starting election) I0312 20:10:59.952185 2191 raft_consensus.cc:458] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "012786a8f1b64995847cf7dacbc029a4" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 43059 } } peers { permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 } } peers { permanent_uuid: "9da22e5afdce42078fe8f94ad166ac3c" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 39175 } } I0312 20:10:59.953146 2191 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [CANDIDATE]: Term 12 pre-election: Requesting pre-vote from peer e3aa58d3045c4a05b94b94eb8bf58ae1 I0312 20:10:59.953500 2191 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [CANDIDATE]: Term 12 pre-election: Requesting pre-vote from peer 9da22e5afdce42078fe8f94ad166ac3c I0312 20:10:59.954198 1989 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "012786a8f1b64995847cf7dacbc029a4" candidate_term: 12 candidate_status { last_received { term: 11 index: 880 } } ignore_live_leader: false dest_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" is_pre_election: true I0312 20:10:59.954684 1989 raft_consensus.cc:2176] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 11 LEADER]: Leader pre-election vote request: Denying vote to candidate 012786a8f1b64995847cf7dacbc029a4 for term 12 because replica is either leader or believes a valid leader to be alive. I0312 20:10:59.954581 1738 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "012786a8f1b64995847cf7dacbc029a4" candidate_term: 12 candidate_status { last_received { term: 11 index: 880 } } ignore_live_leader: false dest_uuid: "9da22e5afdce42078fe8f94ad166ac3c" is_pre_election: true I0312 20:10:59.955068 1738 raft_consensus.cc:2176] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 012786a8f1b64995847cf7dacbc029a4 for term 12 because replica is either leader or believes a valid leader to be alive. I0312 20:10:59.955637 1797 leader_election.cc:377] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [CANDIDATE]: Term 12 pre-election: Vote denied by peer e3aa58d3045c4a05b94b94eb8bf58ae1. Message: Invalid argument: T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 11 LEADER]: Leader pre-election vote request: Denying vote to candidate 012786a8f1b64995847cf7dacbc029a4 for term 12 because replica is either leader or believes a valid leader to be alive. I0312 20:10:59.956076 1797 leader_election.cc:377] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [CANDIDATE]: Term 12 pre-election: Vote denied by peer 9da22e5afdce42078fe8f94ad166ac3c. Message: Invalid argument: T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 012786a8f1b64995847cf7dacbc029a4 for term 12 because replica is either leader or believes a valid leader to be alive. I0312 20:10:59.956260 1797 leader_election.cc:258] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. I0312 20:10:59.956653 2191 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4: Snoozing failure detection for 0.040s (election complete) I0312 20:10:59.956847 2191 raft_consensus.cc:2455] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority Connected to the Master Fetched info from all 3 Tablet Servers Table test-workload is HEALTHY (1 tablet(s) checked) Table Summary Name | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable ---------------+---------+---------------+---------+------------+------------------+------------- test-workload | HEALTHY | 1 | 1 | 0 | 0 | 0 The metadata for 1 table(s) is HEALTHY I0312 20:11:00.101325 2224 raft_consensus.cc:436] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader e3aa58d3045c4a05b94b94eb8bf58ae1) I0312 20:11:00.101516 2224 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4: Snoozing failure detection for 0.067s (starting election) I0312 20:11:00.101749 2224 raft_consensus.cc:458] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "012786a8f1b64995847cf7dacbc029a4" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 43059 } } peers { permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 } } peers { permanent_uuid: "9da22e5afdce42078fe8f94ad166ac3c" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 39175 } } I0312 20:11:00.102632 2224 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [CANDIDATE]: Term 12 pre-election: Requesting pre-vote from peer e3aa58d3045c4a05b94b94eb8bf58ae1 I0312 20:11:00.102916 2224 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [CANDIDATE]: Term 12 pre-election: Requesting pre-vote from peer 9da22e5afdce42078fe8f94ad166ac3c I0312 20:11:00.104022 1989 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "012786a8f1b64995847cf7dacbc029a4" candidate_term: 12 candidate_status { last_received { term: 11 index: 880 } } ignore_live_leader: false dest_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" is_pre_election: true I0312 20:11:00.104398 1989 raft_consensus.cc:2176] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 11 LEADER]: Leader pre-election vote request: Denying vote to candidate 012786a8f1b64995847cf7dacbc029a4 for term 12 because replica is either leader or believes a valid leader to be alive. I0312 20:11:00.105371 1797 leader_election.cc:377] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [CANDIDATE]: Term 12 pre-election: Vote denied by peer e3aa58d3045c4a05b94b94eb8bf58ae1. Message: Invalid argument: T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 11 LEADER]: Leader pre-election vote request: Denying vote to candidate 012786a8f1b64995847cf7dacbc029a4 for term 12 because replica is either leader or believes a valid leader to be alive. I0312 20:11:00.106019 1738 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "012786a8f1b64995847cf7dacbc029a4" candidate_term: 12 candidate_status { last_received { term: 11 index: 880 } } ignore_live_leader: false dest_uuid: "9da22e5afdce42078fe8f94ad166ac3c" is_pre_election: true I0312 20:11:00.106427 1738 raft_consensus.cc:2176] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 012786a8f1b64995847cf7dacbc029a4 for term 12 because replica is either leader or believes a valid leader to be alive. I0312 20:11:00.107112 1797 leader_election.cc:377] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [CANDIDATE]: Term 12 pre-election: Vote denied by peer 9da22e5afdce42078fe8f94ad166ac3c. Message: Invalid argument: T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 012786a8f1b64995847cf7dacbc029a4 for term 12 because replica is either leader or believes a valid leader to be alive. I0312 20:11:00.107311 1797 leader_election.cc:258] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. I0312 20:11:00.107669 2224 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4: Snoozing failure detection for 0.038s (election complete) I0312 20:11:00.107884 2224 raft_consensus.cc:2455] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority Checksum finished in 0s: 0/3 replicas remaining (0B from disk, 2.62k rows summed) ----------------------- test-workload ----------------------- T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 (127.0.53.2:43059): Checksum: 1801605270098 T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c (127.0.53.1:39175): Checksum: 1801605270098 T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 (127.0.53.3:49316): Checksum: 1801605270098 I0312 20:11:00.133154 212 log_verifier.cc:121] Checking tablet 7581a0a77de54ef8943c1521124c7728 I0312 20:11:00.320420 2224 raft_consensus.cc:436] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader e3aa58d3045c4a05b94b94eb8bf58ae1) I0312 20:11:00.320691 2224 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4: Snoozing failure detection for 0.109s (starting election) I0312 20:11:00.320924 2224 raft_consensus.cc:458] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "012786a8f1b64995847cf7dacbc029a4" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 43059 } } peers { permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 } } peers { permanent_uuid: "9da22e5afdce42078fe8f94ad166ac3c" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 39175 } } I0312 20:11:00.322239 2224 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [CANDIDATE]: Term 12 pre-election: Requesting pre-vote from peer e3aa58d3045c4a05b94b94eb8bf58ae1 I0312 20:11:00.322597 2224 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [CANDIDATE]: Term 12 pre-election: Requesting pre-vote from peer 9da22e5afdce42078fe8f94ad166ac3c I0312 20:11:00.324486 1738 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "012786a8f1b64995847cf7dacbc029a4" candidate_term: 12 candidate_status { last_received { term: 11 index: 880 } } ignore_live_leader: false dest_uuid: "9da22e5afdce42078fe8f94ad166ac3c" is_pre_election: true I0312 20:11:00.324941 1738 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c: Snoozing failure detection for 0.027s (vote granted) I0312 20:11:00.325170 1738 raft_consensus.cc:2215] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 012786a8f1b64995847cf7dacbc029a4 in term 11. I0312 20:11:00.325779 1989 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "012786a8f1b64995847cf7dacbc029a4" candidate_term: 12 candidate_status { last_received { term: 11 index: 880 } } ignore_live_leader: false dest_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" is_pre_election: true I0312 20:11:00.326309 1989 raft_consensus.cc:2176] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 11 LEADER]: Leader pre-election vote request: Denying vote to candidate 012786a8f1b64995847cf7dacbc029a4 for term 12 because replica is either leader or believes a valid leader to be alive. I0312 20:11:00.327720 1797 leader_election.cc:363] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [CANDIDATE]: Term 12 pre-election: Vote granted by peer 9da22e5afdce42078fe8f94ad166ac3c I0312 20:11:00.327893 1797 leader_election.cc:258] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. I0312 20:11:00.328368 1797 leader_election.cc:377] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [CANDIDATE]: Term 12 pre-election: Vote denied by peer e3aa58d3045c4a05b94b94eb8bf58ae1. Message: Invalid argument: T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 11 LEADER]: Leader pre-election vote request: Denying vote to candidate 012786a8f1b64995847cf7dacbc029a4 for term 12 because replica is either leader or believes a valid leader to be alive. I0312 20:11:00.328974 2224 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4: Snoozing failure detection for 0.034s (election complete) I0312 20:11:00.329211 2224 raft_consensus.cc:2507] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 11 FOLLOWER]: Leader pre-election won for term 12 I0312 20:11:00.329385 2224 raft_consensus.cc:436] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 11 FOLLOWER]: Starting leader election (detected failure of leader e3aa58d3045c4a05b94b94eb8bf58ae1) I0312 20:11:00.329565 2224 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4: Snoozing failure detection for 0.027s (starting election) I0312 20:11:00.329746 2224 raft_consensus.cc:2752] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 11 FOLLOWER]: Advancing to term 12 I0312 20:11:00.335048 2224 raft_consensus.cc:458] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "012786a8f1b64995847cf7dacbc029a4" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 43059 } } peers { permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 } } peers { permanent_uuid: "9da22e5afdce42078fe8f94ad166ac3c" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 39175 } } I0312 20:11:00.336113 1865 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 12 FOLLOWER]: Rejecting Update request from peer e3aa58d3045c4a05b94b94eb8bf58ae1 for earlier term 11. Current term is 12. Ops: [] I0312 20:11:00.337430 2238 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "012786a8f1b64995847cf7dacbc029a4" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 43059 }, Status: INVALID_TERM, Last received: 11.880, Next index: 881, Last known committed idx: 880, Time since last communication: 0.000s I0312 20:11:00.338130 2224 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [CANDIDATE]: Term 12 election: Requesting vote from peer e3aa58d3045c4a05b94b94eb8bf58ae1 I0312 20:11:00.341606 1865 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 12 FOLLOWER]: Rejecting Update request from peer e3aa58d3045c4a05b94b94eb8bf58ae1 for earlier term 11. Current term is 12. Ops: [] I0312 20:11:00.343233 2245 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "012786a8f1b64995847cf7dacbc029a4" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 43059 }, Status: INVALID_TERM, Last received: 11.880, Next index: 881, Last known committed idx: 880, Time since last communication: 0.000s I0312 20:11:00.345021 1989 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "012786a8f1b64995847cf7dacbc029a4" candidate_term: 12 candidate_status { last_received { term: 11 index: 880 } } ignore_live_leader: false dest_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" I0312 20:11:00.345525 1989 raft_consensus.cc:2176] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 11 LEADER]: Leader election vote request: Denying vote to candidate 012786a8f1b64995847cf7dacbc029a4 for term 12 because replica is either leader or believes a valid leader to be alive. I0312 20:11:00.345572 2224 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [CANDIDATE]: Term 12 election: Requesting vote from peer 9da22e5afdce42078fe8f94ad166ac3c I0312 20:11:00.346457 1797 leader_election.cc:377] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [CANDIDATE]: Term 12 election: Vote denied by peer e3aa58d3045c4a05b94b94eb8bf58ae1. Message: Invalid argument: T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 11 LEADER]: Leader election vote request: Denying vote to candidate 012786a8f1b64995847cf7dacbc029a4 for term 12 because replica is either leader or believes a valid leader to be alive. I0312 20:11:00.347374 1738 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "012786a8f1b64995847cf7dacbc029a4" candidate_term: 12 candidate_status { last_received { term: 11 index: 880 } } ignore_live_leader: false dest_uuid: "9da22e5afdce42078fe8f94ad166ac3c" I0312 20:11:00.347849 1738 raft_consensus.cc:2176] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 012786a8f1b64995847cf7dacbc029a4 for term 12 because replica is either leader or believes a valid leader to be alive. I0312 20:11:00.348762 1797 leader_election.cc:377] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [CANDIDATE]: Term 12 election: Vote denied by peer 9da22e5afdce42078fe8f94ad166ac3c. Message: Invalid argument: T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 012786a8f1b64995847cf7dacbc029a4 for term 12 because replica is either leader or believes a valid leader to be alive. I0312 20:11:00.348964 1797 leader_election.cc:258] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. I0312 20:11:00.349362 2224 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4: Snoozing failure detection for 0.106s (election complete) I0312 20:11:00.349566 2224 raft_consensus.cc:2455] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority I0312 20:11:00.349997 1865 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 12 FOLLOWER]: Rejecting Update request from peer e3aa58d3045c4a05b94b94eb8bf58ae1 for earlier term 11. Current term is 12. Ops: [] I0312 20:11:00.351580 2245 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "012786a8f1b64995847cf7dacbc029a4" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 43059 }, Status: INVALID_TERM, Last received: 11.880, Next index: 881, Last known committed idx: 880, Time since last communication: 0.000s I0312 20:11:00.355051 1865 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 12 FOLLOWER]: Rejecting Update request from peer e3aa58d3045c4a05b94b94eb8bf58ae1 for earlier term 11. Current term is 12. Ops: [] I0312 20:11:00.356297 2238 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "012786a8f1b64995847cf7dacbc029a4" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 43059 }, Status: INVALID_TERM, Last received: 11.880, Next index: 881, Last known committed idx: 880, Time since last communication: 0.000s I0312 20:11:00.359841 1865 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 12 FOLLOWER]: Rejecting Update request from peer e3aa58d3045c4a05b94b94eb8bf58ae1 for earlier term 11. Current term is 12. Ops: [] I0312 20:11:00.361284 2245 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "012786a8f1b64995847cf7dacbc029a4" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 43059 }, Status: INVALID_TERM, Last received: 11.880, Next index: 881, Last known committed idx: 880, Time since last communication: 0.000s I0312 20:11:00.365469 1865 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 12 FOLLOWER]: Rejecting Update request from peer e3aa58d3045c4a05b94b94eb8bf58ae1 for earlier term 11. Current term is 12. Ops: [] I0312 20:11:00.366786 2238 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "012786a8f1b64995847cf7dacbc029a4" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 43059 }, Status: INVALID_TERM, Last received: 11.880, Next index: 881, Last known committed idx: 880, Time since last communication: 0.000s I0312 20:11:00.369247 1865 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 12 FOLLOWER]: Rejecting Update request from peer e3aa58d3045c4a05b94b94eb8bf58ae1 for earlier term 11. Current term is 12. Ops: [] I0312 20:11:00.370690 2245 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "012786a8f1b64995847cf7dacbc029a4" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 43059 }, Status: INVALID_TERM, Last received: 11.880, Next index: 881, Last known committed idx: 880, Time since last communication: 0.000s I0312 20:11:00.372920 1865 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 12 FOLLOWER]: Rejecting Update request from peer e3aa58d3045c4a05b94b94eb8bf58ae1 for earlier term 11. Current term is 12. Ops: [] I0312 20:11:00.374024 2238 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "012786a8f1b64995847cf7dacbc029a4" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 43059 }, Status: INVALID_TERM, Last received: 11.880, Next index: 881, Last known committed idx: 880, Time since last communication: 0.000s I0312 20:11:00.377593 1865 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 12 FOLLOWER]: Rejecting Update request from peer e3aa58d3045c4a05b94b94eb8bf58ae1 for earlier term 11. Current term is 12. Ops: [] I0312 20:11:00.378832 2245 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "012786a8f1b64995847cf7dacbc029a4" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 43059 }, Status: INVALID_TERM, Last received: 11.880, Next index: 881, Last known committed idx: 880, Time since last communication: 0.000s I0312 20:11:00.380764 1865 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 12 FOLLOWER]: Rejecting Update request from peer e3aa58d3045c4a05b94b94eb8bf58ae1 for earlier term 11. Current term is 12. Ops: [] I0312 20:11:00.381569 2245 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "012786a8f1b64995847cf7dacbc029a4" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 43059 }, Status: INVALID_TERM, Last received: 11.880, Next index: 881, Last known committed idx: 880, Time since last communication: 0.000s I0312 20:11:00.382131 2239 raft_consensus.cc:2747] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 11 LEADER]: Stepping down as leader of term 11 I0312 20:11:00.382354 2239 raft_consensus.cc:605] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: e3aa58d3045c4a05b94b94eb8bf58ae1, State: Running, Role: LEADER I0312 20:11:00.382815 2239 consensus_queue.cc:219] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 880, Committed index: 880, Last appended: 11.880, Last appended by leader: 880, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "012786a8f1b64995847cf7dacbc029a4" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 43059 } } peers { permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 } } peers { permanent_uuid: "9da22e5afdce42078fe8f94ad166ac3c" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 39175 } } I0312 20:11:00.383582 2245 consensus_peers.cc:230] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 -> Peer 9da22e5afdce42078fe8f94ad166ac3c (127.0.53.1:39175): Could not obtain request from queue for peer: 9da22e5afdce42078fe8f94ad166ac3c. Status: Not found: Peer not tracked or queue not in leader mode. I0312 20:11:00.383813 2239 consensus_peers.cc:458] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 -> Peer 9da22e5afdce42078fe8f94ad166ac3c (127.0.53.1:39175): Closing peer: 9da22e5afdce42078fe8f94ad166ac3c I0312 20:11:00.383986 2239 consensus_peers.cc:458] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 -> Peer 012786a8f1b64995847cf7dacbc029a4 (127.0.53.2:43059): Closing peer: 012786a8f1b64995847cf7dacbc029a4 I0312 20:11:00.384264 2239 raft_consensus.cc:2752] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 11 FOLLOWER]: Advancing to term 12 I0312 20:11:00.391125 2271 raft_consensus.cc:436] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 11 FOLLOWER]: Starting pre-election (detected failure of leader e3aa58d3045c4a05b94b94eb8bf58ae1) I0312 20:11:00.391329 2271 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c: Snoozing failure detection for 0.029s (starting election) I0312 20:11:00.391510 2271 raft_consensus.cc:458] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "012786a8f1b64995847cf7dacbc029a4" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 43059 } } peers { permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 } } peers { permanent_uuid: "9da22e5afdce42078fe8f94ad166ac3c" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 39175 } } I0312 20:11:00.392217 2271 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [CANDIDATE]: Term 12 pre-election: Requesting pre-vote from peer 012786a8f1b64995847cf7dacbc029a4 I0312 20:11:00.392498 2271 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [CANDIDATE]: Term 12 pre-election: Requesting pre-vote from peer e3aa58d3045c4a05b94b94eb8bf58ae1 I0312 20:11:00.393630 1989 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "9da22e5afdce42078fe8f94ad166ac3c" candidate_term: 12 candidate_status { last_received { term: 11 index: 880 } } ignore_live_leader: false dest_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" is_pre_election: true I0312 20:11:00.393929 1989 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1: Snoozing failure detection for 0.015s (vote granted) I0312 20:11:00.394069 1989 raft_consensus.cc:2215] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9da22e5afdce42078fe8f94ad166ac3c in term 12. I0312 20:11:00.393278 1865 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "9da22e5afdce42078fe8f94ad166ac3c" candidate_term: 12 candidate_status { last_received { term: 11 index: 880 } } ignore_live_leader: false dest_uuid: "012786a8f1b64995847cf7dacbc029a4" is_pre_election: true I0312 20:11:00.394726 1865 raft_consensus.cc:2145] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9da22e5afdce42078fe8f94ad166ac3c in current term 12: Already voted for candidate 012786a8f1b64995847cf7dacbc029a4 in this term. I0312 20:11:00.394810 1663 leader_election.cc:363] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [CANDIDATE]: Term 12 pre-election: Vote granted by peer e3aa58d3045c4a05b94b94eb8bf58ae1 I0312 20:11:00.395015 1663 leader_election.cc:258] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. I0312 20:11:00.395429 2271 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c: Snoozing failure detection for 0.025s (election complete) I0312 20:11:00.395429 1676 leader_election.cc:377] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [CANDIDATE]: Term 12 pre-election: Vote denied by peer 012786a8f1b64995847cf7dacbc029a4. Message: Invalid argument: T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9da22e5afdce42078fe8f94ad166ac3c in current term 12: Already voted for candidate 012786a8f1b64995847cf7dacbc029a4 in this term. I0312 20:11:00.395663 2271 raft_consensus.cc:2507] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 11 FOLLOWER]: Leader pre-election won for term 12 I0312 20:11:00.395830 2271 raft_consensus.cc:436] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 11 FOLLOWER]: Starting leader election (detected failure of leader e3aa58d3045c4a05b94b94eb8bf58ae1) I0312 20:11:00.396004 2271 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c: Snoozing failure detection for 0.031s (starting election) I0312 20:11:00.396159 2271 raft_consensus.cc:2752] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 11 FOLLOWER]: Advancing to term 12 I0312 20:11:00.399240 2271 raft_consensus.cc:458] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "012786a8f1b64995847cf7dacbc029a4" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 43059 } } peers { permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 } } peers { permanent_uuid: "9da22e5afdce42078fe8f94ad166ac3c" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 39175 } } I0312 20:11:00.399922 2271 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [CANDIDATE]: Term 12 election: Requesting vote from peer 012786a8f1b64995847cf7dacbc029a4 I0312 20:11:00.400238 2271 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [CANDIDATE]: Term 12 election: Requesting vote from peer e3aa58d3045c4a05b94b94eb8bf58ae1 I0312 20:11:00.401000 1865 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "9da22e5afdce42078fe8f94ad166ac3c" candidate_term: 12 candidate_status { last_received { term: 11 index: 880 } } ignore_live_leader: false dest_uuid: "012786a8f1b64995847cf7dacbc029a4" I0312 20:11:00.401468 1865 raft_consensus.cc:2145] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 9da22e5afdce42078fe8f94ad166ac3c in current term 12: Already voted for candidate 012786a8f1b64995847cf7dacbc029a4 in this term. I0312 20:11:00.401337 1989 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "9da22e5afdce42078fe8f94ad166ac3c" candidate_term: 12 candidate_status { last_received { term: 11 index: 880 } } ignore_live_leader: false dest_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" I0312 20:11:00.401723 1989 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1: Snoozing failure detection for 0.015s (vote granted) I0312 20:11:00.402078 1676 leader_election.cc:377] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [CANDIDATE]: Term 12 election: Vote denied by peer 012786a8f1b64995847cf7dacbc029a4. Message: Invalid argument: T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 9da22e5afdce42078fe8f94ad166ac3c in current term 12: Already voted for candidate 012786a8f1b64995847cf7dacbc029a4 in this term. I0312 20:11:00.406056 1989 raft_consensus.cc:2215] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9da22e5afdce42078fe8f94ad166ac3c in term 12. I0312 20:11:00.406906 1663 leader_election.cc:363] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [CANDIDATE]: Term 12 election: Vote granted by peer e3aa58d3045c4a05b94b94eb8bf58ae1 I0312 20:11:00.407065 1663 leader_election.cc:258] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. I0312 20:11:00.407372 2271 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c: Snoozing failure detection for 0.015s (election complete) I0312 20:11:00.407586 2271 raft_consensus.cc:2507] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 12 FOLLOWER]: Leader election won for term 12 I0312 20:11:00.421288 2245 raft_consensus.cc:436] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0312 20:11:00.421504 2245 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1: Snoozing failure detection for 0.018s (starting election) I0312 20:11:00.421746 2245 raft_consensus.cc:458] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "012786a8f1b64995847cf7dacbc029a4" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 43059 } } peers { permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 } } peers { permanent_uuid: "9da22e5afdce42078fe8f94ad166ac3c" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 39175 } } I0312 20:11:00.422816 2245 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 13 pre-election: Requesting pre-vote from peer 012786a8f1b64995847cf7dacbc029a4 I0312 20:11:00.423995 1865 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" candidate_term: 13 candidate_status { last_received { term: 11 index: 880 } } ignore_live_leader: false dest_uuid: "012786a8f1b64995847cf7dacbc029a4" is_pre_election: true I0312 20:11:00.424445 1865 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4: Snoozing failure detection for 0.147s (vote granted) I0312 20:11:00.424659 1865 raft_consensus.cc:2215] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3aa58d3045c4a05b94b94eb8bf58ae1 in term 12. I0312 20:11:00.425482 1926 leader_election.cc:363] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 13 pre-election: Vote granted by peer 012786a8f1b64995847cf7dacbc029a4 I0312 20:11:00.425688 1926 leader_election.cc:258] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. I0312 20:11:00.426134 2238 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1: Snoozing failure detection for 0.017s (election complete) I0312 20:11:00.426359 2238 raft_consensus.cc:2507] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 12 FOLLOWER]: Leader pre-election won for term 13 I0312 20:11:00.426550 2238 raft_consensus.cc:436] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0312 20:11:00.426744 2238 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1: Snoozing failure detection for 0.020s (starting election) I0312 20:11:00.426934 2238 raft_consensus.cc:2752] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 12 FOLLOWER]: Advancing to term 13 I0312 20:11:00.431414 2238 raft_consensus.cc:458] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "012786a8f1b64995847cf7dacbc029a4" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 43059 } } peers { permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 } } peers { permanent_uuid: "9da22e5afdce42078fe8f94ad166ac3c" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 39175 } } I0312 20:11:00.432384 2238 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 13 election: Requesting vote from peer 012786a8f1b64995847cf7dacbc029a4 I0312 20:11:00.433575 1864 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" candidate_term: 13 candidate_status { last_received { term: 11 index: 880 } } ignore_live_leader: false dest_uuid: "012786a8f1b64995847cf7dacbc029a4" I0312 20:11:00.433979 1864 raft_consensus.cc:2752] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 12 FOLLOWER]: Advancing to term 13 I0312 20:11:00.434173 1864 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4: Snoozing failure detection for 0.057s (vote granted) I0312 20:11:00.437494 2271 raft_consensus.cc:572] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 12 LEADER]: Becoming Leader. State: Replica: 9da22e5afdce42078fe8f94ad166ac3c, State: Running, Role: LEADER I0312 20:11:00.437902 2271 consensus_queue.cc:196] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 880, Committed index: 880, Last appended: 11.880, Last appended by leader: 880, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "012786a8f1b64995847cf7dacbc029a4" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 43059 } } peers { permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 } } peers { permanent_uuid: "9da22e5afdce42078fe8f94ad166ac3c" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 39175 } } I0312 20:11:00.438550 1864 raft_consensus.cc:2215] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3aa58d3045c4a05b94b94eb8bf58ae1 in term 13. I0312 20:11:00.439350 1926 leader_election.cc:363] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 13 election: Vote granted by peer 012786a8f1b64995847cf7dacbc029a4 I0312 20:11:00.439548 1926 leader_election.cc:258] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. I0312 20:11:00.439781 2272 raft_consensus.cc:419] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 12 LEADER]: Not starting pre-election -- already a leader I0312 20:11:00.440742 2274 raft_consensus.cc:2700] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1: Snoozing failure detection for 0.017s (election complete) I0312 20:11:00.440800 2238 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 13 election: Requesting vote from peer 9da22e5afdce42078fe8f94ad166ac3c I0312 20:11:00.441002 2245 leader_election.cc:231] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 13 pre-election: Requesting pre-vote from peer 9da22e5afdce42078fe8f94ad166ac3c I0312 20:11:00.442210 2274 raft_consensus.cc:2507] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 13 FOLLOWER]: Leader election won for term 13 I0312 20:11:00.442378 1618 catalog_manager.cc:3592] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c reported cstate change: term changed from 11 to 12, leader changed from e3aa58d3045c4a05b94b94eb8bf58ae1 (127.0.53.3) to 9da22e5afdce42078fe8f94ad166ac3c (127.0.53.1). New cstate: current_term: 12 leader_uuid: "9da22e5afdce42078fe8f94ad166ac3c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "012786a8f1b64995847cf7dacbc029a4" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 43059 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9da22e5afdce42078fe8f94ad166ac3c" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 39175 } health_report { overall_health: HEALTHY } } } I0312 20:11:00.443343 2274 raft_consensus.cc:572] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 13 LEADER]: Becoming Leader. State: Replica: e3aa58d3045c4a05b94b94eb8bf58ae1, State: Running, Role: LEADER I0312 20:11:00.443722 2274 consensus_queue.cc:196] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 880, Committed index: 880, Last appended: 11.880, Last appended by leader: 880, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "012786a8f1b64995847cf7dacbc029a4" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 43059 } } peers { permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 } } peers { permanent_uuid: "9da22e5afdce42078fe8f94ad166ac3c" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 39175 } } I0312 20:11:00.450179 1865 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 13 FOLLOWER]: Rejecting Update request from peer 9da22e5afdce42078fe8f94ad166ac3c for earlier term 12. Current term is 13. Ops: [] I0312 20:11:00.450996 1864 raft_consensus.cc:1058] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 13 FOLLOWER]: Refusing update from remote peer e3aa58d3045c4a05b94b94eb8bf58ae1: Log matching property violated. Preceding OpId in replica: term: 11 index: 880. Preceding OpId from leader: term: 13 index: 881. (index mismatch) I0312 20:11:00.451331 2272 consensus_queue.cc:1028] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "012786a8f1b64995847cf7dacbc029a4" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 43059 }, Status: INVALID_TERM, Last received: 11.880, Next index: 881, Last known committed idx: 880, Time since last communication: 0.000s I0312 20:11:00.452189 2238 consensus_queue.cc:1017] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "012786a8f1b64995847cf7dacbc029a4" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 43059 }, Status: LMP_MISMATCH, Last received: 11.880, Next index: 881, Last known committed idx: 880, Time since last communication: 0.000s I0312 20:11:00.454015 1738 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" candidate_term: 13 candidate_status { last_received { term: 11 index: 880 } } ignore_live_leader: false dest_uuid: "9da22e5afdce42078fe8f94ad166ac3c" I0312 20:11:00.454490 1738 raft_consensus.cc:2176] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 12 LEADER]: Leader election vote request: Denying vote to candidate e3aa58d3045c4a05b94b94eb8bf58ae1 for term 13 because replica is either leader or believes a valid leader to be alive. I0312 20:11:00.455411 1738 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "7581a0a77de54ef8943c1521124c7728" candidate_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" candidate_term: 13 candidate_status { last_received { term: 11 index: 880 } } ignore_live_leader: false dest_uuid: "9da22e5afdce42078fe8f94ad166ac3c" is_pre_election: true I0312 20:11:00.455873 1738 raft_consensus.cc:2176] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 12 LEADER]: Leader pre-election vote request: Denying vote to candidate e3aa58d3045c4a05b94b94eb8bf58ae1 for term 13 because replica is either leader or believes a valid leader to be alive. I0312 20:11:00.457134 1922 leader_election.cc:377] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 13 election: Vote denied by peer 9da22e5afdce42078fe8f94ad166ac3c. Message: Invalid argument: T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 12 LEADER]: Leader election vote request: Denying vote to candidate e3aa58d3045c4a05b94b94eb8bf58ae1 for term 13 because replica is either leader or believes a valid leader to be alive. I0312 20:11:00.457769 1922 leader_election.cc:377] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [CANDIDATE]: Term 13 pre-election: Vote denied by peer 9da22e5afdce42078fe8f94ad166ac3c. Message: Invalid argument: T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 12 LEADER]: Leader pre-election vote request: Denying vote to candidate e3aa58d3045c4a05b94b94eb8bf58ae1 for term 13 because replica is either leader or believes a valid leader to be alive. I0312 20:11:00.458683 1738 raft_consensus.cc:2747] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 12 LEADER]: Stepping down as leader of term 12 I0312 20:11:00.458803 1864 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P 012786a8f1b64995847cf7dacbc029a4 [term 13 FOLLOWER]: Rejecting Update request from peer 9da22e5afdce42078fe8f94ad166ac3c for earlier term 12. Current term is 13. Ops: [] I0312 20:11:00.458876 1738 raft_consensus.cc:605] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 9da22e5afdce42078fe8f94ad166ac3c, State: Running, Role: LEADER I0312 20:11:00.459113 1989 raft_consensus.cc:1026] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [term 13 LEADER]: Rejecting Update request from peer 9da22e5afdce42078fe8f94ad166ac3c for earlier term 12. Current term is 13. Ops: [] I0312 20:11:00.459267 1738 consensus_queue.cc:219] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 880, Committed index: 880, Last appended: 12.881, Last appended by leader: 881, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "012786a8f1b64995847cf7dacbc029a4" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 43059 } } peers { permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 } } peers { permanent_uuid: "9da22e5afdce42078fe8f94ad166ac3c" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 39175 } } I0312 20:11:00.459873 1738 consensus_peers.cc:458] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c -> Peer e3aa58d3045c4a05b94b94eb8bf58ae1 (127.0.53.3:49316): Closing peer: e3aa58d3045c4a05b94b94eb8bf58ae1 I0312 20:11:00.460053 1738 consensus_peers.cc:458] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c -> Peer 012786a8f1b64995847cf7dacbc029a4 (127.0.53.2:43059): Closing peer: 012786a8f1b64995847cf7dacbc029a4 I0312 20:11:00.460218 1738 raft_consensus.cc:2752] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 12 FOLLOWER]: Advancing to term 13 I0312 20:11:00.466112 1738 raft_consensus.cc:1058] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 13 FOLLOWER]: Refusing update from remote peer e3aa58d3045c4a05b94b94eb8bf58ae1: Log matching property violated. Preceding OpId in replica: term: 12 index: 881. Preceding OpId from leader: term: 13 index: 881. (term mismatch) I0312 20:11:00.466584 1738 pending_rounds.cc:76] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c: Aborting all transactions after (but not including) 880 I0312 20:11:00.466737 1738 pending_rounds.cc:97] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c: Aborting uncommitted NO_OP operation due to leader change: 12.881 I0312 20:11:00.466920 1738 raft_consensus.cc:2590] T 7581a0a77de54ef8943c1521124c7728 P 9da22e5afdce42078fe8f94ad166ac3c [term 13 FOLLOWER]: NO_OP replication failed: Aborted: Transaction aborted by new leader I0312 20:11:00.468345 2238 consensus_queue.cc:1017] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9da22e5afdce42078fe8f94ad166ac3c" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 39175 }, Status: LMP_MISMATCH, Last received: 11.880, Next index: 881, Last known committed idx: 880, Time since last communication: 0.000s I0312 20:11:00.474792 1617 catalog_manager.cc:3592] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 reported cstate change: term changed from 12 to 13, leader changed from 9da22e5afdce42078fe8f94ad166ac3c (127.0.53.1) to e3aa58d3045c4a05b94b94eb8bf58ae1 (127.0.53.3). New cstate: current_term: 13 leader_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "012786a8f1b64995847cf7dacbc029a4" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 43059 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e3aa58d3045c4a05b94b94eb8bf58ae1" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 49316 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9da22e5afdce42078fe8f94ad166ac3c" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 39175 } health_report { overall_health: UNKNOWN } } } W0312 20:11:00.478981 2239 raft_consensus.cc:747] Couldn't advance consensus term.: Illegal state: Can't advance term to: 12 current term: 13 is higher. W0312 20:11:00.526355 2239 raft_consensus.cc:747] Couldn't advance consensus term.: Illegal state: Can't advance term to: 12 current term: 13 is higher. W0312 20:11:00.569787 2239 raft_consensus.cc:747] Couldn't advance consensus term.: Illegal state: Can't advance term to: 12 current term: 13 is higher. W0312 20:11:00.595376 2239 raft_consensus.cc:747] Couldn't advance consensus term.: Illegal state: Can't advance term to: 12 current term: 13 is higher. W0312 20:11:00.642711 2239 raft_consensus.cc:747] Couldn't advance consensus term.: Illegal state: Can't advance term to: 12 current term: 13 is higher. W0312 20:11:00.663064 2239 raft_consensus.cc:747] Couldn't advance consensus term.: Illegal state: Can't advance term to: 12 current term: 13 is higher. W0312 20:11:00.695385 2239 raft_consensus.cc:747] Couldn't advance consensus term.: Illegal state: Can't advance term to: 12 current term: 13 is higher. W0312 20:11:00.704748 2239 raft_consensus.cc:747] Couldn't advance consensus term.: Illegal state: Can't advance term to: 12 current term: 13 is higher. I0312 20:11:01.095840 212 log_verifier.cc:172] Verified matching terms for 881 ops in tablet 7581a0a77de54ef8943c1521124c7728 I0312 20:11:01.145706 212 external_mini_cluster.cc:997] Killing /tmp/dist-test-task3bwPeE/build/tsan/bin/kudu-tserver with pid 1653 W0312 20:11:01.193962 1797 connection.cc:513] client connection to 127.0.53.1:39175 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0312 20:11:01.194067 1600 connection.cc:513] client connection to 127.0.53.1:39175 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0312 20:11:01.195026 1799 connection.cc:513] server connection from 127.0.53.1:43855 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) I0312 20:11:01.196120 212 external_mini_cluster.cc:997] Killing /tmp/dist-test-task3bwPeE/build/tsan/bin/kudu-tserver with pid 1788 W0312 20:11:01.197262 1602 connection.cc:513] server connection from 127.0.53.1:50514 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0312 20:11:01.198513 1922 connection.cc:513] server connection from 127.0.53.1:52135 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0312 20:11:01.199715 1922 connection.cc:513] client connection to 127.0.53.1:39175 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0312 20:11:01.199997 1922 consensus_peers.cc:436] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 -> Peer 9da22e5afdce42078fe8f94ad166ac3c (127.0.53.1:39175): Couldn't send request to peer 9da22e5afdce42078fe8f94ad166ac3c for tablet 7581a0a77de54ef8943c1521124c7728. Status: Network error: failed to read from TLS socket: Connection reset by peer (error 104). Retrying in the next heartbeat period. Already tried 1 times. W0312 20:11:01.200875 1922 connection.cc:186] Error closing socket: Network error: TlsSocket::Close: Broken pipe W0312 20:11:01.221521 1922 consensus_peers.cc:436] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 -> Peer 9da22e5afdce42078fe8f94ad166ac3c (127.0.53.1:39175): Couldn't send request to peer 9da22e5afdce42078fe8f94ad166ac3c for tablet 7581a0a77de54ef8943c1521124c7728. Status: Network error: Client connection negotiation failed: client connection to 127.0.53.1:39175: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 2 times. W0312 20:11:01.223763 1922 consensus_peers.cc:436] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 -> Peer 9da22e5afdce42078fe8f94ad166ac3c (127.0.53.1:39175): Couldn't send request to peer 9da22e5afdce42078fe8f94ad166ac3c for tablet 7581a0a77de54ef8943c1521124c7728. Status: Network error: Client connection negotiation failed: client connection to 127.0.53.1:39175: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 3 times. W0312 20:11:01.228209 1922 consensus_peers.cc:436] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 -> Peer 9da22e5afdce42078fe8f94ad166ac3c (127.0.53.1:39175): Couldn't send request to peer 9da22e5afdce42078fe8f94ad166ac3c for tablet 7581a0a77de54ef8943c1521124c7728. Status: Network error: Client connection negotiation failed: client connection to 127.0.53.1:39175: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 4 times. W0312 20:11:01.233973 1922 consensus_peers.cc:436] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 -> Peer 9da22e5afdce42078fe8f94ad166ac3c (127.0.53.1:39175): Couldn't send request to peer 9da22e5afdce42078fe8f94ad166ac3c for tablet 7581a0a77de54ef8943c1521124c7728. Status: Network error: Client connection negotiation failed: client connection to 127.0.53.1:39175: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 5 times. W0312 20:11:01.238234 1922 consensus_peers.cc:436] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 -> Peer 9da22e5afdce42078fe8f94ad166ac3c (127.0.53.1:39175): Couldn't send request to peer 9da22e5afdce42078fe8f94ad166ac3c for tablet 7581a0a77de54ef8943c1521124c7728. Status: Network error: Client connection negotiation failed: client connection to 127.0.53.1:39175: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 6 times. W0312 20:11:01.243865 1926 connection.cc:513] client connection to 127.0.53.2:43059 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0312 20:11:01.243965 1604 connection.cc:513] server connection from 127.0.53.2:48706 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0312 20:11:01.244107 1926 consensus_peers.cc:436] T 7581a0a77de54ef8943c1521124c7728 P e3aa58d3045c4a05b94b94eb8bf58ae1 -> Peer 012786a8f1b64995847cf7dacbc029a4 (127.0.53.2:43059): Couldn't send request to peer 012786a8f1b64995847cf7dacbc029a4 for tablet 7581a0a77de54ef8943c1521124c7728. Status: Network error: failed to read from TLS socket: Connection reset by peer (error 104). Retrying in the next heartbeat period. Already tried 1 times. W0312 20:11:01.244592 1604 connection.cc:513] client connection to 127.0.53.2:43059 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0312 20:11:01.244850 1926 connection.cc:186] Error closing socket: Network error: TlsSocket::Close: Broken pipe W0312 20:11:01.245286 1923 connection.cc:513] server connection from 127.0.53.2:45159 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) I0312 20:11:01.246328 212 external_mini_cluster.cc:997] Killing /tmp/dist-test-task3bwPeE/build/tsan/bin/kudu-tserver with pid 1913 W0312 20:11:01.286798 1600 connection.cc:513] client connection to 127.0.53.3:49316 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0312 20:11:01.287585 1600 connection.cc:513] server connection from 127.0.53.3:49501 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) I0312 20:11:01.293277 212 external_mini_cluster.cc:997] Killing /tmp/dist-test-task3bwPeE/build/tsan/bin/kudu-master with pid 1591 W0312 20:11:01.314018 1590 connection.cc:513] client connection to 127.0.53.62:44334 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) [ OK ] RaftConsensusElectionITest.ChurnyElections_WithNotificationLatency (34308 ms) [ RUN ] RaftConsensusElectionITest.ChurnyElections_WithDuplicateKeys I0312 20:11:01.327790 212 test_util.cc:192] Using random seed: -1112891506 I0312 20:11:01.330809 212 ts_itest-base.cc:113] Starting cluster with: I0312 20:11:01.330910 212 ts_itest-base.cc:114] -------------- I0312 20:11:01.330996 212 ts_itest-base.cc:115] 3 tablet servers I0312 20:11:01.331079 212 ts_itest-base.cc:116] 3 replicas per TS I0312 20:11:01.331162 212 ts_itest-base.cc:117] -------------- I0312 20:11:01.333192 212 external_mini_cluster.cc:805] Running /tmp/dist-test-task3bwPeE/build/tsan/bin/kudu-master /tmp/dist-test-task3bwPeE/build/tsan/bin/kudu-master --fs_wal_dir=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ChurnyElections_WithDuplicateKeys.1520885401735786-212/raft_consensus-itest-cluster/master-0/wal --fs_data_dirs=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ChurnyElections_WithDuplicateKeys.1520885401735786-212/raft_consensus-itest-cluster/master-0/data --block_manager=log --webserver_interface=localhost --ipki_ca_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.0.53.62:0 --webserver_interface=127.0.53.62 --webserver_port=0 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ChurnyElections_WithDuplicateKeys.1520885401735786-212/raft_consensus-itest-cluster/master-0/logs --server_dump_info_path=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ChurnyElections_WithDuplicateKeys.1520885401735786-212/raft_consensus-itest-cluster/master-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W0312 20:11:01.452997 2293 flags.cc:412] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0312 20:11:01.453343 2293 flags.cc:412] Enabled unsafe flag: --never_fsync=true W0312 20:11:01.468722 2293 flags.cc:412] Enabled experimental flag: --ipki_ca_key_size=1024 W0312 20:11:01.468863 2293 flags.cc:412] Enabled experimental flag: --ipki_server_key_size=1024 W0312 20:11:01.468974 2293 flags.cc:412] Enabled experimental flag: --tsk_num_rsa_bits=512 I0312 20:11:01.489485 2293 master_main.cc:66] Master server non-default flags: --fs_data_dirs=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ChurnyElections_WithDuplicateKeys.1520885401735786-212/raft_consensus-itest-cluster/master-0/data --fs_wal_dir=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ChurnyElections_WithDuplicateKeys.1520885401735786-212/raft_consensus-itest-cluster/master-0/wal --ipki_ca_key_size=1024 --ipki_server_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.0.53.62:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ChurnyElections_WithDuplicateKeys.1520885401735786-212/raft_consensus-itest-cluster/master-0/data/info.pb --webserver_interface=127.0.53.62 --webserver_port=0 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --log_dir=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ChurnyElections_WithDuplicateKeys.1520885401735786-212/raft_consensus-itest-cluster/master-0/logs --logbuflevel=-1 --logtostderr=true Master server version: kudu 1.7.0-SNAPSHOT revision 13e10c63c46de30ac85f34b543cdd6fb44881f61 build type FASTDEBUG built by None at 12 Mar 2018 20:06:40 UTC on jenkins-slave-vn0g build id 12388 TSAN enabled I0312 20:11:01.498540 2293 master_main.cc:73] Initializing master server... I0312 20:11:01.500038 2293 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 1097776us I0312 20:11:01.500761 2293 fs_manager.cc:260] Metadata directory not provided I0312 20:11:01.500893 2293 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0312 20:11:01.501125 2293 server_base.cc:434] Could not load existing FS layout: Not found: /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ChurnyElections_WithDuplicateKeys.1520885401735786-212/raft_consensus-itest-cluster/master-0/data/instance: No such file or directory (error 2) I0312 20:11:01.501247 2293 server_base.cc:435] Attempting to create new FS layout instead I0312 20:11:01.508546 2293 fs_manager.cc:596] Generated new instance metadata in path /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ChurnyElections_WithDuplicateKeys.1520885401735786-212/raft_consensus-itest-cluster/master-0/data/instance: uuid: "dac3ed6e6366468fbd5bd0caeb40a84d" format_stamp: "Formatted at 2018-03-12 20:11:01 on dist-test-slave-dist-test-slave-f27p" I0312 20:11:01.509250 2293 fs_manager.cc:596] Generated new instance metadata in path /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ChurnyElections_WithDuplicateKeys.1520885401735786-212/raft_consensus-itest-cluster/master-0/wal/instance: uuid: "dac3ed6e6366468fbd5bd0caeb40a84d" format_stamp: "Formatted at 2018-03-12 20:11:01 on dist-test-slave-dist-test-slave-f27p" I0312 20:11:01.517467 2293 fs_manager.cc:496] Time spent creating directory manager: real 0.008s user 0.008s sys 0.000s I0312 20:11:01.518623 2293 env_posix.cc:1620] Not raising this process' open files per process limit of 65536; it is already as high as it can go I0312 20:11:01.518896 2293 file_cache.cc:470] Constructed file cache lbm with capacity 26214 I0312 20:11:01.525245 2293 fs_manager.cc:418] Time spent opening block manager: real 0.001s user 0.004s sys 0.000s I0312 20:11:01.525362 2293 fs_manager.cc:429] Opened local filesystem: /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ChurnyElections_WithDuplicateKeys.1520885401735786-212/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ChurnyElections_WithDuplicateKeys.1520885401735786-212/raft_consensus-itest-cluster/master-0/wal uuid: "dac3ed6e6366468fbd5bd0caeb40a84d" format_stamp: "Formatted at 2018-03-12 20:11:01 on dist-test-slave-dist-test-slave-f27p" I0312 20:11:01.525563 2293 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ChurnyElections_WithDuplicateKeys.1520885401735786-212/raft_consensus-itest-cluster/master-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0312 20:11:01.738524 2293 env_posix.cc:1620] Not raising this process' running threads per effective uid limit of 60284; it is already as high as it can go I0312 20:11:01.739153 2293 master_main.cc:76] Starting Master server... I0312 20:11:01.740391 2310 process_memory.cc:182] Process hard memory limit is 11.777631 GB I0312 20:11:01.740532 2310 process_memory.cc:184] Process soft memory limit is 9.422105 GB I0312 20:11:01.740633 2310 process_memory.cc:187] Process memory pressure threshold is 7.066578 GB I0312 20:11:01.778923 2293 rpc_server.cc:201] RPC server started. Bound to: 127.0.53.62:46983 I0312 20:11:01.804013 2293 webserver.cc:173] Starting webserver on 127.0.53.62:0 I0312 20:11:01.804121 2293 webserver.cc:184] Document root disabled I0312 20:11:01.805366 2293 webserver.cc:311] Webserver started. Bound to: http://127.0.53.62:37647/ I0312 20:11:01.806109 2293 server_base.cc:607] Dumped server information to /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ChurnyElections_WithDuplicateKeys.1520885401735786-212/raft_consensus-itest-cluster/master-0/data/info.pb I0312 20:11:01.806928 212 external_mini_cluster.cc:867] Started /tmp/dist-test-task3bwPeE/build/tsan/bin/kudu-master as pid 2293 I0312 20:11:01.808197 212 external_mini_cluster.cc:805] Running /tmp/dist-test-task3bwPeE/build/tsan/bin/kudu-tserver /tmp/dist-test-task3bwPeE/build/tsan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ChurnyElections_WithDuplicateKeys.1520885401735786-212/raft_consensus-itest-cluster/ts-0/wal --fs_data_dirs=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ChurnyElections_WithDuplicateKeys.1520885401735786-212/raft_consensus-itest-cluster/ts-0/data --block_manager=log --rpc_bind_addresses=127.0.53.1:0 --local_ip_for_outbound_sockets=127.0.53.1 --webserver_interface=127.0.53.1 --webserver_port=0 --tserver_master_addrs=127.0.53.62:46983 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ChurnyElections_WithDuplicateKeys.1520885401735786-212/raft_consensus-itest-cluster/ts-0/logs --server_dump_info_path=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ChurnyElections_WithDuplicateKeys.1520885401735786-212/raft_consensus-itest-cluster/ts-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_heartbeat_interval_ms=5 with env {} I0312 20:11:01.809406 2354 data_dirs.cc:931] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0312 20:11:01.824559 2354 tablet_bootstrap.cc:436] T 00000000000000000000000000000000 P dac3ed6e6366468fbd5bd0caeb40a84d: Bootstrap starting. I0312 20:11:01.826858 2354 tablet_bootstrap.cc:581] T 00000000000000000000000000000000 P dac3ed6e6366468fbd5bd0caeb40a84d: No blocks or log segments found. Creating new log. I0312 20:11:01.828151 2354 log.cc:520] T 00000000000000000000000000000000 P dac3ed6e6366468fbd5bd0caeb40a84d: Log is configured to *not* fsync() on all Append() calls I0312 20:11:01.830595 2354 tablet_bootstrap.cc:436] T 00000000000000000000000000000000 P dac3ed6e6366468fbd5bd0caeb40a84d: No bootstrap required, opened a new log I0312 20:11:01.842694 2354 raft_consensus.cc:304] T 00000000000000000000000000000000 P dac3ed6e6366468fbd5bd0caeb40a84d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dac3ed6e6366468fbd5bd0caeb40a84d" member_type: VOTER } I0312 20:11:01.843006 2354 raft_consensus.cc:330] T 00000000000000000000000000000000 P dac3ed6e6366468fbd5bd0caeb40a84d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0312 20:11:01.843160 2354 raft_consensus.cc:605] T 00000000000000000000000000000000 P dac3ed6e6366468fbd5bd0caeb40a84d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dac3ed6e6366468fbd5bd0caeb40a84d, State: Initialized, Role: FOLLOWER I0312 20:11:01.843683 2354 consensus_queue.cc:219] T 00000000000000000000000000000000 P dac3ed6e6366468fbd5bd0caeb40a84d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dac3ed6e6366468fbd5bd0caeb40a84d" member_type: VOTER } I0312 20:11:01.844054 2354 raft_consensus.cc:344] T 00000000000000000000000000000000 P dac3ed6e6366468fbd5bd0caeb40a84d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0312 20:11:01.844213 2354 raft_consensus.cc:436] T 00000000000000000000000000000000 P dac3ed6e6366468fbd5bd0caeb40a84d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0312 20:11:01.844367 2354 raft_consensus.cc:2700] T 00000000000000000000000000000000 P dac3ed6e6366468fbd5bd0caeb40a84d: Snoozing failure detection for 1.813s (starting election) I0312 20:11:01.844524 2354 raft_consensus.cc:2752] T 00000000000000000000000000000000 P dac3ed6e6366468fbd5bd0caeb40a84d [term 0 FOLLOWER]: Advancing to term 1 I0312 20:11:01.846976 2354 raft_consensus.cc:458] T 00000000000000000000000000000000 P dac3ed6e6366468fbd5bd0caeb40a84d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dac3ed6e6366468fbd5bd0caeb40a84d" member_type: VOTER } I0312 20:11:01.847403 2354 leader_election.cc:258] T 00000000000000000000000000000000 P dac3ed6e6366468fbd5bd0caeb40a84d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0312 20:11:01.848984 2357 raft_consensus.cc:2700] T 00000000000000000000000000000000 P dac3ed6e6366468fbd5bd0caeb40a84d: Snoozing failure detection for 1.961s (election complete) I0312 20:11:01.849262 2357 raft_consensus.cc:2507] T 00000000000000000000000000000000 P dac3ed6e6366468fbd5bd0caeb40a84d [term 1 FOLLOWER]: Leader election won for term 1 I0312 20:11:01.862279 2357 raft_consensus.cc:572] T 00000000000000000000000000000000 P dac3ed6e6366468fbd5bd0caeb40a84d [term 1 LEADER]: Becoming Leader. State: Replica: dac3ed6e6366468fbd5bd0caeb40a84d, State: Running, Role: LEADER I0312 20:11:01.862675 2357 consensus_queue.cc:196] T 00000000000000000000000000000000 P dac3ed6e6366468fbd5bd0caeb40a84d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dac3ed6e6366468fbd5bd0caeb40a84d" member_type: VOTER } I0312 20:11:01.866138 2354 sys_catalog.cc:335] T 00000000000000000000000000000000 P dac3ed6e6366468fbd5bd0caeb40a84d [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 1 leader_uuid: "dac3ed6e6366468fbd5bd0caeb40a84d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dac3ed6e6366468fbd5bd0caeb40a84d" member_type: VOTER } } I0312 20:11:01.866360 2359 sys_catalog.cc:335] T 00000000000000000000000000000000 P dac3ed6e6366468fbd5bd0caeb40a84d [sys.catalog]: SysCatalogTable state changed. Reason: New leader dac3ed6e6366468fbd5bd0caeb40a84d. Latest consensus state: current_term: 1 leader_uuid: "dac3ed6e6366468fbd5bd0caeb40a84d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dac3ed6e6366468fbd5bd0caeb40a84d" member_type: VOTER } } I0312 20:11:01.866542 2354 sys_catalog.cc:338] T 00000000000000000000000000000000 P dac3ed6e6366468fbd5bd0caeb40a84d [sys.catalog]: This master's current role is: LEADER I0312 20:11:01.866675 2359 sys_catalog.cc:338] T 00000000000000000000000000000000 P dac3ed6e6366468fbd5bd0caeb40a84d [sys.catalog]: This master's current role is: LEADER I0312 20:11:01.867352 2358 sys_catalog.cc:335] T 00000000000000000000000000000000 P dac3ed6e6366468fbd5bd0caeb40a84d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dac3ed6e6366468fbd5bd0caeb40a84d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dac3ed6e6366468fbd5bd0caeb40a84d" member_type: VOTER } } I0312 20:11:01.867843 2358 sys_catalog.cc:338] T 00000000000000000000000000000000 P dac3ed6e6366468fbd5bd0caeb40a84d [sys.catalog]: This master's current role is: LEADER I0312 20:11:01.869280 2354 sys_catalog.cc:422] T 00000000000000000000000000000000 P dac3ed6e6366468fbd5bd0caeb40a84d [sys.catalog]: configured and running, proceeding with master startup. I0312 20:11:01.869549 2362 catalog_manager.cc:972] Loading table and tablet metadata into memory... I0312 20:11:01.871069 2293 master_main.cc:79] Master server successfully started. I0312 20:11:01.874199 2362 catalog_manager.cc:983] Initializing Kudu internal certificate authority... I0312 20:11:01.947197 2362 catalog_manager.cc:870] Generated new certificate authority record I0312 20:11:01.948640 2362 catalog_manager.cc:992] Loading token signing keys... I0312 20:11:01.961371 2362 catalog_manager.cc:3868] T 00000000000000000000000000000000 P dac3ed6e6366468fbd5bd0caeb40a84d: Generated new TSK 0 WARNING: Logging before InitGoogleLogging() is written to STDERR W0312 20:11:01.981137 2355 flags.cc:412] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0312 20:11:01.981607 2355 flags.cc:412] Enabled unsafe flag: --never_fsync=true W0312 20:11:02.005671 2355 flags.cc:412] Enabled experimental flag: --ipki_server_key_size=1024 W0312 20:11:02.006147 2355 flags.cc:412] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.53.1 I0312 20:11:02.037005 2355 tablet_server_main.cc:72] Tablet server non-default flags: --raft_heartbeat_interval_ms=5 --fs_data_dirs=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ChurnyElections_WithDuplicateKeys.1520885401735786-212/raft_consensus-itest-cluster/ts-0/data --fs_wal_dir=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ChurnyElections_WithDuplicateKeys.1520885401735786-212/raft_consensus-itest-cluster/ts-0/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.0.53.1:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ChurnyElections_WithDuplicateKeys.1520885401735786-212/raft_consensus-itest-cluster/ts-0/data/info.pb --webserver_interface=127.0.53.1 --webserver_port=0 --tserver_master_addrs=127.0.53.62:46983 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.0.53.1 --log_dir=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ChurnyElections_WithDuplicateKeys.1520885401735786-212/raft_consensus-itest-cluster/ts-0/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.7.0-SNAPSHOT revision 13e10c63c46de30ac85f34b543cdd6fb44881f61 build type FASTDEBUG built by None at 12 Mar 2018 20:06:40 UTC on jenkins-slave-vn0g build id 12388 TSAN enabled I0312 20:11:02.052304 2355 tablet_server_main.cc:79] Initializing tablet server... I0312 20:11:02.054674 2355 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 1098276us I0312 20:11:02.055285 2355 fs_manager.cc:260] Metadata directory not provided I0312 20:11:02.055454 2355 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0312 20:11:02.055729 2355 server_base.cc:434] Could not load existing FS layout: Not found: /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ChurnyElections_WithDuplicateKeys.1520885401735786-212/raft_consensus-itest-cluster/ts-0/data/instance: No such file or directory (error 2) I0312 20:11:02.055893 2355 server_base.cc:435] Attempting to create new FS layout instead I0312 20:11:02.068388 2355 fs_manager.cc:596] Generated new instance metadata in path /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ChurnyElections_WithDuplicateKeys.1520885401735786-212/raft_consensus-itest-cluster/ts-0/data/instance: uuid: "71f80b7649e64a2bafe2eee9b67b14c1" format_stamp: "Formatted at 2018-03-12 20:11:02 on dist-test-slave-dist-test-slave-f27p" I0312 20:11:02.069397 2355 fs_manager.cc:596] Generated new instance metadata in path /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ChurnyElections_WithDuplicateKeys.1520885401735786-212/raft_consensus-itest-cluster/ts-0/wal/instance: uuid: "71f80b7649e64a2bafe2eee9b67b14c1" format_stamp: "Formatted at 2018-03-12 20:11:02 on dist-test-slave-dist-test-slave-f27p" I0312 20:11:02.082428 2355 fs_manager.cc:496] Time spent creating directory manager: real 0.013s user 0.016s sys 0.000s I0312 20:11:02.084131 2355 env_posix.cc:1620] Not raising this process' open files per process limit of 65536; it is already as high as it can go I0312 20:11:02.084511 2355 file_cache.cc:470] Constructed file cache lbm with capacity 26214 I0312 20:11:02.095052 2355 fs_manager.cc:418] Time spent opening block manager: real 0.002s user 0.000s sys 0.004s I0312 20:11:02.095208 2355 fs_manager.cc:429] Opened local filesystem: /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ChurnyElections_WithDuplicateKeys.1520885401735786-212/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ChurnyElections_WithDuplicateKeys.1520885401735786-212/raft_consensus-itest-cluster/ts-0/wal uuid: "71f80b7649e64a2bafe2eee9b67b14c1" format_stamp: "Formatted at 2018-03-12 20:11:02 on dist-test-slave-dist-test-slave-f27p" I0312 20:11:02.095494 2355 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ChurnyElections_WithDuplicateKeys.1520885401735786-212/raft_consensus-itest-cluster/ts-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0312 20:11:02.319006 2355 env_posix.cc:1620] Not raising this process' running threads per effective uid limit of 60284; it is already as high as it can go I0312 20:11:02.320847 2355 ts_tablet_manager.cc:349] Loaded tablet metadata (0 live tablets) I0312 20:11:02.322019 2355 tablet_server_main.cc:84] Starting tablet server... I0312 20:11:02.502179 2355 rpc_server.cc:201] RPC server started. Bound to: 127.0.53.1:52002 I0312 20:11:02.503532 2355 webserver.cc:173] Starting webserver on 127.0.53.1:0 I0312 20:11:02.503681 2355 webserver.cc:184] Document root disabled I0312 20:11:02.505197 2355 webserver.cc:311] Webserver started. Bound to: http://127.0.53.1:38840/ I0312 20:11:02.506153 2355 server_base.cc:607] Dumped server information to /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ChurnyElections_WithDuplicateKeys.1520885401735786-212/raft_consensus-itest-cluster/ts-0/data/info.pb I0312 20:11:02.506814 212 external_mini_cluster.cc:867] Started /tmp/dist-test-task3bwPeE/build/tsan/bin/kudu-tserver as pid 2355 I0312 20:11:02.508184 212 external_mini_cluster.cc:805] Running /tmp/dist-test-task3bwPeE/build/tsan/bin/kudu-tserver /tmp/dist-test-task3bwPeE/build/tsan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ChurnyElections_WithDuplicateKeys.1520885401735786-212/raft_consensus-itest-cluster/ts-1/wal --fs_data_dirs=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ChurnyElections_WithDuplicateKeys.1520885401735786-212/raft_consensus-itest-cluster/ts-1/data --block_manager=log --rpc_bind_addresses=127.0.53.2:0 --local_ip_for_outbound_sockets=127.0.53.2 --webserver_interface=127.0.53.2 --webserver_port=0 --tserver_master_addrs=127.0.53.62:46983 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ChurnyElections_WithDuplicateKeys.1520885401735786-212/raft_consensus-itest-cluster/ts-1/logs --server_dump_info_path=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ChurnyElections_WithDuplicateKeys.1520885401735786-212/raft_consensus-itest-cluster/ts-1/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_heartbeat_interval_ms=5 with env {} I0312 20:11:02.518707 2488 process_memory.cc:182] Process hard memory limit is 11.777631 GB I0312 20:11:02.518915 2488 process_memory.cc:184] Process soft memory limit is 9.422105 GB I0312 20:11:02.519050 2488 process_memory.cc:187] Process memory pressure threshold is 7.066578 GB I0312 20:11:02.518726 2355 tablet_server_main.cc:87] Tablet server successfully started. I0312 20:11:02.576408 2487 heartbeater.cc:346] Connected to a master server at 127.0.53.62:46983 I0312 20:11:02.576685 2487 heartbeater.cc:426] Registering TS with master... I0312 20:11:02.577260 2487 heartbeater.cc:475] Master 127.0.53.62:46983 requested a full tablet report, sending... I0312 20:11:02.579294 2321 ts_manager.cc:81] Registered new tserver with Master: 71f80b7649e64a2bafe2eee9b67b14c1 (127.0.53.1:52002) I0312 20:11:02.581846 2321 master_service.cc:243] Signed X509 certificate for tserver {username='slave'} at 127.0.53.1:48719 WARNING: Logging before InitGoogleLogging() is written to STDERR W0312 20:11:02.647995 2489 flags.cc:412] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0312 20:11:02.648311 2489 flags.cc:412] Enabled unsafe flag: --never_fsync=true W0312 20:11:02.662426 2489 flags.cc:412] Enabled experimental flag: --ipki_server_key_size=1024 W0312 20:11:02.662714 2489 flags.cc:412] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.53.2 I0312 20:11:02.687615 2489 tablet_server_main.cc:72] Tablet server non-default flags: --raft_heartbeat_interval_ms=5 --fs_data_dirs=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ChurnyElections_WithDuplicateKeys.1520885401735786-212/raft_consensus-itest-cluster/ts-1/data --fs_wal_dir=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ChurnyElections_WithDuplicateKeys.1520885401735786-212/raft_consensus-itest-cluster/ts-1/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.0.53.2:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ChurnyElections_WithDuplicateKeys.1520885401735786-212/raft_consensus-itest-cluster/ts-1/data/info.pb --webserver_interface=127.0.53.2 --webserver_port=0 --tserver_master_addrs=127.0.53.62:46983 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.0.53.2 --log_dir=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ChurnyElections_WithDuplicateKeys.1520885401735786-212/raft_consensus-itest-cluster/ts-1/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.7.0-SNAPSHOT revision 13e10c63c46de30ac85f34b543cdd6fb44881f61 build type FASTDEBUG built by None at 12 Mar 2018 20:06:40 UTC on jenkins-slave-vn0g build id 12388 TSAN enabled I0312 20:11:02.709620 2489 tablet_server_main.cc:79] Initializing tablet server... I0312 20:11:02.711349 2489 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 1098276us I0312 20:11:02.711763 2489 fs_manager.cc:260] Metadata directory not provided I0312 20:11:02.711874 2489 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0312 20:11:02.712055 2489 server_base.cc:434] Could not load existing FS layout: Not found: /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ChurnyElections_WithDuplicateKeys.1520885401735786-212/raft_consensus-itest-cluster/ts-1/data/instance: No such file or directory (error 2) I0312 20:11:02.712158 2489 server_base.cc:435] Attempting to create new FS layout instead I0312 20:11:02.719529 2489 fs_manager.cc:596] Generated new instance metadata in path /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ChurnyElections_WithDuplicateKeys.1520885401735786-212/raft_consensus-itest-cluster/ts-1/data/instance: uuid: "75cbb9a95bd343cbad786fcb50d74c94" format_stamp: "Formatted at 2018-03-12 20:11:02 on dist-test-slave-dist-test-slave-f27p" I0312 20:11:02.720192 2489 fs_manager.cc:596] Generated new instance metadata in path /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ChurnyElections_WithDuplicateKeys.1520885401735786-212/raft_consensus-itest-cluster/ts-1/wal/instance: uuid: "75cbb9a95bd343cbad786fcb50d74c94" format_stamp: "Formatted at 2018-03-12 20:11:02 on dist-test-slave-dist-test-slave-f27p" I0312 20:11:02.738251 2489 fs_manager.cc:496] Time spent creating directory manager: real 0.018s user 0.008s sys 0.012s I0312 20:11:02.739930 2489 env_posix.cc:1620] Not raising this process' open files per process limit of 65536; it is already as high as it can go I0312 20:11:02.740314 2489 file_cache.cc:470] Constructed file cache lbm with capacity 26214 I0312 20:11:02.751653 2489 fs_manager.cc:418] Time spent opening block manager: real 0.003s user 0.004s sys 0.004s I0312 20:11:02.751809 2489 fs_manager.cc:429] Opened local filesystem: /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ChurnyElections_WithDuplicateKeys.1520885401735786-212/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ChurnyElections_WithDuplicateKeys.1520885401735786-212/raft_consensus-itest-cluster/ts-1/wal uuid: "75cbb9a95bd343cbad786fcb50d74c94" format_stamp: "Formatted at 2018-03-12 20:11:02 on dist-test-slave-dist-test-slave-f27p" I0312 20:11:02.752090 2489 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ChurnyElections_WithDuplicateKeys.1520885401735786-212/raft_consensus-itest-cluster/ts-1/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0312 20:11:02.975375 2489 env_posix.cc:1620] Not raising this process' running threads per effective uid limit of 60284; it is already as high as it can go I0312 20:11:02.976665 2489 ts_tablet_manager.cc:349] Loaded tablet metadata (0 live tablets) I0312 20:11:02.977849 2489 tablet_server_main.cc:84] Starting tablet server... I0312 20:11:03.082698 2489 rpc_server.cc:201] RPC server started. Bound to: 127.0.53.2:59812 I0312 20:11:03.083600 2489 webserver.cc:173] Starting webserver on 127.0.53.2:0 I0312 20:11:03.083711 2489 webserver.cc:184] Document root disabled I0312 20:11:03.085162 2489 webserver.cc:311] Webserver started. Bound to: http://127.0.53.2:39157/ I0312 20:11:03.085916 2489 server_base.cc:607] Dumped server information to /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ChurnyElections_WithDuplicateKeys.1520885401735786-212/raft_consensus-itest-cluster/ts-1/data/info.pb I0312 20:11:03.095033 212 external_mini_cluster.cc:867] Started /tmp/dist-test-task3bwPeE/build/tsan/bin/kudu-tserver as pid 2489 I0312 20:11:03.096251 212 external_mini_cluster.cc:805] Running /tmp/dist-test-task3bwPeE/build/tsan/bin/kudu-tserver /tmp/dist-test-task3bwPeE/build/tsan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ChurnyElections_WithDuplicateKeys.1520885401735786-212/raft_consensus-itest-cluster/ts-2/wal --fs_data_dirs=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ChurnyElections_WithDuplicateKeys.1520885401735786-212/raft_consensus-itest-cluster/ts-2/data --block_manager=log --rpc_bind_addresses=127.0.53.3:0 --local_ip_for_outbound_sockets=127.0.53.3 --webserver_interface=127.0.53.3 --webserver_port=0 --tserver_master_addrs=127.0.53.62:46983 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ChurnyElections_WithDuplicateKeys.1520885401735786-212/raft_consensus-itest-cluster/ts-2/logs --server_dump_info_path=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ChurnyElections_WithDuplicateKeys.1520885401735786-212/raft_consensus-itest-cluster/ts-2/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_heartbeat_interval_ms=5 with env {} I0312 20:11:03.094688 2613 process_memory.cc:182] Process hard memory limit is 11.777631 GB I0312 20:11:03.102149 2613 process_memory.cc:184] Process soft memory limit is 9.422105 GB I0312 20:11:03.102298 2613 process_memory.cc:187] Process memory pressure threshold is 7.066578 GB I0312 20:11:03.121294 2612 heartbeater.cc:346] Connected to a master server at 127.0.53.62:46983 I0312 20:11:03.121582 2612 heartbeater.cc:426] Registering TS with master... I0312 20:11:03.122117 2612 heartbeater.cc:475] Master 127.0.53.62:46983 requested a full tablet report, sending... I0312 20:11:03.123867 2321 ts_manager.cc:81] Registered new tserver with Master: 75cbb9a95bd343cbad786fcb50d74c94 (127.0.53.2:59812) I0312 20:11:03.125620 2321 master_service.cc:243] Signed X509 certificate for tserver {username='slave'} at 127.0.53.2:57719 I0312 20:11:03.101925 2489 tablet_server_main.cc:87] Tablet server successfully started. WARNING: Logging before InitGoogleLogging() is written to STDERR W0312 20:11:03.247149 2616 flags.cc:412] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0312 20:11:03.247604 2616 flags.cc:412] Enabled unsafe flag: --never_fsync=true W0312 20:11:03.270846 2616 flags.cc:412] Enabled experimental flag: --ipki_server_key_size=1024 W0312 20:11:03.271280 2616 flags.cc:412] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.53.3 I0312 20:11:03.301095 2616 tablet_server_main.cc:72] Tablet server non-default flags: --raft_heartbeat_interval_ms=5 --fs_data_dirs=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ChurnyElections_WithDuplicateKeys.1520885401735786-212/raft_consensus-itest-cluster/ts-2/data --fs_wal_dir=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ChurnyElections_WithDuplicateKeys.1520885401735786-212/raft_consensus-itest-cluster/ts-2/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.0.53.3:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ChurnyElections_WithDuplicateKeys.1520885401735786-212/raft_consensus-itest-cluster/ts-2/data/info.pb --webserver_interface=127.0.53.3 --webserver_port=0 --tserver_master_addrs=127.0.53.62:46983 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.0.53.3 --log_dir=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ChurnyElections_WithDuplicateKeys.1520885401735786-212/raft_consensus-itest-cluster/ts-2/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.7.0-SNAPSHOT revision 13e10c63c46de30ac85f34b543cdd6fb44881f61 build type FASTDEBUG built by None at 12 Mar 2018 20:06:40 UTC on jenkins-slave-vn0g build id 12388 TSAN enabled I0312 20:11:03.315754 2616 tablet_server_main.cc:79] Initializing tablet server... I0312 20:11:03.318053 2616 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 1098776us I0312 20:11:03.318650 2616 fs_manager.cc:260] Metadata directory not provided I0312 20:11:03.318828 2616 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0312 20:11:03.319108 2616 server_base.cc:434] Could not load existing FS layout: Not found: /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ChurnyElections_WithDuplicateKeys.1520885401735786-212/raft_consensus-itest-cluster/ts-2/data/instance: No such file or directory (error 2) I0312 20:11:03.319267 2616 server_base.cc:435] Attempting to create new FS layout instead I0312 20:11:03.331295 2616 fs_manager.cc:596] Generated new instance metadata in path /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ChurnyElections_WithDuplicateKeys.1520885401735786-212/raft_consensus-itest-cluster/ts-2/data/instance: uuid: "a5b79d7057a340678b551ae411266cda" format_stamp: "Formatted at 2018-03-12 20:11:03 on dist-test-slave-dist-test-slave-f27p" I0312 20:11:03.332247 2616 fs_manager.cc:596] Generated new instance metadata in path /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ChurnyElections_WithDuplicateKeys.1520885401735786-212/raft_consensus-itest-cluster/ts-2/wal/instance: uuid: "a5b79d7057a340678b551ae411266cda" format_stamp: "Formatted at 2018-03-12 20:11:03 on dist-test-slave-dist-test-slave-f27p" I0312 20:11:03.344799 2616 fs_manager.cc:496] Time spent creating directory manager: real 0.012s user 0.004s sys 0.008s I0312 20:11:03.346494 2616 env_posix.cc:1620] Not raising this process' open files per process limit of 65536; it is already as high as it can go I0312 20:11:03.346877 2616 file_cache.cc:470] Constructed file cache lbm with capacity 26214 I0312 20:11:03.357224 2616 fs_manager.cc:418] Time spent opening block manager: real 0.002s user 0.000s sys 0.000s I0312 20:11:03.357376 2616 fs_manager.cc:429] Opened local filesystem: /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ChurnyElections_WithDuplicateKeys.1520885401735786-212/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ChurnyElections_WithDuplicateKeys.1520885401735786-212/raft_consensus-itest-cluster/ts-2/wal uuid: "a5b79d7057a340678b551ae411266cda" format_stamp: "Formatted at 2018-03-12 20:11:03 on dist-test-slave-dist-test-slave-f27p" I0312 20:11:03.357637 2616 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ChurnyElections_WithDuplicateKeys.1520885401735786-212/raft_consensus-itest-cluster/ts-2/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0312 20:11:03.532786 2616 env_posix.cc:1620] Not raising this process' running threads per effective uid limit of 60284; it is already as high as it can go I0312 20:11:03.534060 2616 ts_tablet_manager.cc:349] Loaded tablet metadata (0 live tablets) I0312 20:11:03.535043 2616 tablet_server_main.cc:84] Starting tablet server... I0312 20:11:03.584987 2487 heartbeater.cc:467] Master 127.0.53.62:46983 was elected leader, sending a full tablet report... I0312 20:11:03.622634 2616 rpc_server.cc:201] RPC server started. Bound to: 127.0.53.3:55527 I0312 20:11:03.623492 2616 webserver.cc:173] Starting webserver on 127.0.53.3:0 I0312 20:11:03.623596 2616 webserver.cc:184] Document root disabled I0312 20:11:03.624789 2616 webserver.cc:311] Webserver started. Bound to: http://127.0.53.3:55521/ I0312 20:11:03.625514 2616 server_base.cc:607] Dumped server information to /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ChurnyElections_WithDuplicateKeys.1520885401735786-212/raft_consensus-itest-cluster/ts-2/data/info.pb I0312 20:11:03.628057 2616 tablet_server_main.cc:87] Tablet server successfully started. I0312 20:11:03.632313 212 external_mini_cluster.cc:867] Started /tmp/dist-test-task3bwPeE/build/tsan/bin/kudu-tserver as pid 2616 I0312 20:11:03.652926 2737 heartbeater.cc:346] Connected to a master server at 127.0.53.62:46983 I0312 20:11:03.653229 2737 heartbeater.cc:426] Registering TS with master... I0312 20:11:03.653796 2737 heartbeater.cc:475] Master 127.0.53.62:46983 requested a full tablet report, sending... I0312 20:11:03.655304 2321 ts_manager.cc:81] Registered new tserver with Master: a5b79d7057a340678b551ae411266cda (127.0.53.3:55527) I0312 20:11:03.656522 2321 master_service.cc:243] Signed X509 certificate for tserver {username='slave'} at 127.0.53.3:53606 I0312 20:11:03.658380 212 external_mini_cluster.cc:498] 3 TS(s) registered with all masters I0312 20:11:03.630074 2738 process_memory.cc:182] Process hard memory limit is 11.777631 GB I0312 20:11:03.659533 2738 process_memory.cc:184] Process soft memory limit is 9.422105 GB I0312 20:11:03.659611 2738 process_memory.cc:187] Process memory pressure threshold is 7.066578 GB I0312 20:11:03.661264 212 test_util.cc:192] Using random seed: -1110558023 I0312 20:11:03.677798 2321 catalog_manager.cc:1294] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47690: name: "test-workload" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 3 split_rows_range_bounds { } partition_schema { range_schema { columns { name: "key" } } } W0312 20:11:03.686137 2321 catalog_manager.cc:1450] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a replica failure: 4 tablet servers are needed, 3 are alive. Consider bringing up additional tablet server(s) or running both the masters and all tablet servers with --raft_prepare_replacement_before_eviction=false flag (not recommended). I0312 20:11:03.719444 2549 tablet_service.cc:755] Processing CreateTablet for tablet c1432f133b26467b949ad48a19496e3a (table=test-workload [id=990c05cc99f043ccaab4ed2caf8ae553]), partition=RANGE (key) PARTITION UNBOUNDED I0312 20:11:03.720479 2549 data_dirs.cc:931] Could only allocate 1 dirs of requested 3 for tablet c1432f133b26467b949ad48a19496e3a. 1 dirs total, 0 dirs full, 0 dirs failed I0312 20:11:03.721132 2424 tablet_service.cc:755] Processing CreateTablet for tablet c1432f133b26467b949ad48a19496e3a (table=test-workload [id=990c05cc99f043ccaab4ed2caf8ae553]), partition=RANGE (key) PARTITION UNBOUNDED I0312 20:11:03.721895 2424 data_dirs.cc:931] Could only allocate 1 dirs of requested 3 for tablet c1432f133b26467b949ad48a19496e3a. 1 dirs total, 0 dirs full, 0 dirs failed I0312 20:11:03.727439 2674 tablet_service.cc:755] Processing CreateTablet for tablet c1432f133b26467b949ad48a19496e3a (table=test-workload [id=990c05cc99f043ccaab4ed2caf8ae553]), partition=RANGE (key) PARTITION UNBOUNDED I0312 20:11:03.728301 2674 data_dirs.cc:931] Could only allocate 1 dirs of requested 3 for tablet c1432f133b26467b949ad48a19496e3a. 1 dirs total, 0 dirs full, 0 dirs failed I0312 20:11:03.733137 2424 ts_tablet_manager.cc:1163] T c1432f133b26467b949ad48a19496e3a P 71f80b7649e64a2bafe2eee9b67b14c1: Registered tablet (data state: TABLET_DATA_READY) I0312 20:11:03.736304 2549 ts_tablet_manager.cc:1163] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94: Registered tablet (data state: TABLET_DATA_READY) I0312 20:11:03.738255 2759 ts_tablet_manager.cc:1018] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94: Bootstrapping tablet I0312 20:11:03.738672 2759 tablet_bootstrap.cc:436] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94: Bootstrap starting. I0312 20:11:03.739761 2758 ts_tablet_manager.cc:1018] T c1432f133b26467b949ad48a19496e3a P 71f80b7649e64a2bafe2eee9b67b14c1: Bootstrapping tablet I0312 20:11:03.740834 2758 tablet_bootstrap.cc:436] T c1432f133b26467b949ad48a19496e3a P 71f80b7649e64a2bafe2eee9b67b14c1: Bootstrap starting. I0312 20:11:03.741451 2759 tablet_bootstrap.cc:581] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94: No blocks or log segments found. Creating new log. I0312 20:11:03.743075 2759 log.cc:520] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94: Log is configured to *not* fsync() on all Append() calls I0312 20:11:03.744276 2758 tablet_bootstrap.cc:581] T c1432f133b26467b949ad48a19496e3a P 71f80b7649e64a2bafe2eee9b67b14c1: No blocks or log segments found. Creating new log. I0312 20:11:03.745882 2758 log.cc:520] T c1432f133b26467b949ad48a19496e3a P 71f80b7649e64a2bafe2eee9b67b14c1: Log is configured to *not* fsync() on all Append() calls I0312 20:11:03.751237 2674 ts_tablet_manager.cc:1163] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda: Registered tablet (data state: TABLET_DATA_READY) I0312 20:11:03.751363 2758 tablet_bootstrap.cc:436] T c1432f133b26467b949ad48a19496e3a P 71f80b7649e64a2bafe2eee9b67b14c1: No bootstrap required, opened a new log I0312 20:11:03.751755 2759 tablet_bootstrap.cc:436] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94: No bootstrap required, opened a new log I0312 20:11:03.751756 2758 ts_tablet_manager.cc:1035] T c1432f133b26467b949ad48a19496e3a P 71f80b7649e64a2bafe2eee9b67b14c1: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.004s I0312 20:11:03.752182 2759 ts_tablet_manager.cc:1035] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.008s I0312 20:11:03.766834 2759 raft_consensus.cc:304] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71f80b7649e64a2bafe2eee9b67b14c1" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 52002 } } peers { permanent_uuid: "75cbb9a95bd343cbad786fcb50d74c94" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 59812 } } peers { permanent_uuid: "a5b79d7057a340678b551ae411266cda" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 55527 } } I0312 20:11:03.767612 2759 raft_consensus.cc:330] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0312 20:11:03.767109 2758 raft_consensus.cc:304] T c1432f133b26467b949ad48a19496e3a P 71f80b7649e64a2bafe2eee9b67b14c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71f80b7649e64a2bafe2eee9b67b14c1" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 52002 } } peers { permanent_uuid: "75cbb9a95bd343cbad786fcb50d74c94" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 59812 } } peers { permanent_uuid: "a5b79d7057a340678b551ae411266cda" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 55527 } } I0312 20:11:03.767814 2758 raft_consensus.cc:330] T c1432f133b26467b949ad48a19496e3a P 71f80b7649e64a2bafe2eee9b67b14c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0312 20:11:03.767832 2759 raft_consensus.cc:605] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75cbb9a95bd343cbad786fcb50d74c94, State: Initialized, Role: FOLLOWER I0312 20:11:03.768525 2758 raft_consensus.cc:605] T c1432f133b26467b949ad48a19496e3a P 71f80b7649e64a2bafe2eee9b67b14c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71f80b7649e64a2bafe2eee9b67b14c1, State: Initialized, Role: FOLLOWER I0312 20:11:03.768870 2762 ts_tablet_manager.cc:1018] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda: Bootstrapping tablet I0312 20:11:03.768429 2759 consensus_queue.cc:219] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71f80b7649e64a2bafe2eee9b67b14c1" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 52002 } } peers { permanent_uuid: "75cbb9a95bd343cbad786fcb50d74c94" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 59812 } } peers { permanent_uuid: "a5b79d7057a340678b551ae411266cda" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 55527 } } I0312 20:11:03.769215 2758 consensus_queue.cc:219] T c1432f133b26467b949ad48a19496e3a P 71f80b7649e64a2bafe2eee9b67b14c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71f80b7649e64a2bafe2eee9b67b14c1" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 52002 } } peers { permanent_uuid: "75cbb9a95bd343cbad786fcb50d74c94" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 59812 } } peers { permanent_uuid: "a5b79d7057a340678b551ae411266cda" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 55527 } } I0312 20:11:03.770761 2762 tablet_bootstrap.cc:436] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda: Bootstrap starting. I0312 20:11:03.773792 2762 tablet_bootstrap.cc:581] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda: No blocks or log segments found. Creating new log. I0312 20:11:03.775501 2762 log.cc:520] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda: Log is configured to *not* fsync() on all Append() calls I0312 20:11:03.784551 2767 raft_consensus.cc:436] T c1432f133b26467b949ad48a19496e3a P 71f80b7649e64a2bafe2eee9b67b14c1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0312 20:11:03.785182 2758 ts_tablet_manager.cc:1063] T c1432f133b26467b949ad48a19496e3a P 71f80b7649e64a2bafe2eee9b67b14c1: Time spent starting tablet: real 0.033s user 0.020s sys 0.000s I0312 20:11:03.786770 2612 heartbeater.cc:467] Master 127.0.53.62:46983 was elected leader, sending a full tablet report... I0312 20:11:03.786929 2767 raft_consensus.cc:2700] T c1432f133b26467b949ad48a19496e3a P 71f80b7649e64a2bafe2eee9b67b14c1: Snoozing failure detection for 0.020s (starting election) I0312 20:11:03.787204 2767 raft_consensus.cc:458] T c1432f133b26467b949ad48a19496e3a P 71f80b7649e64a2bafe2eee9b67b14c1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71f80b7649e64a2bafe2eee9b67b14c1" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 52002 } } peers { permanent_uuid: "75cbb9a95bd343cbad786fcb50d74c94" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 59812 } } peers { permanent_uuid: "a5b79d7057a340678b551ae411266cda" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 55527 } } I0312 20:11:03.789767 2765 raft_consensus.cc:436] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0312 20:11:03.790051 2765 raft_consensus.cc:2700] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94: Snoozing failure detection for 0.019s (starting election) I0312 20:11:03.790310 2765 raft_consensus.cc:458] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71f80b7649e64a2bafe2eee9b67b14c1" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 52002 } } peers { permanent_uuid: "75cbb9a95bd343cbad786fcb50d74c94" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 59812 } } peers { permanent_uuid: "a5b79d7057a340678b551ae411266cda" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 55527 } } I0312 20:11:03.791235 2765 leader_election.cc:231] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 71f80b7649e64a2bafe2eee9b67b14c1 I0312 20:11:03.791633 2765 leader_election.cc:231] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer a5b79d7057a340678b551ae411266cda I0312 20:11:03.793236 2767 leader_election.cc:231] T c1432f133b26467b949ad48a19496e3a P 71f80b7649e64a2bafe2eee9b67b14c1 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 75cbb9a95bd343cbad786fcb50d74c94 I0312 20:11:03.801555 2759 ts_tablet_manager.cc:1063] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94: Time spent starting tablet: real 0.049s user 0.020s sys 0.000s I0312 20:11:03.802472 2762 tablet_bootstrap.cc:436] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda: No bootstrap required, opened a new log I0312 20:11:03.802850 2762 ts_tablet_manager.cc:1035] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda: Time spent bootstrapping tablet: real 0.032s user 0.008s sys 0.024s I0312 20:11:03.803503 2767 leader_election.cc:231] T c1432f133b26467b949ad48a19496e3a P 71f80b7649e64a2bafe2eee9b67b14c1 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer a5b79d7057a340678b551ae411266cda I0312 20:11:03.808274 2767 raft_consensus.cc:436] T c1432f133b26467b949ad48a19496e3a P 71f80b7649e64a2bafe2eee9b67b14c1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0312 20:11:03.808514 2767 raft_consensus.cc:2700] T c1432f133b26467b949ad48a19496e3a P 71f80b7649e64a2bafe2eee9b67b14c1: Snoozing failure detection for 0.018s (starting election) I0312 20:11:03.808766 2767 raft_consensus.cc:458] T c1432f133b26467b949ad48a19496e3a P 71f80b7649e64a2bafe2eee9b67b14c1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71f80b7649e64a2bafe2eee9b67b14c1" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 52002 } } peers { permanent_uuid: "75cbb9a95bd343cbad786fcb50d74c94" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 59812 } } peers { permanent_uuid: "a5b79d7057a340678b551ae411266cda" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 55527 } } I0312 20:11:03.809624 2767 leader_election.cc:231] T c1432f133b26467b949ad48a19496e3a P 71f80b7649e64a2bafe2eee9b67b14c1 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 75cbb9a95bd343cbad786fcb50d74c94 I0312 20:11:03.809896 2767 leader_election.cc:231] T c1432f133b26467b949ad48a19496e3a P 71f80b7649e64a2bafe2eee9b67b14c1 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer a5b79d7057a340678b551ae411266cda I0312 20:11:03.811825 2763 raft_consensus.cc:436] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) W0312 20:11:03.814725 2379 leader_election.cc:287] T c1432f133b26467b949ad48a19496e3a P 71f80b7649e64a2bafe2eee9b67b14c1 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 75cbb9a95bd343cbad786fcb50d74c94: Timed out: connection negotiation to 127.0.53.2:59812 for RPC RequestConsensusVote timed out after 0.020s (ON_OUTBOUND_QUEUE) I0312 20:11:03.816296 2763 raft_consensus.cc:2700] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94: Snoozing failure detection for 0.019s (starting election) I0312 20:11:03.816629 2763 raft_consensus.cc:458] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71f80b7649e64a2bafe2eee9b67b14c1" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 52002 } } peers { permanent_uuid: "75cbb9a95bd343cbad786fcb50d74c94" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 59812 } } peers { permanent_uuid: "a5b79d7057a340678b551ae411266cda" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 55527 } } W0312 20:11:03.812232 2500 leader_election.cc:287] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer a5b79d7057a340678b551ae411266cda: Timed out: connection negotiation to 127.0.53.3:55527 for RPC RequestConsensusVote timed out after 0.019s (ON_OUTBOUND_QUEUE) W0312 20:11:03.817293 2502 leader_election.cc:287] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 71f80b7649e64a2bafe2eee9b67b14c1: Timed out: connection negotiation to 127.0.53.1:52002 for RPC RequestConsensusVote timed out after 0.019s (ON_OUTBOUND_QUEUE) I0312 20:11:03.818218 2502 leader_election.cc:258] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. I0312 20:11:03.819267 2763 leader_election.cc:231] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 71f80b7649e64a2bafe2eee9b67b14c1 I0312 20:11:03.819284 2765 raft_consensus.cc:2700] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94: Snoozing failure detection for 0.016s (election complete) I0312 20:11:03.819705 2765 raft_consensus.cc:2455] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority I0312 20:11:03.820209 2763 leader_election.cc:231] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer a5b79d7057a340678b551ae411266cda I0312 20:11:03.820521 2762 raft_consensus.cc:304] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71f80b7649e64a2bafe2eee9b67b14c1" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 52002 } } peers { permanent_uuid: "75cbb9a95bd343cbad786fcb50d74c94" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 59812 } } peers { permanent_uuid: "a5b79d7057a340678b551ae411266cda" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 55527 } } I0312 20:11:03.821511 2762 raft_consensus.cc:330] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0312 20:11:03.821702 2762 raft_consensus.cc:605] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5b79d7057a340678b551ae411266cda, State: Initialized, Role: FOLLOWER I0312 20:11:03.822144 2762 consensus_queue.cc:219] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71f80b7649e64a2bafe2eee9b67b14c1" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 52002 } } peers { permanent_uuid: "75cbb9a95bd343cbad786fcb50d74c94" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 59812 } } peers { permanent_uuid: "a5b79d7057a340678b551ae411266cda" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 55527 } } I0312 20:11:03.823415 2694 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "c1432f133b26467b949ad48a19496e3a" candidate_uuid: "71f80b7649e64a2bafe2eee9b67b14c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5b79d7057a340678b551ae411266cda" is_pre_election: true I0312 20:11:03.823997 2694 raft_consensus.cc:2700] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda: Snoozing failure detection for 0.018s (vote granted) I0312 20:11:03.824216 2694 raft_consensus.cc:2215] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71f80b7649e64a2bafe2eee9b67b14c1 in term 0. W0312 20:11:03.824635 2375 leader_election.cc:287] T c1432f133b26467b949ad48a19496e3a P 71f80b7649e64a2bafe2eee9b67b14c1 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer a5b79d7057a340678b551ae411266cda: Timed out: RequestConsensusVote RPC to 127.0.53.3:55527 timed out after 0.020s (SENT) I0312 20:11:03.824873 2375 leader_election.cc:258] T c1432f133b26467b949ad48a19496e3a P 71f80b7649e64a2bafe2eee9b67b14c1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. I0312 20:11:03.825305 2693 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "c1432f133b26467b949ad48a19496e3a" candidate_uuid: "71f80b7649e64a2bafe2eee9b67b14c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5b79d7057a340678b551ae411266cda" is_pre_election: true I0312 20:11:03.825686 2767 raft_consensus.cc:2700] T c1432f133b26467b949ad48a19496e3a P 71f80b7649e64a2bafe2eee9b67b14c1: Snoozing failure detection for 0.018s (election complete) I0312 20:11:03.825742 2693 raft_consensus.cc:2700] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda: Snoozing failure detection for 0.020s (vote granted) I0312 20:11:03.825881 2767 raft_consensus.cc:2455] T c1432f133b26467b949ad48a19496e3a P 71f80b7649e64a2bafe2eee9b67b14c1 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority I0312 20:11:03.825945 2693 raft_consensus.cc:2215] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71f80b7649e64a2bafe2eee9b67b14c1 in term 0. I0312 20:11:03.826324 2444 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "c1432f133b26467b949ad48a19496e3a" candidate_uuid: "75cbb9a95bd343cbad786fcb50d74c94" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71f80b7649e64a2bafe2eee9b67b14c1" is_pre_election: true I0312 20:11:03.826859 2444 raft_consensus.cc:2700] T c1432f133b26467b949ad48a19496e3a P 71f80b7649e64a2bafe2eee9b67b14c1: Snoozing failure detection for 0.032s (vote granted) I0312 20:11:03.827106 2444 raft_consensus.cc:2215] T c1432f133b26467b949ad48a19496e3a P 71f80b7649e64a2bafe2eee9b67b14c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 75cbb9a95bd343cbad786fcb50d74c94 in term 0. I0312 20:11:03.826849 2375 leader_election.cc:363] T c1432f133b26467b949ad48a19496e3a P 71f80b7649e64a2bafe2eee9b67b14c1 [CANDIDATE]: Term 1 pre-election: Vote granted by peer a5b79d7057a340678b551ae411266cda I0312 20:11:03.827599 2375 leader_election.cc:258] T c1432f133b26467b949ad48a19496e3a P 71f80b7649e64a2bafe2eee9b67b14c1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. I0312 20:11:03.827965 2767 raft_consensus.cc:2700] T c1432f133b26467b949ad48a19496e3a P 71f80b7649e64a2bafe2eee9b67b14c1: Snoozing failure detection for 0.028s (election complete) I0312 20:11:03.828138 2737 heartbeater.cc:467] Master 127.0.53.62:46983 was elected leader, sending a full tablet report... W0312 20:11:03.828359 2379 leader_election.cc:287] T c1432f133b26467b949ad48a19496e3a P 71f80b7649e64a2bafe2eee9b67b14c1 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 75cbb9a95bd343cbad786fcb50d74c94: Timed out: connection negotiation to 127.0.53.2:59812 for RPC RequestConsensusVote timed out after 0.018s (ON_OUTBOUND_QUEUE) I0312 20:11:03.828168 2767 raft_consensus.cc:2507] T c1432f133b26467b949ad48a19496e3a P 71f80b7649e64a2bafe2eee9b67b14c1 [term 0 FOLLOWER]: Leader pre-election won for term 1 I0312 20:11:03.828604 2767 raft_consensus.cc:436] T c1432f133b26467b949ad48a19496e3a P 71f80b7649e64a2bafe2eee9b67b14c1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0312 20:11:03.828795 2767 raft_consensus.cc:2700] T c1432f133b26467b949ad48a19496e3a P 71f80b7649e64a2bafe2eee9b67b14c1: Snoozing failure detection for 0.022s (starting election) I0312 20:11:03.829047 2767 raft_consensus.cc:2752] T c1432f133b26467b949ad48a19496e3a P 71f80b7649e64a2bafe2eee9b67b14c1 [term 0 FOLLOWER]: Advancing to term 1 I0312 20:11:03.829334 2762 ts_tablet_manager.cc:1063] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda: Time spent starting tablet: real 0.026s user 0.020s sys 0.000s I0312 20:11:03.828966 2502 leader_election.cc:363] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [CANDIDATE]: Term 1 pre-election: Vote granted by peer 71f80b7649e64a2bafe2eee9b67b14c1 I0312 20:11:03.829643 2502 leader_election.cc:258] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. I0312 20:11:03.830132 2763 raft_consensus.cc:2700] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94: Snoozing failure detection for 0.017s (election complete) I0312 20:11:03.830365 2763 raft_consensus.cc:2507] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [term 0 FOLLOWER]: Leader pre-election won for term 1 I0312 20:11:03.830580 2763 raft_consensus.cc:436] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0312 20:11:03.830833 2763 raft_consensus.cc:2700] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94: Snoozing failure detection for 0.016s (starting election) I0312 20:11:03.831032 2763 raft_consensus.cc:2752] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [term 0 FOLLOWER]: Advancing to term 1 I0312 20:11:03.833511 2767 raft_consensus.cc:458] T c1432f133b26467b949ad48a19496e3a P 71f80b7649e64a2bafe2eee9b67b14c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71f80b7649e64a2bafe2eee9b67b14c1" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 52002 } } peers { permanent_uuid: "75cbb9a95bd343cbad786fcb50d74c94" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 59812 } } peers { permanent_uuid: "a5b79d7057a340678b551ae411266cda" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 55527 } } I0312 20:11:03.834509 2767 leader_election.cc:231] T c1432f133b26467b949ad48a19496e3a P 71f80b7649e64a2bafe2eee9b67b14c1 [CANDIDATE]: Term 1 election: Requesting vote from peer 75cbb9a95bd343cbad786fcb50d74c94 I0312 20:11:03.834800 2767 leader_election.cc:231] T c1432f133b26467b949ad48a19496e3a P 71f80b7649e64a2bafe2eee9b67b14c1 [CANDIDATE]: Term 1 election: Requesting vote from peer a5b79d7057a340678b551ae411266cda I0312 20:11:03.835764 2693 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "c1432f133b26467b949ad48a19496e3a" candidate_uuid: "71f80b7649e64a2bafe2eee9b67b14c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5b79d7057a340678b551ae411266cda" I0312 20:11:03.836058 2693 raft_consensus.cc:2752] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [term 0 FOLLOWER]: Advancing to term 1 I0312 20:11:03.836235 2693 raft_consensus.cc:2700] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda: Snoozing failure detection for 0.017s (vote granted) I0312 20:11:03.839193 2693 raft_consensus.cc:2215] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71f80b7649e64a2bafe2eee9b67b14c1 in term 1. I0312 20:11:03.839851 2375 leader_election.cc:363] T c1432f133b26467b949ad48a19496e3a P 71f80b7649e64a2bafe2eee9b67b14c1 [CANDIDATE]: Term 1 election: Vote granted by peer a5b79d7057a340678b551ae411266cda I0312 20:11:03.840008 2375 leader_election.cc:258] T c1432f133b26467b949ad48a19496e3a P 71f80b7649e64a2bafe2eee9b67b14c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0312 20:11:03.840075 2693 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "c1432f133b26467b949ad48a19496e3a" candidate_uuid: "75cbb9a95bd343cbad786fcb50d74c94" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5b79d7057a340678b551ae411266cda" is_pre_election: true I0312 20:11:03.840306 2767 raft_consensus.cc:2700] T c1432f133b26467b949ad48a19496e3a P 71f80b7649e64a2bafe2eee9b67b14c1: Snoozing failure detection for 0.031s (election complete) I0312 20:11:03.840483 2767 raft_consensus.cc:2507] T c1432f133b26467b949ad48a19496e3a P 71f80b7649e64a2bafe2eee9b67b14c1 [term 1 FOLLOWER]: Leader election won for term 1 I0312 20:11:03.840636 2693 raft_consensus.cc:2145] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 75cbb9a95bd343cbad786fcb50d74c94 in current term 1: Already voted for candidate 71f80b7649e64a2bafe2eee9b67b14c1 in this term. I0312 20:11:03.840704 2767 raft_consensus.cc:572] T c1432f133b26467b949ad48a19496e3a P 71f80b7649e64a2bafe2eee9b67b14c1 [term 1 LEADER]: Becoming Leader. State: Replica: 71f80b7649e64a2bafe2eee9b67b14c1, State: Running, Role: LEADER I0312 20:11:03.841168 2767 consensus_queue.cc:196] T c1432f133b26467b949ad48a19496e3a P 71f80b7649e64a2bafe2eee9b67b14c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71f80b7649e64a2bafe2eee9b67b14c1" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 52002 } } peers { permanent_uuid: "75cbb9a95bd343cbad786fcb50d74c94" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 59812 } } peers { permanent_uuid: "a5b79d7057a340678b551ae411266cda" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 55527 } } I0312 20:11:03.842288 2500 leader_election.cc:377] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [CANDIDATE]: Term 1 pre-election: Vote denied by peer a5b79d7057a340678b551ae411266cda. Message: Invalid argument: T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 75cbb9a95bd343cbad786fcb50d74c94 in current term 1: Already voted for candidate 71f80b7649e64a2bafe2eee9b67b14c1 in this term. I0312 20:11:03.842766 2763 raft_consensus.cc:458] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71f80b7649e64a2bafe2eee9b67b14c1" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 52002 } } peers { permanent_uuid: "75cbb9a95bd343cbad786fcb50d74c94" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 59812 } } peers { permanent_uuid: "a5b79d7057a340678b551ae411266cda" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 55527 } } I0312 20:11:03.843773 2763 leader_election.cc:231] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [CANDIDATE]: Term 1 election: Requesting vote from peer 71f80b7649e64a2bafe2eee9b67b14c1 I0312 20:11:03.844908 2569 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "c1432f133b26467b949ad48a19496e3a" candidate_uuid: "71f80b7649e64a2bafe2eee9b67b14c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75cbb9a95bd343cbad786fcb50d74c94" I0312 20:11:03.845413 2569 raft_consensus.cc:2145] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 71f80b7649e64a2bafe2eee9b67b14c1 in current term 1: Already voted for candidate 75cbb9a95bd343cbad786fcb50d74c94 in this term. I0312 20:11:03.845273 2444 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "c1432f133b26467b949ad48a19496e3a" candidate_uuid: "75cbb9a95bd343cbad786fcb50d74c94" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71f80b7649e64a2bafe2eee9b67b14c1" I0312 20:11:03.845824 2763 leader_election.cc:231] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [CANDIDATE]: Term 1 election: Requesting vote from peer a5b79d7057a340678b551ae411266cda I0312 20:11:03.846343 2379 leader_election.cc:377] T c1432f133b26467b949ad48a19496e3a P 71f80b7649e64a2bafe2eee9b67b14c1 [CANDIDATE]: Term 1 election: Vote denied by peer 75cbb9a95bd343cbad786fcb50d74c94. Message: Invalid argument: T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 71f80b7649e64a2bafe2eee9b67b14c1 in current term 1: Already voted for candidate 75cbb9a95bd343cbad786fcb50d74c94 in this term. I0312 20:11:03.847048 2693 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "c1432f133b26467b949ad48a19496e3a" candidate_uuid: "75cbb9a95bd343cbad786fcb50d74c94" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5b79d7057a340678b551ae411266cda" I0312 20:11:03.847508 2693 raft_consensus.cc:2145] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 75cbb9a95bd343cbad786fcb50d74c94 in current term 1: Already voted for candidate 71f80b7649e64a2bafe2eee9b67b14c1 in this term. I0312 20:11:03.848456 2500 leader_election.cc:377] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [CANDIDATE]: Term 1 election: Vote denied by peer a5b79d7057a340678b551ae411266cda. Message: Invalid argument: T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 75cbb9a95bd343cbad786fcb50d74c94 in current term 1: Already voted for candidate 71f80b7649e64a2bafe2eee9b67b14c1 in this term. I0312 20:11:03.848848 2763 raft_consensus.cc:436] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0312 20:11:03.849107 2763 raft_consensus.cc:2700] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94: Snoozing failure detection for 0.031s (starting election) I0312 20:11:03.849350 2763 raft_consensus.cc:458] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71f80b7649e64a2bafe2eee9b67b14c1" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 52002 } } peers { permanent_uuid: "75cbb9a95bd343cbad786fcb50d74c94" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 59812 } } peers { permanent_uuid: "a5b79d7057a340678b551ae411266cda" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 55527 } } I0312 20:11:03.852190 2321 catalog_manager.cc:3592] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda reported cstate change: term changed from 0 to 1, leader changed from to 71f80b7649e64a2bafe2eee9b67b14c1 (127.0.53.1). New cstate: current_term: 1 leader_uuid: "71f80b7649e64a2bafe2eee9b67b14c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71f80b7649e64a2bafe2eee9b67b14c1" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 52002 } } peers { permanent_uuid: "75cbb9a95bd343cbad786fcb50d74c94" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 59812 } } peers { permanent_uuid: "a5b79d7057a340678b551ae411266cda" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 55527 } } } W0312 20:11:03.853193 2613 tablet.cc:1679] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94: Can't schedule compaction. Clean time has not been advanced past its initial value. I0312 20:11:03.853827 2763 leader_election.cc:231] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 71f80b7649e64a2bafe2eee9b67b14c1 I0312 20:11:03.855166 2443 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "c1432f133b26467b949ad48a19496e3a" candidate_uuid: "75cbb9a95bd343cbad786fcb50d74c94" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71f80b7649e64a2bafe2eee9b67b14c1" is_pre_election: true I0312 20:11:03.855712 2763 leader_election.cc:231] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer a5b79d7057a340678b551ae411266cda I0312 20:11:03.856860 2693 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "c1432f133b26467b949ad48a19496e3a" candidate_uuid: "75cbb9a95bd343cbad786fcb50d74c94" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5b79d7057a340678b551ae411266cda" is_pre_election: true I0312 20:11:03.857744 2693 raft_consensus.cc:2176] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 75cbb9a95bd343cbad786fcb50d74c94 for term 2 because replica is either leader or believes a valid leader to be alive. I0312 20:11:03.858572 2500 leader_election.cc:377] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [CANDIDATE]: Term 2 pre-election: Vote denied by peer a5b79d7057a340678b551ae411266cda. Message: Invalid argument: T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 75cbb9a95bd343cbad786fcb50d74c94 for term 2 because replica is either leader or believes a valid leader to be alive. W0312 20:11:03.861207 2502 leader_election.cc:287] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 71f80b7649e64a2bafe2eee9b67b14c1: Timed out: RequestConsensusVote RPC to 127.0.53.1:52002 timed out after 0.016s (SENT) I0312 20:11:03.861413 2502 leader_election.cc:258] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. I0312 20:11:03.861493 2444 raft_consensus.cc:2176] T c1432f133b26467b949ad48a19496e3a P 71f80b7649e64a2bafe2eee9b67b14c1 [term 1 LEADER]: Leader election vote request: Denying vote to candidate 75cbb9a95bd343cbad786fcb50d74c94 for term 1 because replica is either leader or believes a valid leader to be alive. W0312 20:11:03.862030 2444 rpcz_store.cc:251] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.53.2:46546 (request call id 3) took 17ms (client timeout 16). W0312 20:11:03.862918 2444 rpcz_store.cc:255] Trace: 0312 20:11:03.845025 (+ 0us) service_pool.cc:163] Inserting onto call queue 0312 20:11:03.845151 (+ 126us) service_pool.cc:222] Handling call 0312 20:11:03.861287 (+ 16136us) spinlock_profiling.cc:261] Waited 10.1 ms on lock 0x7b5000080b18. stack: 00007febb18e2c74 00007febb18e2b70 00007febb03d2314 00007febb941f294 00007febb941f229 00007febb7f88ec0 00007febb7fc9627 00007febb945d56b 00007febb3e10b24 00007febb3e10ab1 00007febb3e109c4 00007febb346cc29 00007febb346c6ec 00007febb346dbcd 00007febb346f946 00007febb346f89a 0312 20:11:03.861855 (+ 568us) spinlock_profiling.cc:261] Waited 5.26 ms on lock 0x7b480003c39c. stack: 00007febb18e2c74 00007febb18e2b70 00007febb03d2314 00007febb941f294 00007febb941f229 00007febb7fc9af3 00007febb945d56b 00007febb3e10b24 00007febb3e10ab1 00007febb3e109c4 00007febb346cc29 00007febb346c6ec 00007febb346dbcd 00007febb346f946 00007febb346f89a 00007febb346f823 0312 20:11:03.861985 (+ 130us) inbound_call.cc:157] Queueing success response Metrics: {"spinlock_wait_cycles":35299072} I0312 20:11:03.863731 2443 raft_consensus.cc:2190] T c1432f133b26467b949ad48a19496e3a P 71f80b7649e64a2bafe2eee9b67b14c1 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate 75cbb9a95bd343cbad786fcb50d74c94 for term 2 because replica is already servicing an update from a current leader or another vote. I0312 20:11:03.863867 2763 raft_consensus.cc:2700] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94: Snoozing failure detection for 0.020s (election complete) I0312 20:11:03.864094 2763 raft_consensus.cc:2455] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority I0312 20:11:03.867696 2502 leader_election.cc:377] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 71f80b7649e64a2bafe2eee9b67b14c1. Message: Service unavailable: T c1432f133b26467b949ad48a19496e3a P 71f80b7649e64a2bafe2eee9b67b14c1 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate 75cbb9a95bd343cbad786fcb50d74c94 for term 2 because replica is already servicing an update from a current leader or another vote. I0312 20:11:03.867879 2502 leader_election.cc:258] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. I0312 20:11:03.868182 2763 raft_consensus.cc:2700] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94: Snoozing failure detection for 0.016s (election complete) I0312 20:11:03.868369 2763 raft_consensus.cc:2455] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority I0312 20:11:06.988322 2795 raft_consensus.cc:436] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 71f80b7649e64a2bafe2eee9b67b14c1) I0312 20:11:06.988677 2795 raft_consensus.cc:2700] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94: Snoozing failure detection for 0.041s (starting election) I0312 20:11:06.989022 2795 raft_consensus.cc:458] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71f80b7649e64a2bafe2eee9b67b14c1" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 52002 } } peers { permanent_uuid: "75cbb9a95bd343cbad786fcb50d74c94" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 59812 } } peers { permanent_uuid: "a5b79d7057a340678b551ae411266cda" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 55527 } } I0312 20:11:07.003242 2795 leader_election.cc:231] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 71f80b7649e64a2bafe2eee9b67b14c1 I0312 20:11:07.003628 2795 leader_election.cc:231] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer a5b79d7057a340678b551ae411266cda I0312 20:11:07.005972 2443 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "c1432f133b26467b949ad48a19496e3a" candidate_uuid: "75cbb9a95bd343cbad786fcb50d74c94" candidate_term: 2 candidate_status { last_received { term: 1 index: 524 } } ignore_live_leader: false dest_uuid: "71f80b7649e64a2bafe2eee9b67b14c1" is_pre_election: true I0312 20:11:07.006526 2443 raft_consensus.cc:2176] T c1432f133b26467b949ad48a19496e3a P 71f80b7649e64a2bafe2eee9b67b14c1 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate 75cbb9a95bd343cbad786fcb50d74c94 for term 2 because replica is either leader or believes a valid leader to be alive. I0312 20:11:07.006527 2693 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "c1432f133b26467b949ad48a19496e3a" candidate_uuid: "75cbb9a95bd343cbad786fcb50d74c94" candidate_term: 2 candidate_status { last_received { term: 1 index: 524 } } ignore_live_leader: false dest_uuid: "a5b79d7057a340678b551ae411266cda" is_pre_election: true I0312 20:11:07.007045 2693 raft_consensus.cc:2176] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 75cbb9a95bd343cbad786fcb50d74c94 for term 2 because replica is either leader or believes a valid leader to be alive. I0312 20:11:07.007519 2502 leader_election.cc:377] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 71f80b7649e64a2bafe2eee9b67b14c1. Message: Invalid argument: T c1432f133b26467b949ad48a19496e3a P 71f80b7649e64a2bafe2eee9b67b14c1 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate 75cbb9a95bd343cbad786fcb50d74c94 for term 2 because replica is either leader or believes a valid leader to be alive. I0312 20:11:07.011898 2500 leader_election.cc:377] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [CANDIDATE]: Term 2 pre-election: Vote denied by peer a5b79d7057a340678b551ae411266cda. Message: Invalid argument: T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 75cbb9a95bd343cbad786fcb50d74c94 for term 2 because replica is either leader or believes a valid leader to be alive. I0312 20:11:07.012121 2500 leader_election.cc:258] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. I0312 20:11:07.012775 2795 raft_consensus.cc:2700] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94: Snoozing failure detection for 0.034s (election complete) I0312 20:11:07.012992 2795 raft_consensus.cc:2455] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority I0312 20:11:07.549937 2797 raft_consensus.cc:436] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 71f80b7649e64a2bafe2eee9b67b14c1) I0312 20:11:07.550259 2797 raft_consensus.cc:2700] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda: Snoozing failure detection for 0.019s (starting election) I0312 20:11:07.550458 2797 raft_consensus.cc:458] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71f80b7649e64a2bafe2eee9b67b14c1" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 52002 } } peers { permanent_uuid: "75cbb9a95bd343cbad786fcb50d74c94" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 59812 } } peers { permanent_uuid: "a5b79d7057a340678b551ae411266cda" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 55527 } } I0312 20:11:07.551733 2797 leader_election.cc:231] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 71f80b7649e64a2bafe2eee9b67b14c1 I0312 20:11:07.558611 2797 leader_election.cc:231] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 75cbb9a95bd343cbad786fcb50d74c94 W0312 20:11:07.572702 2627 leader_election.cc:287] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 71f80b7649e64a2bafe2eee9b67b14c1: Timed out: connection negotiation to 127.0.53.1:52002 for RPC RequestConsensusVote timed out after 0.019s (ON_OUTBOUND_QUEUE) W0312 20:11:07.581884 2629 leader_election.cc:287] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 75cbb9a95bd343cbad786fcb50d74c94: Timed out: connection negotiation to 127.0.53.2:59812 for RPC RequestConsensusVote timed out after 0.019s (ON_OUTBOUND_QUEUE) I0312 20:11:07.583480 2629 leader_election.cc:258] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. I0312 20:11:07.584938 2797 raft_consensus.cc:2700] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda: Snoozing failure detection for 0.021s (election complete) I0312 20:11:07.585197 2797 raft_consensus.cc:2455] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority I0312 20:11:07.755874 2803 raft_consensus.cc:436] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 71f80b7649e64a2bafe2eee9b67b14c1) I0312 20:11:07.756199 2803 raft_consensus.cc:2700] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94: Snoozing failure detection for 0.015s (starting election) I0312 20:11:07.756482 2803 raft_consensus.cc:458] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71f80b7649e64a2bafe2eee9b67b14c1" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 52002 } } peers { permanent_uuid: "75cbb9a95bd343cbad786fcb50d74c94" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 59812 } } peers { permanent_uuid: "a5b79d7057a340678b551ae411266cda" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 55527 } } I0312 20:11:07.757596 2803 leader_election.cc:231] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 71f80b7649e64a2bafe2eee9b67b14c1 I0312 20:11:07.757899 2803 leader_election.cc:231] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer a5b79d7057a340678b551ae411266cda I0312 20:11:07.759091 2694 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "c1432f133b26467b949ad48a19496e3a" candidate_uuid: "75cbb9a95bd343cbad786fcb50d74c94" candidate_term: 2 candidate_status { last_received { term: 1 index: 653 } } ignore_live_leader: false dest_uuid: "a5b79d7057a340678b551ae411266cda" is_pre_election: true I0312 20:11:07.759657 2694 raft_consensus.cc:2176] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 75cbb9a95bd343cbad786fcb50d74c94 for term 2 because replica is either leader or believes a valid leader to be alive. I0312 20:11:07.760638 2500 leader_election.cc:377] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [CANDIDATE]: Term 2 pre-election: Vote denied by peer a5b79d7057a340678b551ae411266cda. Message: Invalid argument: T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 75cbb9a95bd343cbad786fcb50d74c94 for term 2 because replica is either leader or believes a valid leader to be alive. I0312 20:11:07.761258 2443 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "c1432f133b26467b949ad48a19496e3a" candidate_uuid: "75cbb9a95bd343cbad786fcb50d74c94" candidate_term: 2 candidate_status { last_received { term: 1 index: 653 } } ignore_live_leader: false dest_uuid: "71f80b7649e64a2bafe2eee9b67b14c1" is_pre_election: true I0312 20:11:07.761781 2443 raft_consensus.cc:2176] T c1432f133b26467b949ad48a19496e3a P 71f80b7649e64a2bafe2eee9b67b14c1 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate 75cbb9a95bd343cbad786fcb50d74c94 for term 2 because replica is either leader or believes a valid leader to be alive. I0312 20:11:07.762676 2502 leader_election.cc:377] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 71f80b7649e64a2bafe2eee9b67b14c1. Message: Invalid argument: T c1432f133b26467b949ad48a19496e3a P 71f80b7649e64a2bafe2eee9b67b14c1 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate 75cbb9a95bd343cbad786fcb50d74c94 for term 2 because replica is either leader or believes a valid leader to be alive. I0312 20:11:07.762887 2502 leader_election.cc:258] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. I0312 20:11:07.764462 2803 raft_consensus.cc:2700] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94: Snoozing failure detection for 0.047s (election complete) I0312 20:11:07.765044 2803 raft_consensus.cc:2455] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority I0312 20:11:08.000381 2803 raft_consensus.cc:436] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 71f80b7649e64a2bafe2eee9b67b14c1) I0312 20:11:08.000625 2803 raft_consensus.cc:2700] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94: Snoozing failure detection for 0.037s (starting election) I0312 20:11:08.000844 2803 raft_consensus.cc:458] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71f80b7649e64a2bafe2eee9b67b14c1" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 52002 } } peers { permanent_uuid: "75cbb9a95bd343cbad786fcb50d74c94" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 59812 } } peers { permanent_uuid: "a5b79d7057a340678b551ae411266cda" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 55527 } } I0312 20:11:08.001891 2803 leader_election.cc:231] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 71f80b7649e64a2bafe2eee9b67b14c1 I0312 20:11:08.002259 2803 leader_election.cc:231] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer a5b79d7057a340678b551ae411266cda I0312 20:11:08.003371 2693 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "c1432f133b26467b949ad48a19496e3a" candidate_uuid: "75cbb9a95bd343cbad786fcb50d74c94" candidate_term: 2 candidate_status { last_received { term: 1 index: 694 } } ignore_live_leader: false dest_uuid: "a5b79d7057a340678b551ae411266cda" is_pre_election: true I0312 20:11:08.005581 2693 raft_consensus.cc:2190] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 75cbb9a95bd343cbad786fcb50d74c94 for term 2 because replica is already servicing an update from a current leader or another vote. I0312 20:11:08.006160 2443 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "c1432f133b26467b949ad48a19496e3a" candidate_uuid: "75cbb9a95bd343cbad786fcb50d74c94" candidate_term: 2 candidate_status { last_received { term: 1 index: 694 } } ignore_live_leader: false dest_uuid: "71f80b7649e64a2bafe2eee9b67b14c1" is_pre_election: true I0312 20:11:08.006688 2443 raft_consensus.cc:2176] T c1432f133b26467b949ad48a19496e3a P 71f80b7649e64a2bafe2eee9b67b14c1 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate 75cbb9a95bd343cbad786fcb50d74c94 for term 2 because replica is either leader or believes a valid leader to be alive. I0312 20:11:08.007511 2500 leader_election.cc:377] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [CANDIDATE]: Term 2 pre-election: Vote denied by peer a5b79d7057a340678b551ae411266cda. Message: Service unavailable: T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 75cbb9a95bd343cbad786fcb50d74c94 for term 2 because replica is already servicing an update from a current leader or another vote. I0312 20:11:08.008185 2502 leader_election.cc:377] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 71f80b7649e64a2bafe2eee9b67b14c1. Message: Invalid argument: T c1432f133b26467b949ad48a19496e3a P 71f80b7649e64a2bafe2eee9b67b14c1 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate 75cbb9a95bd343cbad786fcb50d74c94 for term 2 because replica is either leader or believes a valid leader to be alive. I0312 20:11:08.008409 2502 leader_election.cc:258] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. I0312 20:11:08.010249 2803 raft_consensus.cc:2700] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94: Snoozing failure detection for 0.020s (election complete) I0312 20:11:08.010468 2803 raft_consensus.cc:2455] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority I0312 20:11:08.330746 2803 raft_consensus.cc:436] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 71f80b7649e64a2bafe2eee9b67b14c1) I0312 20:11:08.331007 2803 raft_consensus.cc:2700] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94: Snoozing failure detection for 0.167s (starting election) I0312 20:11:08.331233 2803 raft_consensus.cc:458] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71f80b7649e64a2bafe2eee9b67b14c1" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 52002 } } peers { permanent_uuid: "75cbb9a95bd343cbad786fcb50d74c94" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 59812 } } peers { permanent_uuid: "a5b79d7057a340678b551ae411266cda" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 55527 } } I0312 20:11:08.332247 2803 leader_election.cc:231] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 71f80b7649e64a2bafe2eee9b67b14c1 I0312 20:11:08.332597 2803 leader_election.cc:231] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer a5b79d7057a340678b551ae411266cda I0312 20:11:08.333976 2443 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "c1432f133b26467b949ad48a19496e3a" candidate_uuid: "75cbb9a95bd343cbad786fcb50d74c94" candidate_term: 2 candidate_status { last_received { term: 1 index: 750 } } ignore_live_leader: false dest_uuid: "71f80b7649e64a2bafe2eee9b67b14c1" is_pre_election: true I0312 20:11:08.334501 2443 raft_consensus.cc:2176] T c1432f133b26467b949ad48a19496e3a P 71f80b7649e64a2bafe2eee9b67b14c1 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate 75cbb9a95bd343cbad786fcb50d74c94 for term 2 because replica is either leader or believes a valid leader to be alive. I0312 20:11:08.335412 2693 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "c1432f133b26467b949ad48a19496e3a" candidate_uuid: "75cbb9a95bd343cbad786fcb50d74c94" candidate_term: 2 candidate_status { last_received { term: 1 index: 750 } } ignore_live_leader: false dest_uuid: "a5b79d7057a340678b551ae411266cda" is_pre_election: true I0312 20:11:08.335981 2693 raft_consensus.cc:2190] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 75cbb9a95bd343cbad786fcb50d74c94 for term 2 because replica is already servicing an update from a current leader or another vote. I0312 20:11:08.336216 2502 leader_election.cc:377] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 71f80b7649e64a2bafe2eee9b67b14c1. Message: Invalid argument: T c1432f133b26467b949ad48a19496e3a P 71f80b7649e64a2bafe2eee9b67b14c1 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate 75cbb9a95bd343cbad786fcb50d74c94 for term 2 because replica is either leader or believes a valid leader to be alive. I0312 20:11:08.336977 2500 leader_election.cc:377] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [CANDIDATE]: Term 2 pre-election: Vote denied by peer a5b79d7057a340678b551ae411266cda. Message: Service unavailable: T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 75cbb9a95bd343cbad786fcb50d74c94 for term 2 because replica is already servicing an update from a current leader or another vote. I0312 20:11:08.337232 2500 leader_election.cc:258] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. I0312 20:11:08.337873 2803 raft_consensus.cc:2700] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94: Snoozing failure detection for 0.020s (election complete) I0312 20:11:08.338088 2803 raft_consensus.cc:2455] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority I0312 20:11:08.990109 2804 raft_consensus.cc:436] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 71f80b7649e64a2bafe2eee9b67b14c1) I0312 20:11:08.990439 2804 raft_consensus.cc:2700] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda: Snoozing failure detection for 0.028s (starting election) I0312 20:11:08.993597 2804 raft_consensus.cc:458] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71f80b7649e64a2bafe2eee9b67b14c1" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 52002 } } peers { permanent_uuid: "75cbb9a95bd343cbad786fcb50d74c94" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 59812 } } peers { permanent_uuid: "a5b79d7057a340678b551ae411266cda" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 55527 } } I0312 20:11:09.002749 2804 leader_election.cc:231] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 71f80b7649e64a2bafe2eee9b67b14c1 I0312 20:11:09.003962 2804 leader_election.cc:231] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 75cbb9a95bd343cbad786fcb50d74c94 I0312 20:11:09.004535 2443 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "c1432f133b26467b949ad48a19496e3a" candidate_uuid: "a5b79d7057a340678b551ae411266cda" candidate_term: 2 candidate_status { last_received { term: 1 index: 861 } } ignore_live_leader: false dest_uuid: "71f80b7649e64a2bafe2eee9b67b14c1" is_pre_election: true I0312 20:11:09.005234 2569 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "c1432f133b26467b949ad48a19496e3a" candidate_uuid: "a5b79d7057a340678b551ae411266cda" candidate_term: 2 candidate_status { last_received { term: 1 index: 861 } } ignore_live_leader: false dest_uuid: "75cbb9a95bd343cbad786fcb50d74c94" is_pre_election: true I0312 20:11:09.005630 2569 raft_consensus.cc:2176] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a5b79d7057a340678b551ae411266cda for term 2 because replica is either leader or believes a valid leader to be alive. I0312 20:11:09.006556 2629 leader_election.cc:377] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [CANDIDATE]: Term 2 pre-election: Vote denied by peer 75cbb9a95bd343cbad786fcb50d74c94. Message: Invalid argument: T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a5b79d7057a340678b551ae411266cda for term 2 because replica is either leader or believes a valid leader to be alive. I0312 20:11:09.008946 2443 raft_consensus.cc:2176] T c1432f133b26467b949ad48a19496e3a P 71f80b7649e64a2bafe2eee9b67b14c1 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate a5b79d7057a340678b551ae411266cda for term 2 because replica is either leader or believes a valid leader to be alive. I0312 20:11:09.010397 2627 leader_election.cc:377] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [CANDIDATE]: Term 2 pre-election: Vote denied by peer 71f80b7649e64a2bafe2eee9b67b14c1. Message: Invalid argument: T c1432f133b26467b949ad48a19496e3a P 71f80b7649e64a2bafe2eee9b67b14c1 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate a5b79d7057a340678b551ae411266cda for term 2 because replica is either leader or believes a valid leader to be alive. I0312 20:11:09.010619 2627 leader_election.cc:258] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. I0312 20:11:09.011808 2804 raft_consensus.cc:2700] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda: Snoozing failure detection for 0.028s (election complete) I0312 20:11:09.012040 2804 raft_consensus.cc:2455] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority I0312 20:11:10.006971 2808 raft_consensus.cc:436] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 71f80b7649e64a2bafe2eee9b67b14c1) I0312 20:11:10.007321 2808 raft_consensus.cc:2700] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94: Snoozing failure detection for 0.107s (starting election) I0312 20:11:10.007627 2808 raft_consensus.cc:458] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71f80b7649e64a2bafe2eee9b67b14c1" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 52002 } } peers { permanent_uuid: "75cbb9a95bd343cbad786fcb50d74c94" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 59812 } } peers { permanent_uuid: "a5b79d7057a340678b551ae411266cda" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 55527 } } I0312 20:11:10.011270 2808 leader_election.cc:231] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 71f80b7649e64a2bafe2eee9b67b14c1 I0312 20:11:10.011656 2808 leader_election.cc:231] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer a5b79d7057a340678b551ae411266cda I0312 20:11:10.013605 2443 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "c1432f133b26467b949ad48a19496e3a" candidate_uuid: "75cbb9a95bd343cbad786fcb50d74c94" candidate_term: 2 candidate_status { last_received { term: 1 index: 1027 } } ignore_live_leader: false dest_uuid: "71f80b7649e64a2bafe2eee9b67b14c1" is_pre_election: true I0312 20:11:10.014135 2443 raft_consensus.cc:2190] T c1432f133b26467b949ad48a19496e3a P 71f80b7649e64a2bafe2eee9b67b14c1 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate 75cbb9a95bd343cbad786fcb50d74c94 for term 2 because replica is already servicing an update from a current leader or another vote. I0312 20:11:10.015259 2502 leader_election.cc:377] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 71f80b7649e64a2bafe2eee9b67b14c1. Message: Service unavailable: T c1432f133b26467b949ad48a19496e3a P 71f80b7649e64a2bafe2eee9b67b14c1 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate 75cbb9a95bd343cbad786fcb50d74c94 for term 2 because replica is already servicing an update from a current leader or another vote. I0312 20:11:10.016187 2693 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "c1432f133b26467b949ad48a19496e3a" candidate_uuid: "75cbb9a95bd343cbad786fcb50d74c94" candidate_term: 2 candidate_status { last_received { term: 1 index: 1027 } } ignore_live_leader: false dest_uuid: "a5b79d7057a340678b551ae411266cda" is_pre_election: true I0312 20:11:10.016607 2693 raft_consensus.cc:2176] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 75cbb9a95bd343cbad786fcb50d74c94 for term 2 because replica is either leader or believes a valid leader to be alive. I0312 20:11:10.017437 2500 leader_election.cc:377] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [CANDIDATE]: Term 2 pre-election: Vote denied by peer a5b79d7057a340678b551ae411266cda. Message: Invalid argument: T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 75cbb9a95bd343cbad786fcb50d74c94 for term 2 because replica is either leader or believes a valid leader to be alive. I0312 20:11:10.017660 2500 leader_election.cc:258] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. I0312 20:11:10.018292 2808 raft_consensus.cc:2700] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94: Snoozing failure detection for 0.059s (election complete) I0312 20:11:10.018532 2808 raft_consensus.cc:2455] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority I0312 20:11:11.889612 2815 raft_consensus.cc:436] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 71f80b7649e64a2bafe2eee9b67b14c1) I0312 20:11:11.889961 2815 raft_consensus.cc:2700] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda: Snoozing failure detection for 0.047s (starting election) I0312 20:11:11.890247 2815 raft_consensus.cc:458] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71f80b7649e64a2bafe2eee9b67b14c1" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 52002 } } peers { permanent_uuid: "75cbb9a95bd343cbad786fcb50d74c94" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 59812 } } peers { permanent_uuid: "a5b79d7057a340678b551ae411266cda" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 55527 } } I0312 20:11:11.891153 2815 leader_election.cc:231] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 71f80b7649e64a2bafe2eee9b67b14c1 I0312 20:11:11.893204 2443 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "c1432f133b26467b949ad48a19496e3a" candidate_uuid: "a5b79d7057a340678b551ae411266cda" candidate_term: 2 candidate_status { last_received { term: 1 index: 1336 } } ignore_live_leader: false dest_uuid: "71f80b7649e64a2bafe2eee9b67b14c1" is_pre_election: true I0312 20:11:11.894117 2815 leader_election.cc:231] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 75cbb9a95bd343cbad786fcb50d74c94 I0312 20:11:11.895506 2443 raft_consensus.cc:2176] T c1432f133b26467b949ad48a19496e3a P 71f80b7649e64a2bafe2eee9b67b14c1 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate a5b79d7057a340678b551ae411266cda for term 2 because replica is either leader or believes a valid leader to be alive. I0312 20:11:11.898089 2627 leader_election.cc:377] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [CANDIDATE]: Term 2 pre-election: Vote denied by peer 71f80b7649e64a2bafe2eee9b67b14c1. Message: Invalid argument: T c1432f133b26467b949ad48a19496e3a P 71f80b7649e64a2bafe2eee9b67b14c1 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate a5b79d7057a340678b551ae411266cda for term 2 because replica is either leader or believes a valid leader to be alive. I0312 20:11:11.898639 2569 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "c1432f133b26467b949ad48a19496e3a" candidate_uuid: "a5b79d7057a340678b551ae411266cda" candidate_term: 2 candidate_status { last_received { term: 1 index: 1336 } } ignore_live_leader: false dest_uuid: "75cbb9a95bd343cbad786fcb50d74c94" is_pre_election: true I0312 20:11:11.899148 2569 raft_consensus.cc:2176] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a5b79d7057a340678b551ae411266cda for term 2 because replica is either leader or believes a valid leader to be alive. I0312 20:11:11.900084 2629 leader_election.cc:377] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [CANDIDATE]: Term 2 pre-election: Vote denied by peer 75cbb9a95bd343cbad786fcb50d74c94. Message: Invalid argument: T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a5b79d7057a340678b551ae411266cda for term 2 because replica is either leader or believes a valid leader to be alive. I0312 20:11:11.900305 2629 leader_election.cc:258] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. I0312 20:11:11.901856 2815 raft_consensus.cc:2700] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda: Snoozing failure detection for 0.034s (election complete) I0312 20:11:11.902015 2815 raft_consensus.cc:2455] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority I0312 20:11:12.026829 2818 raft_consensus.cc:436] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 71f80b7649e64a2bafe2eee9b67b14c1) I0312 20:11:12.027325 2818 raft_consensus.cc:2700] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94: Snoozing failure detection for 0.221s (starting election) I0312 20:11:12.027621 2818 raft_consensus.cc:458] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71f80b7649e64a2bafe2eee9b67b14c1" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 52002 } } peers { permanent_uuid: "75cbb9a95bd343cbad786fcb50d74c94" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 59812 } } peers { permanent_uuid: "a5b79d7057a340678b551ae411266cda" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 55527 } } I0312 20:11:12.028689 2818 leader_election.cc:231] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 71f80b7649e64a2bafe2eee9b67b14c1 I0312 20:11:12.029090 2818 leader_election.cc:231] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer a5b79d7057a340678b551ae411266cda I0312 20:11:12.030453 2443 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "c1432f133b26467b949ad48a19496e3a" candidate_uuid: "75cbb9a95bd343cbad786fcb50d74c94" candidate_term: 2 candidate_status { last_received { term: 1 index: 1352 } } ignore_live_leader: false dest_uuid: "71f80b7649e64a2bafe2eee9b67b14c1" is_pre_election: true I0312 20:11:12.030987 2443 raft_consensus.cc:2176] T c1432f133b26467b949ad48a19496e3a P 71f80b7649e64a2bafe2eee9b67b14c1 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate 75cbb9a95bd343cbad786fcb50d74c94 for term 2 because replica is either leader or believes a valid leader to be alive. I0312 20:11:12.031966 2693 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "c1432f133b26467b949ad48a19496e3a" candidate_uuid: "75cbb9a95bd343cbad786fcb50d74c94" candidate_term: 2 candidate_status { last_received { term: 1 index: 1352 } } ignore_live_leader: false dest_uuid: "a5b79d7057a340678b551ae411266cda" is_pre_election: true I0312 20:11:12.032469 2693 raft_consensus.cc:2176] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 75cbb9a95bd343cbad786fcb50d74c94 for term 2 because replica is either leader or believes a valid leader to be alive. I0312 20:11:12.033470 2500 leader_election.cc:377] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [CANDIDATE]: Term 2 pre-election: Vote denied by peer a5b79d7057a340678b551ae411266cda. Message: Invalid argument: T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 75cbb9a95bd343cbad786fcb50d74c94 for term 2 because replica is either leader or believes a valid leader to be alive. I0312 20:11:12.034533 2502 leader_election.cc:377] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 71f80b7649e64a2bafe2eee9b67b14c1. Message: Invalid argument: T c1432f133b26467b949ad48a19496e3a P 71f80b7649e64a2bafe2eee9b67b14c1 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate 75cbb9a95bd343cbad786fcb50d74c94 for term 2 because replica is either leader or believes a valid leader to be alive. I0312 20:11:12.034731 2502 leader_election.cc:258] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. I0312 20:11:12.035284 2818 raft_consensus.cc:2700] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94: Snoozing failure detection for 0.187s (election complete) I0312 20:11:12.035517 2818 raft_consensus.cc:2455] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority I0312 20:11:12.752581 2822 raft_consensus.cc:436] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 71f80b7649e64a2bafe2eee9b67b14c1) I0312 20:11:12.752899 2822 raft_consensus.cc:2700] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda: Snoozing failure detection for 0.020s (starting election) I0312 20:11:12.753142 2822 raft_consensus.cc:458] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71f80b7649e64a2bafe2eee9b67b14c1" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 52002 } } peers { permanent_uuid: "75cbb9a95bd343cbad786fcb50d74c94" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 59812 } } peers { permanent_uuid: "a5b79d7057a340678b551ae411266cda" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 55527 } } I0312 20:11:12.753973 2822 leader_election.cc:231] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 71f80b7649e64a2bafe2eee9b67b14c1 I0312 20:11:12.755244 2443 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "c1432f133b26467b949ad48a19496e3a" candidate_uuid: "a5b79d7057a340678b551ae411266cda" candidate_term: 2 candidate_status { last_received { term: 1 index: 1477 } } ignore_live_leader: false dest_uuid: "71f80b7649e64a2bafe2eee9b67b14c1" is_pre_election: true I0312 20:11:12.755775 2443 raft_consensus.cc:2176] T c1432f133b26467b949ad48a19496e3a P 71f80b7649e64a2bafe2eee9b67b14c1 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate a5b79d7057a340678b551ae411266cda for term 2 because replica is either leader or believes a valid leader to be alive. I0312 20:11:12.756269 2822 leader_election.cc:231] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 75cbb9a95bd343cbad786fcb50d74c94 I0312 20:11:12.757030 2627 leader_election.cc:377] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [CANDIDATE]: Term 2 pre-election: Vote denied by peer 71f80b7649e64a2bafe2eee9b67b14c1. Message: Invalid argument: T c1432f133b26467b949ad48a19496e3a P 71f80b7649e64a2bafe2eee9b67b14c1 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate a5b79d7057a340678b551ae411266cda for term 2 because replica is either leader or believes a valid leader to be alive. I0312 20:11:12.758870 2569 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "c1432f133b26467b949ad48a19496e3a" candidate_uuid: "a5b79d7057a340678b551ae411266cda" candidate_term: 2 candidate_status { last_received { term: 1 index: 1477 } } ignore_live_leader: false dest_uuid: "75cbb9a95bd343cbad786fcb50d74c94" is_pre_election: true I0312 20:11:12.761749 2569 raft_consensus.cc:2190] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a5b79d7057a340678b551ae411266cda for term 2 because replica is already servicing an update from a current leader or another vote. I0312 20:11:12.763878 2629 leader_election.cc:377] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [CANDIDATE]: Term 2 pre-election: Vote denied by peer 75cbb9a95bd343cbad786fcb50d74c94. Message: Service unavailable: T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a5b79d7057a340678b551ae411266cda for term 2 because replica is already servicing an update from a current leader or another vote. I0312 20:11:12.764117 2629 leader_election.cc:258] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. I0312 20:11:12.767482 2822 raft_consensus.cc:2700] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda: Snoozing failure detection for 0.046s (election complete) I0312 20:11:12.767643 2822 raft_consensus.cc:2455] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority I0312 20:11:13.001657 2823 raft_consensus.cc:436] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 71f80b7649e64a2bafe2eee9b67b14c1) I0312 20:11:13.001992 2823 raft_consensus.cc:2700] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94: Snoozing failure detection for 0.552s (starting election) I0312 20:11:13.002279 2823 raft_consensus.cc:458] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71f80b7649e64a2bafe2eee9b67b14c1" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 52002 } } peers { permanent_uuid: "75cbb9a95bd343cbad786fcb50d74c94" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 59812 } } peers { permanent_uuid: "a5b79d7057a340678b551ae411266cda" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 55527 } } I0312 20:11:13.004149 2823 leader_election.cc:231] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 71f80b7649e64a2bafe2eee9b67b14c1 I0312 20:11:13.004551 2823 leader_election.cc:231] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer a5b79d7057a340678b551ae411266cda I0312 20:11:13.005954 2443 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "c1432f133b26467b949ad48a19496e3a" candidate_uuid: "75cbb9a95bd343cbad786fcb50d74c94" candidate_term: 2 candidate_status { last_received { term: 1 index: 1518 } } ignore_live_leader: false dest_uuid: "71f80b7649e64a2bafe2eee9b67b14c1" is_pre_election: true I0312 20:11:13.006619 2443 raft_consensus.cc:2176] T c1432f133b26467b949ad48a19496e3a P 71f80b7649e64a2bafe2eee9b67b14c1 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate 75cbb9a95bd343cbad786fcb50d74c94 for term 2 because replica is either leader or believes a valid leader to be alive. I0312 20:11:13.007627 2502 leader_election.cc:377] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 71f80b7649e64a2bafe2eee9b67b14c1. Message: Invalid argument: T c1432f133b26467b949ad48a19496e3a P 71f80b7649e64a2bafe2eee9b67b14c1 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate 75cbb9a95bd343cbad786fcb50d74c94 for term 2 because replica is either leader or believes a valid leader to be alive. I0312 20:11:13.007055 2692 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "c1432f133b26467b949ad48a19496e3a" candidate_uuid: "75cbb9a95bd343cbad786fcb50d74c94" candidate_term: 2 candidate_status { last_received { term: 1 index: 1518 } } ignore_live_leader: false dest_uuid: "a5b79d7057a340678b551ae411266cda" is_pre_election: true I0312 20:11:13.008340 2692 raft_consensus.cc:2176] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 75cbb9a95bd343cbad786fcb50d74c94 for term 2 because replica is either leader or believes a valid leader to be alive. I0312 20:11:13.009203 2500 leader_election.cc:377] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [CANDIDATE]: Term 2 pre-election: Vote denied by peer a5b79d7057a340678b551ae411266cda. Message: Invalid argument: T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 75cbb9a95bd343cbad786fcb50d74c94 for term 2 because replica is either leader or believes a valid leader to be alive. I0312 20:11:13.009392 2500 leader_election.cc:258] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. I0312 20:11:13.009726 2823 raft_consensus.cc:2700] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94: Snoozing failure detection for 0.431s (election complete) I0312 20:11:13.009960 2823 raft_consensus.cc:2455] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority I0312 20:11:13.183176 2822 raft_consensus.cc:436] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 71f80b7649e64a2bafe2eee9b67b14c1) I0312 20:11:13.183609 2822 raft_consensus.cc:2700] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda: Snoozing failure detection for 0.036s (starting election) I0312 20:11:13.183918 2822 raft_consensus.cc:458] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71f80b7649e64a2bafe2eee9b67b14c1" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 52002 } } peers { permanent_uuid: "75cbb9a95bd343cbad786fcb50d74c94" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 59812 } } peers { permanent_uuid: "a5b79d7057a340678b551ae411266cda" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 55527 } } I0312 20:11:13.185093 2822 leader_election.cc:231] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 71f80b7649e64a2bafe2eee9b67b14c1 I0312 20:11:13.185528 2822 leader_election.cc:231] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 75cbb9a95bd343cbad786fcb50d74c94 I0312 20:11:13.188201 2443 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "c1432f133b26467b949ad48a19496e3a" candidate_uuid: "a5b79d7057a340678b551ae411266cda" candidate_term: 2 candidate_status { last_received { term: 1 index: 1550 } } ignore_live_leader: false dest_uuid: "71f80b7649e64a2bafe2eee9b67b14c1" is_pre_election: true I0312 20:11:13.188740 2443 raft_consensus.cc:2176] T c1432f133b26467b949ad48a19496e3a P 71f80b7649e64a2bafe2eee9b67b14c1 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate a5b79d7057a340678b551ae411266cda for term 2 because replica is either leader or believes a valid leader to be alive. I0312 20:11:13.190428 2627 leader_election.cc:377] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [CANDIDATE]: Term 2 pre-election: Vote denied by peer 71f80b7649e64a2bafe2eee9b67b14c1. Message: Invalid argument: T c1432f133b26467b949ad48a19496e3a P 71f80b7649e64a2bafe2eee9b67b14c1 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate a5b79d7057a340678b551ae411266cda for term 2 because replica is either leader or believes a valid leader to be alive. I0312 20:11:13.191082 2568 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "c1432f133b26467b949ad48a19496e3a" candidate_uuid: "a5b79d7057a340678b551ae411266cda" candidate_term: 2 candidate_status { last_received { term: 1 index: 1550 } } ignore_live_leader: false dest_uuid: "75cbb9a95bd343cbad786fcb50d74c94" is_pre_election: true I0312 20:11:13.194931 2568 raft_consensus.cc:2190] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a5b79d7057a340678b551ae411266cda for term 2 because replica is already servicing an update from a current leader or another vote. I0312 20:11:13.196715 2629 leader_election.cc:377] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [CANDIDATE]: Term 2 pre-election: Vote denied by peer 75cbb9a95bd343cbad786fcb50d74c94. Message: Service unavailable: T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a5b79d7057a340678b551ae411266cda for term 2 because replica is already servicing an update from a current leader or another vote. I0312 20:11:13.196952 2629 leader_election.cc:258] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. I0312 20:11:13.197667 2822 raft_consensus.cc:2700] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda: Snoozing failure detection for 0.106s (election complete) I0312 20:11:13.197871 2822 raft_consensus.cc:2455] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority I0312 20:11:13.554257 2825 raft_consensus.cc:436] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 71f80b7649e64a2bafe2eee9b67b14c1) I0312 20:11:13.554563 2825 raft_consensus.cc:2700] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94: Snoozing failure detection for 0.796s (starting election) I0312 20:11:13.554858 2825 raft_consensus.cc:458] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71f80b7649e64a2bafe2eee9b67b14c1" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 52002 } } peers { permanent_uuid: "75cbb9a95bd343cbad786fcb50d74c94" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 59812 } } peers { permanent_uuid: "a5b79d7057a340678b551ae411266cda" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 55527 } } I0312 20:11:13.557549 2825 leader_election.cc:231] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 71f80b7649e64a2bafe2eee9b67b14c1 I0312 20:11:13.557921 2825 leader_election.cc:231] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer a5b79d7057a340678b551ae411266cda I0312 20:11:13.559798 2443 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "c1432f133b26467b949ad48a19496e3a" candidate_uuid: "75cbb9a95bd343cbad786fcb50d74c94" candidate_term: 2 candidate_status { last_received { term: 1 index: 1613 } } ignore_live_leader: false dest_uuid: "71f80b7649e64a2bafe2eee9b67b14c1" is_pre_election: true I0312 20:11:13.559311 2693 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "c1432f133b26467b949ad48a19496e3a" candidate_uuid: "75cbb9a95bd343cbad786fcb50d74c94" candidate_term: 2 candidate_status { last_received { term: 1 index: 1613 } } ignore_live_leader: false dest_uuid: "a5b79d7057a340678b551ae411266cda" is_pre_election: true I0312 20:11:13.560792 2693 raft_consensus.cc:2176] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 75cbb9a95bd343cbad786fcb50d74c94 for term 2 because replica is either leader or believes a valid leader to be alive. I0312 20:11:13.562036 2500 leader_election.cc:377] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [CANDIDATE]: Term 2 pre-election: Vote denied by peer a5b79d7057a340678b551ae411266cda. Message: Invalid argument: T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 75cbb9a95bd343cbad786fcb50d74c94 for term 2 because replica is either leader or believes a valid leader to be alive. I0312 20:11:13.562533 2443 raft_consensus.cc:2176] T c1432f133b26467b949ad48a19496e3a P 71f80b7649e64a2bafe2eee9b67b14c1 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate 75cbb9a95bd343cbad786fcb50d74c94 for term 2 because replica is either leader or believes a valid leader to be alive. I0312 20:11:13.564972 2502 leader_election.cc:377] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 71f80b7649e64a2bafe2eee9b67b14c1. Message: Invalid argument: T c1432f133b26467b949ad48a19496e3a P 71f80b7649e64a2bafe2eee9b67b14c1 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate 75cbb9a95bd343cbad786fcb50d74c94 for term 2 because replica is either leader or believes a valid leader to be alive. I0312 20:11:13.565158 2502 leader_election.cc:258] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. I0312 20:11:13.570454 2825 raft_consensus.cc:2700] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94: Snoozing failure detection for 0.177s (election complete) I0312 20:11:13.570816 2825 raft_consensus.cc:2455] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority I0312 20:11:14.339308 2827 raft_consensus.cc:436] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 71f80b7649e64a2bafe2eee9b67b14c1) I0312 20:11:14.339772 2827 raft_consensus.cc:2700] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94: Snoozing failure detection for 1.040s (starting election) I0312 20:11:14.340039 2827 raft_consensus.cc:458] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71f80b7649e64a2bafe2eee9b67b14c1" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 52002 } } peers { permanent_uuid: "75cbb9a95bd343cbad786fcb50d74c94" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 59812 } } peers { permanent_uuid: "a5b79d7057a340678b551ae411266cda" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 55527 } } I0312 20:11:14.341285 2827 leader_election.cc:231] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 71f80b7649e64a2bafe2eee9b67b14c1 I0312 20:11:14.342466 2443 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "c1432f133b26467b949ad48a19496e3a" candidate_uuid: "75cbb9a95bd343cbad786fcb50d74c94" candidate_term: 2 candidate_status { last_received { term: 1 index: 1747 } } ignore_live_leader: false dest_uuid: "71f80b7649e64a2bafe2eee9b67b14c1" is_pre_election: true I0312 20:11:14.342979 2443 raft_consensus.cc:2176] T c1432f133b26467b949ad48a19496e3a P 71f80b7649e64a2bafe2eee9b67b14c1 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate 75cbb9a95bd343cbad786fcb50d74c94 for term 2 because replica is either leader or believes a valid leader to be alive. I0312 20:11:14.343914 2502 leader_election.cc:377] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 71f80b7649e64a2bafe2eee9b67b14c1. Message: Invalid argument: T c1432f133b26467b949ad48a19496e3a P 71f80b7649e64a2bafe2eee9b67b14c1 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate 75cbb9a95bd343cbad786fcb50d74c94 for term 2 because replica is either leader or believes a valid leader to be alive. I0312 20:11:14.344332 2827 leader_election.cc:231] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer a5b79d7057a340678b551ae411266cda I0312 20:11:14.345577 2693 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "c1432f133b26467b949ad48a19496e3a" candidate_uuid: "75cbb9a95bd343cbad786fcb50d74c94" candidate_term: 2 candidate_status { last_received { term: 1 index: 1747 } } ignore_live_leader: false dest_uuid: "a5b79d7057a340678b551ae411266cda" is_pre_election: true I0312 20:11:14.346120 2693 raft_consensus.cc:2176] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 75cbb9a95bd343cbad786fcb50d74c94 for term 2 because replica is either leader or believes a valid leader to be alive. I0312 20:11:14.346971 2500 leader_election.cc:377] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [CANDIDATE]: Term 2 pre-election: Vote denied by peer a5b79d7057a340678b551ae411266cda. Message: Invalid argument: T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 75cbb9a95bd343cbad786fcb50d74c94 for term 2 because replica is either leader or believes a valid leader to be alive. I0312 20:11:14.347187 2500 leader_election.cc:258] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. I0312 20:11:14.348496 2827 raft_consensus.cc:2700] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94: Snoozing failure detection for 1.147s (election complete) I0312 20:11:14.348728 2827 raft_consensus.cc:2455] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority I0312 20:11:15.001523 2832 raft_consensus.cc:436] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 71f80b7649e64a2bafe2eee9b67b14c1) I0312 20:11:15.001718 2832 raft_consensus.cc:2700] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94: Snoozing failure detection for 0.488s (starting election) I0312 20:11:15.001909 2832 raft_consensus.cc:458] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71f80b7649e64a2bafe2eee9b67b14c1" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 52002 } } peers { permanent_uuid: "75cbb9a95bd343cbad786fcb50d74c94" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 59812 } } peers { permanent_uuid: "a5b79d7057a340678b551ae411266cda" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 55527 } } I0312 20:11:15.004878 2832 leader_election.cc:231] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 71f80b7649e64a2bafe2eee9b67b14c1 I0312 20:11:15.005187 2832 leader_election.cc:231] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer a5b79d7057a340678b551ae411266cda I0312 20:11:15.006419 2692 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "c1432f133b26467b949ad48a19496e3a" candidate_uuid: "75cbb9a95bd343cbad786fcb50d74c94" candidate_term: 2 candidate_status { last_received { term: 1 index: 1857 } } ignore_live_leader: false dest_uuid: "a5b79d7057a340678b551ae411266cda" is_pre_election: true I0312 20:11:15.006973 2692 raft_consensus.cc:2190] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 75cbb9a95bd343cbad786fcb50d74c94 for term 2 because replica is already servicing an update from a current leader or another vote. I0312 20:11:15.008023 2500 leader_election.cc:377] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [CANDIDATE]: Term 2 pre-election: Vote denied by peer a5b79d7057a340678b551ae411266cda. Message: Service unavailable: T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 75cbb9a95bd343cbad786fcb50d74c94 for term 2 because replica is already servicing an update from a current leader or another vote. I0312 20:11:15.009789 2443 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "c1432f133b26467b949ad48a19496e3a" candidate_uuid: "75cbb9a95bd343cbad786fcb50d74c94" candidate_term: 2 candidate_status { last_received { term: 1 index: 1857 } } ignore_live_leader: false dest_uuid: "71f80b7649e64a2bafe2eee9b67b14c1" is_pre_election: true I0312 20:11:15.010313 2443 raft_consensus.cc:2176] T c1432f133b26467b949ad48a19496e3a P 71f80b7649e64a2bafe2eee9b67b14c1 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate 75cbb9a95bd343cbad786fcb50d74c94 for term 2 because replica is either leader or believes a valid leader to be alive. I0312 20:11:15.012159 2502 leader_election.cc:377] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 71f80b7649e64a2bafe2eee9b67b14c1. Message: Invalid argument: T c1432f133b26467b949ad48a19496e3a P 71f80b7649e64a2bafe2eee9b67b14c1 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate 75cbb9a95bd343cbad786fcb50d74c94 for term 2 because replica is either leader or believes a valid leader to be alive. I0312 20:11:15.012377 2502 leader_election.cc:258] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. I0312 20:11:15.016635 2832 raft_consensus.cc:2700] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94: Snoozing failure detection for 1.677s (election complete) I0312 20:11:15.016845 2832 raft_consensus.cc:2455] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority I0312 20:11:15.979569 2835 raft_consensus.cc:436] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 71f80b7649e64a2bafe2eee9b67b14c1) I0312 20:11:15.979774 2835 raft_consensus.cc:2700] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94: Snoozing failure detection for 1.735s (starting election) I0312 20:11:15.979943 2835 raft_consensus.cc:458] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71f80b7649e64a2bafe2eee9b67b14c1" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 52002 } } peers { permanent_uuid: "75cbb9a95bd343cbad786fcb50d74c94" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 59812 } } peers { permanent_uuid: "a5b79d7057a340678b551ae411266cda" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 55527 } } I0312 20:11:15.992007 2835 leader_election.cc:231] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 71f80b7649e64a2bafe2eee9b67b14c1 I0312 20:11:15.997012 2835 leader_election.cc:231] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer a5b79d7057a340678b551ae411266cda I0312 20:11:15.997851 2443 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "c1432f133b26467b949ad48a19496e3a" candidate_uuid: "75cbb9a95bd343cbad786fcb50d74c94" candidate_term: 2 candidate_status { last_received { term: 1 index: 2026 } } ignore_live_leader: false dest_uuid: "71f80b7649e64a2bafe2eee9b67b14c1" is_pre_election: true I0312 20:11:15.998383 2443 raft_consensus.cc:2176] T c1432f133b26467b949ad48a19496e3a P 71f80b7649e64a2bafe2eee9b67b14c1 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate 75cbb9a95bd343cbad786fcb50d74c94 for term 2 because replica is either leader or believes a valid leader to be alive. I0312 20:11:15.999318 2502 leader_election.cc:377] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 71f80b7649e64a2bafe2eee9b67b14c1. Message: Invalid argument: T c1432f133b26467b949ad48a19496e3a P 71f80b7649e64a2bafe2eee9b67b14c1 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate 75cbb9a95bd343cbad786fcb50d74c94 for term 2 because replica is either leader or believes a valid leader to be alive. I0312 20:11:16.000262 2693 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "c1432f133b26467b949ad48a19496e3a" candidate_uuid: "75cbb9a95bd343cbad786fcb50d74c94" candidate_term: 2 candidate_status { last_received { term: 1 index: 2026 } } ignore_live_leader: false dest_uuid: "a5b79d7057a340678b551ae411266cda" is_pre_election: true I0312 20:11:16.000640 2693 raft_consensus.cc:2176] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 75cbb9a95bd343cbad786fcb50d74c94 for term 2 because replica is either leader or believes a valid leader to be alive. I0312 20:11:16.001307 2500 leader_election.cc:377] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [CANDIDATE]: Term 2 pre-election: Vote denied by peer a5b79d7057a340678b551ae411266cda. Message: Invalid argument: T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 75cbb9a95bd343cbad786fcb50d74c94 for term 2 because replica is either leader or believes a valid leader to be alive. I0312 20:11:16.001494 2500 leader_election.cc:258] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. I0312 20:11:16.013793 2835 raft_consensus.cc:2700] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94: Snoozing failure detection for 1.057s (election complete) I0312 20:11:16.014039 2835 raft_consensus.cc:2455] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority I0312 20:11:16.339927 2835 raft_consensus.cc:436] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 71f80b7649e64a2bafe2eee9b67b14c1) I0312 20:11:16.340173 2835 raft_consensus.cc:2700] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94: Snoozing failure detection for 3.047s (starting election) I0312 20:11:16.340399 2835 raft_consensus.cc:458] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71f80b7649e64a2bafe2eee9b67b14c1" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 52002 } } peers { permanent_uuid: "75cbb9a95bd343cbad786fcb50d74c94" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 59812 } } peers { permanent_uuid: "a5b79d7057a340678b551ae411266cda" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 55527 } } I0312 20:11:16.341418 2835 leader_election.cc:231] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 71f80b7649e64a2bafe2eee9b67b14c1 I0312 20:11:16.341787 2835 leader_election.cc:231] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer a5b79d7057a340678b551ae411266cda I0312 20:11:16.342483 2443 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "c1432f133b26467b949ad48a19496e3a" candidate_uuid: "75cbb9a95bd343cbad786fcb50d74c94" candidate_term: 2 candidate_status { last_received { term: 1 index: 2083 } } ignore_live_leader: false dest_uuid: "71f80b7649e64a2bafe2eee9b67b14c1" is_pre_election: true I0312 20:11:16.343021 2443 raft_consensus.cc:2176] T c1432f133b26467b949ad48a19496e3a P 71f80b7649e64a2bafe2eee9b67b14c1 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate 75cbb9a95bd343cbad786fcb50d74c94 for term 2 because replica is either leader or believes a valid leader to be alive. I0312 20:11:16.342981 2692 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "c1432f133b26467b949ad48a19496e3a" candidate_uuid: "75cbb9a95bd343cbad786fcb50d74c94" candidate_term: 2 candidate_status { last_received { term: 1 index: 2083 } } ignore_live_leader: false dest_uuid: "a5b79d7057a340678b551ae411266cda" is_pre_election: true I0312 20:11:16.343513 2692 raft_consensus.cc:2176] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 75cbb9a95bd343cbad786fcb50d74c94 for term 2 because replica is either leader or believes a valid leader to be alive. I0312 20:11:16.344151 2502 leader_election.cc:377] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 71f80b7649e64a2bafe2eee9b67b14c1. Message: Invalid argument: T c1432f133b26467b949ad48a19496e3a P 71f80b7649e64a2bafe2eee9b67b14c1 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate 75cbb9a95bd343cbad786fcb50d74c94 for term 2 because replica is either leader or believes a valid leader to be alive. I0312 20:11:16.344403 2500 leader_election.cc:377] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [CANDIDATE]: Term 2 pre-election: Vote denied by peer a5b79d7057a340678b551ae411266cda. Message: Invalid argument: T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 75cbb9a95bd343cbad786fcb50d74c94 for term 2 because replica is either leader or believes a valid leader to be alive. I0312 20:11:16.344601 2500 leader_election.cc:258] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. I0312 20:11:16.344931 2835 raft_consensus.cc:2700] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94: Snoozing failure detection for 0.964s (election complete) I0312 20:11:16.345135 2835 raft_consensus.cc:2455] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority I0312 20:11:17.763550 2840 raft_consensus.cc:436] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 71f80b7649e64a2bafe2eee9b67b14c1) I0312 20:11:17.763881 2840 raft_consensus.cc:2700] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94: Snoozing failure detection for 0.015s (starting election) I0312 20:11:17.764322 2840 raft_consensus.cc:458] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71f80b7649e64a2bafe2eee9b67b14c1" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 52002 } } peers { permanent_uuid: "75cbb9a95bd343cbad786fcb50d74c94" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 59812 } } peers { permanent_uuid: "a5b79d7057a340678b551ae411266cda" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 55527 } } I0312 20:11:17.765579 2840 leader_election.cc:231] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 71f80b7649e64a2bafe2eee9b67b14c1 I0312 20:11:17.765967 2840 leader_election.cc:231] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer a5b79d7057a340678b551ae411266cda I0312 20:11:17.766829 2443 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "c1432f133b26467b949ad48a19496e3a" candidate_uuid: "75cbb9a95bd343cbad786fcb50d74c94" candidate_term: 2 candidate_status { last_received { term: 1 index: 2324 } } ignore_live_leader: false dest_uuid: "71f80b7649e64a2bafe2eee9b67b14c1" is_pre_election: true I0312 20:11:17.767186 2693 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "c1432f133b26467b949ad48a19496e3a" candidate_uuid: "75cbb9a95bd343cbad786fcb50d74c94" candidate_term: 2 candidate_status { last_received { term: 1 index: 2324 } } ignore_live_leader: false dest_uuid: "a5b79d7057a340678b551ae411266cda" is_pre_election: true I0312 20:11:17.767863 2443 raft_consensus.cc:2190] T c1432f133b26467b949ad48a19496e3a P 71f80b7649e64a2bafe2eee9b67b14c1 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate 75cbb9a95bd343cbad786fcb50d74c94 for term 2 because replica is already servicing an update from a current leader or another vote. I0312 20:11:17.768857 2502 leader_election.cc:377] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 71f80b7649e64a2bafe2eee9b67b14c1. Message: Service unavailable: T c1432f133b26467b949ad48a19496e3a P 71f80b7649e64a2bafe2eee9b67b14c1 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate 75cbb9a95bd343cbad786fcb50d74c94 for term 2 because replica is already servicing an update from a current leader or another vote. I0312 20:11:17.769608 2693 raft_consensus.cc:2190] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 75cbb9a95bd343cbad786fcb50d74c94 for term 2 because replica is already servicing an update from a current leader or another vote. I0312 20:11:17.770665 2500 leader_election.cc:377] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [CANDIDATE]: Term 2 pre-election: Vote denied by peer a5b79d7057a340678b551ae411266cda. Message: Service unavailable: T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 75cbb9a95bd343cbad786fcb50d74c94 for term 2 because replica is already servicing an update from a current leader or another vote. I0312 20:11:17.770850 2500 leader_election.cc:258] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. I0312 20:11:17.771167 2840 raft_consensus.cc:2700] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94: Snoozing failure detection for 5.264s (election complete) I0312 20:11:17.771404 2840 raft_consensus.cc:2455] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority I0312 20:11:18.557600 2842 raft_consensus.cc:436] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 71f80b7649e64a2bafe2eee9b67b14c1) I0312 20:11:18.557941 2842 raft_consensus.cc:2700] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94: Snoozing failure detection for 2.326s (starting election) I0312 20:11:18.558193 2842 raft_consensus.cc:458] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71f80b7649e64a2bafe2eee9b67b14c1" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 52002 } } peers { permanent_uuid: "75cbb9a95bd343cbad786fcb50d74c94" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 59812 } } peers { permanent_uuid: "a5b79d7057a340678b551ae411266cda" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 55527 } } I0312 20:11:18.560482 2842 leader_election.cc:231] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 71f80b7649e64a2bafe2eee9b67b14c1 I0312 20:11:18.562568 2842 leader_election.cc:231] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer a5b79d7057a340678b551ae411266cda I0312 20:11:18.562726 2443 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "c1432f133b26467b949ad48a19496e3a" candidate_uuid: "75cbb9a95bd343cbad786fcb50d74c94" candidate_term: 2 candidate_status { last_received { term: 1 index: 2458 } } ignore_live_leader: false dest_uuid: "71f80b7649e64a2bafe2eee9b67b14c1" is_pre_election: true I0312 20:11:18.565248 2443 raft_consensus.cc:2190] T c1432f133b26467b949ad48a19496e3a P 71f80b7649e64a2bafe2eee9b67b14c1 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate 75cbb9a95bd343cbad786fcb50d74c94 for term 2 because replica is already servicing an update from a current leader or another vote. I0312 20:11:18.566998 2502 leader_election.cc:377] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 71f80b7649e64a2bafe2eee9b67b14c1. Message: Service unavailable: T c1432f133b26467b949ad48a19496e3a P 71f80b7649e64a2bafe2eee9b67b14c1 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate 75cbb9a95bd343cbad786fcb50d74c94 for term 2 because replica is already servicing an update from a current leader or another vote. I0312 20:11:18.571002 2692 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "c1432f133b26467b949ad48a19496e3a" candidate_uuid: "75cbb9a95bd343cbad786fcb50d74c94" candidate_term: 2 candidate_status { last_received { term: 1 index: 2458 } } ignore_live_leader: false dest_uuid: "a5b79d7057a340678b551ae411266cda" is_pre_election: true I0312 20:11:18.571748 2692 raft_consensus.cc:2190] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 75cbb9a95bd343cbad786fcb50d74c94 for term 2 because replica is already servicing an update from a current leader or another vote. I0312 20:11:18.572794 2500 leader_election.cc:377] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [CANDIDATE]: Term 2 pre-election: Vote denied by peer a5b79d7057a340678b551ae411266cda. Message: Service unavailable: T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 75cbb9a95bd343cbad786fcb50d74c94 for term 2 because replica is already servicing an update from a current leader or another vote. I0312 20:11:18.573010 2500 leader_election.cc:258] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. I0312 20:11:18.574463 2842 raft_consensus.cc:2700] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94: Snoozing failure detection for 4.212s (election complete) I0312 20:11:18.574682 2842 raft_consensus.cc:2455] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority I0312 20:11:20.013259 2844 raft_consensus.cc:436] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 71f80b7649e64a2bafe2eee9b67b14c1) I0312 20:11:20.013604 2844 raft_consensus.cc:2700] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94: Snoozing failure detection for 8.101s (starting election) I0312 20:11:20.013895 2844 raft_consensus.cc:458] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71f80b7649e64a2bafe2eee9b67b14c1" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 52002 } } peers { permanent_uuid: "75cbb9a95bd343cbad786fcb50d74c94" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 59812 } } peers { permanent_uuid: "a5b79d7057a340678b551ae411266cda" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 55527 } } I0312 20:11:20.018483 2844 leader_election.cc:231] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 71f80b7649e64a2bafe2eee9b67b14c1 I0312 20:11:20.018771 2844 leader_election.cc:231] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer a5b79d7057a340678b551ae411266cda I0312 20:11:20.020033 2443 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "c1432f133b26467b949ad48a19496e3a" candidate_uuid: "75cbb9a95bd343cbad786fcb50d74c94" candidate_term: 2 candidate_status { last_received { term: 1 index: 2705 } } ignore_live_leader: false dest_uuid: "71f80b7649e64a2bafe2eee9b67b14c1" is_pre_election: true I0312 20:11:20.020329 2692 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "c1432f133b26467b949ad48a19496e3a" candidate_uuid: "75cbb9a95bd343cbad786fcb50d74c94" candidate_term: 2 candidate_status { last_received { term: 1 index: 2705 } } ignore_live_leader: false dest_uuid: "a5b79d7057a340678b551ae411266cda" is_pre_election: true I0312 20:11:20.020566 2443 raft_consensus.cc:2176] T c1432f133b26467b949ad48a19496e3a P 71f80b7649e64a2bafe2eee9b67b14c1 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate 75cbb9a95bd343cbad786fcb50d74c94 for term 2 because replica is either leader or believes a valid leader to be alive. I0312 20:11:20.020762 2692 raft_consensus.cc:2176] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 75cbb9a95bd343cbad786fcb50d74c94 for term 2 because replica is either leader or believes a valid leader to be alive. I0312 20:11:20.021636 2500 leader_election.cc:377] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [CANDIDATE]: Term 2 pre-election: Vote denied by peer a5b79d7057a340678b551ae411266cda. Message: Invalid argument: T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 75cbb9a95bd343cbad786fcb50d74c94 for term 2 because replica is either leader or believes a valid leader to be alive. I0312 20:11:20.022007 2502 leader_election.cc:377] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 71f80b7649e64a2bafe2eee9b67b14c1. Message: Invalid argument: T c1432f133b26467b949ad48a19496e3a P 71f80b7649e64a2bafe2eee9b67b14c1 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate 75cbb9a95bd343cbad786fcb50d74c94 for term 2 because replica is either leader or believes a valid leader to be alive. I0312 20:11:20.022168 2502 leader_election.cc:258] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. I0312 20:11:20.022691 2844 raft_consensus.cc:2700] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94: Snoozing failure detection for 0.943s (election complete) I0312 20:11:20.022917 2844 raft_consensus.cc:2455] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority I0312 20:11:20.990409 2848 raft_consensus.cc:436] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 71f80b7649e64a2bafe2eee9b67b14c1) I0312 20:11:20.990742 2848 raft_consensus.cc:2700] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94: Snoozing failure detection for 7.567s (starting election) I0312 20:11:20.991009 2848 raft_consensus.cc:458] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71f80b7649e64a2bafe2eee9b67b14c1" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 52002 } } peers { permanent_uuid: "75cbb9a95bd343cbad786fcb50d74c94" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 59812 } } peers { permanent_uuid: "a5b79d7057a340678b551ae411266cda" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 55527 } } I0312 20:11:20.992516 2848 leader_election.cc:231] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 71f80b7649e64a2bafe2eee9b67b14c1 I0312 20:11:20.993273 2848 leader_election.cc:231] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer a5b79d7057a340678b551ae411266cda I0312 20:11:20.994117 2443 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "c1432f133b26467b949ad48a19496e3a" candidate_uuid: "75cbb9a95bd343cbad786fcb50d74c94" candidate_term: 2 candidate_status { last_received { term: 1 index: 2869 } } ignore_live_leader: false dest_uuid: "71f80b7649e64a2bafe2eee9b67b14c1" is_pre_election: true I0312 20:11:20.994645 2443 raft_consensus.cc:2176] T c1432f133b26467b949ad48a19496e3a P 71f80b7649e64a2bafe2eee9b67b14c1 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate 75cbb9a95bd343cbad786fcb50d74c94 for term 2 because replica is either leader or believes a valid leader to be alive. I0312 20:11:20.995368 2694 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "c1432f133b26467b949ad48a19496e3a" candidate_uuid: "75cbb9a95bd343cbad786fcb50d74c94" candidate_term: 2 candidate_status { last_received { term: 1 index: 2869 } } ignore_live_leader: false dest_uuid: "a5b79d7057a340678b551ae411266cda" is_pre_election: true I0312 20:11:20.995877 2694 raft_consensus.cc:2190] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 75cbb9a95bd343cbad786fcb50d74c94 for term 2 because replica is already servicing an update from a current leader or another vote. I0312 20:11:20.996495 2502 leader_election.cc:377] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 71f80b7649e64a2bafe2eee9b67b14c1. Message: Invalid argument: T c1432f133b26467b949ad48a19496e3a P 71f80b7649e64a2bafe2eee9b67b14c1 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate 75cbb9a95bd343cbad786fcb50d74c94 for term 2 because replica is either leader or believes a valid leader to be alive. I0312 20:11:20.997802 2500 leader_election.cc:377] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [CANDIDATE]: Term 2 pre-election: Vote denied by peer a5b79d7057a340678b551ae411266cda. Message: Service unavailable: T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 75cbb9a95bd343cbad786fcb50d74c94 for term 2 because replica is already servicing an update from a current leader or another vote. I0312 20:11:20.998029 2500 leader_election.cc:258] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. I0312 20:11:21.019528 2848 raft_consensus.cc:2700] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94: Snoozing failure detection for 15.628s (election complete) I0312 20:11:21.019729 2848 raft_consensus.cc:2455] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority I0312 20:11:21.021528 2849 raft_consensus.cc:436] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 71f80b7649e64a2bafe2eee9b67b14c1) I0312 20:11:21.021872 2849 raft_consensus.cc:2700] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94: Snoozing failure detection for 8.197s (starting election) I0312 20:11:21.022128 2849 raft_consensus.cc:458] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71f80b7649e64a2bafe2eee9b67b14c1" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 52002 } } peers { permanent_uuid: "75cbb9a95bd343cbad786fcb50d74c94" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 59812 } } peers { permanent_uuid: "a5b79d7057a340678b551ae411266cda" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 55527 } } I0312 20:11:21.024612 2849 leader_election.cc:231] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 71f80b7649e64a2bafe2eee9b67b14c1 I0312 20:11:21.025001 2849 leader_election.cc:231] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer a5b79d7057a340678b551ae411266cda I0312 20:11:21.025796 2443 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "c1432f133b26467b949ad48a19496e3a" candidate_uuid: "75cbb9a95bd343cbad786fcb50d74c94" candidate_term: 2 candidate_status { last_received { term: 1 index: 2873 } } ignore_live_leader: false dest_uuid: "71f80b7649e64a2bafe2eee9b67b14c1" is_pre_election: true I0312 20:11:21.026168 2694 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "c1432f133b26467b949ad48a19496e3a" candidate_uuid: "75cbb9a95bd343cbad786fcb50d74c94" candidate_term: 2 candidate_status { last_received { term: 1 index: 2873 } } ignore_live_leader: false dest_uuid: "a5b79d7057a340678b551ae411266cda" is_pre_election: true I0312 20:11:21.026589 2443 raft_consensus.cc:2190] T c1432f133b26467b949ad48a19496e3a P 71f80b7649e64a2bafe2eee9b67b14c1 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate 75cbb9a95bd343cbad786fcb50d74c94 for term 2 because replica is already servicing an update from a current leader or another vote. I0312 20:11:21.026618 2694 raft_consensus.cc:2190] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 75cbb9a95bd343cbad786fcb50d74c94 for term 2 because replica is already servicing an update from a current leader or another vote. I0312 20:11:21.027586 2500 leader_election.cc:377] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [CANDIDATE]: Term 2 pre-election: Vote denied by peer a5b79d7057a340678b551ae411266cda. Message: Service unavailable: T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 75cbb9a95bd343cbad786fcb50d74c94 for term 2 because replica is already servicing an update from a current leader or another vote. I0312 20:11:21.030270 2502 leader_election.cc:377] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 71f80b7649e64a2bafe2eee9b67b14c1. Message: Service unavailable: T c1432f133b26467b949ad48a19496e3a P 71f80b7649e64a2bafe2eee9b67b14c1 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate 75cbb9a95bd343cbad786fcb50d74c94 for term 2 because replica is already servicing an update from a current leader or another vote. I0312 20:11:21.030484 2502 leader_election.cc:258] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. I0312 20:11:21.041710 2849 raft_consensus.cc:2700] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94: Snoozing failure detection for 11.819s (election complete) I0312 20:11:21.041939 2849 raft_consensus.cc:2455] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority I0312 20:11:22.998497 2859 raft_consensus.cc:436] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 71f80b7649e64a2bafe2eee9b67b14c1) I0312 20:11:22.998878 2859 raft_consensus.cc:2700] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda: Snoozing failure detection for 0.027s (starting election) I0312 20:11:22.999151 2859 raft_consensus.cc:458] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71f80b7649e64a2bafe2eee9b67b14c1" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 52002 } } peers { permanent_uuid: "75cbb9a95bd343cbad786fcb50d74c94" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 59812 } } peers { permanent_uuid: "a5b79d7057a340678b551ae411266cda" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 55527 } } I0312 20:11:23.000416 2859 leader_election.cc:231] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 71f80b7649e64a2bafe2eee9b67b14c1 I0312 20:11:23.001780 2859 leader_election.cc:231] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 75cbb9a95bd343cbad786fcb50d74c94 I0312 20:11:23.002001 2443 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "c1432f133b26467b949ad48a19496e3a" candidate_uuid: "a5b79d7057a340678b551ae411266cda" candidate_term: 2 candidate_status { last_received { term: 1 index: 3204 } } ignore_live_leader: false dest_uuid: "71f80b7649e64a2bafe2eee9b67b14c1" is_pre_election: true I0312 20:11:23.003150 2569 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "c1432f133b26467b949ad48a19496e3a" candidate_uuid: "a5b79d7057a340678b551ae411266cda" candidate_term: 2 candidate_status { last_received { term: 1 index: 3204 } } ignore_live_leader: false dest_uuid: "75cbb9a95bd343cbad786fcb50d74c94" is_pre_election: true I0312 20:11:23.003806 2569 raft_consensus.cc:2176] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a5b79d7057a340678b551ae411266cda for term 2 because replica is either leader or believes a valid leader to be alive. I0312 20:11:23.004191 2443 raft_consensus.cc:2176] T c1432f133b26467b949ad48a19496e3a P 71f80b7649e64a2bafe2eee9b67b14c1 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate a5b79d7057a340678b551ae411266cda for term 2 because replica is either leader or believes a valid leader to be alive. I0312 20:11:23.005336 2627 leader_election.cc:377] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [CANDIDATE]: Term 2 pre-election: Vote denied by peer 71f80b7649e64a2bafe2eee9b67b14c1. Message: Invalid argument: T c1432f133b26467b949ad48a19496e3a P 71f80b7649e64a2bafe2eee9b67b14c1 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate a5b79d7057a340678b551ae411266cda for term 2 because replica is either leader or believes a valid leader to be alive. I0312 20:11:23.005692 2629 leader_election.cc:377] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [CANDIDATE]: Term 2 pre-election: Vote denied by peer 75cbb9a95bd343cbad786fcb50d74c94. Message: Invalid argument: T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a5b79d7057a340678b551ae411266cda for term 2 because replica is either leader or believes a valid leader to be alive. I0312 20:11:23.005901 2629 leader_election.cc:258] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. I0312 20:11:23.011504 2859 raft_consensus.cc:2700] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda: Snoozing failure detection for 0.060s (election complete) I0312 20:11:23.011668 2859 raft_consensus.cc:2455] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority I0312 20:11:23.223901 2860 raft_consensus.cc:436] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 71f80b7649e64a2bafe2eee9b67b14c1) I0312 20:11:23.224284 2860 raft_consensus.cc:2700] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda: Snoozing failure detection for 0.081s (starting election) I0312 20:11:23.224519 2860 raft_consensus.cc:458] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71f80b7649e64a2bafe2eee9b67b14c1" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 52002 } } peers { permanent_uuid: "75cbb9a95bd343cbad786fcb50d74c94" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 59812 } } peers { permanent_uuid: "a5b79d7057a340678b551ae411266cda" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 55527 } } I0312 20:11:23.225528 2860 leader_election.cc:231] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 71f80b7649e64a2bafe2eee9b67b14c1 I0312 20:11:23.225926 2860 leader_election.cc:231] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 75cbb9a95bd343cbad786fcb50d74c94 I0312 20:11:23.228117 2443 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "c1432f133b26467b949ad48a19496e3a" candidate_uuid: "a5b79d7057a340678b551ae411266cda" candidate_term: 2 candidate_status { last_received { term: 1 index: 3240 } } ignore_live_leader: false dest_uuid: "71f80b7649e64a2bafe2eee9b67b14c1" is_pre_election: true I0312 20:11:23.228166 2568 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "c1432f133b26467b949ad48a19496e3a" candidate_uuid: "a5b79d7057a340678b551ae411266cda" candidate_term: 2 candidate_status { last_received { term: 1 index: 3240 } } ignore_live_leader: false dest_uuid: "75cbb9a95bd343cbad786fcb50d74c94" is_pre_election: true I0312 20:11:23.228518 2568 raft_consensus.cc:2176] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a5b79d7057a340678b551ae411266cda for term 2 because replica is either leader or believes a valid leader to be alive. I0312 20:11:23.228561 2443 raft_consensus.cc:2176] T c1432f133b26467b949ad48a19496e3a P 71f80b7649e64a2bafe2eee9b67b14c1 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate a5b79d7057a340678b551ae411266cda for term 2 because replica is either leader or believes a valid leader to be alive. I0312 20:11:23.229389 2627 leader_election.cc:377] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [CANDIDATE]: Term 2 pre-election: Vote denied by peer 71f80b7649e64a2bafe2eee9b67b14c1. Message: Invalid argument: T c1432f133b26467b949ad48a19496e3a P 71f80b7649e64a2bafe2eee9b67b14c1 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate a5b79d7057a340678b551ae411266cda for term 2 because replica is either leader or believes a valid leader to be alive. I0312 20:11:23.229833 2629 leader_election.cc:377] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [CANDIDATE]: Term 2 pre-election: Vote denied by peer 75cbb9a95bd343cbad786fcb50d74c94. Message: Invalid argument: T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a5b79d7057a340678b551ae411266cda for term 2 because replica is either leader or believes a valid leader to be alive. I0312 20:11:23.229991 2629 leader_election.cc:258] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. I0312 20:11:23.232058 2860 raft_consensus.cc:2700] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda: Snoozing failure detection for 0.215s (election complete) I0312 20:11:23.232316 2860 raft_consensus.cc:2455] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority I0312 20:11:23.554292 2860 raft_consensus.cc:436] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 71f80b7649e64a2bafe2eee9b67b14c1) I0312 20:11:23.554548 2860 raft_consensus.cc:2700] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda: Snoozing failure detection for 0.248s (starting election) I0312 20:11:23.554769 2860 raft_consensus.cc:458] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71f80b7649e64a2bafe2eee9b67b14c1" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 52002 } } peers { permanent_uuid: "75cbb9a95bd343cbad786fcb50d74c94" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 59812 } } peers { permanent_uuid: "a5b79d7057a340678b551ae411266cda" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 55527 } } I0312 20:11:23.555940 2860 leader_election.cc:231] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 71f80b7649e64a2bafe2eee9b67b14c1 I0312 20:11:23.556290 2860 leader_election.cc:231] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 75cbb9a95bd343cbad786fcb50d74c94 I0312 20:11:23.557687 2443 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "c1432f133b26467b949ad48a19496e3a" candidate_uuid: "a5b79d7057a340678b551ae411266cda" candidate_term: 2 candidate_status { last_received { term: 1 index: 3296 } } ignore_live_leader: false dest_uuid: "71f80b7649e64a2bafe2eee9b67b14c1" is_pre_election: true I0312 20:11:23.558205 2443 raft_consensus.cc:2176] T c1432f133b26467b949ad48a19496e3a P 71f80b7649e64a2bafe2eee9b67b14c1 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate a5b79d7057a340678b551ae411266cda for term 2 because replica is either leader or believes a valid leader to be alive. I0312 20:11:23.559422 2568 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "c1432f133b26467b949ad48a19496e3a" candidate_uuid: "a5b79d7057a340678b551ae411266cda" candidate_term: 2 candidate_status { last_received { term: 1 index: 3296 } } ignore_live_leader: false dest_uuid: "75cbb9a95bd343cbad786fcb50d74c94" is_pre_election: true I0312 20:11:23.560204 2568 raft_consensus.cc:2190] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a5b79d7057a340678b551ae411266cda for term 2 because replica is already servicing an update from a current leader or another vote. I0312 20:11:23.560607 2627 leader_election.cc:377] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [CANDIDATE]: Term 2 pre-election: Vote denied by peer 71f80b7649e64a2bafe2eee9b67b14c1. Message: Invalid argument: T c1432f133b26467b949ad48a19496e3a P 71f80b7649e64a2bafe2eee9b67b14c1 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate a5b79d7057a340678b551ae411266cda for term 2 because replica is either leader or believes a valid leader to be alive. I0312 20:11:23.561538 2629 leader_election.cc:377] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [CANDIDATE]: Term 2 pre-election: Vote denied by peer 75cbb9a95bd343cbad786fcb50d74c94. Message: Service unavailable: T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a5b79d7057a340678b551ae411266cda for term 2 because replica is already servicing an update from a current leader or another vote. I0312 20:11:23.561714 2629 leader_election.cc:258] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. I0312 20:11:23.562690 2860 raft_consensus.cc:2700] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda: Snoozing failure detection for 0.016s (election complete) I0312 20:11:23.562886 2860 raft_consensus.cc:2455] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority I0312 20:11:23.754664 2863 raft_consensus.cc:436] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 71f80b7649e64a2bafe2eee9b67b14c1) I0312 20:11:23.755003 2863 raft_consensus.cc:2700] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94: Snoozing failure detection for 2.901s (starting election) I0312 20:11:23.755288 2863 raft_consensus.cc:458] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71f80b7649e64a2bafe2eee9b67b14c1" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 52002 } } peers { permanent_uuid: "75cbb9a95bd343cbad786fcb50d74c94" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 59812 } } peers { permanent_uuid: "a5b79d7057a340678b551ae411266cda" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 55527 } } I0312 20:11:23.756539 2863 leader_election.cc:231] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 71f80b7649e64a2bafe2eee9b67b14c1 I0312 20:11:23.756940 2863 leader_election.cc:231] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer a5b79d7057a340678b551ae411266cda I0312 20:11:23.758473 2693 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "c1432f133b26467b949ad48a19496e3a" candidate_uuid: "75cbb9a95bd343cbad786fcb50d74c94" candidate_term: 2 candidate_status { last_received { term: 1 index: 3330 } } ignore_live_leader: false dest_uuid: "a5b79d7057a340678b551ae411266cda" is_pre_election: true I0312 20:11:23.759048 2693 raft_consensus.cc:2190] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 75cbb9a95bd343cbad786fcb50d74c94 for term 2 because replica is already servicing an update from a current leader or another vote. I0312 20:11:23.760529 2500 leader_election.cc:377] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [CANDIDATE]: Term 2 pre-election: Vote denied by peer a5b79d7057a340678b551ae411266cda. Message: Service unavailable: T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 75cbb9a95bd343cbad786fcb50d74c94 for term 2 because replica is already servicing an update from a current leader or another vote. I0312 20:11:23.761142 2443 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "c1432f133b26467b949ad48a19496e3a" candidate_uuid: "75cbb9a95bd343cbad786fcb50d74c94" candidate_term: 2 candidate_status { last_received { term: 1 index: 3330 } } ignore_live_leader: false dest_uuid: "71f80b7649e64a2bafe2eee9b67b14c1" is_pre_election: true I0312 20:11:23.761526 2443 raft_consensus.cc:2176] T c1432f133b26467b949ad48a19496e3a P 71f80b7649e64a2bafe2eee9b67b14c1 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate 75cbb9a95bd343cbad786fcb50d74c94 for term 2 because replica is either leader or believes a valid leader to be alive. I0312 20:11:23.763190 2502 leader_election.cc:377] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 71f80b7649e64a2bafe2eee9b67b14c1. Message: Invalid argument: T c1432f133b26467b949ad48a19496e3a P 71f80b7649e64a2bafe2eee9b67b14c1 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate 75cbb9a95bd343cbad786fcb50d74c94 for term 2 because replica is either leader or believes a valid leader to be alive. I0312 20:11:23.763417 2502 leader_election.cc:258] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. I0312 20:11:23.766433 2863 raft_consensus.cc:2700] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94: Snoozing failure detection for 10.409s (election complete) I0312 20:11:23.766657 2863 raft_consensus.cc:2455] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority I0312 20:11:23.989089 2863 raft_consensus.cc:436] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 71f80b7649e64a2bafe2eee9b67b14c1) I0312 20:11:23.989354 2863 raft_consensus.cc:2700] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94: Snoozing failure detection for 13.376s (starting election) I0312 20:11:23.989574 2863 raft_consensus.cc:458] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71f80b7649e64a2bafe2eee9b67b14c1" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 52002 } } peers { permanent_uuid: "75cbb9a95bd343cbad786fcb50d74c94" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 59812 } } peers { permanent_uuid: "a5b79d7057a340678b551ae411266cda" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 55527 } } I0312 20:11:23.990643 2863 leader_election.cc:231] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 71f80b7649e64a2bafe2eee9b67b14c1 I0312 20:11:23.991010 2863 leader_election.cc:231] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer a5b79d7057a340678b551ae411266cda I0312 20:11:23.992156 2693 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "c1432f133b26467b949ad48a19496e3a" candidate_uuid: "75cbb9a95bd343cbad786fcb50d74c94" candidate_term: 2 candidate_status { last_received { term: 1 index: 3373 } } ignore_live_leader: false dest_uuid: "a5b79d7057a340678b551ae411266cda" is_pre_election: true I0312 20:11:23.992400 2443 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "c1432f133b26467b949ad48a19496e3a" candidate_uuid: "75cbb9a95bd343cbad786fcb50d74c94" candidate_term: 2 candidate_status { last_received { term: 1 index: 3373 } } ignore_live_leader: false dest_uuid: "71f80b7649e64a2bafe2eee9b67b14c1" is_pre_election: true I0312 20:11:23.992668 2693 raft_consensus.cc:2176] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 75cbb9a95bd343cbad786fcb50d74c94 for term 2 because replica is either leader or believes a valid leader to be alive. I0312 20:11:23.992918 2443 raft_consensus.cc:2176] T c1432f133b26467b949ad48a19496e3a P 71f80b7649e64a2bafe2eee9b67b14c1 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate 75cbb9a95bd343cbad786fcb50d74c94 for term 2 because replica is either leader or believes a valid leader to be alive. I0312 20:11:23.993669 2500 leader_election.cc:377] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [CANDIDATE]: Term 2 pre-election: Vote denied by peer a5b79d7057a340678b551ae411266cda. Message: Invalid argument: T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 75cbb9a95bd343cbad786fcb50d74c94 for term 2 because replica is either leader or believes a valid leader to be alive. I0312 20:11:23.994002 2502 leader_election.cc:377] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 71f80b7649e64a2bafe2eee9b67b14c1. Message: Invalid argument: T c1432f133b26467b949ad48a19496e3a P 71f80b7649e64a2bafe2eee9b67b14c1 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate 75cbb9a95bd343cbad786fcb50d74c94 for term 2 because replica is either leader or believes a valid leader to be alive. I0312 20:11:23.994215 2502 leader_election.cc:258] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. I0312 20:11:23.994573 2863 raft_consensus.cc:2700] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94: Snoozing failure detection for 2.979s (election complete) I0312 20:11:23.994781 2863 raft_consensus.cc:2455] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority I0312 20:11:25.334609 2868 raft_consensus.cc:436] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 71f80b7649e64a2bafe2eee9b67b14c1) I0312 20:11:25.334909 2868 raft_consensus.cc:2700] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94: Snoozing failure detection for 5.663s (starting election) I0312 20:11:25.335108 2868 raft_consensus.cc:458] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71f80b7649e64a2bafe2eee9b67b14c1" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 52002 } } peers { permanent_uuid: "75cbb9a95bd343cbad786fcb50d74c94" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 59812 } } peers { permanent_uuid: "a5b79d7057a340678b551ae411266cda" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 55527 } } I0312 20:11:25.336310 2868 leader_election.cc:231] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 71f80b7649e64a2bafe2eee9b67b14c1 I0312 20:11:25.337043 2868 leader_election.cc:231] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer a5b79d7057a340678b551ae411266cda I0312 20:11:25.338366 2694 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "c1432f133b26467b949ad48a19496e3a" candidate_uuid: "75cbb9a95bd343cbad786fcb50d74c94" candidate_term: 2 candidate_status { last_received { term: 1 index: 3601 } } ignore_live_leader: false dest_uuid: "a5b79d7057a340678b551ae411266cda" is_pre_election: true I0312 20:11:25.338966 2694 raft_consensus.cc:2176] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 75cbb9a95bd343cbad786fcb50d74c94 for term 2 because replica is either leader or believes a valid leader to be alive. I0312 20:11:25.339890 2500 leader_election.cc:377] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [CANDIDATE]: Term 2 pre-election: Vote denied by peer a5b79d7057a340678b551ae411266cda. Message: Invalid argument: T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 75cbb9a95bd343cbad786fcb50d74c94 for term 2 because replica is either leader or believes a valid leader to be alive. I0312 20:11:25.341019 2443 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "c1432f133b26467b949ad48a19496e3a" candidate_uuid: "75cbb9a95bd343cbad786fcb50d74c94" candidate_term: 2 candidate_status { last_received { term: 1 index: 3601 } } ignore_live_leader: false dest_uuid: "71f80b7649e64a2bafe2eee9b67b14c1" is_pre_election: true I0312 20:11:25.341564 2443 raft_consensus.cc:2176] T c1432f133b26467b949ad48a19496e3a P 71f80b7649e64a2bafe2eee9b67b14c1 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate 75cbb9a95bd343cbad786fcb50d74c94 for term 2 because replica is either leader or believes a valid leader to be alive. I0312 20:11:25.342528 2502 leader_election.cc:377] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 71f80b7649e64a2bafe2eee9b67b14c1. Message: Invalid argument: T c1432f133b26467b949ad48a19496e3a P 71f80b7649e64a2bafe2eee9b67b14c1 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate 75cbb9a95bd343cbad786fcb50d74c94 for term 2 because replica is either leader or believes a valid leader to be alive. I0312 20:11:25.342746 2502 leader_election.cc:258] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. I0312 20:11:25.343495 2868 raft_consensus.cc:2700] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94: Snoozing failure detection for 10.627s (election complete) I0312 20:11:25.343720 2868 raft_consensus.cc:2455] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority I0312 20:11:26.017129 2871 raft_consensus.cc:436] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 71f80b7649e64a2bafe2eee9b67b14c1) I0312 20:11:26.017387 2871 raft_consensus.cc:2700] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda: Snoozing failure detection for 0.527s (starting election) I0312 20:11:26.017612 2871 raft_consensus.cc:458] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71f80b7649e64a2bafe2eee9b67b14c1" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 52002 } } peers { permanent_uuid: "75cbb9a95bd343cbad786fcb50d74c94" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 59812 } } peers { permanent_uuid: "a5b79d7057a340678b551ae411266cda" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 55527 } } I0312 20:11:26.018759 2871 leader_election.cc:231] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 71f80b7649e64a2bafe2eee9b67b14c1 I0312 20:11:26.019106 2871 leader_election.cc:231] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 75cbb9a95bd343cbad786fcb50d74c94 I0312 20:11:26.019486 2871 raft_consensus.cc:436] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 71f80b7649e64a2bafe2eee9b67b14c1) I0312 20:11:26.019692 2871 raft_consensus.cc:2700] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda: Snoozing failure detection for 0.554s (starting election) I0312 20:11:26.019879 2871 raft_consensus.cc:458] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71f80b7649e64a2bafe2eee9b67b14c1" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 52002 } } peers { permanent_uuid: "75cbb9a95bd343cbad786fcb50d74c94" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 59812 } } peers { permanent_uuid: "a5b79d7057a340678b551ae411266cda" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 55527 } } I0312 20:11:26.020593 2871 leader_election.cc:231] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 71f80b7649e64a2bafe2eee9b67b14c1 I0312 20:11:26.020912 2871 leader_election.cc:231] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 75cbb9a95bd343cbad786fcb50d74c94 I0312 20:11:26.022680 2443 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "c1432f133b26467b949ad48a19496e3a" candidate_uuid: "a5b79d7057a340678b551ae411266cda" candidate_term: 2 candidate_status { last_received { term: 1 index: 3715 } } ignore_live_leader: false dest_uuid: "71f80b7649e64a2bafe2eee9b67b14c1" is_pre_election: true I0312 20:11:26.023293 2443 raft_consensus.cc:2176] T c1432f133b26467b949ad48a19496e3a P 71f80b7649e64a2bafe2eee9b67b14c1 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate a5b79d7057a340678b551ae411266cda for term 2 because replica is either leader or believes a valid leader to be alive. I0312 20:11:26.023073 2569 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "c1432f133b26467b949ad48a19496e3a" candidate_uuid: "a5b79d7057a340678b551ae411266cda" candidate_term: 2 candidate_status { last_received { term: 1 index: 3715 } } ignore_live_leader: false dest_uuid: "75cbb9a95bd343cbad786fcb50d74c94" is_pre_election: true I0312 20:11:26.023591 2569 raft_consensus.cc:2176] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a5b79d7057a340678b551ae411266cda for term 2 because replica is either leader or believes a valid leader to be alive. I0312 20:11:26.024581 2629 leader_election.cc:377] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [CANDIDATE]: Term 2 pre-election: Vote denied by peer 75cbb9a95bd343cbad786fcb50d74c94. Message: Invalid argument: T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a5b79d7057a340678b551ae411266cda for term 2 because replica is either leader or believes a valid leader to be alive. I0312 20:11:26.025496 2443 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "c1432f133b26467b949ad48a19496e3a" candidate_uuid: "a5b79d7057a340678b551ae411266cda" candidate_term: 2 candidate_status { last_received { term: 1 index: 3715 } } ignore_live_leader: false dest_uuid: "71f80b7649e64a2bafe2eee9b67b14c1" is_pre_election: true I0312 20:11:26.025849 2627 leader_election.cc:377] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [CANDIDATE]: Term 2 pre-election: Vote denied by peer 71f80b7649e64a2bafe2eee9b67b14c1. Message: Invalid argument: T c1432f133b26467b949ad48a19496e3a P 71f80b7649e64a2bafe2eee9b67b14c1 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate a5b79d7057a340678b551ae411266cda for term 2 because replica is either leader or believes a valid leader to be alive. I0312 20:11:26.025964 2443 raft_consensus.cc:2176] T c1432f133b26467b949ad48a19496e3a P 71f80b7649e64a2bafe2eee9b67b14c1 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate a5b79d7057a340678b551ae411266cda for term 2 because replica is either leader or believes a valid leader to be alive. I0312 20:11:26.026059 2627 leader_election.cc:258] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. I0312 20:11:26.026819 2627 leader_election.cc:377] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [CANDIDATE]: Term 2 pre-election: Vote denied by peer 71f80b7649e64a2bafe2eee9b67b14c1. Message: Invalid argument: T c1432f133b26467b949ad48a19496e3a P 71f80b7649e64a2bafe2eee9b67b14c1 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate a5b79d7057a340678b551ae411266cda for term 2 because replica is either leader or believes a valid leader to be alive. I0312 20:11:26.028584 2569 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "c1432f133b26467b949ad48a19496e3a" candidate_uuid: "a5b79d7057a340678b551ae411266cda" candidate_term: 2 candidate_status { last_received { term: 1 index: 3715 } } ignore_live_leader: false dest_uuid: "75cbb9a95bd343cbad786fcb50d74c94" is_pre_election: true I0312 20:11:26.029083 2569 raft_consensus.cc:2176] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a5b79d7057a340678b551ae411266cda for term 2 because replica is either leader or believes a valid leader to be alive. I0312 20:11:26.029912 2629 leader_election.cc:377] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [CANDIDATE]: Term 2 pre-election: Vote denied by peer 75cbb9a95bd343cbad786fcb50d74c94. Message: Invalid argument: T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a5b79d7057a340678b551ae411266cda for term 2 because replica is either leader or believes a valid leader to be alive. I0312 20:11:26.030124 2629 leader_election.cc:258] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. I0312 20:11:26.031103 2871 raft_consensus.cc:2700] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda: Snoozing failure detection for 0.546s (election complete) I0312 20:11:26.031334 2871 raft_consensus.cc:2455] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority I0312 20:11:26.031631 2874 raft_consensus.cc:2700] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda: Snoozing failure detection for 0.653s (election complete) I0312 20:11:26.031886 2874 raft_consensus.cc:2455] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority I0312 20:11:26.044708 2875 raft_consensus.cc:436] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 71f80b7649e64a2bafe2eee9b67b14c1) I0312 20:11:26.044980 2875 raft_consensus.cc:2700] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94: Snoozing failure detection for 3.338s (starting election) I0312 20:11:26.045269 2875 raft_consensus.cc:458] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71f80b7649e64a2bafe2eee9b67b14c1" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 52002 } } peers { permanent_uuid: "75cbb9a95bd343cbad786fcb50d74c94" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 59812 } } peers { permanent_uuid: "a5b79d7057a340678b551ae411266cda" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 55527 } } I0312 20:11:26.047397 2875 leader_election.cc:231] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 71f80b7649e64a2bafe2eee9b67b14c1 I0312 20:11:26.050822 2875 leader_election.cc:231] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer a5b79d7057a340678b551ae411266cda I0312 20:11:26.050776 2443 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "c1432f133b26467b949ad48a19496e3a" candidate_uuid: "75cbb9a95bd343cbad786fcb50d74c94" candidate_term: 2 candidate_status { last_received { term: 1 index: 3715 } } ignore_live_leader: false dest_uuid: "71f80b7649e64a2bafe2eee9b67b14c1" is_pre_election: true I0312 20:11:26.051336 2443 raft_consensus.cc:2176] T c1432f133b26467b949ad48a19496e3a P 71f80b7649e64a2bafe2eee9b67b14c1 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate 75cbb9a95bd343cbad786fcb50d74c94 for term 2 because replica is either leader or believes a valid leader to be alive. I0312 20:11:26.052191 2502 leader_election.cc:377] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 71f80b7649e64a2bafe2eee9b67b14c1. Message: Invalid argument: T c1432f133b26467b949ad48a19496e3a P 71f80b7649e64a2bafe2eee9b67b14c1 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate 75cbb9a95bd343cbad786fcb50d74c94 for term 2 because replica is either leader or believes a valid leader to be alive. I0312 20:11:26.052925 2694 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "c1432f133b26467b949ad48a19496e3a" candidate_uuid: "75cbb9a95bd343cbad786fcb50d74c94" candidate_term: 2 candidate_status { last_received { term: 1 index: 3715 } } ignore_live_leader: false dest_uuid: "a5b79d7057a340678b551ae411266cda" is_pre_election: true I0312 20:11:26.056557 2694 raft_consensus.cc:2190] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 75cbb9a95bd343cbad786fcb50d74c94 for term 2 because replica is already servicing an update from a current leader or another vote. I0312 20:11:26.057523 2500 leader_election.cc:377] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [CANDIDATE]: Term 2 pre-election: Vote denied by peer a5b79d7057a340678b551ae411266cda. Message: Service unavailable: T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 75cbb9a95bd343cbad786fcb50d74c94 for term 2 because replica is already servicing an update from a current leader or another vote. I0312 20:11:26.057750 2500 leader_election.cc:258] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. I0312 20:11:26.059113 2875 raft_consensus.cc:2700] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94: Snoozing failure detection for 15.677s (election complete) I0312 20:11:26.059664 2875 raft_consensus.cc:2455] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority I0312 20:11:26.754114 2879 raft_consensus.cc:436] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 71f80b7649e64a2bafe2eee9b67b14c1) I0312 20:11:26.754586 2879 raft_consensus.cc:2700] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94: Snoozing failure detection for 17.419s (starting election) I0312 20:11:26.754989 2879 raft_consensus.cc:458] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71f80b7649e64a2bafe2eee9b67b14c1" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 52002 } } peers { permanent_uuid: "75cbb9a95bd343cbad786fcb50d74c94" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 59812 } } peers { permanent_uuid: "a5b79d7057a340678b551ae411266cda" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 55527 } } I0312 20:11:26.756036 2879 leader_election.cc:231] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 71f80b7649e64a2bafe2eee9b67b14c1 I0312 20:11:26.756417 2879 leader_election.cc:231] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer a5b79d7057a340678b551ae411266cda I0312 20:11:26.758049 2693 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "c1432f133b26467b949ad48a19496e3a" candidate_uuid: "75cbb9a95bd343cbad786fcb50d74c94" candidate_term: 2 candidate_status { last_received { term: 1 index: 3838 } } ignore_live_leader: false dest_uuid: "a5b79d7057a340678b551ae411266cda" is_pre_election: true I0312 20:11:26.758479 2693 raft_consensus.cc:2176] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 75cbb9a95bd343cbad786fcb50d74c94 for term 2 because replica is either leader or believes a valid leader to be alive. I0312 20:11:26.760063 2500 leader_election.cc:377] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [CANDIDATE]: Term 2 pre-election: Vote denied by peer a5b79d7057a340678b551ae411266cda. Message: Invalid argument: T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 75cbb9a95bd343cbad786fcb50d74c94 for term 2 because replica is either leader or believes a valid leader to be alive. I0312 20:11:26.761431 2443 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "c1432f133b26467b949ad48a19496e3a" candidate_uuid: "75cbb9a95bd343cbad786fcb50d74c94" candidate_term: 2 candidate_status { last_received { term: 1 index: 3838 } } ignore_live_leader: false dest_uuid: "71f80b7649e64a2bafe2eee9b67b14c1" is_pre_election: true I0312 20:11:26.761857 2443 raft_consensus.cc:2176] T c1432f133b26467b949ad48a19496e3a P 71f80b7649e64a2bafe2eee9b67b14c1 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate 75cbb9a95bd343cbad786fcb50d74c94 for term 2 because replica is either leader or believes a valid leader to be alive. I0312 20:11:26.762744 2502 leader_election.cc:377] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 71f80b7649e64a2bafe2eee9b67b14c1. Message: Invalid argument: T c1432f133b26467b949ad48a19496e3a P 71f80b7649e64a2bafe2eee9b67b14c1 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate 75cbb9a95bd343cbad786fcb50d74c94 for term 2 because replica is either leader or believes a valid leader to be alive. I0312 20:11:26.762920 2502 leader_election.cc:258] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. I0312 20:11:26.764524 2879 raft_consensus.cc:2700] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94: Snoozing failure detection for 13.500s (election complete) I0312 20:11:26.764745 2879 raft_consensus.cc:2455] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority I0312 20:11:28.003160 2881 raft_consensus.cc:436] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 71f80b7649e64a2bafe2eee9b67b14c1) I0312 20:11:28.005604 2881 raft_consensus.cc:2700] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94: Snoozing failure detection for 19.376s (starting election) I0312 20:11:28.005810 2881 raft_consensus.cc:458] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71f80b7649e64a2bafe2eee9b67b14c1" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 52002 } } peers { permanent_uuid: "75cbb9a95bd343cbad786fcb50d74c94" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 59812 } } peers { permanent_uuid: "a5b79d7057a340678b551ae411266cda" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 55527 } } I0312 20:11:28.006544 2881 leader_election.cc:231] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 71f80b7649e64a2bafe2eee9b67b14c1 I0312 20:11:28.006865 2882 raft_consensus.cc:436] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 71f80b7649e64a2bafe2eee9b67b14c1) I0312 20:11:28.007174 2882 raft_consensus.cc:2700] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda: Snoozing failure detection for 0.701s (starting election) I0312 20:11:28.007439 2882 raft_consensus.cc:458] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71f80b7649e64a2bafe2eee9b67b14c1" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 52002 } } peers { permanent_uuid: "75cbb9a95bd343cbad786fcb50d74c94" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 59812 } } peers { permanent_uuid: "a5b79d7057a340678b551ae411266cda" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 55527 } } I0312 20:11:28.008595 2882 leader_election.cc:231] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 71f80b7649e64a2bafe2eee9b67b14c1 I0312 20:11:28.009863 2443 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "c1432f133b26467b949ad48a19496e3a" candidate_uuid: "a5b79d7057a340678b551ae411266cda" candidate_term: 2 candidate_status { last_received { term: 1 index: 4055 } } ignore_live_leader: false dest_uuid: "71f80b7649e64a2bafe2eee9b67b14c1" is_pre_election: true I0312 20:11:28.010401 2443 raft_consensus.cc:2176] T c1432f133b26467b949ad48a19496e3a P 71f80b7649e64a2bafe2eee9b67b14c1 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate a5b79d7057a340678b551ae411266cda for term 2 because replica is either leader or believes a valid leader to be alive. I0312 20:11:28.010488 2881 leader_election.cc:231] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer a5b79d7057a340678b551ae411266cda I0312 20:11:28.011502 2627 leader_election.cc:377] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [CANDIDATE]: Term 2 pre-election: Vote denied by peer 71f80b7649e64a2bafe2eee9b67b14c1. Message: Invalid argument: T c1432f133b26467b949ad48a19496e3a P 71f80b7649e64a2bafe2eee9b67b14c1 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate a5b79d7057a340678b551ae411266cda for term 2 because replica is either leader or believes a valid leader to be alive. I0312 20:11:28.011811 2443 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "c1432f133b26467b949ad48a19496e3a" candidate_uuid: "75cbb9a95bd343cbad786fcb50d74c94" candidate_term: 2 candidate_status { last_received { term: 1 index: 4052 } } ignore_live_leader: false dest_uuid: "71f80b7649e64a2bafe2eee9b67b14c1" is_pre_election: true I0312 20:11:28.012303 2882 leader_election.cc:231] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 75cbb9a95bd343cbad786fcb50d74c94 I0312 20:11:28.013555 2568 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "c1432f133b26467b949ad48a19496e3a" candidate_uuid: "a5b79d7057a340678b551ae411266cda" candidate_term: 2 candidate_status { last_received { term: 1 index: 4055 } } ignore_live_leader: false dest_uuid: "75cbb9a95bd343cbad786fcb50d74c94" is_pre_election: true I0312 20:11:28.013607 2694 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "c1432f133b26467b949ad48a19496e3a" candidate_uuid: "75cbb9a95bd343cbad786fcb50d74c94" candidate_term: 2 candidate_status { last_received { term: 1 index: 4052 } } ignore_live_leader: false dest_uuid: "a5b79d7057a340678b551ae411266cda" is_pre_election: true I0312 20:11:28.014031 2568 raft_consensus.cc:2190] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a5b79d7057a340678b551ae411266cda for term 2 because replica is already servicing an update from a current leader or another vote. I0312 20:11:28.014096 2694 raft_consensus.cc:2176] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 75cbb9a95bd343cbad786fcb50d74c94 for term 2 because replica is either leader or believes a valid leader to be alive. I0312 20:11:28.015139 2500 leader_election.cc:377] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [CANDIDATE]: Term 2 pre-election: Vote denied by peer a5b79d7057a340678b551ae411266cda. Message: Invalid argument: T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 75cbb9a95bd343cbad786fcb50d74c94 for term 2 because replica is either leader or believes a valid leader to be alive. I0312 20:11:28.016361 2629 leader_election.cc:377] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [CANDIDATE]: Term 2 pre-election: Vote denied by peer 75cbb9a95bd343cbad786fcb50d74c94. Message: Service unavailable: T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a5b79d7057a340678b551ae411266cda for term 2 because replica is already servicing an update from a current leader or another vote. I0312 20:11:28.016575 2629 leader_election.cc:258] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. I0312 20:11:28.017694 2443 raft_consensus.cc:2190] T c1432f133b26467b949ad48a19496e3a P 71f80b7649e64a2bafe2eee9b67b14c1 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate 75cbb9a95bd343cbad786fcb50d74c94 for term 2 because replica is already servicing an update from a current leader or another vote. I0312 20:11:28.018985 2502 leader_election.cc:377] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 71f80b7649e64a2bafe2eee9b67b14c1. Message: Service unavailable: T c1432f133b26467b949ad48a19496e3a P 71f80b7649e64a2bafe2eee9b67b14c1 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate 75cbb9a95bd343cbad786fcb50d74c94 for term 2 because replica is already servicing an update from a current leader or another vote. I0312 20:11:28.019198 2502 leader_election.cc:258] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. I0312 20:11:28.020669 2881 raft_consensus.cc:2700] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94: Snoozing failure detection for 12.047s (election complete) I0312 20:11:28.020929 2881 raft_consensus.cc:2455] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority I0312 20:11:28.022476 2883 raft_consensus.cc:2700] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda: Snoozing failure detection for 0.563s (election complete) I0312 20:11:28.022711 2883 raft_consensus.cc:2455] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority I0312 20:11:29.734493 2890 raft_consensus.cc:436] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 71f80b7649e64a2bafe2eee9b67b14c1) I0312 20:11:29.734774 2890 raft_consensus.cc:2700] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda: Snoozing failure detection for 0.993s (starting election) I0312 20:11:29.735004 2890 raft_consensus.cc:458] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71f80b7649e64a2bafe2eee9b67b14c1" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 52002 } } peers { permanent_uuid: "75cbb9a95bd343cbad786fcb50d74c94" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 59812 } } peers { permanent_uuid: "a5b79d7057a340678b551ae411266cda" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 55527 } } I0312 20:11:29.736008 2890 leader_election.cc:231] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 71f80b7649e64a2bafe2eee9b67b14c1 I0312 20:11:29.738812 2890 leader_election.cc:231] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 75cbb9a95bd343cbad786fcb50d74c94 I0312 20:11:29.738804 2891 raft_consensus.cc:436] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 71f80b7649e64a2bafe2eee9b67b14c1) I0312 20:11:29.739104 2891 raft_consensus.cc:2700] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94: Snoozing failure detection for 15.068s (starting election) I0312 20:11:29.739363 2891 raft_consensus.cc:458] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71f80b7649e64a2bafe2eee9b67b14c1" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 52002 } } peers { permanent_uuid: "75cbb9a95bd343cbad786fcb50d74c94" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 59812 } } peers { permanent_uuid: "a5b79d7057a340678b551ae411266cda" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 55527 } } I0312 20:11:29.740087 2568 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "c1432f133b26467b949ad48a19496e3a" candidate_uuid: "a5b79d7057a340678b551ae411266cda" candidate_term: 2 candidate_status { last_received { term: 1 index: 4344 } } ignore_live_leader: false dest_uuid: "75cbb9a95bd343cbad786fcb50d74c94" is_pre_election: true I0312 20:11:29.740363 2891 leader_election.cc:231] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 71f80b7649e64a2bafe2eee9b67b14c1 I0312 20:11:29.741011 2568 raft_consensus.cc:2190] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a5b79d7057a340678b551ae411266cda for term 2 because replica is already servicing an update from a current leader or another vote. I0312 20:11:29.741441 2891 leader_election.cc:231] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer a5b79d7057a340678b551ae411266cda I0312 20:11:29.741875 2443 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "c1432f133b26467b949ad48a19496e3a" candidate_uuid: "75cbb9a95bd343cbad786fcb50d74c94" candidate_term: 2 candidate_status { last_received { term: 1 index: 4344 } } ignore_live_leader: false dest_uuid: "71f80b7649e64a2bafe2eee9b67b14c1" is_pre_election: true I0312 20:11:29.742391 2443 raft_consensus.cc:2176] T c1432f133b26467b949ad48a19496e3a P 71f80b7649e64a2bafe2eee9b67b14c1 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate 75cbb9a95bd343cbad786fcb50d74c94 for term 2 because replica is either leader or believes a valid leader to be alive. I0312 20:11:29.743266 2629 leader_election.cc:377] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [CANDIDATE]: Term 2 pre-election: Vote denied by peer 75cbb9a95bd343cbad786fcb50d74c94. Message: Service unavailable: T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a5b79d7057a340678b551ae411266cda for term 2 because replica is already servicing an update from a current leader or another vote. I0312 20:11:29.746553 2502 leader_election.cc:377] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 71f80b7649e64a2bafe2eee9b67b14c1. Message: Invalid argument: T c1432f133b26467b949ad48a19496e3a P 71f80b7649e64a2bafe2eee9b67b14c1 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate 75cbb9a95bd343cbad786fcb50d74c94 for term 2 because replica is either leader or believes a valid leader to be alive. I0312 20:11:29.748105 2443 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "c1432f133b26467b949ad48a19496e3a" candidate_uuid: "a5b79d7057a340678b551ae411266cda" candidate_term: 2 candidate_status { last_received { term: 1 index: 4344 } } ignore_live_leader: false dest_uuid: "71f80b7649e64a2bafe2eee9b67b14c1" is_pre_election: true I0312 20:11:29.749264 2443 raft_consensus.cc:2190] T c1432f133b26467b949ad48a19496e3a P 71f80b7649e64a2bafe2eee9b67b14c1 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate a5b79d7057a340678b551ae411266cda for term 2 because replica is already servicing an update from a current leader or another vote. I0312 20:11:29.750077 2627 leader_election.cc:377] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [CANDIDATE]: Term 2 pre-election: Vote denied by peer 71f80b7649e64a2bafe2eee9b67b14c1. Message: Service unavailable: T c1432f133b26467b949ad48a19496e3a P 71f80b7649e64a2bafe2eee9b67b14c1 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate a5b79d7057a340678b551ae411266cda for term 2 because replica is already servicing an update from a current leader or another vote. I0312 20:11:29.750291 2627 leader_election.cc:258] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. I0312 20:11:29.757305 2693 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "c1432f133b26467b949ad48a19496e3a" candidate_uuid: "75cbb9a95bd343cbad786fcb50d74c94" candidate_term: 2 candidate_status { last_received { term: 1 index: 4344 } } ignore_live_leader: false dest_uuid: "a5b79d7057a340678b551ae411266cda" is_pre_election: true I0312 20:11:29.757468 2890 raft_consensus.cc:2700] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda: Snoozing failure detection for 0.748s (election complete) I0312 20:11:29.757668 2890 raft_consensus.cc:2455] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority I0312 20:11:29.757928 2693 raft_consensus.cc:2176] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 75cbb9a95bd343cbad786fcb50d74c94 for term 2 because replica is either leader or believes a valid leader to be alive. I0312 20:11:29.758960 2500 leader_election.cc:377] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [CANDIDATE]: Term 2 pre-election: Vote denied by peer a5b79d7057a340678b551ae411266cda. Message: Invalid argument: T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 75cbb9a95bd343cbad786fcb50d74c94 for term 2 because replica is either leader or believes a valid leader to be alive. I0312 20:11:29.759187 2500 leader_election.cc:258] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. I0312 20:11:29.759582 2891 raft_consensus.cc:2700] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94: Snoozing failure detection for 16.293s (election complete) I0312 20:11:29.759737 2891 raft_consensus.cc:2455] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority I0312 20:11:29.992892 2891 raft_consensus.cc:436] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 71f80b7649e64a2bafe2eee9b67b14c1) I0312 20:11:29.993173 2891 raft_consensus.cc:2700] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94: Snoozing failure detection for 10.468s (starting election) I0312 20:11:29.993384 2891 raft_consensus.cc:458] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71f80b7649e64a2bafe2eee9b67b14c1" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 52002 } } peers { permanent_uuid: "75cbb9a95bd343cbad786fcb50d74c94" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 59812 } } peers { permanent_uuid: "a5b79d7057a340678b551ae411266cda" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 55527 } } I0312 20:11:29.994367 2891 leader_election.cc:231] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 71f80b7649e64a2bafe2eee9b67b14c1 I0312 20:11:29.994717 2891 leader_election.cc:231] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer a5b79d7057a340678b551ae411266cda I0312 20:11:29.996615 2694 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "c1432f133b26467b949ad48a19496e3a" candidate_uuid: "75cbb9a95bd343cbad786fcb50d74c94" candidate_term: 2 candidate_status { last_received { term: 1 index: 4386 } } ignore_live_leader: false dest_uuid: "a5b79d7057a340678b551ae411266cda" is_pre_election: true I0312 20:11:29.996937 2443 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "c1432f133b26467b949ad48a19496e3a" candidate_uuid: "75cbb9a95bd343cbad786fcb50d74c94" candidate_term: 2 candidate_status { last_received { term: 1 index: 4386 } } ignore_live_leader: false dest_uuid: "71f80b7649e64a2bafe2eee9b67b14c1" is_pre_election: true I0312 20:11:29.997349 2443 raft_consensus.cc:2176] T c1432f133b26467b949ad48a19496e3a P 71f80b7649e64a2bafe2eee9b67b14c1 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate 75cbb9a95bd343cbad786fcb50d74c94 for term 2 because replica is either leader or believes a valid leader to be alive. I0312 20:11:29.998219 2502 leader_election.cc:377] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 71f80b7649e64a2bafe2eee9b67b14c1. Message: Invalid argument: T c1432f133b26467b949ad48a19496e3a P 71f80b7649e64a2bafe2eee9b67b14c1 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate 75cbb9a95bd343cbad786fcb50d74c94 for term 2 because replica is either leader or believes a valid leader to be alive. I0312 20:11:29.999075 2694 raft_consensus.cc:2190] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 75cbb9a95bd343cbad786fcb50d74c94 for term 2 because replica is already servicing an update from a current leader or another vote. I0312 20:11:30.000011 2500 leader_election.cc:377] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [CANDIDATE]: Term 2 pre-election: Vote denied by peer a5b79d7057a340678b551ae411266cda. Message: Service unavailable: T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 75cbb9a95bd343cbad786fcb50d74c94 for term 2 because replica is already servicing an update from a current leader or another vote. I0312 20:11:30.000195 2500 leader_election.cc:258] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. I0312 20:11:30.000538 2891 raft_consensus.cc:2700] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94: Snoozing failure detection for 19.008s (election complete) I0312 20:11:30.000743 2891 raft_consensus.cc:2455] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority W0312 20:11:30.724553 2380 debug-util.cc:396] Leaking SignalData structure 0x7b080002b080 after lost signal to thread 2360 W0312 20:11:30.725569 2380 debug-util.cc:396] Leaking SignalData structure 0x7b080009c700 after lost signal to thread 2485 I0312 20:11:30.738200 2893 raft_consensus.cc:436] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 71f80b7649e64a2bafe2eee9b67b14c1) I0312 20:11:30.738622 2893 raft_consensus.cc:2700] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94: Snoozing failure detection for 10.521s (starting election) I0312 20:11:30.738903 2893 raft_consensus.cc:458] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71f80b7649e64a2bafe2eee9b67b14c1" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 52002 } } peers { permanent_uuid: "75cbb9a95bd343cbad786fcb50d74c94" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 59812 } } peers { permanent_uuid: "a5b79d7057a340678b551ae411266cda" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 55527 } } I0312 20:11:30.740061 2893 leader_election.cc:231] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 71f80b7649e64a2bafe2eee9b67b14c1 I0312 20:11:30.740435 2893 leader_election.cc:231] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer a5b79d7057a340678b551ae411266cda I0312 20:11:30.741602 2443 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "c1432f133b26467b949ad48a19496e3a" candidate_uuid: "75cbb9a95bd343cbad786fcb50d74c94" candidate_term: 2 candidate_status { last_received { term: 1 index: 4514 } } ignore_live_leader: false dest_uuid: "71f80b7649e64a2bafe2eee9b67b14c1" is_pre_election: true I0312 20:11:30.742127 2443 raft_consensus.cc:2176] T c1432f133b26467b949ad48a19496e3a P 71f80b7649e64a2bafe2eee9b67b14c1 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate 75cbb9a95bd343cbad786fcb50d74c94 for term 2 because replica is either leader or believes a valid leader to be alive. I0312 20:11:30.742697 2693 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "c1432f133b26467b949ad48a19496e3a" candidate_uuid: "75cbb9a95bd343cbad786fcb50d74c94" candidate_term: 2 candidate_status { last_received { term: 1 index: 4514 } } ignore_live_leader: false dest_uuid: "a5b79d7057a340678b551ae411266cda" is_pre_election: true I0312 20:11:30.742965 2502 leader_election.cc:377] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 71f80b7649e64a2bafe2eee9b67b14c1. Message: Invalid argument: T c1432f133b26467b949ad48a19496e3a P 71f80b7649e64a2bafe2eee9b67b14c1 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate 75cbb9a95bd343cbad786fcb50d74c94 for term 2 because replica is either leader or believes a valid leader to be alive. I0312 20:11:30.743245 2693 raft_consensus.cc:2176] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 75cbb9a95bd343cbad786fcb50d74c94 for term 2 because replica is either leader or believes a valid leader to be alive. I0312 20:11:30.744082 2500 leader_election.cc:377] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [CANDIDATE]: Term 2 pre-election: Vote denied by peer a5b79d7057a340678b551ae411266cda. Message: Invalid argument: T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 75cbb9a95bd343cbad786fcb50d74c94 for term 2 because replica is either leader or believes a valid leader to be alive. I0312 20:11:30.744303 2500 leader_election.cc:258] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. I0312 20:11:30.749909 2893 raft_consensus.cc:2700] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94: Snoozing failure detection for 2.947s (election complete) I0312 20:11:30.750380 2893 raft_consensus.cc:2455] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority I0312 20:11:31.125876 2894 raft_consensus.cc:436] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 71f80b7649e64a2bafe2eee9b67b14c1) I0312 20:11:31.126144 2894 raft_consensus.cc:2700] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94: Snoozing failure detection for 6.733s (starting election) I0312 20:11:31.126374 2894 raft_consensus.cc:458] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71f80b7649e64a2bafe2eee9b67b14c1" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 52002 } } peers { permanent_uuid: "75cbb9a95bd343cbad786fcb50d74c94" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 59812 } } peers { permanent_uuid: "a5b79d7057a340678b551ae411266cda" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 55527 } } I0312 20:11:31.127388 2894 leader_election.cc:231] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 71f80b7649e64a2bafe2eee9b67b14c1 I0312 20:11:31.127733 2894 leader_election.cc:231] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer a5b79d7057a340678b551ae411266cda I0312 20:11:31.129416 2693 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "c1432f133b26467b949ad48a19496e3a" candidate_uuid: "75cbb9a95bd343cbad786fcb50d74c94" candidate_term: 2 candidate_status { last_received { term: 1 index: 4564 } } ignore_live_leader: false dest_uuid: "a5b79d7057a340678b551ae411266cda" is_pre_election: true I0312 20:11:31.130084 2693 raft_consensus.cc:2162] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 75cbb9a95bd343cbad786fcb50d74c94 for term 2 because replica has last-logged OpId of term: 1 index: 4566, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4564. I0312 20:11:31.131523 2443 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "c1432f133b26467b949ad48a19496e3a" candidate_uuid: "75cbb9a95bd343cbad786fcb50d74c94" candidate_term: 2 candidate_status { last_received { term: 1 index: 4564 } } ignore_live_leader: false dest_uuid: "71f80b7649e64a2bafe2eee9b67b14c1" is_pre_election: true I0312 20:11:31.131969 2443 raft_consensus.cc:2190] T c1432f133b26467b949ad48a19496e3a P 71f80b7649e64a2bafe2eee9b67b14c1 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate 75cbb9a95bd343cbad786fcb50d74c94 for term 2 because replica is already servicing an update from a current leader or another vote. I0312 20:11:31.132067 2500 leader_election.cc:377] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [CANDIDATE]: Term 2 pre-election: Vote denied by peer a5b79d7057a340678b551ae411266cda. Message: Invalid argument: T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 75cbb9a95bd343cbad786fcb50d74c94 for term 2 because replica has last-logged OpId of term: 1 index: 4566, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4564. I0312 20:11:31.134217 2897 raft_consensus.cc:436] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 71f80b7649e64a2bafe2eee9b67b14c1) I0312 20:11:31.134459 2897 raft_consensus.cc:2700] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda: Snoozing failure detection for 2.588s (starting election) I0312 20:11:31.134685 2897 raft_consensus.cc:458] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71f80b7649e64a2bafe2eee9b67b14c1" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 52002 } } peers { permanent_uuid: "75cbb9a95bd343cbad786fcb50d74c94" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 59812 } } peers { permanent_uuid: "a5b79d7057a340678b551ae411266cda" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 55527 } } I0312 20:11:31.135033 2502 leader_election.cc:377] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 71f80b7649e64a2bafe2eee9b67b14c1. Message: Service unavailable: T c1432f133b26467b949ad48a19496e3a P 71f80b7649e64a2bafe2eee9b67b14c1 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate 75cbb9a95bd343cbad786fcb50d74c94 for term 2 because replica is already servicing an update from a current leader or another vote. I0312 20:11:31.135259 2502 leader_election.cc:258] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. I0312 20:11:31.135699 2897 leader_election.cc:231] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 71f80b7649e64a2bafe2eee9b67b14c1 I0312 20:11:31.135861 2894 raft_consensus.cc:2700] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94: Snoozing failure detection for 9.091s (election complete) I0312 20:11:31.136046 2897 leader_election.cc:231] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 75cbb9a95bd343cbad786fcb50d74c94 I0312 20:11:31.136051 2894 raft_consensus.cc:2455] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority I0312 20:11:31.137692 2569 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "c1432f133b26467b949ad48a19496e3a" candidate_uuid: "a5b79d7057a340678b551ae411266cda" candidate_term: 2 candidate_status { last_received { term: 1 index: 4566 } } ignore_live_leader: false dest_uuid: "75cbb9a95bd343cbad786fcb50d74c94" is_pre_election: true I0312 20:11:31.138181 2443 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "c1432f133b26467b949ad48a19496e3a" candidate_uuid: "a5b79d7057a340678b551ae411266cda" candidate_term: 2 candidate_status { last_received { term: 1 index: 4566 } } ignore_live_leader: false dest_uuid: "71f80b7649e64a2bafe2eee9b67b14c1" is_pre_election: true I0312 20:11:31.138519 2569 raft_consensus.cc:2190] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a5b79d7057a340678b551ae411266cda for term 2 because replica is already servicing an update from a current leader or another vote. I0312 20:11:31.138685 2443 raft_consensus.cc:2176] T c1432f133b26467b949ad48a19496e3a P 71f80b7649e64a2bafe2eee9b67b14c1 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate a5b79d7057a340678b551ae411266cda for term 2 because replica is either leader or believes a valid leader to be alive. I0312 20:11:31.139441 2629 leader_election.cc:377] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [CANDIDATE]: Term 2 pre-election: Vote denied by peer 75cbb9a95bd343cbad786fcb50d74c94. Message: Service unavailable: T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a5b79d7057a340678b551ae411266cda for term 2 because replica is already servicing an update from a current leader or another vote. I0312 20:11:31.140413 2627 leader_election.cc:377] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [CANDIDATE]: Term 2 pre-election: Vote denied by peer 71f80b7649e64a2bafe2eee9b67b14c1. Message: Invalid argument: T c1432f133b26467b949ad48a19496e3a P 71f80b7649e64a2bafe2eee9b67b14c1 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate a5b79d7057a340678b551ae411266cda for term 2 because replica is either leader or believes a valid leader to be alive. I0312 20:11:31.140592 2627 leader_election.cc:258] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. I0312 20:11:31.140926 2897 raft_consensus.cc:2700] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda: Snoozing failure detection for 2.237s (election complete) I0312 20:11:31.141160 2897 raft_consensus.cc:2455] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority I0312 20:11:31.471035 2894 raft_consensus.cc:436] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 71f80b7649e64a2bafe2eee9b67b14c1) I0312 20:11:31.471251 2894 raft_consensus.cc:2700] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94: Snoozing failure detection for 7.037s (starting election) I0312 20:11:31.471446 2894 raft_consensus.cc:458] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71f80b7649e64a2bafe2eee9b67b14c1" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 52002 } } peers { permanent_uuid: "75cbb9a95bd343cbad786fcb50d74c94" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 59812 } } peers { permanent_uuid: "a5b79d7057a340678b551ae411266cda" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 55527 } } I0312 20:11:31.472162 2894 leader_election.cc:231] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 71f80b7649e64a2bafe2eee9b67b14c1 I0312 20:11:31.472492 2894 leader_election.cc:231] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer a5b79d7057a340678b551ae411266cda I0312 20:11:31.474018 2693 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "c1432f133b26467b949ad48a19496e3a" candidate_uuid: "75cbb9a95bd343cbad786fcb50d74c94" candidate_term: 2 candidate_status { last_received { term: 1 index: 4613 } } ignore_live_leader: false dest_uuid: "a5b79d7057a340678b551ae411266cda" is_pre_election: true I0312 20:11:31.474519 2693 raft_consensus.cc:2176] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 75cbb9a95bd343cbad786fcb50d74c94 for term 2 because replica is either leader or believes a valid leader to be alive. I0312 20:11:31.475054 2443 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "c1432f133b26467b949ad48a19496e3a" candidate_uuid: "75cbb9a95bd343cbad786fcb50d74c94" candidate_term: 2 candidate_status { last_received { term: 1 index: 4613 } } ignore_live_leader: false dest_uuid: "71f80b7649e64a2bafe2eee9b67b14c1" is_pre_election: true I0312 20:11:31.475481 2500 leader_election.cc:377] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [CANDIDATE]: Term 2 pre-election: Vote denied by peer a5b79d7057a340678b551ae411266cda. Message: Invalid argument: T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 75cbb9a95bd343cbad786fcb50d74c94 for term 2 because replica is either leader or believes a valid leader to be alive. I0312 20:11:31.477118 2443 raft_consensus.cc:2190] T c1432f133b26467b949ad48a19496e3a P 71f80b7649e64a2bafe2eee9b67b14c1 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate 75cbb9a95bd343cbad786fcb50d74c94 for term 2 because replica is already servicing an update from a current leader or another vote. I0312 20:11:31.478224 2502 leader_election.cc:377] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 71f80b7649e64a2bafe2eee9b67b14c1. Message: Service unavailable: T c1432f133b26467b949ad48a19496e3a P 71f80b7649e64a2bafe2eee9b67b14c1 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate 75cbb9a95bd343cbad786fcb50d74c94 for term 2 because replica is already servicing an update from a current leader or another vote. I0312 20:11:31.478435 2502 leader_election.cc:258] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. I0312 20:11:31.484624 2894 raft_consensus.cc:2700] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94: Snoozing failure detection for 15.711s (election complete) I0312 20:11:31.484824 2894 raft_consensus.cc:2455] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority I0312 20:11:31.586047 2894 raft_consensus.cc:436] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 71f80b7649e64a2bafe2eee9b67b14c1) I0312 20:11:31.586258 2894 raft_consensus.cc:2700] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94: Snoozing failure detection for 3.180s (starting election) I0312 20:11:31.586460 2894 raft_consensus.cc:458] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71f80b7649e64a2bafe2eee9b67b14c1" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 52002 } } peers { permanent_uuid: "75cbb9a95bd343cbad786fcb50d74c94" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 59812 } } peers { permanent_uuid: "a5b79d7057a340678b551ae411266cda" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 55527 } } I0312 20:11:31.587326 2894 leader_election.cc:231] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 71f80b7649e64a2bafe2eee9b67b14c1 I0312 20:11:31.587678 2894 leader_election.cc:231] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer a5b79d7057a340678b551ae411266cda I0312 20:11:31.589589 2443 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "c1432f133b26467b949ad48a19496e3a" candidate_uuid: "75cbb9a95bd343cbad786fcb50d74c94" candidate_term: 2 candidate_status { last_received { term: 1 index: 4627 } } ignore_live_leader: false dest_uuid: "71f80b7649e64a2bafe2eee9b67b14c1" is_pre_election: true I0312 20:11:31.589609 2694 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "c1432f133b26467b949ad48a19496e3a" candidate_uuid: "75cbb9a95bd343cbad786fcb50d74c94" candidate_term: 2 candidate_status { last_received { term: 1 index: 4627 } } ignore_live_leader: false dest_uuid: "a5b79d7057a340678b551ae411266cda" is_pre_election: true I0312 20:11:31.590100 2443 raft_consensus.cc:2190] T c1432f133b26467b949ad48a19496e3a P 71f80b7649e64a2bafe2eee9b67b14c1 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate 75cbb9a95bd343cbad786fcb50d74c94 for term 2 because replica is already servicing an update from a current leader or another vote. I0312 20:11:31.590113 2694 raft_consensus.cc:2176] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 75cbb9a95bd343cbad786fcb50d74c94 for term 2 because replica is either leader or believes a valid leader to be alive. I0312 20:11:31.590863 2500 leader_election.cc:377] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [CANDIDATE]: Term 2 pre-election: Vote denied by peer a5b79d7057a340678b551ae411266cda. Message: Invalid argument: T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 75cbb9a95bd343cbad786fcb50d74c94 for term 2 because replica is either leader or believes a valid leader to be alive. I0312 20:11:31.591444 2502 leader_election.cc:377] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 71f80b7649e64a2bafe2eee9b67b14c1. Message: Service unavailable: T c1432f133b26467b949ad48a19496e3a P 71f80b7649e64a2bafe2eee9b67b14c1 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate 75cbb9a95bd343cbad786fcb50d74c94 for term 2 because replica is already servicing an update from a current leader or another vote. I0312 20:11:31.591656 2502 leader_election.cc:258] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. I0312 20:11:31.592237 2894 raft_consensus.cc:2700] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94: Snoozing failure detection for 15.036s (election complete) I0312 20:11:31.592392 2894 raft_consensus.cc:2455] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority I0312 20:11:31.612864 2894 raft_consensus.cc:436] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 71f80b7649e64a2bafe2eee9b67b14c1) I0312 20:11:31.613059 2894 raft_consensus.cc:2700] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94: Snoozing failure detection for 15.116s (starting election) I0312 20:11:31.613251 2894 raft_consensus.cc:458] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71f80b7649e64a2bafe2eee9b67b14c1" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 52002 } } peers { permanent_uuid: "75cbb9a95bd343cbad786fcb50d74c94" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 59812 } } peers { permanent_uuid: "a5b79d7057a340678b551ae411266cda" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 55527 } } I0312 20:11:31.613917 2894 leader_election.cc:231] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 71f80b7649e64a2bafe2eee9b67b14c1 I0312 20:11:31.614189 2894 leader_election.cc:231] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer a5b79d7057a340678b551ae411266cda I0312 20:11:31.615247 2693 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "c1432f133b26467b949ad48a19496e3a" candidate_uuid: "75cbb9a95bd343cbad786fcb50d74c94" candidate_term: 2 candidate_status { last_received { term: 1 index: 4630 } } ignore_live_leader: false dest_uuid: "a5b79d7057a340678b551ae411266cda" is_pre_election: true I0312 20:11:31.615950 2693 raft_consensus.cc:2190] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 75cbb9a95bd343cbad786fcb50d74c94 for term 2 because replica is already servicing an update from a current leader or another vote. I0312 20:11:31.616825 2500 leader_election.cc:377] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [CANDIDATE]: Term 2 pre-election: Vote denied by peer a5b79d7057a340678b551ae411266cda. Message: Service unavailable: T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 75cbb9a95bd343cbad786fcb50d74c94 for term 2 because replica is already servicing an update from a current leader or another vote. I0312 20:11:31.618175 2443 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "c1432f133b26467b949ad48a19496e3a" candidate_uuid: "75cbb9a95bd343cbad786fcb50d74c94" candidate_term: 2 candidate_status { last_received { term: 1 index: 4630 } } ignore_live_leader: false dest_uuid: "71f80b7649e64a2bafe2eee9b67b14c1" is_pre_election: true I0312 20:11:31.618671 2443 raft_consensus.cc:2176] T c1432f133b26467b949ad48a19496e3a P 71f80b7649e64a2bafe2eee9b67b14c1 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate 75cbb9a95bd343cbad786fcb50d74c94 for term 2 because replica is either leader or believes a valid leader to be alive. I0312 20:11:31.622759 2502 leader_election.cc:377] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 71f80b7649e64a2bafe2eee9b67b14c1. Message: Invalid argument: T c1432f133b26467b949ad48a19496e3a P 71f80b7649e64a2bafe2eee9b67b14c1 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate 75cbb9a95bd343cbad786fcb50d74c94 for term 2 because replica is either leader or believes a valid leader to be alive. I0312 20:11:31.622969 2502 leader_election.cc:258] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. I0312 20:11:31.623528 2894 raft_consensus.cc:2700] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94: Snoozing failure detection for 6.020s (election complete) I0312 20:11:31.623713 2894 raft_consensus.cc:2455] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority I0312 20:11:31.960299 2894 raft_consensus.cc:436] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 71f80b7649e64a2bafe2eee9b67b14c1) I0312 20:11:31.960510 2894 raft_consensus.cc:2700] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94: Snoozing failure detection for 2.650s (starting election) I0312 20:11:31.960693 2894 raft_consensus.cc:458] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71f80b7649e64a2bafe2eee9b67b14c1" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 52002 } } peers { permanent_uuid: "75cbb9a95bd343cbad786fcb50d74c94" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 59812 } } peers { permanent_uuid: "a5b79d7057a340678b551ae411266cda" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 55527 } } I0312 20:11:31.961592 2894 leader_election.cc:231] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 71f80b7649e64a2bafe2eee9b67b14c1 I0312 20:11:31.961900 2894 leader_election.cc:231] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer a5b79d7057a340678b551ae411266cda I0312 20:11:31.963021 2443 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "c1432f133b26467b949ad48a19496e3a" candidate_uuid: "75cbb9a95bd343cbad786fcb50d74c94" candidate_term: 2 candidate_status { last_received { term: 1 index: 4675 } } ignore_live_leader: false dest_uuid: "71f80b7649e64a2bafe2eee9b67b14c1" is_pre_election: true I0312 20:11:31.963739 2443 raft_consensus.cc:2176] T c1432f133b26467b949ad48a19496e3a P 71f80b7649e64a2bafe2eee9b67b14c1 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate 75cbb9a95bd343cbad786fcb50d74c94 for term 2 because replica is either leader or believes a valid leader to be alive. I0312 20:11:31.964668 2502 leader_election.cc:377] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 71f80b7649e64a2bafe2eee9b67b14c1. Message: Invalid argument: T c1432f133b26467b949ad48a19496e3a P 71f80b7649e64a2bafe2eee9b67b14c1 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate 75cbb9a95bd343cbad786fcb50d74c94 for term 2 because replica is either leader or believes a valid leader to be alive. I0312 20:11:31.965592 2900 raft_consensus.cc:436] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 71f80b7649e64a2bafe2eee9b67b14c1) I0312 20:11:31.965929 2900 raft_consensus.cc:2700] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda: Snoozing failure detection for 1.252s (starting election) I0312 20:11:31.965934 2693 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "c1432f133b26467b949ad48a19496e3a" candidate_uuid: "75cbb9a95bd343cbad786fcb50d74c94" candidate_term: 2 candidate_status { last_received { term: 1 index: 4675 } } ignore_live_leader: false dest_uuid: "a5b79d7057a340678b551ae411266cda" is_pre_election: true I0312 20:11:31.966240 2900 raft_consensus.cc:458] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71f80b7649e64a2bafe2eee9b67b14c1" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 52002 } } peers { permanent_uuid: "75cbb9a95bd343cbad786fcb50d74c94" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 59812 } } peers { permanent_uuid: "a5b79d7057a340678b551ae411266cda" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 55527 } } I0312 20:11:31.967289 2693 raft_consensus.cc:2700] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda: Snoozing failure detection for 2.225s (vote granted) I0312 20:11:31.967489 2693 raft_consensus.cc:2215] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 75cbb9a95bd343cbad786fcb50d74c94 in term 1. I0312 20:11:31.968134 2900 leader_election.cc:231] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 71f80b7649e64a2bafe2eee9b67b14c1 I0312 20:11:31.968628 2500 leader_election.cc:363] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [CANDIDATE]: Term 2 pre-election: Vote granted by peer a5b79d7057a340678b551ae411266cda I0312 20:11:31.968816 2500 leader_election.cc:258] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. I0312 20:11:31.968935 2900 leader_election.cc:231] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 75cbb9a95bd343cbad786fcb50d74c94 I0312 20:11:31.969446 2894 raft_consensus.cc:2700] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94: Snoozing failure detection for 7.893s (election complete) I0312 20:11:31.969651 2894 raft_consensus.cc:2507] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [term 1 FOLLOWER]: Leader pre-election won for term 2 I0312 20:11:31.969825 2894 raft_consensus.cc:436] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 71f80b7649e64a2bafe2eee9b67b14c1) I0312 20:11:31.970003 2894 raft_consensus.cc:2700] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94: Snoozing failure detection for 16.626s (starting election) I0312 20:11:31.970168 2894 raft_consensus.cc:2752] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [term 1 FOLLOWER]: Advancing to term 2 I0312 20:11:31.970165 2443 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "c1432f133b26467b949ad48a19496e3a" candidate_uuid: "a5b79d7057a340678b551ae411266cda" candidate_term: 2 candidate_status { last_received { term: 1 index: 4675 } } ignore_live_leader: false dest_uuid: "71f80b7649e64a2bafe2eee9b67b14c1" is_pre_election: true I0312 20:11:31.970640 2443 raft_consensus.cc:2176] T c1432f133b26467b949ad48a19496e3a P 71f80b7649e64a2bafe2eee9b67b14c1 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate a5b79d7057a340678b551ae411266cda for term 2 because replica is either leader or believes a valid leader to be alive. I0312 20:11:31.971606 2627 leader_election.cc:377] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [CANDIDATE]: Term 2 pre-election: Vote denied by peer 71f80b7649e64a2bafe2eee9b67b14c1. Message: Invalid argument: T c1432f133b26467b949ad48a19496e3a P 71f80b7649e64a2bafe2eee9b67b14c1 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate a5b79d7057a340678b551ae411266cda for term 2 because replica is either leader or believes a valid leader to be alive. I0312 20:11:31.972028 2569 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "c1432f133b26467b949ad48a19496e3a" candidate_uuid: "a5b79d7057a340678b551ae411266cda" candidate_term: 2 candidate_status { last_received { term: 1 index: 4675 } } ignore_live_leader: false dest_uuid: "75cbb9a95bd343cbad786fcb50d74c94" is_pre_election: true I0312 20:11:31.977547 2894 raft_consensus.cc:458] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71f80b7649e64a2bafe2eee9b67b14c1" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 52002 } } peers { permanent_uuid: "75cbb9a95bd343cbad786fcb50d74c94" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 59812 } } peers { permanent_uuid: "a5b79d7057a340678b551ae411266cda" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 55527 } } I0312 20:11:31.980836 2894 leader_election.cc:231] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [CANDIDATE]: Term 2 election: Requesting vote from peer 71f80b7649e64a2bafe2eee9b67b14c1 I0312 20:11:31.980988 2569 raft_consensus.cc:2176] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a5b79d7057a340678b551ae411266cda for term 2 because replica is either leader or believes a valid leader to be alive. I0312 20:11:31.982156 2568 raft_consensus.cc:1026] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [term 2 FOLLOWER]: Rejecting Update request from peer 71f80b7649e64a2bafe2eee9b67b14c1 for earlier term 1. Current term is 2. Ops: [] I0312 20:11:31.982385 2894 leader_election.cc:231] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [CANDIDATE]: Term 2 election: Requesting vote from peer a5b79d7057a340678b551ae411266cda I0312 20:11:31.983510 2693 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "c1432f133b26467b949ad48a19496e3a" candidate_uuid: "75cbb9a95bd343cbad786fcb50d74c94" candidate_term: 2 candidate_status { last_received { term: 1 index: 4675 } } ignore_live_leader: false dest_uuid: "a5b79d7057a340678b551ae411266cda" I0312 20:11:31.983963 2693 raft_consensus.cc:2176] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 75cbb9a95bd343cbad786fcb50d74c94 for term 2 because replica is either leader or believes a valid leader to be alive. I0312 20:11:31.983808 2896 consensus_queue.cc:1028] T c1432f133b26467b949ad48a19496e3a P 71f80b7649e64a2bafe2eee9b67b14c1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "75cbb9a95bd343cbad786fcb50d74c94" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 59812 }, Status: INVALID_TERM, Last received: 1.4675, Next index: 4676, Last known committed idx: 4675, Time since last communication: 0.000s I0312 20:11:31.984570 2895 raft_consensus.cc:2747] T c1432f133b26467b949ad48a19496e3a P 71f80b7649e64a2bafe2eee9b67b14c1 [term 1 LEADER]: Stepping down as leader of term 1 I0312 20:11:31.984760 2895 raft_consensus.cc:605] T c1432f133b26467b949ad48a19496e3a P 71f80b7649e64a2bafe2eee9b67b14c1 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 71f80b7649e64a2bafe2eee9b67b14c1, State: Running, Role: LEADER I0312 20:11:31.984925 2500 leader_election.cc:377] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [CANDIDATE]: Term 2 election: Vote denied by peer a5b79d7057a340678b551ae411266cda. Message: Invalid argument: T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 75cbb9a95bd343cbad786fcb50d74c94 for term 2 because replica is either leader or believes a valid leader to be alive. I0312 20:11:31.985180 2895 consensus_queue.cc:219] T c1432f133b26467b949ad48a19496e3a P 71f80b7649e64a2bafe2eee9b67b14c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4677, Committed index: 4677, Last appended: 1.4677, Last appended by leader: 4677, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71f80b7649e64a2bafe2eee9b67b14c1" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 52002 } } peers { permanent_uuid: "75cbb9a95bd343cbad786fcb50d74c94" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 59812 } } peers { permanent_uuid: "a5b79d7057a340678b551ae411266cda" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 55527 } } I0312 20:11:31.985834 2895 consensus_peers.cc:458] T c1432f133b26467b949ad48a19496e3a P 71f80b7649e64a2bafe2eee9b67b14c1 -> Peer a5b79d7057a340678b551ae411266cda (127.0.53.3:55527): Closing peer: a5b79d7057a340678b551ae411266cda I0312 20:11:31.986472 2443 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "c1432f133b26467b949ad48a19496e3a" candidate_uuid: "75cbb9a95bd343cbad786fcb50d74c94" candidate_term: 2 candidate_status { last_received { term: 1 index: 4675 } } ignore_live_leader: false dest_uuid: "71f80b7649e64a2bafe2eee9b67b14c1" I0312 20:11:31.987499 2895 consensus_peers.cc:458] T c1432f133b26467b949ad48a19496e3a P 71f80b7649e64a2bafe2eee9b67b14c1 -> Peer 75cbb9a95bd343cbad786fcb50d74c94 (127.0.53.2:59812): Closing peer: 75cbb9a95bd343cbad786fcb50d74c94 I0312 20:11:31.987828 2895 raft_consensus.cc:2752] T c1432f133b26467b949ad48a19496e3a P 71f80b7649e64a2bafe2eee9b67b14c1 [term 1 FOLLOWER]: Advancing to term 2 I0312 20:11:31.991381 2900 raft_consensus.cc:436] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 71f80b7649e64a2bafe2eee9b67b14c1) I0312 20:11:31.991590 2900 raft_consensus.cc:2700] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda: Snoozing failure detection for 0.539s (starting election) I0312 20:11:31.991792 2900 raft_consensus.cc:458] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71f80b7649e64a2bafe2eee9b67b14c1" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 52002 } } peers { permanent_uuid: "75cbb9a95bd343cbad786fcb50d74c94" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 59812 } } peers { permanent_uuid: "a5b79d7057a340678b551ae411266cda" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 55527 } } I0312 20:11:31.992710 2900 leader_election.cc:231] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 71f80b7649e64a2bafe2eee9b67b14c1 I0312 20:11:31.992843 2443 raft_consensus.cc:2162] T c1432f133b26467b949ad48a19496e3a P 71f80b7649e64a2bafe2eee9b67b14c1 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 75cbb9a95bd343cbad786fcb50d74c94 for term 2 because replica has last-logged OpId of term: 1 index: 4677, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4675. I0312 20:11:31.995684 2900 leader_election.cc:231] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 75cbb9a95bd343cbad786fcb50d74c94 I0312 20:11:31.996920 2568 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "c1432f133b26467b949ad48a19496e3a" candidate_uuid: "a5b79d7057a340678b551ae411266cda" candidate_term: 2 candidate_status { last_received { term: 1 index: 4677 } } ignore_live_leader: false dest_uuid: "75cbb9a95bd343cbad786fcb50d74c94" is_pre_election: true I0312 20:11:31.997447 2568 raft_consensus.cc:2145] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a5b79d7057a340678b551ae411266cda in current term 2: Already voted for candidate 75cbb9a95bd343cbad786fcb50d74c94 in this term. I0312 20:11:31.997560 2443 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "c1432f133b26467b949ad48a19496e3a" candidate_uuid: "a5b79d7057a340678b551ae411266cda" candidate_term: 2 candidate_status { last_received { term: 1 index: 4677 } } ignore_live_leader: false dest_uuid: "71f80b7649e64a2bafe2eee9b67b14c1" is_pre_election: true I0312 20:11:31.997947 2443 raft_consensus.cc:2700] T c1432f133b26467b949ad48a19496e3a P 71f80b7649e64a2bafe2eee9b67b14c1: Snoozing failure detection for 0.020s (vote granted) I0312 20:11:31.998003 2502 leader_election.cc:377] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [CANDIDATE]: Term 2 election: Vote denied by peer 71f80b7649e64a2bafe2eee9b67b14c1. Message: Invalid argument: T c1432f133b26467b949ad48a19496e3a P 71f80b7649e64a2bafe2eee9b67b14c1 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 75cbb9a95bd343cbad786fcb50d74c94 for term 2 because replica has last-logged OpId of term: 1 index: 4677, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4675. I0312 20:11:31.998138 2443 raft_consensus.cc:2215] T c1432f133b26467b949ad48a19496e3a P 71f80b7649e64a2bafe2eee9b67b14c1 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5b79d7057a340678b551ae411266cda in term 2. I0312 20:11:31.998217 2502 leader_election.cc:258] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. I0312 20:11:31.998809 2894 raft_consensus.cc:2700] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94: Snoozing failure detection for 14.428s (election complete) I0312 20:11:31.999037 2894 raft_consensus.cc:2455] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority I0312 20:11:31.999142 2627 leader_election.cc:363] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [CANDIDATE]: Term 2 pre-election: Vote granted by peer 71f80b7649e64a2bafe2eee9b67b14c1 I0312 20:11:31.999341 2627 leader_election.cc:258] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. I0312 20:11:32.000772 2903 raft_consensus.cc:2700] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda: Snoozing failure detection for 0.545s (election complete) I0312 20:11:32.001029 2903 raft_consensus.cc:2507] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [term 1 FOLLOWER]: Leader pre-election won for term 2 I0312 20:11:32.001001 2629 leader_election.cc:377] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [CANDIDATE]: Term 2 pre-election: Vote denied by peer 75cbb9a95bd343cbad786fcb50d74c94. Message: Invalid argument: T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a5b79d7057a340678b551ae411266cda in current term 2: Already voted for candidate 75cbb9a95bd343cbad786fcb50d74c94 in this term. I0312 20:11:32.001608 2629 leader_election.cc:377] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [CANDIDATE]: Term 2 pre-election: Vote denied by peer 75cbb9a95bd343cbad786fcb50d74c94. Message: Invalid argument: T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a5b79d7057a340678b551ae411266cda for term 2 because replica is either leader or believes a valid leader to be alive. I0312 20:11:32.001817 2629 leader_election.cc:258] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. I0312 20:11:32.002478 2903 raft_consensus.cc:436] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [term 1 FOLLOWER]: Starting leader election (detected failure of leader 71f80b7649e64a2bafe2eee9b67b14c1) I0312 20:11:32.002813 2903 raft_consensus.cc:2700] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda: Snoozing failure detection for 3.704s (starting election) I0312 20:11:32.003351 2903 raft_consensus.cc:2752] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [term 1 FOLLOWER]: Advancing to term 2 I0312 20:11:32.008051 2903 raft_consensus.cc:458] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71f80b7649e64a2bafe2eee9b67b14c1" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 52002 } } peers { permanent_uuid: "75cbb9a95bd343cbad786fcb50d74c94" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 59812 } } peers { permanent_uuid: "a5b79d7057a340678b551ae411266cda" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 55527 } } I0312 20:11:32.009023 2900 raft_consensus.cc:2700] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda: Snoozing failure detection for 0.611s (election complete) I0312 20:11:32.009258 2900 raft_consensus.cc:2455] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority I0312 20:11:32.009649 2903 leader_election.cc:231] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [CANDIDATE]: Term 2 election: Requesting vote from peer 71f80b7649e64a2bafe2eee9b67b14c1 I0312 20:11:32.010074 2903 leader_election.cc:231] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [CANDIDATE]: Term 2 election: Requesting vote from peer 75cbb9a95bd343cbad786fcb50d74c94 I0312 20:11:32.010874 2443 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "c1432f133b26467b949ad48a19496e3a" candidate_uuid: "a5b79d7057a340678b551ae411266cda" candidate_term: 2 candidate_status { last_received { term: 1 index: 4677 } } ignore_live_leader: false dest_uuid: "71f80b7649e64a2bafe2eee9b67b14c1" I0312 20:11:32.011256 2443 raft_consensus.cc:2700] T c1432f133b26467b949ad48a19496e3a P 71f80b7649e64a2bafe2eee9b67b14c1: Snoozing failure detection for 0.020s (vote granted) I0312 20:11:32.014259 2569 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "c1432f133b26467b949ad48a19496e3a" candidate_uuid: "a5b79d7057a340678b551ae411266cda" candidate_term: 2 candidate_status { last_received { term: 1 index: 4677 } } ignore_live_leader: false dest_uuid: "75cbb9a95bd343cbad786fcb50d74c94" I0312 20:11:32.014752 2569 raft_consensus.cc:2145] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate a5b79d7057a340678b551ae411266cda in current term 2: Already voted for candidate 75cbb9a95bd343cbad786fcb50d74c94 in this term. I0312 20:11:32.015180 2443 raft_consensus.cc:2215] T c1432f133b26467b949ad48a19496e3a P 71f80b7649e64a2bafe2eee9b67b14c1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5b79d7057a340678b551ae411266cda in term 2. I0312 20:11:32.015693 2629 leader_election.cc:377] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [CANDIDATE]: Term 2 election: Vote denied by peer 75cbb9a95bd343cbad786fcb50d74c94. Message: Invalid argument: T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate a5b79d7057a340678b551ae411266cda in current term 2: Already voted for candidate 75cbb9a95bd343cbad786fcb50d74c94 in this term. I0312 20:11:32.015943 2627 leader_election.cc:363] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [CANDIDATE]: Term 2 election: Vote granted by peer 71f80b7649e64a2bafe2eee9b67b14c1 I0312 20:11:32.016146 2627 leader_election.cc:258] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. I0312 20:11:32.016551 2903 raft_consensus.cc:2700] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda: Snoozing failure detection for 4.656s (election complete) I0312 20:11:32.016779 2903 raft_consensus.cc:2507] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [term 2 FOLLOWER]: Leader election won for term 2 I0312 20:11:32.017223 2903 raft_consensus.cc:572] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [term 2 LEADER]: Becoming Leader. State: Replica: a5b79d7057a340678b551ae411266cda, State: Running, Role: LEADER I0312 20:11:32.017843 2903 consensus_queue.cc:196] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4677, Committed index: 4677, Last appended: 1.4677, Last appended by leader: 4677, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71f80b7649e64a2bafe2eee9b67b14c1" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 52002 } } peers { permanent_uuid: "75cbb9a95bd343cbad786fcb50d74c94" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 59812 } } peers { permanent_uuid: "a5b79d7057a340678b551ae411266cda" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 55527 } } I0312 20:11:32.022603 2321 catalog_manager.cc:3592] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda reported cstate change: term changed from 1 to 2, leader changed from 71f80b7649e64a2bafe2eee9b67b14c1 (127.0.53.1) to a5b79d7057a340678b551ae411266cda (127.0.53.3). New cstate: current_term: 2 leader_uuid: "a5b79d7057a340678b551ae411266cda" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71f80b7649e64a2bafe2eee9b67b14c1" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 52002 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "75cbb9a95bd343cbad786fcb50d74c94" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 59812 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5b79d7057a340678b551ae411266cda" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 55527 } health_report { overall_health: HEALTHY } } } I0312 20:11:32.027655 2443 raft_consensus.cc:1058] T c1432f133b26467b949ad48a19496e3a P 71f80b7649e64a2bafe2eee9b67b14c1 [term 2 FOLLOWER]: Refusing update from remote peer a5b79d7057a340678b551ae411266cda: Log matching property violated. Preceding OpId in replica: term: 1 index: 4677. Preceding OpId from leader: term: 2 index: 4678. (index mismatch) I0312 20:11:32.029157 2900 consensus_queue.cc:1017] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [LEADER]: Connected to new peer: Peer: permanent_uuid: "71f80b7649e64a2bafe2eee9b67b14c1" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 52002 }, Status: LMP_MISMATCH, Last received: 1.4677, Next index: 4678, Last known committed idx: 4677, Time since last communication: 0.000s I0312 20:11:32.029671 2569 raft_consensus.cc:1058] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [term 2 FOLLOWER]: Refusing update from remote peer a5b79d7057a340678b551ae411266cda: Log matching property violated. Preceding OpId in replica: term: 1 index: 4675. Preceding OpId from leader: term: 2 index: 4678. (index mismatch) I0312 20:11:32.033571 2900 consensus_queue.cc:1017] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [LEADER]: Connected to new peer: Peer: permanent_uuid: "75cbb9a95bd343cbad786fcb50d74c94" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 59812 }, Status: LMP_MISMATCH, Last received: 1.4675, Next index: 4676, Last known committed idx: 4675, Time since last communication: 0.000s I0312 20:11:32.201335 2895 raft_consensus.cc:436] T c1432f133b26467b949ad48a19496e3a P 71f80b7649e64a2bafe2eee9b67b14c1 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a5b79d7057a340678b551ae411266cda) I0312 20:11:32.201581 2895 raft_consensus.cc:2700] T c1432f133b26467b949ad48a19496e3a P 71f80b7649e64a2bafe2eee9b67b14c1: Snoozing failure detection for 0.016s (starting election) I0312 20:11:32.201742 2895 raft_consensus.cc:458] T c1432f133b26467b949ad48a19496e3a P 71f80b7649e64a2bafe2eee9b67b14c1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71f80b7649e64a2bafe2eee9b67b14c1" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 52002 } } peers { permanent_uuid: "75cbb9a95bd343cbad786fcb50d74c94" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 59812 } } peers { permanent_uuid: "a5b79d7057a340678b551ae411266cda" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 55527 } } I0312 20:11:32.203032 2895 leader_election.cc:231] T c1432f133b26467b949ad48a19496e3a P 71f80b7649e64a2bafe2eee9b67b14c1 [CANDIDATE]: Term 3 pre-election: Requesting pre-vote from peer 75cbb9a95bd343cbad786fcb50d74c94 I0312 20:11:32.203405 2895 leader_election.cc:231] T c1432f133b26467b949ad48a19496e3a P 71f80b7649e64a2bafe2eee9b67b14c1 [CANDIDATE]: Term 3 pre-election: Requesting pre-vote from peer a5b79d7057a340678b551ae411266cda I0312 20:11:32.204376 2569 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "c1432f133b26467b949ad48a19496e3a" candidate_uuid: "71f80b7649e64a2bafe2eee9b67b14c1" candidate_term: 3 candidate_status { last_received { term: 2 index: 4697 } } ignore_live_leader: false dest_uuid: "75cbb9a95bd343cbad786fcb50d74c94" is_pre_election: true I0312 20:11:32.204885 2569 raft_consensus.cc:2176] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 71f80b7649e64a2bafe2eee9b67b14c1 for term 3 because replica is either leader or believes a valid leader to be alive. I0312 20:11:32.204746 2694 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "c1432f133b26467b949ad48a19496e3a" candidate_uuid: "71f80b7649e64a2bafe2eee9b67b14c1" candidate_term: 3 candidate_status { last_received { term: 2 index: 4697 } } ignore_live_leader: false dest_uuid: "a5b79d7057a340678b551ae411266cda" is_pre_election: true I0312 20:11:32.205379 2694 raft_consensus.cc:2176] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [term 2 LEADER]: Leader pre-election vote request: Denying vote to candidate 71f80b7649e64a2bafe2eee9b67b14c1 for term 3 because replica is either leader or believes a valid leader to be alive. I0312 20:11:32.206015 2379 leader_election.cc:377] T c1432f133b26467b949ad48a19496e3a P 71f80b7649e64a2bafe2eee9b67b14c1 [CANDIDATE]: Term 3 pre-election: Vote denied by peer 75cbb9a95bd343cbad786fcb50d74c94. Message: Invalid argument: T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 71f80b7649e64a2bafe2eee9b67b14c1 for term 3 because replica is either leader or believes a valid leader to be alive. I0312 20:11:32.206348 2375 leader_election.cc:377] T c1432f133b26467b949ad48a19496e3a P 71f80b7649e64a2bafe2eee9b67b14c1 [CANDIDATE]: Term 3 pre-election: Vote denied by peer a5b79d7057a340678b551ae411266cda. Message: Invalid argument: T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [term 2 LEADER]: Leader pre-election vote request: Denying vote to candidate 71f80b7649e64a2bafe2eee9b67b14c1 for term 3 because replica is either leader or believes a valid leader to be alive. I0312 20:11:32.206496 2375 leader_election.cc:258] T c1432f133b26467b949ad48a19496e3a P 71f80b7649e64a2bafe2eee9b67b14c1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. I0312 20:11:32.207118 2895 raft_consensus.cc:2700] T c1432f133b26467b949ad48a19496e3a P 71f80b7649e64a2bafe2eee9b67b14c1: Snoozing failure detection for 0.016s (election complete) I0312 20:11:32.207325 2895 raft_consensus.cc:2455] T c1432f133b26467b949ad48a19496e3a P 71f80b7649e64a2bafe2eee9b67b14c1 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority I0312 20:11:32.674473 2914 raft_consensus.cc:436] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a5b79d7057a340678b551ae411266cda) I0312 20:11:32.674834 2914 raft_consensus.cc:2700] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94: Snoozing failure detection for 0.019s (starting election) I0312 20:11:32.675058 2914 raft_consensus.cc:458] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71f80b7649e64a2bafe2eee9b67b14c1" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 52002 } } peers { permanent_uuid: "75cbb9a95bd343cbad786fcb50d74c94" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 59812 } } peers { permanent_uuid: "a5b79d7057a340678b551ae411266cda" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 55527 } } I0312 20:11:32.676151 2914 leader_election.cc:231] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [CANDIDATE]: Term 3 pre-election: Requesting pre-vote from peer 71f80b7649e64a2bafe2eee9b67b14c1 I0312 20:11:32.678694 2914 leader_election.cc:231] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [CANDIDATE]: Term 3 pre-election: Requesting pre-vote from peer a5b79d7057a340678b551ae411266cda I0312 20:11:32.679821 2443 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "c1432f133b26467b949ad48a19496e3a" candidate_uuid: "75cbb9a95bd343cbad786fcb50d74c94" candidate_term: 3 candidate_status { last_received { term: 2 index: 4780 } } ignore_live_leader: false dest_uuid: "71f80b7649e64a2bafe2eee9b67b14c1" is_pre_election: true I0312 20:11:32.679924 2694 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "c1432f133b26467b949ad48a19496e3a" candidate_uuid: "75cbb9a95bd343cbad786fcb50d74c94" candidate_term: 3 candidate_status { last_received { term: 2 index: 4780 } } ignore_live_leader: false dest_uuid: "a5b79d7057a340678b551ae411266cda" is_pre_election: true I0312 20:11:32.680199 2443 raft_consensus.cc:2176] T c1432f133b26467b949ad48a19496e3a P 71f80b7649e64a2bafe2eee9b67b14c1 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 75cbb9a95bd343cbad786fcb50d74c94 for term 3 because replica is either leader or believes a valid leader to be alive. I0312 20:11:32.680459 2694 raft_consensus.cc:2176] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [term 2 LEADER]: Leader pre-election vote request: Denying vote to candidate 75cbb9a95bd343cbad786fcb50d74c94 for term 3 because replica is either leader or believes a valid leader to be alive. I0312 20:11:32.681085 2502 leader_election.cc:377] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [CANDIDATE]: Term 3 pre-election: Vote denied by peer 71f80b7649e64a2bafe2eee9b67b14c1. Message: Invalid argument: T c1432f133b26467b949ad48a19496e3a P 71f80b7649e64a2bafe2eee9b67b14c1 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 75cbb9a95bd343cbad786fcb50d74c94 for term 3 because replica is either leader or believes a valid leader to be alive. I0312 20:11:32.682018 2500 leader_election.cc:377] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [CANDIDATE]: Term 3 pre-election: Vote denied by peer a5b79d7057a340678b551ae411266cda. Message: Invalid argument: T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [term 2 LEADER]: Leader pre-election vote request: Denying vote to candidate 75cbb9a95bd343cbad786fcb50d74c94 for term 3 because replica is either leader or believes a valid leader to be alive. I0312 20:11:32.682265 2500 leader_election.cc:258] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. I0312 20:11:32.689635 2914 raft_consensus.cc:2700] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94: Snoozing failure detection for 0.016s (election complete) I0312 20:11:32.689841 2914 raft_consensus.cc:2455] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority I0312 20:11:32.992280 2917 raft_consensus.cc:436] T c1432f133b26467b949ad48a19496e3a P 71f80b7649e64a2bafe2eee9b67b14c1 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a5b79d7057a340678b551ae411266cda) I0312 20:11:32.992545 2917 raft_consensus.cc:2700] T c1432f133b26467b949ad48a19496e3a P 71f80b7649e64a2bafe2eee9b67b14c1: Snoozing failure detection for 0.019s (starting election) I0312 20:11:32.992801 2917 raft_consensus.cc:458] T c1432f133b26467b949ad48a19496e3a P 71f80b7649e64a2bafe2eee9b67b14c1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71f80b7649e64a2bafe2eee9b67b14c1" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 52002 } } peers { permanent_uuid: "75cbb9a95bd343cbad786fcb50d74c94" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 59812 } } peers { permanent_uuid: "a5b79d7057a340678b551ae411266cda" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 55527 } } I0312 20:11:32.993898 2917 leader_election.cc:231] T c1432f133b26467b949ad48a19496e3a P 71f80b7649e64a2bafe2eee9b67b14c1 [CANDIDATE]: Term 3 pre-election: Requesting pre-vote from peer 75cbb9a95bd343cbad786fcb50d74c94 I0312 20:11:32.996748 2569 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "c1432f133b26467b949ad48a19496e3a" candidate_uuid: "71f80b7649e64a2bafe2eee9b67b14c1" candidate_term: 3 candidate_status { last_received { term: 2 index: 4836 } } ignore_live_leader: false dest_uuid: "75cbb9a95bd343cbad786fcb50d74c94" is_pre_election: true I0312 20:11:32.997133 2917 leader_election.cc:231] T c1432f133b26467b949ad48a19496e3a P 71f80b7649e64a2bafe2eee9b67b14c1 [CANDIDATE]: Term 3 pre-election: Requesting pre-vote from peer a5b79d7057a340678b551ae411266cda I0312 20:11:32.997462 2569 raft_consensus.cc:2176] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 71f80b7649e64a2bafe2eee9b67b14c1 for term 3 because replica is either leader or believes a valid leader to be alive. I0312 20:11:32.998283 2694 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "c1432f133b26467b949ad48a19496e3a" candidate_uuid: "71f80b7649e64a2bafe2eee9b67b14c1" candidate_term: 3 candidate_status { last_received { term: 2 index: 4836 } } ignore_live_leader: false dest_uuid: "a5b79d7057a340678b551ae411266cda" is_pre_election: true I0312 20:11:32.998733 2379 leader_election.cc:377] T c1432f133b26467b949ad48a19496e3a P 71f80b7649e64a2bafe2eee9b67b14c1 [CANDIDATE]: Term 3 pre-election: Vote denied by peer 75cbb9a95bd343cbad786fcb50d74c94. Message: Invalid argument: T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 71f80b7649e64a2bafe2eee9b67b14c1 for term 3 because replica is either leader or believes a valid leader to be alive. I0312 20:11:32.998809 2694 raft_consensus.cc:2176] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [term 2 LEADER]: Leader pre-election vote request: Denying vote to candidate 71f80b7649e64a2bafe2eee9b67b14c1 for term 3 because replica is either leader or believes a valid leader to be alive. I0312 20:11:33.000430 2375 leader_election.cc:377] T c1432f133b26467b949ad48a19496e3a P 71f80b7649e64a2bafe2eee9b67b14c1 [CANDIDATE]: Term 3 pre-election: Vote denied by peer a5b79d7057a340678b551ae411266cda. Message: Invalid argument: T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [term 2 LEADER]: Leader pre-election vote request: Denying vote to candidate 71f80b7649e64a2bafe2eee9b67b14c1 for term 3 because replica is either leader or believes a valid leader to be alive. I0312 20:11:33.000639 2375 leader_election.cc:258] T c1432f133b26467b949ad48a19496e3a P 71f80b7649e64a2bafe2eee9b67b14c1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. I0312 20:11:33.017184 2917 raft_consensus.cc:2700] T c1432f133b26467b949ad48a19496e3a P 71f80b7649e64a2bafe2eee9b67b14c1: Snoozing failure detection for 0.025s (election complete) I0312 20:11:33.017957 2917 raft_consensus.cc:2455] T c1432f133b26467b949ad48a19496e3a P 71f80b7649e64a2bafe2eee9b67b14c1 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority Connected to the Master Fetched info from all 3 Tablet Servers Table test-workload is HEALTHY (1 tablet(s) checked) Table Summary Name | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable ---------------+---------+---------------+---------+------------+------------------+------------- test-workload | HEALTHY | 1 | 1 | 0 | 0 | 0 The metadata for 1 table(s) is HEALTHY I0312 20:11:33.990090 2946 raft_consensus.cc:436] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a5b79d7057a340678b551ae411266cda) I0312 20:11:33.990434 2946 raft_consensus.cc:2700] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94: Snoozing failure detection for 0.017s (starting election) I0312 20:11:33.990726 2946 raft_consensus.cc:458] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71f80b7649e64a2bafe2eee9b67b14c1" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 52002 } } peers { permanent_uuid: "75cbb9a95bd343cbad786fcb50d74c94" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 59812 } } peers { permanent_uuid: "a5b79d7057a340678b551ae411266cda" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 55527 } } I0312 20:11:33.991806 2946 leader_election.cc:231] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [CANDIDATE]: Term 3 pre-election: Requesting pre-vote from peer 71f80b7649e64a2bafe2eee9b67b14c1 I0312 20:11:33.992260 2946 leader_election.cc:231] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [CANDIDATE]: Term 3 pre-election: Requesting pre-vote from peer a5b79d7057a340678b551ae411266cda I0312 20:11:33.993562 2947 raft_consensus.cc:436] T c1432f133b26467b949ad48a19496e3a P 71f80b7649e64a2bafe2eee9b67b14c1 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a5b79d7057a340678b551ae411266cda) I0312 20:11:33.993840 2947 raft_consensus.cc:2700] T c1432f133b26467b949ad48a19496e3a P 71f80b7649e64a2bafe2eee9b67b14c1: Snoozing failure detection for 0.031s (starting election) I0312 20:11:33.993654 2694 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "c1432f133b26467b949ad48a19496e3a" candidate_uuid: "75cbb9a95bd343cbad786fcb50d74c94" candidate_term: 3 candidate_status { last_received { term: 2 index: 4989 } } ignore_live_leader: false dest_uuid: "a5b79d7057a340678b551ae411266cda" is_pre_election: true I0312 20:11:33.994366 2694 raft_consensus.cc:2176] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [term 2 LEADER]: Leader pre-election vote request: Denying vote to candidate 75cbb9a95bd343cbad786fcb50d74c94 for term 3 because replica is either leader or believes a valid leader to be alive. Checksum finished in 0s: 0/3 replicas remaining (0B from disk, 60 rows summed) ----------------------- test-workload ----------------------- T c1432f133b26467b949ad48a19496e3a P 71f80b7649e64a2bafe2eee9b67b14c1 (127.0.53.1:52002): Checksum: 33349065875 T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 (127.0.53.2:59812): Checksum: 33349065875 T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda (127.0.53.3:55527): Checksum: 33349065875 I0312 20:11:33.995577 2500 leader_election.cc:377] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [CANDIDATE]: Term 3 pre-election: Vote denied by peer a5b79d7057a340678b551ae411266cda. Message: Invalid argument: T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [term 2 LEADER]: Leader pre-election vote request: Denying vote to candidate 75cbb9a95bd343cbad786fcb50d74c94 for term 3 because replica is either leader or believes a valid leader to be alive. I0312 20:11:33.996517 2443 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "c1432f133b26467b949ad48a19496e3a" candidate_uuid: "75cbb9a95bd343cbad786fcb50d74c94" candidate_term: 3 candidate_status { last_received { term: 2 index: 4989 } } ignore_live_leader: false dest_uuid: "71f80b7649e64a2bafe2eee9b67b14c1" is_pre_election: true I0312 20:11:34.006142 212 log_verifier.cc:121] Checking tablet c1432f133b26467b949ad48a19496e3a I0312 20:11:33.994088 2947 raft_consensus.cc:458] T c1432f133b26467b949ad48a19496e3a P 71f80b7649e64a2bafe2eee9b67b14c1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71f80b7649e64a2bafe2eee9b67b14c1" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 52002 } } peers { permanent_uuid: "75cbb9a95bd343cbad786fcb50d74c94" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 59812 } } peers { permanent_uuid: "a5b79d7057a340678b551ae411266cda" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 55527 } } I0312 20:11:34.008147 2443 raft_consensus.cc:2700] T c1432f133b26467b949ad48a19496e3a P 71f80b7649e64a2bafe2eee9b67b14c1: Snoozing failure detection for 0.018s (vote granted) I0312 20:11:34.008371 2443 raft_consensus.cc:2215] T c1432f133b26467b949ad48a19496e3a P 71f80b7649e64a2bafe2eee9b67b14c1 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 75cbb9a95bd343cbad786fcb50d74c94 in term 2. W0312 20:11:34.009869 2502 leader_election.cc:287] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 71f80b7649e64a2bafe2eee9b67b14c1: Timed out: RequestConsensusVote RPC to 127.0.53.1:52002 timed out after 0.017s (SENT) I0312 20:11:34.010092 2502 leader_election.cc:258] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. I0312 20:11:34.010931 2946 raft_consensus.cc:2700] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94: Snoozing failure detection for 0.027s (election complete) I0312 20:11:34.011183 2946 raft_consensus.cc:2455] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority I0312 20:11:34.011689 2947 leader_election.cc:231] T c1432f133b26467b949ad48a19496e3a P 71f80b7649e64a2bafe2eee9b67b14c1 [CANDIDATE]: Term 3 pre-election: Requesting pre-vote from peer 75cbb9a95bd343cbad786fcb50d74c94 I0312 20:11:34.012825 2947 leader_election.cc:231] T c1432f133b26467b949ad48a19496e3a P 71f80b7649e64a2bafe2eee9b67b14c1 [CANDIDATE]: Term 3 pre-election: Requesting pre-vote from peer a5b79d7057a340678b551ae411266cda I0312 20:11:34.014035 2568 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "c1432f133b26467b949ad48a19496e3a" candidate_uuid: "71f80b7649e64a2bafe2eee9b67b14c1" candidate_term: 3 candidate_status { last_received { term: 2 index: 4989 } } ignore_live_leader: false dest_uuid: "75cbb9a95bd343cbad786fcb50d74c94" is_pre_election: true I0312 20:11:34.014169 2694 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "c1432f133b26467b949ad48a19496e3a" candidate_uuid: "71f80b7649e64a2bafe2eee9b67b14c1" candidate_term: 3 candidate_status { last_received { term: 2 index: 4989 } } ignore_live_leader: false dest_uuid: "a5b79d7057a340678b551ae411266cda" is_pre_election: true I0312 20:11:34.014643 2694 raft_consensus.cc:2176] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [term 2 LEADER]: Leader pre-election vote request: Denying vote to candidate 71f80b7649e64a2bafe2eee9b67b14c1 for term 3 because replica is either leader or believes a valid leader to be alive. I0312 20:11:34.014581 2568 raft_consensus.cc:2176] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 71f80b7649e64a2bafe2eee9b67b14c1 for term 3 because replica is either leader or believes a valid leader to be alive. I0312 20:11:34.015640 2375 leader_election.cc:377] T c1432f133b26467b949ad48a19496e3a P 71f80b7649e64a2bafe2eee9b67b14c1 [CANDIDATE]: Term 3 pre-election: Vote denied by peer a5b79d7057a340678b551ae411266cda. Message: Invalid argument: T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [term 2 LEADER]: Leader pre-election vote request: Denying vote to candidate 71f80b7649e64a2bafe2eee9b67b14c1 for term 3 because replica is either leader or believes a valid leader to be alive. I0312 20:11:34.016485 2379 leader_election.cc:377] T c1432f133b26467b949ad48a19496e3a P 71f80b7649e64a2bafe2eee9b67b14c1 [CANDIDATE]: Term 3 pre-election: Vote denied by peer 75cbb9a95bd343cbad786fcb50d74c94. Message: Invalid argument: T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 71f80b7649e64a2bafe2eee9b67b14c1 for term 3 because replica is either leader or believes a valid leader to be alive. I0312 20:11:34.016713 2379 leader_election.cc:258] T c1432f133b26467b949ad48a19496e3a P 71f80b7649e64a2bafe2eee9b67b14c1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. I0312 20:11:34.022109 2947 raft_consensus.cc:2700] T c1432f133b26467b949ad48a19496e3a P 71f80b7649e64a2bafe2eee9b67b14c1: Snoozing failure detection for 0.041s (election complete) I0312 20:11:34.022992 2947 raft_consensus.cc:2455] T c1432f133b26467b949ad48a19496e3a P 71f80b7649e64a2bafe2eee9b67b14c1 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority I0312 20:11:34.217852 2946 raft_consensus.cc:436] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a5b79d7057a340678b551ae411266cda) I0312 20:11:34.218118 2946 raft_consensus.cc:2700] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94: Snoozing failure detection for 0.022s (starting election) I0312 20:11:34.218348 2946 raft_consensus.cc:458] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71f80b7649e64a2bafe2eee9b67b14c1" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 52002 } } peers { permanent_uuid: "75cbb9a95bd343cbad786fcb50d74c94" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 59812 } } peers { permanent_uuid: "a5b79d7057a340678b551ae411266cda" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 55527 } } I0312 20:11:34.219426 2946 leader_election.cc:231] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [CANDIDATE]: Term 3 pre-election: Requesting pre-vote from peer 71f80b7649e64a2bafe2eee9b67b14c1 I0312 20:11:34.219808 2946 leader_election.cc:231] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [CANDIDATE]: Term 3 pre-election: Requesting pre-vote from peer a5b79d7057a340678b551ae411266cda I0312 20:11:34.220715 2444 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "c1432f133b26467b949ad48a19496e3a" candidate_uuid: "75cbb9a95bd343cbad786fcb50d74c94" candidate_term: 3 candidate_status { last_received { term: 2 index: 4989 } } ignore_live_leader: false dest_uuid: "71f80b7649e64a2bafe2eee9b67b14c1" is_pre_election: true I0312 20:11:34.221263 2444 raft_consensus.cc:2176] T c1432f133b26467b949ad48a19496e3a P 71f80b7649e64a2bafe2eee9b67b14c1 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 75cbb9a95bd343cbad786fcb50d74c94 for term 3 because replica is either leader or believes a valid leader to be alive. I0312 20:11:34.221262 2694 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "c1432f133b26467b949ad48a19496e3a" candidate_uuid: "75cbb9a95bd343cbad786fcb50d74c94" candidate_term: 3 candidate_status { last_received { term: 2 index: 4989 } } ignore_live_leader: false dest_uuid: "a5b79d7057a340678b551ae411266cda" is_pre_election: true I0312 20:11:34.221770 2694 raft_consensus.cc:2176] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [term 2 LEADER]: Leader pre-election vote request: Denying vote to candidate 75cbb9a95bd343cbad786fcb50d74c94 for term 3 because replica is either leader or believes a valid leader to be alive. I0312 20:11:34.222406 2502 leader_election.cc:377] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [CANDIDATE]: Term 3 pre-election: Vote denied by peer 71f80b7649e64a2bafe2eee9b67b14c1. Message: Invalid argument: T c1432f133b26467b949ad48a19496e3a P 71f80b7649e64a2bafe2eee9b67b14c1 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 75cbb9a95bd343cbad786fcb50d74c94 for term 3 because replica is either leader or believes a valid leader to be alive. I0312 20:11:34.223233 2500 leader_election.cc:377] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [CANDIDATE]: Term 3 pre-election: Vote denied by peer a5b79d7057a340678b551ae411266cda. Message: Invalid argument: T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [term 2 LEADER]: Leader pre-election vote request: Denying vote to candidate 75cbb9a95bd343cbad786fcb50d74c94 for term 3 because replica is either leader or believes a valid leader to be alive. I0312 20:11:34.223444 2500 leader_election.cc:258] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. I0312 20:11:34.224344 2946 raft_consensus.cc:2700] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94: Snoozing failure detection for 0.038s (election complete) I0312 20:11:34.224547 2946 raft_consensus.cc:2455] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority I0312 20:11:34.233609 2948 raft_consensus.cc:436] T c1432f133b26467b949ad48a19496e3a P 71f80b7649e64a2bafe2eee9b67b14c1 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a5b79d7057a340678b551ae411266cda) I0312 20:11:34.233871 2948 raft_consensus.cc:2700] T c1432f133b26467b949ad48a19496e3a P 71f80b7649e64a2bafe2eee9b67b14c1: Snoozing failure detection for 0.065s (starting election) I0312 20:11:34.234109 2948 raft_consensus.cc:458] T c1432f133b26467b949ad48a19496e3a P 71f80b7649e64a2bafe2eee9b67b14c1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71f80b7649e64a2bafe2eee9b67b14c1" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 52002 } } peers { permanent_uuid: "75cbb9a95bd343cbad786fcb50d74c94" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 59812 } } peers { permanent_uuid: "a5b79d7057a340678b551ae411266cda" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 55527 } } I0312 20:11:34.235183 2948 leader_election.cc:231] T c1432f133b26467b949ad48a19496e3a P 71f80b7649e64a2bafe2eee9b67b14c1 [CANDIDATE]: Term 3 pre-election: Requesting pre-vote from peer 75cbb9a95bd343cbad786fcb50d74c94 I0312 20:11:34.235543 2948 leader_election.cc:231] T c1432f133b26467b949ad48a19496e3a P 71f80b7649e64a2bafe2eee9b67b14c1 [CANDIDATE]: Term 3 pre-election: Requesting pre-vote from peer a5b79d7057a340678b551ae411266cda I0312 20:11:34.238000 2694 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "c1432f133b26467b949ad48a19496e3a" candidate_uuid: "71f80b7649e64a2bafe2eee9b67b14c1" candidate_term: 3 candidate_status { last_received { term: 2 index: 4989 } } ignore_live_leader: false dest_uuid: "a5b79d7057a340678b551ae411266cda" is_pre_election: true I0312 20:11:34.238479 2694 raft_consensus.cc:2176] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [term 2 LEADER]: Leader pre-election vote request: Denying vote to candidate 71f80b7649e64a2bafe2eee9b67b14c1 for term 3 because replica is either leader or believes a valid leader to be alive. I0312 20:11:34.239171 2568 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "c1432f133b26467b949ad48a19496e3a" candidate_uuid: "71f80b7649e64a2bafe2eee9b67b14c1" candidate_term: 3 candidate_status { last_received { term: 2 index: 4989 } } ignore_live_leader: false dest_uuid: "75cbb9a95bd343cbad786fcb50d74c94" is_pre_election: true I0312 20:11:34.239665 2568 raft_consensus.cc:2176] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 71f80b7649e64a2bafe2eee9b67b14c1 for term 3 because replica is either leader or believes a valid leader to be alive. I0312 20:11:34.241128 2379 leader_election.cc:377] T c1432f133b26467b949ad48a19496e3a P 71f80b7649e64a2bafe2eee9b67b14c1 [CANDIDATE]: Term 3 pre-election: Vote denied by peer 75cbb9a95bd343cbad786fcb50d74c94. Message: Invalid argument: T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 71f80b7649e64a2bafe2eee9b67b14c1 for term 3 because replica is either leader or believes a valid leader to be alive. I0312 20:11:34.241859 2375 leader_election.cc:377] T c1432f133b26467b949ad48a19496e3a P 71f80b7649e64a2bafe2eee9b67b14c1 [CANDIDATE]: Term 3 pre-election: Vote denied by peer a5b79d7057a340678b551ae411266cda. Message: Invalid argument: T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [term 2 LEADER]: Leader pre-election vote request: Denying vote to candidate 71f80b7649e64a2bafe2eee9b67b14c1 for term 3 because replica is either leader or believes a valid leader to be alive. I0312 20:11:34.242071 2375 leader_election.cc:258] T c1432f133b26467b949ad48a19496e3a P 71f80b7649e64a2bafe2eee9b67b14c1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. I0312 20:11:34.242970 2948 raft_consensus.cc:2700] T c1432f133b26467b949ad48a19496e3a P 71f80b7649e64a2bafe2eee9b67b14c1: Snoozing failure detection for 0.070s (election complete) I0312 20:11:34.243185 2948 raft_consensus.cc:2455] T c1432f133b26467b949ad48a19496e3a P 71f80b7649e64a2bafe2eee9b67b14c1 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority I0312 20:11:37.757206 2949 raft_consensus.cc:436] T c1432f133b26467b949ad48a19496e3a P 71f80b7649e64a2bafe2eee9b67b14c1 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a5b79d7057a340678b551ae411266cda) I0312 20:11:37.757553 2949 raft_consensus.cc:2700] T c1432f133b26467b949ad48a19496e3a P 71f80b7649e64a2bafe2eee9b67b14c1: Snoozing failure detection for 0.050s (starting election) I0312 20:11:37.757855 2949 raft_consensus.cc:458] T c1432f133b26467b949ad48a19496e3a P 71f80b7649e64a2bafe2eee9b67b14c1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71f80b7649e64a2bafe2eee9b67b14c1" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 52002 } } peers { permanent_uuid: "75cbb9a95bd343cbad786fcb50d74c94" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 59812 } } peers { permanent_uuid: "a5b79d7057a340678b551ae411266cda" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 55527 } } I0312 20:11:37.760453 2949 leader_election.cc:231] T c1432f133b26467b949ad48a19496e3a P 71f80b7649e64a2bafe2eee9b67b14c1 [CANDIDATE]: Term 3 pre-election: Requesting pre-vote from peer 75cbb9a95bd343cbad786fcb50d74c94 I0312 20:11:37.760861 2949 leader_election.cc:231] T c1432f133b26467b949ad48a19496e3a P 71f80b7649e64a2bafe2eee9b67b14c1 [CANDIDATE]: Term 3 pre-election: Requesting pre-vote from peer a5b79d7057a340678b551ae411266cda I0312 20:11:37.761883 2569 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "c1432f133b26467b949ad48a19496e3a" candidate_uuid: "71f80b7649e64a2bafe2eee9b67b14c1" candidate_term: 3 candidate_status { last_received { term: 2 index: 4989 } } ignore_live_leader: false dest_uuid: "75cbb9a95bd343cbad786fcb50d74c94" is_pre_election: true I0312 20:11:37.762369 2569 raft_consensus.cc:2176] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 71f80b7649e64a2bafe2eee9b67b14c1 for term 3 because replica is either leader or believes a valid leader to be alive. I0312 20:11:37.762281 2694 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "c1432f133b26467b949ad48a19496e3a" candidate_uuid: "71f80b7649e64a2bafe2eee9b67b14c1" candidate_term: 3 candidate_status { last_received { term: 2 index: 4989 } } ignore_live_leader: false dest_uuid: "a5b79d7057a340678b551ae411266cda" is_pre_election: true I0312 20:11:37.762840 2694 raft_consensus.cc:2176] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [term 2 LEADER]: Leader pre-election vote request: Denying vote to candidate 71f80b7649e64a2bafe2eee9b67b14c1 for term 3 because replica is either leader or believes a valid leader to be alive. I0312 20:11:37.763295 2379 leader_election.cc:377] T c1432f133b26467b949ad48a19496e3a P 71f80b7649e64a2bafe2eee9b67b14c1 [CANDIDATE]: Term 3 pre-election: Vote denied by peer 75cbb9a95bd343cbad786fcb50d74c94. Message: Invalid argument: T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 71f80b7649e64a2bafe2eee9b67b14c1 for term 3 because replica is either leader or believes a valid leader to be alive. I0312 20:11:37.763836 2375 leader_election.cc:377] T c1432f133b26467b949ad48a19496e3a P 71f80b7649e64a2bafe2eee9b67b14c1 [CANDIDATE]: Term 3 pre-election: Vote denied by peer a5b79d7057a340678b551ae411266cda. Message: Invalid argument: T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [term 2 LEADER]: Leader pre-election vote request: Denying vote to candidate 71f80b7649e64a2bafe2eee9b67b14c1 for term 3 because replica is either leader or believes a valid leader to be alive. I0312 20:11:37.764081 2375 leader_election.cc:258] T c1432f133b26467b949ad48a19496e3a P 71f80b7649e64a2bafe2eee9b67b14c1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. I0312 20:11:37.764943 2949 raft_consensus.cc:2700] T c1432f133b26467b949ad48a19496e3a P 71f80b7649e64a2bafe2eee9b67b14c1: Snoozing failure detection for 0.101s (election complete) I0312 20:11:37.765210 2949 raft_consensus.cc:2455] T c1432f133b26467b949ad48a19496e3a P 71f80b7649e64a2bafe2eee9b67b14c1 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority W0312 20:11:39.800114 2307 debug-util.cc:396] Leaking SignalData structure 0x7b080002b060 after lost signal to thread 2294 W0312 20:11:39.800513 2307 debug-util.cc:396] Leaking SignalData structure 0x7b080007a680 after lost signal to thread 2352 I0312 20:11:40.125625 2950 raft_consensus.cc:436] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a5b79d7057a340678b551ae411266cda) I0312 20:11:40.125893 2950 raft_consensus.cc:2700] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94: Snoozing failure detection for 0.017s (starting election) I0312 20:11:40.126132 2950 raft_consensus.cc:458] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71f80b7649e64a2bafe2eee9b67b14c1" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 52002 } } peers { permanent_uuid: "75cbb9a95bd343cbad786fcb50d74c94" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 59812 } } peers { permanent_uuid: "a5b79d7057a340678b551ae411266cda" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 55527 } } I0312 20:11:40.130995 2950 leader_election.cc:231] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [CANDIDATE]: Term 3 pre-election: Requesting pre-vote from peer 71f80b7649e64a2bafe2eee9b67b14c1 I0312 20:11:40.131427 2950 leader_election.cc:231] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [CANDIDATE]: Term 3 pre-election: Requesting pre-vote from peer a5b79d7057a340678b551ae411266cda I0312 20:11:40.132231 2444 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "c1432f133b26467b949ad48a19496e3a" candidate_uuid: "75cbb9a95bd343cbad786fcb50d74c94" candidate_term: 3 candidate_status { last_received { term: 2 index: 4989 } } ignore_live_leader: false dest_uuid: "71f80b7649e64a2bafe2eee9b67b14c1" is_pre_election: true I0312 20:11:40.132740 2444 raft_consensus.cc:2176] T c1432f133b26467b949ad48a19496e3a P 71f80b7649e64a2bafe2eee9b67b14c1 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 75cbb9a95bd343cbad786fcb50d74c94 for term 3 because replica is either leader or believes a valid leader to be alive. I0312 20:11:40.133705 2502 leader_election.cc:377] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [CANDIDATE]: Term 3 pre-election: Vote denied by peer 71f80b7649e64a2bafe2eee9b67b14c1. Message: Invalid argument: T c1432f133b26467b949ad48a19496e3a P 71f80b7649e64a2bafe2eee9b67b14c1 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 75cbb9a95bd343cbad786fcb50d74c94 for term 3 because replica is either leader or believes a valid leader to be alive. I0312 20:11:40.137323 2694 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "c1432f133b26467b949ad48a19496e3a" candidate_uuid: "75cbb9a95bd343cbad786fcb50d74c94" candidate_term: 3 candidate_status { last_received { term: 2 index: 4989 } } ignore_live_leader: false dest_uuid: "a5b79d7057a340678b551ae411266cda" is_pre_election: true I0312 20:11:40.137873 2694 raft_consensus.cc:2176] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [term 2 LEADER]: Leader pre-election vote request: Denying vote to candidate 75cbb9a95bd343cbad786fcb50d74c94 for term 3 because replica is either leader or believes a valid leader to be alive. I0312 20:11:40.140630 2500 leader_election.cc:377] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [CANDIDATE]: Term 3 pre-election: Vote denied by peer a5b79d7057a340678b551ae411266cda. Message: Invalid argument: T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda [term 2 LEADER]: Leader pre-election vote request: Denying vote to candidate 75cbb9a95bd343cbad786fcb50d74c94 for term 3 because replica is either leader or believes a valid leader to be alive. I0312 20:11:40.141162 2500 leader_election.cc:258] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. I0312 20:11:40.141935 2950 raft_consensus.cc:2700] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94: Snoozing failure detection for 0.054s (election complete) I0312 20:11:40.142189 2950 raft_consensus.cc:2455] T c1432f133b26467b949ad48a19496e3a P 75cbb9a95bd343cbad786fcb50d74c94 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority I0312 20:11:40.951930 212 log_verifier.cc:172] Verified matching terms for 4989 ops in tablet c1432f133b26467b949ad48a19496e3a I0312 20:11:41.014152 212 external_mini_cluster.cc:997] Killing /tmp/dist-test-task3bwPeE/build/tsan/bin/kudu-tserver with pid 2355 W0312 20:11:41.059890 2502 connection.cc:513] client connection to 127.0.53.1:52002 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0312 20:11:41.059911 2304 connection.cc:513] client connection to 127.0.53.1:52002 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0312 20:11:41.061022 2501 connection.cc:513] server connection from 127.0.53.1:48348 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0312 20:11:41.061048 2302 connection.cc:513] server connection from 127.0.53.1:48719 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0312 20:11:41.062069 2625 connection.cc:513] server connection from 127.0.53.1:36081 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0312 20:11:41.062377 2627 connection.cc:513] client connection to 127.0.53.1:52002 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0312 20:11:41.062631 2627 consensus_peers.cc:436] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda -> Peer 71f80b7649e64a2bafe2eee9b67b14c1 (127.0.53.1:52002): Couldn't send request to peer 71f80b7649e64a2bafe2eee9b67b14c1 for tablet c1432f133b26467b949ad48a19496e3a. Status: Network error: failed to read from TLS socket: Connection reset by peer (error 104). Retrying in the next heartbeat period. Already tried 1 times. I0312 20:11:41.063563 212 external_mini_cluster.cc:997] Killing /tmp/dist-test-task3bwPeE/build/tsan/bin/kudu-tserver with pid 2489 W0312 20:11:41.078213 2627 consensus_peers.cc:436] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda -> Peer 71f80b7649e64a2bafe2eee9b67b14c1 (127.0.53.1:52002): Couldn't send request to peer 71f80b7649e64a2bafe2eee9b67b14c1 for tablet c1432f133b26467b949ad48a19496e3a. Status: Network error: Client connection negotiation failed: client connection to 127.0.53.1:52002: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 2 times. W0312 20:11:41.083971 2627 consensus_peers.cc:436] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda -> Peer 71f80b7649e64a2bafe2eee9b67b14c1 (127.0.53.1:52002): Couldn't send request to peer 71f80b7649e64a2bafe2eee9b67b14c1 for tablet c1432f133b26467b949ad48a19496e3a. Status: Network error: Client connection negotiation failed: client connection to 127.0.53.1:52002: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 3 times. W0312 20:11:41.087345 2627 consensus_peers.cc:436] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda -> Peer 71f80b7649e64a2bafe2eee9b67b14c1 (127.0.53.1:52002): Couldn't send request to peer 71f80b7649e64a2bafe2eee9b67b14c1 for tablet c1432f133b26467b949ad48a19496e3a. Status: Network error: Client connection negotiation failed: client connection to 127.0.53.1:52002: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 4 times. W0312 20:11:41.090792 2627 consensus_peers.cc:436] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda -> Peer 71f80b7649e64a2bafe2eee9b67b14c1 (127.0.53.1:52002): Couldn't send request to peer 71f80b7649e64a2bafe2eee9b67b14c1 for tablet c1432f133b26467b949ad48a19496e3a. Status: Network error: Client connection negotiation failed: client connection to 127.0.53.1:52002: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 5 times. W0312 20:11:41.095273 2627 consensus_peers.cc:436] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda -> Peer 71f80b7649e64a2bafe2eee9b67b14c1 (127.0.53.1:52002): Couldn't send request to peer 71f80b7649e64a2bafe2eee9b67b14c1 for tablet c1432f133b26467b949ad48a19496e3a. Status: Network error: Client connection negotiation failed: client connection to 127.0.53.1:52002: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 6 times. W0312 20:11:41.098758 2627 consensus_peers.cc:436] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda -> Peer 71f80b7649e64a2bafe2eee9b67b14c1 (127.0.53.1:52002): Couldn't send request to peer 71f80b7649e64a2bafe2eee9b67b14c1 for tablet c1432f133b26467b949ad48a19496e3a. Status: Network error: Client connection negotiation failed: client connection to 127.0.53.1:52002: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 7 times. W0312 20:11:41.102164 2627 consensus_peers.cc:436] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda -> Peer 71f80b7649e64a2bafe2eee9b67b14c1 (127.0.53.1:52002): Couldn't send request to peer 71f80b7649e64a2bafe2eee9b67b14c1 for tablet c1432f133b26467b949ad48a19496e3a. Status: Network error: Client connection negotiation failed: client connection to 127.0.53.1:52002: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 8 times. W0312 20:11:41.104231 2304 connection.cc:513] server connection from 127.0.53.2:57719 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0312 20:11:41.104264 2626 connection.cc:513] server connection from 127.0.53.2:51172 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0312 20:11:41.104290 2306 connection.cc:513] client connection to 127.0.53.2:59812 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0312 20:11:41.105084 2629 connection.cc:513] client connection to 127.0.53.2:59812 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0312 20:11:41.105335 2629 consensus_peers.cc:436] T c1432f133b26467b949ad48a19496e3a P a5b79d7057a340678b551ae411266cda -> Peer 75cbb9a95bd343cbad786fcb50d74c94 (127.0.53.2:59812): Couldn't send request to peer 75cbb9a95bd343cbad786fcb50d74c94 for tablet c1432f133b26467b949ad48a19496e3a. Status: Network error: failed to read from TLS socket: Connection reset by peer (error 104). Retrying in the next heartbeat period. Already tried 1 times. W0312 20:11:41.105916 2629 connection.cc:186] Error closing socket: Network error: TlsSocket::Close: Broken pipe I0312 20:11:41.106551 212 external_mini_cluster.cc:997] Killing /tmp/dist-test-task3bwPeE/build/tsan/bin/kudu-tserver with pid 2616 W0312 20:11:41.154512 2302 connection.cc:513] client connection to 127.0.53.3:55527 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0312 20:11:41.154603 2303 connection.cc:513] server connection from 127.0.53.3:53606 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) I0312 20:11:41.155048 212 external_mini_cluster.cc:997] Killing /tmp/dist-test-task3bwPeE/build/tsan/bin/kudu-master with pid 2293 W0312 20:11:41.183287 2292 connection.cc:513] client connection to 127.0.53.62:46983 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) [ OK ] RaftConsensusElectionITest.ChurnyElections_WithDuplicateKeys (39874 ms) [ RUN ] RaftConsensusElectionITest.AutomaticLeaderElection I0312 20:11:41.202221 212 test_util.cc:192] Using random seed: -1073017067 I0312 20:11:41.205054 212 ts_itest-base.cc:113] Starting cluster with: I0312 20:11:41.205185 212 ts_itest-base.cc:114] -------------- I0312 20:11:41.205260 212 ts_itest-base.cc:115] 3 tablet servers I0312 20:11:41.205339 212 ts_itest-base.cc:116] 3 replicas per TS I0312 20:11:41.205412 212 ts_itest-base.cc:117] -------------- I0312 20:11:41.207602 212 external_mini_cluster.cc:805] Running /tmp/dist-test-task3bwPeE/build/tsan/bin/kudu-master /tmp/dist-test-task3bwPeE/build/tsan/bin/kudu-master --fs_wal_dir=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.AutomaticLeaderElection.1520885401735786-212/raft_consensus-itest-cluster/master-0/wal --fs_data_dirs=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.AutomaticLeaderElection.1520885401735786-212/raft_consensus-itest-cluster/master-0/data --block_manager=log --webserver_interface=localhost --ipki_ca_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.0.53.62:0 --webserver_interface=127.0.53.62 --webserver_port=0 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.AutomaticLeaderElection.1520885401735786-212/raft_consensus-itest-cluster/master-0/logs --server_dump_info_path=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.AutomaticLeaderElection.1520885401735786-212/raft_consensus-itest-cluster/master-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W0312 20:11:41.328829 2967 flags.cc:412] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0312 20:11:41.329159 2967 flags.cc:412] Enabled unsafe flag: --never_fsync=true W0312 20:11:41.344465 2967 flags.cc:412] Enabled experimental flag: --ipki_ca_key_size=1024 W0312 20:11:41.344588 2967 flags.cc:412] Enabled experimental flag: --ipki_server_key_size=1024 W0312 20:11:41.344684 2967 flags.cc:412] Enabled experimental flag: --tsk_num_rsa_bits=512 I0312 20:11:41.364923 2967 master_main.cc:66] Master server non-default flags: --fs_data_dirs=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.AutomaticLeaderElection.1520885401735786-212/raft_consensus-itest-cluster/master-0/data --fs_wal_dir=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.AutomaticLeaderElection.1520885401735786-212/raft_consensus-itest-cluster/master-0/wal --ipki_ca_key_size=1024 --ipki_server_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.0.53.62:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.AutomaticLeaderElection.1520885401735786-212/raft_consensus-itest-cluster/master-0/data/info.pb --webserver_interface=127.0.53.62 --webserver_port=0 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --log_dir=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.AutomaticLeaderElection.1520885401735786-212/raft_consensus-itest-cluster/master-0/logs --logbuflevel=-1 --logtostderr=true Master server version: kudu 1.7.0-SNAPSHOT revision 13e10c63c46de30ac85f34b543cdd6fb44881f61 build type FASTDEBUG built by None at 12 Mar 2018 20:06:40 UTC on jenkins-slave-vn0g build id 12388 TSAN enabled I0312 20:11:41.374071 2967 master_main.cc:73] Initializing master server... I0312 20:11:41.375564 2967 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 1117776us I0312 20:11:41.376291 2967 fs_manager.cc:260] Metadata directory not provided I0312 20:11:41.376418 2967 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0312 20:11:41.376608 2967 server_base.cc:434] Could not load existing FS layout: Not found: /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.AutomaticLeaderElection.1520885401735786-212/raft_consensus-itest-cluster/master-0/data/instance: No such file or directory (error 2) I0312 20:11:41.376713 2967 server_base.cc:435] Attempting to create new FS layout instead I0312 20:11:41.383941 2967 fs_manager.cc:596] Generated new instance metadata in path /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.AutomaticLeaderElection.1520885401735786-212/raft_consensus-itest-cluster/master-0/data/instance: uuid: "0fcaa2522f6746a3ad845af22399e8ae" format_stamp: "Formatted at 2018-03-12 20:11:41 on dist-test-slave-dist-test-slave-f27p" I0312 20:11:41.384594 2967 fs_manager.cc:596] Generated new instance metadata in path /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.AutomaticLeaderElection.1520885401735786-212/raft_consensus-itest-cluster/master-0/wal/instance: uuid: "0fcaa2522f6746a3ad845af22399e8ae" format_stamp: "Formatted at 2018-03-12 20:11:41 on dist-test-slave-dist-test-slave-f27p" I0312 20:11:41.392752 2967 fs_manager.cc:496] Time spent creating directory manager: real 0.008s user 0.004s sys 0.004s I0312 20:11:41.393887 2967 env_posix.cc:1620] Not raising this process' open files per process limit of 65536; it is already as high as it can go I0312 20:11:41.394150 2967 file_cache.cc:470] Constructed file cache lbm with capacity 26214 I0312 20:11:41.400509 2967 fs_manager.cc:418] Time spent opening block manager: real 0.001s user 0.004s sys 0.000s I0312 20:11:41.400619 2967 fs_manager.cc:429] Opened local filesystem: /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.AutomaticLeaderElection.1520885401735786-212/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.AutomaticLeaderElection.1520885401735786-212/raft_consensus-itest-cluster/master-0/wal uuid: "0fcaa2522f6746a3ad845af22399e8ae" format_stamp: "Formatted at 2018-03-12 20:11:41 on dist-test-slave-dist-test-slave-f27p" I0312 20:11:41.400801 2967 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.AutomaticLeaderElection.1520885401735786-212/raft_consensus-itest-cluster/master-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0312 20:11:41.594374 2967 env_posix.cc:1620] Not raising this process' running threads per effective uid limit of 60284; it is already as high as it can go I0312 20:11:41.594980 2967 master_main.cc:76] Starting Master server... I0312 20:11:41.596184 2984 process_memory.cc:182] Process hard memory limit is 11.777631 GB I0312 20:11:41.596323 2984 process_memory.cc:184] Process soft memory limit is 9.422105 GB I0312 20:11:41.596418 2984 process_memory.cc:187] Process memory pressure threshold is 7.066578 GB I0312 20:11:41.635300 2967 rpc_server.cc:201] RPC server started. Bound to: 127.0.53.62:48469 I0312 20:11:41.660426 2967 webserver.cc:173] Starting webserver on 127.0.53.62:0 I0312 20:11:41.660529 2967 webserver.cc:184] Document root disabled I0312 20:11:41.661810 2967 webserver.cc:311] Webserver started. Bound to: http://127.0.53.62:60760/ I0312 20:11:41.662537 2967 server_base.cc:607] Dumped server information to /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.AutomaticLeaderElection.1520885401735786-212/raft_consensus-itest-cluster/master-0/data/info.pb I0312 20:11:41.665196 3028 data_dirs.cc:931] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0312 20:11:41.672158 212 external_mini_cluster.cc:867] Started /tmp/dist-test-task3bwPeE/build/tsan/bin/kudu-master as pid 2967 I0312 20:11:41.673110 212 external_mini_cluster.cc:805] Running /tmp/dist-test-task3bwPeE/build/tsan/bin/kudu-tserver /tmp/dist-test-task3bwPeE/build/tsan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.AutomaticLeaderElection.1520885401735786-212/raft_consensus-itest-cluster/ts-0/wal --fs_data_dirs=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.AutomaticLeaderElection.1520885401735786-212/raft_consensus-itest-cluster/ts-0/data --block_manager=log --rpc_bind_addresses=127.0.53.1:0 --local_ip_for_outbound_sockets=127.0.53.1 --webserver_interface=127.0.53.1 --webserver_port=0 --tserver_master_addrs=127.0.53.62:48469 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.AutomaticLeaderElection.1520885401735786-212/raft_consensus-itest-cluster/ts-0/logs --server_dump_info_path=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.AutomaticLeaderElection.1520885401735786-212/raft_consensus-itest-cluster/ts-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --log_cache_size_limit_mb=10 --consensus_rpc_timeout_ms=50 with env {} I0312 20:11:41.678807 3028 tablet_bootstrap.cc:436] T 00000000000000000000000000000000 P 0fcaa2522f6746a3ad845af22399e8ae: Bootstrap starting. I0312 20:11:41.681125 3028 tablet_bootstrap.cc:581] T 00000000000000000000000000000000 P 0fcaa2522f6746a3ad845af22399e8ae: No blocks or log segments found. Creating new log. I0312 20:11:41.682317 3028 log.cc:520] T 00000000000000000000000000000000 P 0fcaa2522f6746a3ad845af22399e8ae: Log is configured to *not* fsync() on all Append() calls I0312 20:11:41.684720 3028 tablet_bootstrap.cc:436] T 00000000000000000000000000000000 P 0fcaa2522f6746a3ad845af22399e8ae: No bootstrap required, opened a new log I0312 20:11:41.696665 3028 raft_consensus.cc:304] T 00000000000000000000000000000000 P 0fcaa2522f6746a3ad845af22399e8ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fcaa2522f6746a3ad845af22399e8ae" member_type: VOTER } I0312 20:11:41.696987 3028 raft_consensus.cc:330] T 00000000000000000000000000000000 P 0fcaa2522f6746a3ad845af22399e8ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0312 20:11:41.697151 3028 raft_consensus.cc:605] T 00000000000000000000000000000000 P 0fcaa2522f6746a3ad845af22399e8ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fcaa2522f6746a3ad845af22399e8ae, State: Initialized, Role: FOLLOWER I0312 20:11:41.697592 3028 consensus_queue.cc:219] T 00000000000000000000000000000000 P 0fcaa2522f6746a3ad845af22399e8ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fcaa2522f6746a3ad845af22399e8ae" member_type: VOTER } I0312 20:11:41.697933 3028 raft_consensus.cc:344] T 00000000000000000000000000000000 P 0fcaa2522f6746a3ad845af22399e8ae [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0312 20:11:41.698115 3028 raft_consensus.cc:436] T 00000000000000000000000000000000 P 0fcaa2522f6746a3ad845af22399e8ae [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0312 20:11:41.698282 3028 raft_consensus.cc:2700] T 00000000000000000000000000000000 P 0fcaa2522f6746a3ad845af22399e8ae: Snoozing failure detection for 1.944s (starting election) I0312 20:11:41.698443 3028 raft_consensus.cc:2752] T 00000000000000000000000000000000 P 0fcaa2522f6746a3ad845af22399e8ae [term 0 FOLLOWER]: Advancing to term 1 I0312 20:11:41.700963 3028 raft_consensus.cc:458] T 00000000000000000000000000000000 P 0fcaa2522f6746a3ad845af22399e8ae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fcaa2522f6746a3ad845af22399e8ae" member_type: VOTER } I0312 20:11:41.701408 3028 leader_election.cc:258] T 00000000000000000000000000000000 P 0fcaa2522f6746a3ad845af22399e8ae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0312 20:11:41.703066 3031 raft_consensus.cc:2700] T 00000000000000000000000000000000 P 0fcaa2522f6746a3ad845af22399e8ae: Snoozing failure detection for 1.610s (election complete) I0312 20:11:41.703353 3031 raft_consensus.cc:2507] T 00000000000000000000000000000000 P 0fcaa2522f6746a3ad845af22399e8ae [term 1 FOLLOWER]: Leader election won for term 1 I0312 20:11:41.717593 3031 raft_consensus.cc:572] T 00000000000000000000000000000000 P 0fcaa2522f6746a3ad845af22399e8ae [term 1 LEADER]: Becoming Leader. State: Replica: 0fcaa2522f6746a3ad845af22399e8ae, State: Running, Role: LEADER I0312 20:11:41.718004 3031 consensus_queue.cc:196] T 00000000000000000000000000000000 P 0fcaa2522f6746a3ad845af22399e8ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fcaa2522f6746a3ad845af22399e8ae" member_type: VOTER } I0312 20:11:41.721104 3028 sys_catalog.cc:335] T 00000000000000000000000000000000 P 0fcaa2522f6746a3ad845af22399e8ae [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 1 leader_uuid: "0fcaa2522f6746a3ad845af22399e8ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fcaa2522f6746a3ad845af22399e8ae" member_type: VOTER } } I0312 20:11:41.721521 3028 sys_catalog.cc:338] T 00000000000000000000000000000000 P 0fcaa2522f6746a3ad845af22399e8ae [sys.catalog]: This master's current role is: LEADER I0312 20:11:41.728327 3028 sys_catalog.cc:422] T 00000000000000000000000000000000 P 0fcaa2522f6746a3ad845af22399e8ae [sys.catalog]: configured and running, proceeding with master startup. I0312 20:11:41.728910 3033 sys_catalog.cc:335] T 00000000000000000000000000000000 P 0fcaa2522f6746a3ad845af22399e8ae [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0fcaa2522f6746a3ad845af22399e8ae. Latest consensus state: current_term: 1 leader_uuid: "0fcaa2522f6746a3ad845af22399e8ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fcaa2522f6746a3ad845af22399e8ae" member_type: VOTER } } I0312 20:11:41.729272 3033 sys_catalog.cc:338] T 00000000000000000000000000000000 P 0fcaa2522f6746a3ad845af22399e8ae [sys.catalog]: This master's current role is: LEADER I0312 20:11:41.728746 3032 sys_catalog.cc:335] T 00000000000000000000000000000000 P 0fcaa2522f6746a3ad845af22399e8ae [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0fcaa2522f6746a3ad845af22399e8ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fcaa2522f6746a3ad845af22399e8ae" member_type: VOTER } } I0312 20:11:41.729955 3032 sys_catalog.cc:338] T 00000000000000000000000000000000 P 0fcaa2522f6746a3ad845af22399e8ae [sys.catalog]: This master's current role is: LEADER I0312 20:11:41.731108 2967 master_main.cc:79] Master server successfully started. W0312 20:11:41.734387 3038 catalog_manager.cc:1018] T 00000000000000000000000000000000 P 0fcaa2522f6746a3ad845af22399e8ae: acquiring CA information for follower catalog manager: Not found: root CA entry not found W0312 20:11:41.734633 3038 catalog_manager.cc:560] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry I0312 20:11:41.734925 3036 catalog_manager.cc:972] Loading table and tablet metadata into memory... I0312 20:11:41.737792 3036 catalog_manager.cc:983] Initializing Kudu internal certificate authority... I0312 20:11:41.795594 3036 catalog_manager.cc:870] Generated new certificate authority record I0312 20:11:41.797850 3036 catalog_manager.cc:992] Loading token signing keys... I0312 20:11:41.815273 3036 catalog_manager.cc:3868] T 00000000000000000000000000000000 P 0fcaa2522f6746a3ad845af22399e8ae: Generated new TSK 0 WARNING: Logging before InitGoogleLogging() is written to STDERR W0312 20:11:41.821135 3029 flags.cc:412] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0312 20:11:41.821493 3029 flags.cc:412] Enabled unsafe flag: --never_fsync=true W0312 20:11:41.835764 3029 flags.cc:412] Enabled experimental flag: --ipki_server_key_size=1024 W0312 20:11:41.836077 3029 flags.cc:412] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.53.1 I0312 20:11:41.859660 3029 tablet_server_main.cc:72] Tablet server non-default flags: --consensus_rpc_timeout_ms=50 --log_cache_size_limit_mb=10 --fs_data_dirs=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.AutomaticLeaderElection.1520885401735786-212/raft_consensus-itest-cluster/ts-0/data --fs_wal_dir=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.AutomaticLeaderElection.1520885401735786-212/raft_consensus-itest-cluster/ts-0/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.0.53.1:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.AutomaticLeaderElection.1520885401735786-212/raft_consensus-itest-cluster/ts-0/data/info.pb --webserver_interface=127.0.53.1 --webserver_port=0 --tserver_master_addrs=127.0.53.62:48469 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.0.53.1 --log_dir=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.AutomaticLeaderElection.1520885401735786-212/raft_consensus-itest-cluster/ts-0/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.7.0-SNAPSHOT revision 13e10c63c46de30ac85f34b543cdd6fb44881f61 build type FASTDEBUG built by None at 12 Mar 2018 20:06:40 UTC on jenkins-slave-vn0g build id 12388 TSAN enabled I0312 20:11:41.889410 3029 tablet_server_main.cc:79] Initializing tablet server... I0312 20:11:41.891155 3029 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 1117776us I0312 20:11:41.891610 3029 fs_manager.cc:260] Metadata directory not provided I0312 20:11:41.891736 3029 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0312 20:11:41.891942 3029 server_base.cc:434] Could not load existing FS layout: Not found: /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.AutomaticLeaderElection.1520885401735786-212/raft_consensus-itest-cluster/ts-0/data/instance: No such file or directory (error 2) I0312 20:11:41.892055 3029 server_base.cc:435] Attempting to create new FS layout instead I0312 20:11:41.899286 3029 fs_manager.cc:596] Generated new instance metadata in path /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.AutomaticLeaderElection.1520885401735786-212/raft_consensus-itest-cluster/ts-0/data/instance: uuid: "1e9afc22e84f401ba29b1e7ec68ec8c6" format_stamp: "Formatted at 2018-03-12 20:11:41 on dist-test-slave-dist-test-slave-f27p" I0312 20:11:41.899986 3029 fs_manager.cc:596] Generated new instance metadata in path /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.AutomaticLeaderElection.1520885401735786-212/raft_consensus-itest-cluster/ts-0/wal/instance: uuid: "1e9afc22e84f401ba29b1e7ec68ec8c6" format_stamp: "Formatted at 2018-03-12 20:11:41 on dist-test-slave-dist-test-slave-f27p" I0312 20:11:41.910531 3029 fs_manager.cc:496] Time spent creating directory manager: real 0.010s user 0.008s sys 0.004s I0312 20:11:41.911739 3029 env_posix.cc:1620] Not raising this process' open files per process limit of 65536; it is already as high as it can go I0312 20:11:41.912016 3029 file_cache.cc:470] Constructed file cache lbm with capacity 26214 I0312 20:11:41.926200 3029 fs_manager.cc:418] Time spent opening block manager: real 0.009s user 0.004s sys 0.008s I0312 20:11:41.926327 3029 fs_manager.cc:429] Opened local filesystem: /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.AutomaticLeaderElection.1520885401735786-212/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.AutomaticLeaderElection.1520885401735786-212/raft_consensus-itest-cluster/ts-0/wal uuid: "1e9afc22e84f401ba29b1e7ec68ec8c6" format_stamp: "Formatted at 2018-03-12 20:11:41 on dist-test-slave-dist-test-slave-f27p" I0312 20:11:41.926534 3029 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.AutomaticLeaderElection.1520885401735786-212/raft_consensus-itest-cluster/ts-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0312 20:11:42.160414 3029 env_posix.cc:1620] Not raising this process' running threads per effective uid limit of 60284; it is already as high as it can go I0312 20:11:42.161747 3029 ts_tablet_manager.cc:349] Loaded tablet metadata (0 live tablets) I0312 20:11:42.173436 3029 tablet_server_main.cc:84] Starting tablet server... I0312 20:11:42.403836 3029 rpc_server.cc:201] RPC server started. Bound to: 127.0.53.1:33856 I0312 20:11:42.404736 3029 webserver.cc:173] Starting webserver on 127.0.53.1:0 I0312 20:11:42.404842 3029 webserver.cc:184] Document root disabled I0312 20:11:42.406306 3029 webserver.cc:311] Webserver started. Bound to: http://127.0.53.1:38008/ I0312 20:11:42.407034 3029 server_base.cc:607] Dumped server information to /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.AutomaticLeaderElection.1520885401735786-212/raft_consensus-itest-cluster/ts-0/data/info.pb I0312 20:11:42.412467 3163 process_memory.cc:182] Process hard memory limit is 11.777631 GB I0312 20:11:42.412611 3163 process_memory.cc:184] Process soft memory limit is 9.422105 GB I0312 20:11:42.412708 3163 process_memory.cc:187] Process memory pressure threshold is 7.066578 GB I0312 20:11:42.412822 212 external_mini_cluster.cc:867] Started /tmp/dist-test-task3bwPeE/build/tsan/bin/kudu-tserver as pid 3029 I0312 20:11:42.413540 3029 tablet_server_main.cc:87] Tablet server successfully started. I0312 20:11:42.414276 212 external_mini_cluster.cc:805] Running /tmp/dist-test-task3bwPeE/build/tsan/bin/kudu-tserver /tmp/dist-test-task3bwPeE/build/tsan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.AutomaticLeaderElection.1520885401735786-212/raft_consensus-itest-cluster/ts-1/wal --fs_data_dirs=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.AutomaticLeaderElection.1520885401735786-212/raft_consensus-itest-cluster/ts-1/data --block_manager=log --rpc_bind_addresses=127.0.53.2:0 --local_ip_for_outbound_sockets=127.0.53.2 --webserver_interface=127.0.53.2 --webserver_port=0 --tserver_master_addrs=127.0.53.62:48469 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.AutomaticLeaderElection.1520885401735786-212/raft_consensus-itest-cluster/ts-1/logs --server_dump_info_path=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.AutomaticLeaderElection.1520885401735786-212/raft_consensus-itest-cluster/ts-1/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --log_cache_size_limit_mb=10 --consensus_rpc_timeout_ms=50 with env {} I0312 20:11:42.448026 3162 heartbeater.cc:346] Connected to a master server at 127.0.53.62:48469 I0312 20:11:42.448326 3162 heartbeater.cc:426] Registering TS with master... I0312 20:11:42.448895 3162 heartbeater.cc:475] Master 127.0.53.62:48469 requested a full tablet report, sending... I0312 20:11:42.450922 2995 ts_manager.cc:81] Registered new tserver with Master: 1e9afc22e84f401ba29b1e7ec68ec8c6 (127.0.53.1:33856) I0312 20:11:42.453661 2995 master_service.cc:243] Signed X509 certificate for tserver {username='slave'} at 127.0.53.1:47622 WARNING: Logging before InitGoogleLogging() is written to STDERR W0312 20:11:42.535210 3166 flags.cc:412] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0312 20:11:42.535574 3166 flags.cc:412] Enabled unsafe flag: --never_fsync=true W0312 20:11:42.550019 3166 flags.cc:412] Enabled experimental flag: --ipki_server_key_size=1024 W0312 20:11:42.550336 3166 flags.cc:412] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.53.2 I0312 20:11:42.569820 3166 tablet_server_main.cc:72] Tablet server non-default flags: --consensus_rpc_timeout_ms=50 --log_cache_size_limit_mb=10 --fs_data_dirs=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.AutomaticLeaderElection.1520885401735786-212/raft_consensus-itest-cluster/ts-1/data --fs_wal_dir=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.AutomaticLeaderElection.1520885401735786-212/raft_consensus-itest-cluster/ts-1/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.0.53.2:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.AutomaticLeaderElection.1520885401735786-212/raft_consensus-itest-cluster/ts-1/data/info.pb --webserver_interface=127.0.53.2 --webserver_port=0 --tserver_master_addrs=127.0.53.62:48469 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.0.53.2 --log_dir=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.AutomaticLeaderElection.1520885401735786-212/raft_consensus-itest-cluster/ts-1/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.7.0-SNAPSHOT revision 13e10c63c46de30ac85f34b543cdd6fb44881f61 build type FASTDEBUG built by None at 12 Mar 2018 20:06:40 UTC on jenkins-slave-vn0g build id 12388 TSAN enabled I0312 20:11:42.580247 3166 tablet_server_main.cc:79] Initializing tablet server... I0312 20:11:42.581938 3166 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 1118276us I0312 20:11:42.582372 3166 fs_manager.cc:260] Metadata directory not provided I0312 20:11:42.582497 3166 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0312 20:11:42.582701 3166 server_base.cc:434] Could not load existing FS layout: Not found: /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.AutomaticLeaderElection.1520885401735786-212/raft_consensus-itest-cluster/ts-1/data/instance: No such file or directory (error 2) I0312 20:11:42.582813 3166 server_base.cc:435] Attempting to create new FS layout instead I0312 20:11:42.590029 3166 fs_manager.cc:596] Generated new instance metadata in path /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.AutomaticLeaderElection.1520885401735786-212/raft_consensus-itest-cluster/ts-1/data/instance: uuid: "c305fa84777a4d6ea71fe631360fde01" format_stamp: "Formatted at 2018-03-12 20:11:42 on dist-test-slave-dist-test-slave-f27p" I0312 20:11:42.590730 3166 fs_manager.cc:596] Generated new instance metadata in path /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.AutomaticLeaderElection.1520885401735786-212/raft_consensus-itest-cluster/ts-1/wal/instance: uuid: "c305fa84777a4d6ea71fe631360fde01" format_stamp: "Formatted at 2018-03-12 20:11:42 on dist-test-slave-dist-test-slave-f27p" I0312 20:11:42.599766 3166 fs_manager.cc:496] Time spent creating directory manager: real 0.009s user 0.008s sys 0.000s I0312 20:11:42.600896 3166 env_posix.cc:1620] Not raising this process' open files per process limit of 65536; it is already as high as it can go I0312 20:11:42.601231 3166 file_cache.cc:470] Constructed file cache lbm with capacity 26214 I0312 20:11:42.607683 3166 fs_manager.cc:418] Time spent opening block manager: real 0.002s user 0.000s sys 0.000s I0312 20:11:42.607798 3166 fs_manager.cc:429] Opened local filesystem: /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.AutomaticLeaderElection.1520885401735786-212/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.AutomaticLeaderElection.1520885401735786-212/raft_consensus-itest-cluster/ts-1/wal uuid: "c305fa84777a4d6ea71fe631360fde01" format_stamp: "Formatted at 2018-03-12 20:11:42 on dist-test-slave-dist-test-slave-f27p" I0312 20:11:42.607998 3166 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.AutomaticLeaderElection.1520885401735786-212/raft_consensus-itest-cluster/ts-1/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0312 20:11:42.833220 3166 env_posix.cc:1620] Not raising this process' running threads per effective uid limit of 60284; it is already as high as it can go I0312 20:11:42.834511 3166 ts_tablet_manager.cc:349] Loaded tablet metadata (0 live tablets) I0312 20:11:42.835635 3166 tablet_server_main.cc:84] Starting tablet server... I0312 20:11:42.936609 3166 rpc_server.cc:201] RPC server started. Bound to: 127.0.53.2:39347 I0312 20:11:42.937500 3166 webserver.cc:173] Starting webserver on 127.0.53.2:0 I0312 20:11:42.937602 3166 webserver.cc:184] Document root disabled I0312 20:11:42.938807 3166 webserver.cc:311] Webserver started. Bound to: http://127.0.53.2:47579/ I0312 20:11:42.939575 3166 server_base.cc:607] Dumped server information to /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.AutomaticLeaderElection.1520885401735786-212/raft_consensus-itest-cluster/ts-1/data/info.pb I0312 20:11:42.943485 212 external_mini_cluster.cc:867] Started /tmp/dist-test-task3bwPeE/build/tsan/bin/kudu-tserver as pid 3166 I0312 20:11:42.944659 212 external_mini_cluster.cc:805] Running /tmp/dist-test-task3bwPeE/build/tsan/bin/kudu-tserver /tmp/dist-test-task3bwPeE/build/tsan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.AutomaticLeaderElection.1520885401735786-212/raft_consensus-itest-cluster/ts-2/wal --fs_data_dirs=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.AutomaticLeaderElection.1520885401735786-212/raft_consensus-itest-cluster/ts-2/data --block_manager=log --rpc_bind_addresses=127.0.53.3:0 --local_ip_for_outbound_sockets=127.0.53.3 --webserver_interface=127.0.53.3 --webserver_port=0 --tserver_master_addrs=127.0.53.62:48469 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.AutomaticLeaderElection.1520885401735786-212/raft_consensus-itest-cluster/ts-2/logs --server_dump_info_path=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.AutomaticLeaderElection.1520885401735786-212/raft_consensus-itest-cluster/ts-2/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --log_cache_size_limit_mb=10 --consensus_rpc_timeout_ms=50 with env {} I0312 20:11:42.952282 3288 process_memory.cc:182] Process hard memory limit is 11.777631 GB I0312 20:11:42.953297 3288 process_memory.cc:184] Process soft memory limit is 9.422105 GB I0312 20:11:42.953444 3288 process_memory.cc:187] Process memory pressure threshold is 7.066578 GB I0312 20:11:42.963765 3287 heartbeater.cc:346] Connected to a master server at 127.0.53.62:48469 I0312 20:11:42.964049 3287 heartbeater.cc:426] Registering TS with master... I0312 20:11:42.964565 3287 heartbeater.cc:475] Master 127.0.53.62:48469 requested a full tablet report, sending... I0312 20:11:42.966254 2995 ts_manager.cc:81] Registered new tserver with Master: c305fa84777a4d6ea71fe631360fde01 (127.0.53.2:39347) I0312 20:11:42.967813 2995 master_service.cc:243] Signed X509 certificate for tserver {username='slave'} at 127.0.53.2:54620 I0312 20:11:42.952724 3166 tablet_server_main.cc:87] Tablet server successfully started. WARNING: Logging before InitGoogleLogging() is written to STDERR W0312 20:11:43.067842 3290 flags.cc:412] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0312 20:11:43.068169 3290 flags.cc:412] Enabled unsafe flag: --never_fsync=true W0312 20:11:43.082185 3290 flags.cc:412] Enabled experimental flag: --ipki_server_key_size=1024 W0312 20:11:43.082494 3290 flags.cc:412] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.53.3 I0312 20:11:43.101083 3290 tablet_server_main.cc:72] Tablet server non-default flags: --consensus_rpc_timeout_ms=50 --log_cache_size_limit_mb=10 --fs_data_dirs=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.AutomaticLeaderElection.1520885401735786-212/raft_consensus-itest-cluster/ts-2/data --fs_wal_dir=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.AutomaticLeaderElection.1520885401735786-212/raft_consensus-itest-cluster/ts-2/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.0.53.3:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.AutomaticLeaderElection.1520885401735786-212/raft_consensus-itest-cluster/ts-2/data/info.pb --webserver_interface=127.0.53.3 --webserver_port=0 --tserver_master_addrs=127.0.53.62:48469 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.0.53.3 --log_dir=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.AutomaticLeaderElection.1520885401735786-212/raft_consensus-itest-cluster/ts-2/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.7.0-SNAPSHOT revision 13e10c63c46de30ac85f34b543cdd6fb44881f61 build type FASTDEBUG built by None at 12 Mar 2018 20:06:40 UTC on jenkins-slave-vn0g build id 12388 TSAN enabled I0312 20:11:43.110934 3290 tablet_server_main.cc:79] Initializing tablet server... I0312 20:11:43.112613 3290 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 1118776us I0312 20:11:43.113034 3290 fs_manager.cc:260] Metadata directory not provided I0312 20:11:43.113196 3290 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0312 20:11:43.113406 3290 server_base.cc:434] Could not load existing FS layout: Not found: /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.AutomaticLeaderElection.1520885401735786-212/raft_consensus-itest-cluster/ts-2/data/instance: No such file or directory (error 2) I0312 20:11:43.113519 3290 server_base.cc:435] Attempting to create new FS layout instead I0312 20:11:43.120754 3290 fs_manager.cc:596] Generated new instance metadata in path /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.AutomaticLeaderElection.1520885401735786-212/raft_consensus-itest-cluster/ts-2/data/instance: uuid: "d1a250fbb6a14dca8c8bb69a2b97065e" format_stamp: "Formatted at 2018-03-12 20:11:43 on dist-test-slave-dist-test-slave-f27p" I0312 20:11:43.121474 3290 fs_manager.cc:596] Generated new instance metadata in path /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.AutomaticLeaderElection.1520885401735786-212/raft_consensus-itest-cluster/ts-2/wal/instance: uuid: "d1a250fbb6a14dca8c8bb69a2b97065e" format_stamp: "Formatted at 2018-03-12 20:11:43 on dist-test-slave-dist-test-slave-f27p" I0312 20:11:43.130020 3290 fs_manager.cc:496] Time spent creating directory manager: real 0.008s user 0.004s sys 0.004s I0312 20:11:43.131157 3290 env_posix.cc:1620] Not raising this process' open files per process limit of 65536; it is already as high as it can go I0312 20:11:43.131430 3290 file_cache.cc:470] Constructed file cache lbm with capacity 26214 I0312 20:11:43.137831 3290 fs_manager.cc:418] Time spent opening block manager: real 0.001s user 0.000s sys 0.004s I0312 20:11:43.137948 3290 fs_manager.cc:429] Opened local filesystem: /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.AutomaticLeaderElection.1520885401735786-212/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.AutomaticLeaderElection.1520885401735786-212/raft_consensus-itest-cluster/ts-2/wal uuid: "d1a250fbb6a14dca8c8bb69a2b97065e" format_stamp: "Formatted at 2018-03-12 20:11:43 on dist-test-slave-dist-test-slave-f27p" I0312 20:11:43.138142 3290 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.AutomaticLeaderElection.1520885401735786-212/raft_consensus-itest-cluster/ts-2/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0312 20:11:43.311560 3290 env_posix.cc:1620] Not raising this process' running threads per effective uid limit of 60284; it is already as high as it can go I0312 20:11:43.312832 3290 ts_tablet_manager.cc:349] Loaded tablet metadata (0 live tablets) I0312 20:11:43.313812 3290 tablet_server_main.cc:84] Starting tablet server... I0312 20:11:43.399960 3290 rpc_server.cc:201] RPC server started. Bound to: 127.0.53.3:35242 I0312 20:11:43.400837 3290 webserver.cc:173] Starting webserver on 127.0.53.3:0 I0312 20:11:43.400944 3290 webserver.cc:184] Document root disabled I0312 20:11:43.402201 3290 webserver.cc:311] Webserver started. Bound to: http://127.0.53.3:55882/ I0312 20:11:43.402920 3290 server_base.cc:607] Dumped server information to /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.AutomaticLeaderElection.1520885401735786-212/raft_consensus-itest-cluster/ts-2/data/info.pb I0312 20:11:43.403295 212 external_mini_cluster.cc:867] Started /tmp/dist-test-task3bwPeE/build/tsan/bin/kudu-tserver as pid 3290 I0312 20:11:43.409168 3290 tablet_server_main.cc:87] Tablet server successfully started. I0312 20:11:43.409791 3412 process_memory.cc:182] Process hard memory limit is 11.777631 GB I0312 20:11:43.432700 3412 process_memory.cc:184] Process soft memory limit is 9.422105 GB I0312 20:11:43.432826 3412 process_memory.cc:187] Process memory pressure threshold is 7.066578 GB I0312 20:11:43.432898 3411 heartbeater.cc:346] Connected to a master server at 127.0.53.62:48469 I0312 20:11:43.433221 3411 heartbeater.cc:426] Registering TS with master... I0312 20:11:43.433651 3411 heartbeater.cc:475] Master 127.0.53.62:48469 requested a full tablet report, sending... I0312 20:11:43.434986 2995 ts_manager.cc:81] Registered new tserver with Master: d1a250fbb6a14dca8c8bb69a2b97065e (127.0.53.3:35242) I0312 20:11:43.436223 2995 master_service.cc:243] Signed X509 certificate for tserver {username='slave'} at 127.0.53.3:32902 I0312 20:11:43.437147 212 external_mini_cluster.cc:498] 3 TS(s) registered with all masters I0312 20:11:43.456863 3162 heartbeater.cc:467] Master 127.0.53.62:48469 was elected leader, sending a full tablet report... I0312 20:11:43.454988 2995 catalog_manager.cc:1294] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53280: name: "TestTable" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 3 split_rows_range_bounds { } partition_schema { range_schema { columns { name: "key" } } } W0312 20:11:43.463723 2995 catalog_manager.cc:1450] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a replica failure: 4 tablet servers are needed, 3 are alive. Consider bringing up additional tablet server(s) or running both the masters and all tablet servers with --raft_prepare_replacement_before_eviction=false flag (not recommended). I0312 20:11:43.492880 3224 tablet_service.cc:755] Processing CreateTablet for tablet b020e04781ed46b99331105c6af00dfe (table=TestTable [id=75b7c430a7254c2e9683606937659c40]), partition=RANGE (key) PARTITION UNBOUNDED I0312 20:11:43.493603 3224 data_dirs.cc:931] Could only allocate 1 dirs of requested 3 for tablet b020e04781ed46b99331105c6af00dfe. 1 dirs total, 0 dirs full, 0 dirs failed I0312 20:11:43.494065 3099 tablet_service.cc:755] Processing CreateTablet for tablet b020e04781ed46b99331105c6af00dfe (table=TestTable [id=75b7c430a7254c2e9683606937659c40]), partition=RANGE (key) PARTITION UNBOUNDED I0312 20:11:43.494918 3099 data_dirs.cc:931] Could only allocate 1 dirs of requested 3 for tablet b020e04781ed46b99331105c6af00dfe. 1 dirs total, 0 dirs full, 0 dirs failed I0312 20:11:43.503588 3348 tablet_service.cc:755] Processing CreateTablet for tablet b020e04781ed46b99331105c6af00dfe (table=TestTable [id=75b7c430a7254c2e9683606937659c40]), partition=RANGE (key) PARTITION UNBOUNDED I0312 20:11:43.504446 3348 data_dirs.cc:931] Could only allocate 1 dirs of requested 3 for tablet b020e04781ed46b99331105c6af00dfe. 1 dirs total, 0 dirs full, 0 dirs failed I0312 20:11:43.506126 3099 ts_tablet_manager.cc:1163] T b020e04781ed46b99331105c6af00dfe P 1e9afc22e84f401ba29b1e7ec68ec8c6: Registered tablet (data state: TABLET_DATA_READY) I0312 20:11:43.506975 3224 ts_tablet_manager.cc:1163] T b020e04781ed46b99331105c6af00dfe P c305fa84777a4d6ea71fe631360fde01: Registered tablet (data state: TABLET_DATA_READY) I0312 20:11:43.508611 3432 ts_tablet_manager.cc:1018] T b020e04781ed46b99331105c6af00dfe P c305fa84777a4d6ea71fe631360fde01: Bootstrapping tablet I0312 20:11:43.509105 3432 tablet_bootstrap.cc:436] T b020e04781ed46b99331105c6af00dfe P c305fa84777a4d6ea71fe631360fde01: Bootstrap starting. I0312 20:11:43.512045 3432 tablet_bootstrap.cc:581] T b020e04781ed46b99331105c6af00dfe P c305fa84777a4d6ea71fe631360fde01: No blocks or log segments found. Creating new log. I0312 20:11:43.513620 3432 log.cc:520] T b020e04781ed46b99331105c6af00dfe P c305fa84777a4d6ea71fe631360fde01: Log is configured to *not* fsync() on all Append() calls I0312 20:11:43.516753 3348 ts_tablet_manager.cc:1163] T b020e04781ed46b99331105c6af00dfe P d1a250fbb6a14dca8c8bb69a2b97065e: Registered tablet (data state: TABLET_DATA_READY) I0312 20:11:43.518923 3434 ts_tablet_manager.cc:1018] T b020e04781ed46b99331105c6af00dfe P d1a250fbb6a14dca8c8bb69a2b97065e: Bootstrapping tablet I0312 20:11:43.519464 3431 ts_tablet_manager.cc:1018] T b020e04781ed46b99331105c6af00dfe P 1e9afc22e84f401ba29b1e7ec68ec8c6: Bootstrapping tablet I0312 20:11:43.519815 3431 tablet_bootstrap.cc:436] T b020e04781ed46b99331105c6af00dfe P 1e9afc22e84f401ba29b1e7ec68ec8c6: Bootstrap starting. I0312 20:11:43.520143 3434 tablet_bootstrap.cc:436] T b020e04781ed46b99331105c6af00dfe P d1a250fbb6a14dca8c8bb69a2b97065e: Bootstrap starting. I0312 20:11:43.522320 3431 tablet_bootstrap.cc:581] T b020e04781ed46b99331105c6af00dfe P 1e9afc22e84f401ba29b1e7ec68ec8c6: No blocks or log segments found. Creating new log. I0312 20:11:43.522538 3434 tablet_bootstrap.cc:581] T b020e04781ed46b99331105c6af00dfe P d1a250fbb6a14dca8c8bb69a2b97065e: No blocks or log segments found. Creating new log. I0312 20:11:43.523712 3434 log.cc:520] T b020e04781ed46b99331105c6af00dfe P d1a250fbb6a14dca8c8bb69a2b97065e: Log is configured to *not* fsync() on all Append() calls I0312 20:11:43.523958 3432 tablet_bootstrap.cc:436] T b020e04781ed46b99331105c6af00dfe P c305fa84777a4d6ea71fe631360fde01: No bootstrap required, opened a new log I0312 20:11:43.524246 3432 ts_tablet_manager.cc:1035] T b020e04781ed46b99331105c6af00dfe P c305fa84777a4d6ea71fe631360fde01: Time spent bootstrapping tablet: real 0.015s user 0.000s sys 0.008s I0312 20:11:43.526079 3434 tablet_bootstrap.cc:436] T b020e04781ed46b99331105c6af00dfe P d1a250fbb6a14dca8c8bb69a2b97065e: No bootstrap required, opened a new log I0312 20:11:43.526357 3434 ts_tablet_manager.cc:1035] T b020e04781ed46b99331105c6af00dfe P d1a250fbb6a14dca8c8bb69a2b97065e: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s I0312 20:11:43.535320 3432 raft_consensus.cc:304] T b020e04781ed46b99331105c6af00dfe P c305fa84777a4d6ea71fe631360fde01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c305fa84777a4d6ea71fe631360fde01" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 39347 } } peers { permanent_uuid: "1e9afc22e84f401ba29b1e7ec68ec8c6" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 33856 } } peers { permanent_uuid: "d1a250fbb6a14dca8c8bb69a2b97065e" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 35242 } } I0312 20:11:43.536106 3432 raft_consensus.cc:330] T b020e04781ed46b99331105c6af00dfe P c305fa84777a4d6ea71fe631360fde01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0312 20:11:43.535807 3434 raft_consensus.cc:304] T b020e04781ed46b99331105c6af00dfe P d1a250fbb6a14dca8c8bb69a2b97065e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c305fa84777a4d6ea71fe631360fde01" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 39347 } } peers { permanent_uuid: "1e9afc22e84f401ba29b1e7ec68ec8c6" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 33856 } } peers { permanent_uuid: "d1a250fbb6a14dca8c8bb69a2b97065e" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 35242 } } I0312 20:11:43.536305 3432 raft_consensus.cc:605] T b020e04781ed46b99331105c6af00dfe P c305fa84777a4d6ea71fe631360fde01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c305fa84777a4d6ea71fe631360fde01, State: Initialized, Role: FOLLOWER I0312 20:11:43.536366 3434 raft_consensus.cc:330] T b020e04781ed46b99331105c6af00dfe P d1a250fbb6a14dca8c8bb69a2b97065e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0312 20:11:43.536507 3434 raft_consensus.cc:605] T b020e04781ed46b99331105c6af00dfe P d1a250fbb6a14dca8c8bb69a2b97065e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1a250fbb6a14dca8c8bb69a2b97065e, State: Initialized, Role: FOLLOWER I0312 20:11:43.536761 3432 consensus_queue.cc:219] T b020e04781ed46b99331105c6af00dfe P c305fa84777a4d6ea71fe631360fde01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c305fa84777a4d6ea71fe631360fde01" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 39347 } } peers { permanent_uuid: "1e9afc22e84f401ba29b1e7ec68ec8c6" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 33856 } } peers { permanent_uuid: "d1a250fbb6a14dca8c8bb69a2b97065e" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 35242 } } I0312 20:11:43.536950 3434 consensus_queue.cc:219] T b020e04781ed46b99331105c6af00dfe P d1a250fbb6a14dca8c8bb69a2b97065e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c305fa84777a4d6ea71fe631360fde01" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 39347 } } peers { permanent_uuid: "1e9afc22e84f401ba29b1e7ec68ec8c6" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 33856 } } peers { permanent_uuid: "d1a250fbb6a14dca8c8bb69a2b97065e" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 35242 } } I0312 20:11:43.538902 3411 heartbeater.cc:467] Master 127.0.53.62:48469 was elected leader, sending a full tablet report... I0312 20:11:43.539425 3431 log.cc:520] T b020e04781ed46b99331105c6af00dfe P 1e9afc22e84f401ba29b1e7ec68ec8c6: Log is configured to *not* fsync() on all Append() calls I0312 20:11:43.539512 3434 ts_tablet_manager.cc:1063] T b020e04781ed46b99331105c6af00dfe P d1a250fbb6a14dca8c8bb69a2b97065e: Time spent starting tablet: real 0.013s user 0.008s sys 0.008s I0312 20:11:43.540483 3287 heartbeater.cc:467] Master 127.0.53.62:48469 was elected leader, sending a full tablet report... I0312 20:11:43.541085 3432 ts_tablet_manager.cc:1063] T b020e04781ed46b99331105c6af00dfe P c305fa84777a4d6ea71fe631360fde01: Time spent starting tablet: real 0.017s user 0.016s sys 0.000s I0312 20:11:43.542431 3431 tablet_bootstrap.cc:436] T b020e04781ed46b99331105c6af00dfe P 1e9afc22e84f401ba29b1e7ec68ec8c6: No bootstrap required, opened a new log I0312 20:11:43.542834 3431 ts_tablet_manager.cc:1035] T b020e04781ed46b99331105c6af00dfe P 1e9afc22e84f401ba29b1e7ec68ec8c6: Time spent bootstrapping tablet: real 0.023s user 0.004s sys 0.000s I0312 20:11:43.554167 3431 raft_consensus.cc:304] T b020e04781ed46b99331105c6af00dfe P 1e9afc22e84f401ba29b1e7ec68ec8c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c305fa84777a4d6ea71fe631360fde01" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 39347 } } peers { permanent_uuid: "1e9afc22e84f401ba29b1e7ec68ec8c6" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 33856 } } peers { permanent_uuid: "d1a250fbb6a14dca8c8bb69a2b97065e" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 35242 } } I0312 20:11:43.554682 3431 raft_consensus.cc:330] T b020e04781ed46b99331105c6af00dfe P 1e9afc22e84f401ba29b1e7ec68ec8c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0312 20:11:43.554814 3431 raft_consensus.cc:605] T b020e04781ed46b99331105c6af00dfe P 1e9afc22e84f401ba29b1e7ec68ec8c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e9afc22e84f401ba29b1e7ec68ec8c6, State: Initialized, Role: FOLLOWER I0312 20:11:43.555207 3431 consensus_queue.cc:219] T b020e04781ed46b99331105c6af00dfe P 1e9afc22e84f401ba29b1e7ec68ec8c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c305fa84777a4d6ea71fe631360fde01" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 39347 } } peers { permanent_uuid: "1e9afc22e84f401ba29b1e7ec68ec8c6" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 33856 } } peers { permanent_uuid: "d1a250fbb6a14dca8c8bb69a2b97065e" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 35242 } } I0312 20:11:43.557332 3431 ts_tablet_manager.cc:1063] T b020e04781ed46b99331105c6af00dfe P 1e9afc22e84f401ba29b1e7ec68ec8c6: Time spent starting tablet: real 0.014s user 0.016s sys 0.000s W0312 20:11:43.683224 3412 tablet.cc:1679] T b020e04781ed46b99331105c6af00dfe P d1a250fbb6a14dca8c8bb69a2b97065e: Can't schedule compaction. Clean time has not been advanced past its initial value. W0312 20:11:43.704033 3288 tablet.cc:1679] T b020e04781ed46b99331105c6af00dfe P c305fa84777a4d6ea71fe631360fde01: Can't schedule compaction. Clean time has not been advanced past its initial value. I0312 20:11:43.707545 3436 raft_consensus.cc:436] T b020e04781ed46b99331105c6af00dfe P c305fa84777a4d6ea71fe631360fde01 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0312 20:11:43.707861 3436 raft_consensus.cc:2700] T b020e04781ed46b99331105c6af00dfe P c305fa84777a4d6ea71fe631360fde01: Snoozing failure detection for 2.148s (starting election) I0312 20:11:43.708153 3436 raft_consensus.cc:458] T b020e04781ed46b99331105c6af00dfe P c305fa84777a4d6ea71fe631360fde01 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c305fa84777a4d6ea71fe631360fde01" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 39347 } } peers { permanent_uuid: "1e9afc22e84f401ba29b1e7ec68ec8c6" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 33856 } } peers { permanent_uuid: "d1a250fbb6a14dca8c8bb69a2b97065e" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 35242 } } I0312 20:11:43.709540 3436 leader_election.cc:231] T b020e04781ed46b99331105c6af00dfe P c305fa84777a4d6ea71fe631360fde01 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 1e9afc22e84f401ba29b1e7ec68ec8c6 I0312 20:11:43.710077 3436 leader_election.cc:231] T b020e04781ed46b99331105c6af00dfe P c305fa84777a4d6ea71fe631360fde01 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer d1a250fbb6a14dca8c8bb69a2b97065e I0312 20:11:43.724948 3119 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "b020e04781ed46b99331105c6af00dfe" candidate_uuid: "c305fa84777a4d6ea71fe631360fde01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e9afc22e84f401ba29b1e7ec68ec8c6" is_pre_election: true I0312 20:11:43.725420 3119 raft_consensus.cc:2700] T b020e04781ed46b99331105c6af00dfe P 1e9afc22e84f401ba29b1e7ec68ec8c6: Snoozing failure detection for 1.758s (vote granted) I0312 20:11:43.725617 3119 raft_consensus.cc:2215] T b020e04781ed46b99331105c6af00dfe P 1e9afc22e84f401ba29b1e7ec68ec8c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c305fa84777a4d6ea71fe631360fde01 in term 0. I0312 20:11:43.726526 3175 leader_election.cc:363] T b020e04781ed46b99331105c6af00dfe P c305fa84777a4d6ea71fe631360fde01 [CANDIDATE]: Term 1 pre-election: Vote granted by peer 1e9afc22e84f401ba29b1e7ec68ec8c6 I0312 20:11:43.726704 3175 leader_election.cc:258] T b020e04781ed46b99331105c6af00dfe P c305fa84777a4d6ea71fe631360fde01 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. I0312 20:11:43.727094 3436 raft_consensus.cc:2700] T b020e04781ed46b99331105c6af00dfe P c305fa84777a4d6ea71fe631360fde01: Snoozing failure detection for 1.879s (election complete) I0312 20:11:43.727273 3436 raft_consensus.cc:2507] T b020e04781ed46b99331105c6af00dfe P c305fa84777a4d6ea71fe631360fde01 [term 0 FOLLOWER]: Leader pre-election won for term 1 I0312 20:11:43.727485 3436 raft_consensus.cc:436] T b020e04781ed46b99331105c6af00dfe P c305fa84777a4d6ea71fe631360fde01 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0312 20:11:43.727668 3436 raft_consensus.cc:2700] T b020e04781ed46b99331105c6af00dfe P c305fa84777a4d6ea71fe631360fde01: Snoozing failure detection for 2.220s (starting election) I0312 20:11:43.727882 3436 raft_consensus.cc:2752] T b020e04781ed46b99331105c6af00dfe P c305fa84777a4d6ea71fe631360fde01 [term 0 FOLLOWER]: Advancing to term 1 I0312 20:11:43.728641 3368 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "b020e04781ed46b99331105c6af00dfe" candidate_uuid: "c305fa84777a4d6ea71fe631360fde01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1a250fbb6a14dca8c8bb69a2b97065e" is_pre_election: true I0312 20:11:43.729238 3368 raft_consensus.cc:2700] T b020e04781ed46b99331105c6af00dfe P d1a250fbb6a14dca8c8bb69a2b97065e: Snoozing failure detection for 2.127s (vote granted) I0312 20:11:43.729485 3368 raft_consensus.cc:2215] T b020e04781ed46b99331105c6af00dfe P d1a250fbb6a14dca8c8bb69a2b97065e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c305fa84777a4d6ea71fe631360fde01 in term 0. I0312 20:11:43.730715 3176 leader_election.cc:363] T b020e04781ed46b99331105c6af00dfe P c305fa84777a4d6ea71fe631360fde01 [CANDIDATE]: Term 1 pre-election: Vote granted by peer d1a250fbb6a14dca8c8bb69a2b97065e I0312 20:11:43.731904 3436 raft_consensus.cc:458] T b020e04781ed46b99331105c6af00dfe P c305fa84777a4d6ea71fe631360fde01 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c305fa84777a4d6ea71fe631360fde01" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 39347 } } peers { permanent_uuid: "1e9afc22e84f401ba29b1e7ec68ec8c6" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 33856 } } peers { permanent_uuid: "d1a250fbb6a14dca8c8bb69a2b97065e" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 35242 } } I0312 20:11:43.732630 3436 leader_election.cc:231] T b020e04781ed46b99331105c6af00dfe P c305fa84777a4d6ea71fe631360fde01 [CANDIDATE]: Term 1 election: Requesting vote from peer 1e9afc22e84f401ba29b1e7ec68ec8c6 I0312 20:11:43.732914 3436 leader_election.cc:231] T b020e04781ed46b99331105c6af00dfe P c305fa84777a4d6ea71fe631360fde01 [CANDIDATE]: Term 1 election: Requesting vote from peer d1a250fbb6a14dca8c8bb69a2b97065e I0312 20:11:43.733757 3119 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "b020e04781ed46b99331105c6af00dfe" candidate_uuid: "c305fa84777a4d6ea71fe631360fde01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e9afc22e84f401ba29b1e7ec68ec8c6" I0312 20:11:43.734153 3119 raft_consensus.cc:2752] T b020e04781ed46b99331105c6af00dfe P 1e9afc22e84f401ba29b1e7ec68ec8c6 [term 0 FOLLOWER]: Advancing to term 1 I0312 20:11:43.734021 3368 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "b020e04781ed46b99331105c6af00dfe" candidate_uuid: "c305fa84777a4d6ea71fe631360fde01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1a250fbb6a14dca8c8bb69a2b97065e" I0312 20:11:43.734398 3119 raft_consensus.cc:2700] T b020e04781ed46b99331105c6af00dfe P 1e9afc22e84f401ba29b1e7ec68ec8c6: Snoozing failure detection for 2.193s (vote granted) I0312 20:11:43.734398 3368 raft_consensus.cc:2752] T b020e04781ed46b99331105c6af00dfe P d1a250fbb6a14dca8c8bb69a2b97065e [term 0 FOLLOWER]: Advancing to term 1 I0312 20:11:43.734627 3368 raft_consensus.cc:2700] T b020e04781ed46b99331105c6af00dfe P d1a250fbb6a14dca8c8bb69a2b97065e: Snoozing failure detection for 2.089s (vote granted) I0312 20:11:43.738574 3119 raft_consensus.cc:2215] T b020e04781ed46b99331105c6af00dfe P 1e9afc22e84f401ba29b1e7ec68ec8c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c305fa84777a4d6ea71fe631360fde01 in term 1. I0312 20:11:43.739440 3175 leader_election.cc:363] T b020e04781ed46b99331105c6af00dfe P c305fa84777a4d6ea71fe631360fde01 [CANDIDATE]: Term 1 election: Vote granted by peer 1e9afc22e84f401ba29b1e7ec68ec8c6 I0312 20:11:43.739593 3175 leader_election.cc:258] T b020e04781ed46b99331105c6af00dfe P c305fa84777a4d6ea71fe631360fde01 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0312 20:11:43.739935 3436 raft_consensus.cc:2700] T b020e04781ed46b99331105c6af00dfe P c305fa84777a4d6ea71fe631360fde01: Snoozing failure detection for 1.580s (election complete) I0312 20:11:43.740007 3368 raft_consensus.cc:2215] T b020e04781ed46b99331105c6af00dfe P d1a250fbb6a14dca8c8bb69a2b97065e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c305fa84777a4d6ea71fe631360fde01 in term 1. I0312 20:11:43.740167 3436 raft_consensus.cc:2507] T b020e04781ed46b99331105c6af00dfe P c305fa84777a4d6ea71fe631360fde01 [term 1 FOLLOWER]: Leader election won for term 1 I0312 20:11:43.740850 3176 leader_election.cc:363] T b020e04781ed46b99331105c6af00dfe P c305fa84777a4d6ea71fe631360fde01 [CANDIDATE]: Term 1 election: Vote granted by peer d1a250fbb6a14dca8c8bb69a2b97065e I0312 20:11:43.742236 3436 raft_consensus.cc:572] T b020e04781ed46b99331105c6af00dfe P c305fa84777a4d6ea71fe631360fde01 [term 1 LEADER]: Becoming Leader. State: Replica: c305fa84777a4d6ea71fe631360fde01, State: Running, Role: LEADER I0312 20:11:43.742602 3436 consensus_queue.cc:196] T b020e04781ed46b99331105c6af00dfe P c305fa84777a4d6ea71fe631360fde01 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c305fa84777a4d6ea71fe631360fde01" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 39347 } } peers { permanent_uuid: "1e9afc22e84f401ba29b1e7ec68ec8c6" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 33856 } } peers { permanent_uuid: "d1a250fbb6a14dca8c8bb69a2b97065e" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 35242 } } I0312 20:11:43.747928 2995 catalog_manager.cc:3592] T b020e04781ed46b99331105c6af00dfe P c305fa84777a4d6ea71fe631360fde01 reported cstate change: term changed from 0 to 1, leader changed from to c305fa84777a4d6ea71fe631360fde01 (127.0.53.2). New cstate: current_term: 1 leader_uuid: "c305fa84777a4d6ea71fe631360fde01" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c305fa84777a4d6ea71fe631360fde01" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 39347 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1e9afc22e84f401ba29b1e7ec68ec8c6" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 33856 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1a250fbb6a14dca8c8bb69a2b97065e" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 35242 } health_report { overall_health: UNKNOWN } } } W0312 20:11:43.663568 3163 tablet.cc:1679] T b020e04781ed46b99331105c6af00dfe P 1e9afc22e84f401ba29b1e7ec68ec8c6: Can't schedule compaction. Clean time has not been advanced past its initial value. I0312 20:11:43.810942 212 external_mini_cluster.cc:498] 3 TS(s) registered with all masters I0312 20:11:43.812969 212 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 1e9afc22e84f401ba29b1e7ec68ec8c6 to finish bootstrapping I0312 20:11:43.822412 212 ts_itest-base.cc:233] Waiting for 1 tablets on tserver c305fa84777a4d6ea71fe631360fde01 to finish bootstrapping I0312 20:11:43.831003 212 ts_itest-base.cc:233] Waiting for 1 tablets on tserver d1a250fbb6a14dca8c8bb69a2b97065e to finish bootstrapping I0312 20:11:43.860087 212 raft_consensus_election-itest.cc:240] Writing data to leader of 3-node config (3 alive)... I0312 20:11:43.887823 3368 raft_consensus.cc:1058] T b020e04781ed46b99331105c6af00dfe P d1a250fbb6a14dca8c8bb69a2b97065e [term 1 FOLLOWER]: Refusing update from remote peer c305fa84777a4d6ea71fe631360fde01: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0312 20:11:43.888211 3119 raft_consensus.cc:1058] T b020e04781ed46b99331105c6af00dfe P 1e9afc22e84f401ba29b1e7ec68ec8c6 [term 1 FOLLOWER]: Refusing update from remote peer c305fa84777a4d6ea71fe631360fde01: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0312 20:11:43.889098 3442 consensus_queue.cc:1017] T b020e04781ed46b99331105c6af00dfe P c305fa84777a4d6ea71fe631360fde01 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1a250fbb6a14dca8c8bb69a2b97065e" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 35242 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0312 20:11:43.889559 3436 consensus_queue.cc:1017] T b020e04781ed46b99331105c6af00dfe P c305fa84777a4d6ea71fe631360fde01 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e9afc22e84f401ba29b1e7ec68ec8c6" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 33856 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0312 20:11:43.952061 212 raft_consensus_election-itest.cc:251] Killing current leader c305fa84777a4d6ea71fe631360fde01... I0312 20:11:43.952308 212 external_mini_cluster.cc:997] Killing /tmp/dist-test-task3bwPeE/build/tsan/bin/kudu-tserver with pid 3166 W0312 20:11:43.972789 2977 connection.cc:513] client connection to 127.0.53.2:39347 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0312 20:11:43.972810 2966 connection.cc:513] client connection to 127.0.53.2:39347 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0312 20:11:43.972789 2978 connection.cc:513] server connection from 127.0.53.2:54620 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0312 20:11:43.973800 3054 connection.cc:513] server connection from 127.0.53.2:60182 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0312 20:11:43.974491 3054 connection.cc:186] Error closing socket: Network error: TlsSocket::Close: Broken pipe W0312 20:11:43.975061 3299 connection.cc:513] server connection from 127.0.53.2:37294 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0312 20:11:43.975073 2963 connection.cc:513] client connection to 127.0.53.2:39347 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) I0312 20:11:43.975229 212 raft_consensus_election-itest.cc:255] Waiting for new guy to be elected leader. W0312 20:11:43.975895 3417 connection.cc:513] client connection to 127.0.53.2:39347 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) I0312 20:11:45.452131 3455 raft_consensus.cc:436] T b020e04781ed46b99331105c6af00dfe P d1a250fbb6a14dca8c8bb69a2b97065e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c305fa84777a4d6ea71fe631360fde01) I0312 20:11:45.452371 3455 raft_consensus.cc:2700] T b020e04781ed46b99331105c6af00dfe P d1a250fbb6a14dca8c8bb69a2b97065e: Snoozing failure detection for 2.019s (starting election) I0312 20:11:45.452594 3455 raft_consensus.cc:458] T b020e04781ed46b99331105c6af00dfe P d1a250fbb6a14dca8c8bb69a2b97065e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c305fa84777a4d6ea71fe631360fde01" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 39347 } } peers { permanent_uuid: "1e9afc22e84f401ba29b1e7ec68ec8c6" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 33856 } } peers { permanent_uuid: "d1a250fbb6a14dca8c8bb69a2b97065e" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 35242 } } I0312 20:11:45.453593 3455 leader_election.cc:231] T b020e04781ed46b99331105c6af00dfe P d1a250fbb6a14dca8c8bb69a2b97065e [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer c305fa84777a4d6ea71fe631360fde01 I0312 20:11:45.453991 3455 leader_election.cc:231] T b020e04781ed46b99331105c6af00dfe P d1a250fbb6a14dca8c8bb69a2b97065e [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 1e9afc22e84f401ba29b1e7ec68ec8c6 I0312 20:11:45.455015 3456 raft_consensus.cc:436] T b020e04781ed46b99331105c6af00dfe P 1e9afc22e84f401ba29b1e7ec68ec8c6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c305fa84777a4d6ea71fe631360fde01) I0312 20:11:45.457007 3456 raft_consensus.cc:2700] T b020e04781ed46b99331105c6af00dfe P 1e9afc22e84f401ba29b1e7ec68ec8c6: Snoozing failure detection for 1.950s (starting election) I0312 20:11:45.457257 3456 raft_consensus.cc:458] T b020e04781ed46b99331105c6af00dfe P 1e9afc22e84f401ba29b1e7ec68ec8c6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c305fa84777a4d6ea71fe631360fde01" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 39347 } } peers { permanent_uuid: "1e9afc22e84f401ba29b1e7ec68ec8c6" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 33856 } } peers { permanent_uuid: "d1a250fbb6a14dca8c8bb69a2b97065e" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 35242 } } W0312 20:11:45.457929 3300 leader_election.cc:287] T b020e04781ed46b99331105c6af00dfe P d1a250fbb6a14dca8c8bb69a2b97065e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c305fa84777a4d6ea71fe631360fde01: Network error: Client connection negotiation failed: client connection to 127.0.53.2:39347: connect: Connection refused (error 111) I0312 20:11:45.458236 3456 leader_election.cc:231] T b020e04781ed46b99331105c6af00dfe P 1e9afc22e84f401ba29b1e7ec68ec8c6 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer c305fa84777a4d6ea71fe631360fde01 I0312 20:11:45.458612 3456 leader_election.cc:231] T b020e04781ed46b99331105c6af00dfe P 1e9afc22e84f401ba29b1e7ec68ec8c6 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer d1a250fbb6a14dca8c8bb69a2b97065e W0312 20:11:45.461730 3050 leader_election.cc:287] T b020e04781ed46b99331105c6af00dfe P 1e9afc22e84f401ba29b1e7ec68ec8c6 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c305fa84777a4d6ea71fe631360fde01: Network error: Client connection negotiation failed: client connection to 127.0.53.2:39347: connect: Connection refused (error 111) I0312 20:11:45.465603 3119 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "b020e04781ed46b99331105c6af00dfe" candidate_uuid: "d1a250fbb6a14dca8c8bb69a2b97065e" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "1e9afc22e84f401ba29b1e7ec68ec8c6" is_pre_election: true I0312 20:11:45.465976 3119 raft_consensus.cc:2700] T b020e04781ed46b99331105c6af00dfe P 1e9afc22e84f401ba29b1e7ec68ec8c6: Snoozing failure detection for 1.855s (vote granted) I0312 20:11:45.466135 3119 raft_consensus.cc:2215] T b020e04781ed46b99331105c6af00dfe P 1e9afc22e84f401ba29b1e7ec68ec8c6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1a250fbb6a14dca8c8bb69a2b97065e in term 1. I0312 20:11:45.466922 3299 leader_election.cc:363] T b020e04781ed46b99331105c6af00dfe P d1a250fbb6a14dca8c8bb69a2b97065e [CANDIDATE]: Term 2 pre-election: Vote granted by peer 1e9afc22e84f401ba29b1e7ec68ec8c6 I0312 20:11:45.467095 3299 leader_election.cc:258] T b020e04781ed46b99331105c6af00dfe P d1a250fbb6a14dca8c8bb69a2b97065e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. I0312 20:11:45.467494 3455 raft_consensus.cc:2700] T b020e04781ed46b99331105c6af00dfe P d1a250fbb6a14dca8c8bb69a2b97065e: Snoozing failure detection for 1.905s (election complete) I0312 20:11:45.467707 3455 raft_consensus.cc:2507] T b020e04781ed46b99331105c6af00dfe P d1a250fbb6a14dca8c8bb69a2b97065e [term 1 FOLLOWER]: Leader pre-election won for term 2 I0312 20:11:45.467834 3455 raft_consensus.cc:436] T b020e04781ed46b99331105c6af00dfe P d1a250fbb6a14dca8c8bb69a2b97065e [term 1 FOLLOWER]: Starting leader election (detected failure of leader c305fa84777a4d6ea71fe631360fde01) I0312 20:11:45.467958 3455 raft_consensus.cc:2700] T b020e04781ed46b99331105c6af00dfe P d1a250fbb6a14dca8c8bb69a2b97065e: Snoozing failure detection for 1.978s (starting election) I0312 20:11:45.468075 3455 raft_consensus.cc:2752] T b020e04781ed46b99331105c6af00dfe P d1a250fbb6a14dca8c8bb69a2b97065e [term 1 FOLLOWER]: Advancing to term 2 I0312 20:11:45.469875 3367 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "b020e04781ed46b99331105c6af00dfe" candidate_uuid: "1e9afc22e84f401ba29b1e7ec68ec8c6" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "d1a250fbb6a14dca8c8bb69a2b97065e" is_pre_election: true I0312 20:11:45.471323 3455 raft_consensus.cc:458] T b020e04781ed46b99331105c6af00dfe P d1a250fbb6a14dca8c8bb69a2b97065e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c305fa84777a4d6ea71fe631360fde01" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 39347 } } peers { permanent_uuid: "1e9afc22e84f401ba29b1e7ec68ec8c6" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 33856 } } peers { permanent_uuid: "d1a250fbb6a14dca8c8bb69a2b97065e" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 35242 } } I0312 20:11:45.472025 3455 leader_election.cc:231] T b020e04781ed46b99331105c6af00dfe P d1a250fbb6a14dca8c8bb69a2b97065e [CANDIDATE]: Term 2 election: Requesting vote from peer c305fa84777a4d6ea71fe631360fde01 I0312 20:11:45.472167 3367 raft_consensus.cc:2145] T b020e04781ed46b99331105c6af00dfe P d1a250fbb6a14dca8c8bb69a2b97065e [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e9afc22e84f401ba29b1e7ec68ec8c6 in current term 2: Already voted for candidate d1a250fbb6a14dca8c8bb69a2b97065e in this term. I0312 20:11:45.472298 3455 leader_election.cc:231] T b020e04781ed46b99331105c6af00dfe P d1a250fbb6a14dca8c8bb69a2b97065e [CANDIDATE]: Term 2 election: Requesting vote from peer 1e9afc22e84f401ba29b1e7ec68ec8c6 W0312 20:11:45.473290 3300 leader_election.cc:287] T b020e04781ed46b99331105c6af00dfe P d1a250fbb6a14dca8c8bb69a2b97065e [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer c305fa84777a4d6ea71fe631360fde01: Network error: Client connection negotiation failed: client connection to 127.0.53.2:39347: connect: Connection refused (error 111) I0312 20:11:45.473420 3050 leader_election.cc:377] T b020e04781ed46b99331105c6af00dfe P 1e9afc22e84f401ba29b1e7ec68ec8c6 [CANDIDATE]: Term 2 pre-election: Vote denied by peer d1a250fbb6a14dca8c8bb69a2b97065e. Message: Invalid argument: T b020e04781ed46b99331105c6af00dfe P d1a250fbb6a14dca8c8bb69a2b97065e [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e9afc22e84f401ba29b1e7ec68ec8c6 in current term 2: Already voted for candidate d1a250fbb6a14dca8c8bb69a2b97065e in this term. I0312 20:11:45.473620 3050 leader_election.cc:258] T b020e04781ed46b99331105c6af00dfe P 1e9afc22e84f401ba29b1e7ec68ec8c6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. I0312 20:11:45.473531 3119 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "b020e04781ed46b99331105c6af00dfe" candidate_uuid: "d1a250fbb6a14dca8c8bb69a2b97065e" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "1e9afc22e84f401ba29b1e7ec68ec8c6" I0312 20:11:45.473896 3119 raft_consensus.cc:2752] T b020e04781ed46b99331105c6af00dfe P 1e9afc22e84f401ba29b1e7ec68ec8c6 [term 1 FOLLOWER]: Advancing to term 2 I0312 20:11:45.474081 3119 raft_consensus.cc:2700] T b020e04781ed46b99331105c6af00dfe P 1e9afc22e84f401ba29b1e7ec68ec8c6: Snoozing failure detection for 1.556s (vote granted) I0312 20:11:45.476678 3119 raft_consensus.cc:2215] T b020e04781ed46b99331105c6af00dfe P 1e9afc22e84f401ba29b1e7ec68ec8c6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1a250fbb6a14dca8c8bb69a2b97065e in term 2. I0312 20:11:45.476894 3456 raft_consensus.cc:2700] T b020e04781ed46b99331105c6af00dfe P 1e9afc22e84f401ba29b1e7ec68ec8c6: Snoozing failure detection for 1.784s (election complete) I0312 20:11:45.477059 3456 raft_consensus.cc:2455] T b020e04781ed46b99331105c6af00dfe P 1e9afc22e84f401ba29b1e7ec68ec8c6 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority I0312 20:11:45.477419 3299 leader_election.cc:363] T b020e04781ed46b99331105c6af00dfe P d1a250fbb6a14dca8c8bb69a2b97065e [CANDIDATE]: Term 2 election: Vote granted by peer 1e9afc22e84f401ba29b1e7ec68ec8c6 I0312 20:11:45.477612 3299 leader_election.cc:258] T b020e04781ed46b99331105c6af00dfe P d1a250fbb6a14dca8c8bb69a2b97065e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. I0312 20:11:45.477918 3455 raft_consensus.cc:2700] T b020e04781ed46b99331105c6af00dfe P d1a250fbb6a14dca8c8bb69a2b97065e: Snoozing failure detection for 1.621s (election complete) I0312 20:11:45.478107 3455 raft_consensus.cc:2507] T b020e04781ed46b99331105c6af00dfe P d1a250fbb6a14dca8c8bb69a2b97065e [term 2 FOLLOWER]: Leader election won for term 2 I0312 20:11:45.479125 3455 raft_consensus.cc:572] T b020e04781ed46b99331105c6af00dfe P d1a250fbb6a14dca8c8bb69a2b97065e [term 2 LEADER]: Becoming Leader. State: Replica: d1a250fbb6a14dca8c8bb69a2b97065e, State: Running, Role: LEADER I0312 20:11:45.479480 3455 consensus_queue.cc:196] T b020e04781ed46b99331105c6af00dfe P d1a250fbb6a14dca8c8bb69a2b97065e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c305fa84777a4d6ea71fe631360fde01" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 39347 } } peers { permanent_uuid: "1e9afc22e84f401ba29b1e7ec68ec8c6" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 33856 } } peers { permanent_uuid: "d1a250fbb6a14dca8c8bb69a2b97065e" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 35242 } } I0312 20:11:45.483821 2993 catalog_manager.cc:3592] T b020e04781ed46b99331105c6af00dfe P d1a250fbb6a14dca8c8bb69a2b97065e reported cstate change: term changed from 1 to 2, leader changed from c305fa84777a4d6ea71fe631360fde01 (127.0.53.2) to d1a250fbb6a14dca8c8bb69a2b97065e (127.0.53.3). New cstate: current_term: 2 leader_uuid: "d1a250fbb6a14dca8c8bb69a2b97065e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c305fa84777a4d6ea71fe631360fde01" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 39347 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e9afc22e84f401ba29b1e7ec68ec8c6" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 33856 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1a250fbb6a14dca8c8bb69a2b97065e" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 35242 } health_report { overall_health: HEALTHY } } } I0312 20:11:45.531141 212 raft_consensus_election-itest.cc:240] Writing data to leader of 3-node config (2 alive)... W0312 20:11:45.536078 3417 meta_cache.cc:218] Tablet b020e04781ed46b99331105c6af00dfe: Replica c305fa84777a4d6ea71fe631360fde01 (127.0.53.2:39347) has failed: Network error: Client connection negotiation failed: client connection to 127.0.53.2:39347: connect: Connection refused (error 111) I0312 20:11:45.570741 3119 raft_consensus.cc:1058] T b020e04781ed46b99331105c6af00dfe P 1e9afc22e84f401ba29b1e7ec68ec8c6 [term 2 FOLLOWER]: Refusing update from remote peer d1a250fbb6a14dca8c8bb69a2b97065e: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 8. (index mismatch) W0312 20:11:45.570897 3300 consensus_peers.cc:436] T b020e04781ed46b99331105c6af00dfe P d1a250fbb6a14dca8c8bb69a2b97065e -> Peer c305fa84777a4d6ea71fe631360fde01 (127.0.53.2:39347): Couldn't send request to peer c305fa84777a4d6ea71fe631360fde01 for tablet b020e04781ed46b99331105c6af00dfe. Status: Network error: Client connection negotiation failed: client connection to 127.0.53.2:39347: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 1 times. I0312 20:11:45.571864 3463 consensus_queue.cc:1017] T b020e04781ed46b99331105c6af00dfe P d1a250fbb6a14dca8c8bb69a2b97065e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e9afc22e84f401ba29b1e7ec68ec8c6" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 33856 }, Status: LMP_MISMATCH, Last received: 1.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s I0312 20:11:45.627481 212 external_mini_cluster.cc:805] Running /tmp/dist-test-task3bwPeE/build/tsan/bin/kudu-tserver /tmp/dist-test-task3bwPeE/build/tsan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.AutomaticLeaderElection.1520885401735786-212/raft_consensus-itest-cluster/ts-1/wal --fs_data_dirs=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.AutomaticLeaderElection.1520885401735786-212/raft_consensus-itest-cluster/ts-1/data --block_manager=log --rpc_bind_addresses=127.0.53.2:39347 --local_ip_for_outbound_sockets=127.0.53.2 --webserver_port=47579 --webserver_interface=127.0.53.2 --tserver_master_addrs=127.0.53.62:48469 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.AutomaticLeaderElection.1520885401735786-212/raft_consensus-itest-cluster/ts-1/logs --server_dump_info_path=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.AutomaticLeaderElection.1520885401735786-212/raft_consensus-itest-cluster/ts-1/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --log_cache_size_limit_mb=10 --consensus_rpc_timeout_ms=50 with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W0312 20:11:45.744232 3477 flags.cc:412] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0312 20:11:45.744601 3477 flags.cc:412] Enabled unsafe flag: --never_fsync=true W0312 20:11:45.758893 3477 flags.cc:412] Enabled experimental flag: --ipki_server_key_size=1024 W0312 20:11:45.759227 3477 flags.cc:412] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.53.2 I0312 20:11:45.778731 3477 tablet_server_main.cc:72] Tablet server non-default flags: --consensus_rpc_timeout_ms=50 --log_cache_size_limit_mb=10 --fs_data_dirs=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.AutomaticLeaderElection.1520885401735786-212/raft_consensus-itest-cluster/ts-1/data --fs_wal_dir=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.AutomaticLeaderElection.1520885401735786-212/raft_consensus-itest-cluster/ts-1/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.0.53.2:39347 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.AutomaticLeaderElection.1520885401735786-212/raft_consensus-itest-cluster/ts-1/data/info.pb --webserver_interface=127.0.53.2 --webserver_port=47579 --tserver_master_addrs=127.0.53.62:48469 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.0.53.2 --log_dir=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.AutomaticLeaderElection.1520885401735786-212/raft_consensus-itest-cluster/ts-1/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.7.0-SNAPSHOT revision 13e10c63c46de30ac85f34b543cdd6fb44881f61 build type FASTDEBUG built by None at 12 Mar 2018 20:06:40 UTC on jenkins-slave-vn0g build id 12388 TSAN enabled I0312 20:11:45.788646 3477 tablet_server_main.cc:79] Initializing tablet server... I0312 20:11:45.790374 3477 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 1119776us I0312 20:11:45.790833 3477 fs_manager.cc:260] Metadata directory not provided I0312 20:11:45.790977 3477 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0312 20:11:45.795514 3477 fs_manager.cc:406] Time spent opening directory manager: real 0.002s user 0.000s sys 0.000s I0312 20:11:45.795792 3477 env_posix.cc:1620] Not raising this process' open files per process limit of 65536; it is already as high as it can go I0312 20:11:45.796080 3477 file_cache.cc:470] Constructed file cache lbm with capacity 26214 I0312 20:11:45.802412 3477 fs_manager.cc:418] Time spent opening block manager: real 0.001s user 0.008s sys 0.000s I0312 20:11:45.802536 3477 fs_manager.cc:429] Opened local filesystem: /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.AutomaticLeaderElection.1520885401735786-212/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.AutomaticLeaderElection.1520885401735786-212/raft_consensus-itest-cluster/ts-1/wal uuid: "c305fa84777a4d6ea71fe631360fde01" format_stamp: "Formatted at 2018-03-12 20:11:42 on dist-test-slave-dist-test-slave-f27p" I0312 20:11:45.803692 3477 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.AutomaticLeaderElection.1520885401735786-212/raft_consensus-itest-cluster/ts-1/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) W0312 20:11:45.954233 3300 consensus_peers.cc:436] T b020e04781ed46b99331105c6af00dfe P d1a250fbb6a14dca8c8bb69a2b97065e -> Peer c305fa84777a4d6ea71fe631360fde01 (127.0.53.2:39347): Couldn't send request to peer c305fa84777a4d6ea71fe631360fde01 for tablet b020e04781ed46b99331105c6af00dfe. Status: Network error: Client connection negotiation failed: client connection to 127.0.53.2:39347: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 2 times. I0312 20:11:45.972349 3477 env_posix.cc:1620] Not raising this process' running threads per effective uid limit of 60284; it is already as high as it can go I0312 20:11:45.973649 3477 ts_tablet_manager.cc:337] Loading tablet metadata (0/1 complete) I0312 20:11:45.973776 3477 ts_tablet_manager.cc:995] T b020e04781ed46b99331105c6af00dfe P c305fa84777a4d6ea71fe631360fde01: Loading tablet metadata I0312 20:11:45.977926 3477 ts_tablet_manager.cc:349] Loaded tablet metadata (1 live tablets) I0312 20:11:45.980813 3477 ts_tablet_manager.cc:1163] T b020e04781ed46b99331105c6af00dfe P c305fa84777a4d6ea71fe631360fde01: Registered tablet (data state: TABLET_DATA_READY) I0312 20:11:45.983464 3494 ts_tablet_manager.cc:1018] T b020e04781ed46b99331105c6af00dfe P c305fa84777a4d6ea71fe631360fde01: Bootstrapping tablet I0312 20:11:45.983887 3494 tablet_bootstrap.cc:436] T b020e04781ed46b99331105c6af00dfe P c305fa84777a4d6ea71fe631360fde01: Bootstrap starting. I0312 20:11:46.009343 3494 log.cc:520] T b020e04781ed46b99331105c6af00dfe P c305fa84777a4d6ea71fe631360fde01: Log is configured to *not* fsync() on all Append() calls I0312 20:11:45.984407 3477 tablet_server_main.cc:84] Starting tablet server... I0312 20:11:46.051640 3494 tablet_bootstrap.cc:436] T b020e04781ed46b99331105c6af00dfe P c305fa84777a4d6ea71fe631360fde01: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=50 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I0312 20:11:46.052371 3494 tablet_bootstrap.cc:436] T b020e04781ed46b99331105c6af00dfe P c305fa84777a4d6ea71fe631360fde01: Bootstrap complete. I0312 20:11:46.054102 3494 ts_tablet_manager.cc:1035] T b020e04781ed46b99331105c6af00dfe P c305fa84777a4d6ea71fe631360fde01: Time spent bootstrapping tablet: real 0.070s user 0.048s sys 0.004s I0312 20:11:46.070204 3494 raft_consensus.cc:304] T b020e04781ed46b99331105c6af00dfe P c305fa84777a4d6ea71fe631360fde01 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c305fa84777a4d6ea71fe631360fde01" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 39347 } } peers { permanent_uuid: "1e9afc22e84f401ba29b1e7ec68ec8c6" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 33856 } } peers { permanent_uuid: "d1a250fbb6a14dca8c8bb69a2b97065e" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 35242 } } I0312 20:11:46.076586 3494 raft_consensus.cc:605] T b020e04781ed46b99331105c6af00dfe P c305fa84777a4d6ea71fe631360fde01 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c305fa84777a4d6ea71fe631360fde01, State: Initialized, Role: FOLLOWER I0312 20:11:46.077455 3494 consensus_queue.cc:219] T b020e04781ed46b99331105c6af00dfe P c305fa84777a4d6ea71fe631360fde01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c305fa84777a4d6ea71fe631360fde01" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 39347 } } peers { permanent_uuid: "1e9afc22e84f401ba29b1e7ec68ec8c6" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 33856 } } peers { permanent_uuid: "d1a250fbb6a14dca8c8bb69a2b97065e" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 35242 } } I0312 20:11:46.086593 3494 ts_tablet_manager.cc:1063] T b020e04781ed46b99331105c6af00dfe P c305fa84777a4d6ea71fe631360fde01: Time spent starting tablet: real 0.032s user 0.012s sys 0.016s I0312 20:11:46.148665 3477 rpc_server.cc:201] RPC server started. Bound to: 127.0.53.2:39347 I0312 20:11:46.149585 3477 webserver.cc:173] Starting webserver on 127.0.53.2:47579 I0312 20:11:46.149686 3477 webserver.cc:184] Document root disabled I0312 20:11:46.150943 3477 webserver.cc:311] Webserver started. Bound to: http://127.0.53.2:47579/ I0312 20:11:46.153683 3477 server_base.cc:607] Dumped server information to /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.AutomaticLeaderElection.1520885401735786-212/raft_consensus-itest-cluster/ts-1/data/info.pb I0312 20:11:46.156249 3477 tablet_server_main.cc:87] Tablet server successfully started. I0312 20:11:46.158846 3602 process_memory.cc:182] Process hard memory limit is 11.777631 GB I0312 20:11:46.159024 3602 process_memory.cc:184] Process soft memory limit is 9.422105 GB I0312 20:11:46.159140 3602 process_memory.cc:187] Process memory pressure threshold is 7.066578 GB I0312 20:11:46.163813 212 external_mini_cluster.cc:867] Started /tmp/dist-test-task3bwPeE/build/tsan/bin/kudu-tserver as pid 3477 I0312 20:11:46.170063 3601 heartbeater.cc:346] Connected to a master server at 127.0.53.62:48469 I0312 20:11:46.170331 3601 heartbeater.cc:426] Registering TS with master... I0312 20:11:46.170800 3601 heartbeater.cc:475] Master 127.0.53.62:48469 requested a full tablet report, sending... I0312 20:11:46.172765 2995 ts_manager.cc:87] Re-registered known tserver with Master: c305fa84777a4d6ea71fe631360fde01 (127.0.53.2:39347) I0312 20:11:46.174679 2995 master_service.cc:243] Signed X509 certificate for tserver {username='slave'} at 127.0.53.2:52327 Connected to the Master Fetched info from all 3 Tablet Servers Tablet b020e04781ed46b99331105c6af00dfe of table 'TestTable' is conflicted: Tablet b020e04781ed46b99331105c6af00dfe of table 'TestTable' replicas' active configs disagree with the master's 1e9afc22e84f401ba29b1e7ec68ec8c6 (127.0.53.1:33856): RUNNING c305fa84777a4d6ea71fe631360fde01 (127.0.53.2:39347): RUNNING d1a250fbb6a14dca8c8bb69a2b97065e (127.0.53.3:35242): RUNNING [LEADER] All the peers reported by the master and tablet servers are: A = 1e9afc22e84f401ba29b1e7ec68ec8c6 B = c305fa84777a4d6ea71fe631360fde01 C = d1a250fbb6a14dca8c8bb69a2b97065e The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+--------------+--------------+--------------+------------ master | A B C* | | | Yes A | A B C* | 2 | -1 | Yes B | A B C | 1 | -1 | Yes C | A B C* | 2 | -1 | Yes Table TestTable has 1 tablet(s) with mismatched consensus Table Summary Name | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable -----------+-------------+---------------+---------+------------+------------------+------------- TestTable | UNAVAILABLE | 1 | 0 | 0 | 0 | 1 I0312 20:11:46.266911 212 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Corruption: 1 out of 1 table(s) are not healthy Connected to the Master Fetched info from all 3 Tablet Servers Tablet b020e04781ed46b99331105c6af00dfe of table 'TestTable' is conflicted: Tablet b020e04781ed46b99331105c6af00dfe of table 'TestTable' replicas' active configs disagree with the master's 1e9afc22e84f401ba29b1e7ec68ec8c6 (127.0.53.1:33856): RUNNING c305fa84777a4d6ea71fe631360fde01 (127.0.53.2:39347): RUNNING d1a250fbb6a14dca8c8bb69a2b97065e (127.0.53.3:35242): RUNNING [LEADER] All the peers reported by the master and tablet servers are: A = 1e9afc22e84f401ba29b1e7ec68ec8c6 B = c305fa84777a4d6ea71fe631360fde01 C = d1a250fbb6a14dca8c8bb69a2b97065e The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+--------------+--------------+--------------+------------ master | A B C* | | | Yes A | A B C* | 2 | -1 | Yes B | A B C | 1 | -1 | Yes C | A B C* | 2 | -1 | Yes Table TestTable has 1 tablet(s) with mismatched consensus Table Summary I0312 20:11:46.480967 3558 raft_consensus.cc:2752] T b020e04781ed46b99331105c6af00dfe P c305fa84777a4d6ea71fe631360fde01 [term 1 FOLLOWER]: Advancing to term 2 Name | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable -----------+-------------+---------------+---------+------------+------------------+------------- TestTable | UNAVAILABLE | 1 | 0 | 0 | 0 | 1 I0312 20:11:46.484376 212 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Corruption: 1 out of 1 table(s) are not healthy I0312 20:11:46.484872 3601 heartbeater.cc:467] Master 127.0.53.62:48469 was elected leader, sending a full tablet report... Connected to the Master Fetched info from all 3 Tablet Servers Table TestTable is HEALTHY (1 tablet(s) checked) Table Summary Name | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable -----------+---------+---------------+---------+------------+------------------+------------- TestTable | HEALTHY | 1 | 1 | 0 | 0 | 0 The metadata for 1 table(s) is HEALTHY Checksum finished in 0s: 0/3 replicas remaining (0B from disk, 300 rows summed) ----------------------- TestTable ----------------------- T b020e04781ed46b99331105c6af00dfe P d1a250fbb6a14dca8c8bb69a2b97065e (127.0.53.3:35242): Checksum: 214634478652 T b020e04781ed46b99331105c6af00dfe P 1e9afc22e84f401ba29b1e7ec68ec8c6 (127.0.53.1:33856): Checksum: 214634478652 T b020e04781ed46b99331105c6af00dfe P c305fa84777a4d6ea71fe631360fde01 (127.0.53.2:39347): Checksum: 214634478652 I0312 20:11:46.813446 212 log_verifier.cc:121] Checking tablet b020e04781ed46b99331105c6af00dfe I0312 20:11:46.922789 212 log_verifier.cc:172] Verified matching terms for 12 ops in tablet b020e04781ed46b99331105c6af00dfe I0312 20:11:47.022052 212 external_mini_cluster.cc:997] Killing /tmp/dist-test-task3bwPeE/build/tsan/bin/kudu-tserver with pid 3029 W0312 20:11:47.047232 2976 connection.cc:513] client connection to 127.0.53.1:33856 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0312 20:11:47.048194 2976 connection.cc:513] server connection from 127.0.53.1:47622 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0312 20:11:47.048782 3299 connection.cc:513] client connection to 127.0.53.1:33856 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0312 20:11:47.049664 2966 connection.cc:513] client connection to 127.0.53.1:33856 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0312 20:11:47.050704 2962 connection.cc:513] client connection to 127.0.53.1:33856 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0312 20:11:47.051607 3303 connection.cc:513] server connection from 127.0.53.1:43208 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) I0312 20:11:47.052613 212 external_mini_cluster.cc:997] Killing /tmp/dist-test-task3bwPeE/build/tsan/bin/kudu-tserver with pid 3477 W0312 20:11:47.078199 2977 connection.cc:513] server connection from 127.0.53.2:52327 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0312 20:11:47.078199 3300 connection.cc:513] client connection to 127.0.53.2:39347 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) I0312 20:11:47.079061 212 external_mini_cluster.cc:997] Killing /tmp/dist-test-task3bwPeE/build/tsan/bin/kudu-tserver with pid 3290 W0312 20:11:47.107503 2977 connection.cc:513] server connection from 127.0.53.3:32902 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0312 20:11:47.107514 2966 connection.cc:513] client connection to 127.0.53.3:35242 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0312 20:11:47.107930 2977 connection.cc:513] client connection to 127.0.53.3:35242 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0312 20:11:47.108563 2963 connection.cc:513] client connection to 127.0.53.3:35242 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) I0312 20:11:47.109235 212 external_mini_cluster.cc:997] Killing /tmp/dist-test-task3bwPeE/build/tsan/bin/kudu-master with pid 2967 W0312 20:11:47.135973 2966 connection.cc:513] client connection to 127.0.53.62:48469 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) [ OK ] RaftConsensusElectionITest.AutomaticLeaderElection (5947 ms) [ RUN ] RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica I0312 20:11:47.149055 212 test_util.cc:192] Using random seed: -1067070232 I0312 20:11:47.151887 212 ts_itest-base.cc:113] Starting cluster with: I0312 20:11:47.151988 212 ts_itest-base.cc:114] -------------- I0312 20:11:47.152079 212 ts_itest-base.cc:115] 1 tablet servers I0312 20:11:47.152166 212 ts_itest-base.cc:116] 1 replicas per TS I0312 20:11:47.152254 212 ts_itest-base.cc:117] -------------- I0312 20:11:47.154376 212 external_mini_cluster.cc:805] Running /tmp/dist-test-task3bwPeE/build/tsan/bin/kudu-master /tmp/dist-test-task3bwPeE/build/tsan/bin/kudu-master --fs_wal_dir=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1520885401735786-212/raft_consensus-itest-cluster/master-0/wal --fs_data_dirs=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1520885401735786-212/raft_consensus-itest-cluster/master-0/data --block_manager=log --webserver_interface=localhost --ipki_ca_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.0.53.62:0 --webserver_interface=127.0.53.62 --webserver_port=0 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1520885401735786-212/raft_consensus-itest-cluster/master-0/logs --server_dump_info_path=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1520885401735786-212/raft_consensus-itest-cluster/master-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W0312 20:11:47.274101 3693 flags.cc:412] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0312 20:11:47.274415 3693 flags.cc:412] Enabled unsafe flag: --never_fsync=true W0312 20:11:47.289819 3693 flags.cc:412] Enabled experimental flag: --ipki_ca_key_size=1024 W0312 20:11:47.289945 3693 flags.cc:412] Enabled experimental flag: --ipki_server_key_size=1024 W0312 20:11:47.290041 3693 flags.cc:412] Enabled experimental flag: --tsk_num_rsa_bits=512 I0312 20:11:47.310333 3693 master_main.cc:66] Master server non-default flags: --fs_data_dirs=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1520885401735786-212/raft_consensus-itest-cluster/master-0/data --fs_wal_dir=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1520885401735786-212/raft_consensus-itest-cluster/master-0/wal --ipki_ca_key_size=1024 --ipki_server_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.0.53.62:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1520885401735786-212/raft_consensus-itest-cluster/master-0/data/info.pb --webserver_interface=127.0.53.62 --webserver_port=0 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --log_dir=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1520885401735786-212/raft_consensus-itest-cluster/master-0/logs --logbuflevel=-1 --logtostderr=true Master server version: kudu 1.7.0-SNAPSHOT revision 13e10c63c46de30ac85f34b543cdd6fb44881f61 build type FASTDEBUG built by None at 12 Mar 2018 20:06:40 UTC on jenkins-slave-vn0g build id 12388 TSAN enabled I0312 20:11:47.319401 3693 master_main.cc:73] Initializing master server... I0312 20:11:47.320883 3693 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 1120776us I0312 20:11:47.321614 3693 fs_manager.cc:260] Metadata directory not provided I0312 20:11:47.321740 3693 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0312 20:11:47.321928 3693 server_base.cc:434] Could not load existing FS layout: Not found: /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1520885401735786-212/raft_consensus-itest-cluster/master-0/data/instance: No such file or directory (error 2) I0312 20:11:47.322031 3693 server_base.cc:435] Attempting to create new FS layout instead I0312 20:11:47.329310 3693 fs_manager.cc:596] Generated new instance metadata in path /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1520885401735786-212/raft_consensus-itest-cluster/master-0/data/instance: uuid: "017653fcf9844579bc3695e824a2daa2" format_stamp: "Formatted at 2018-03-12 20:11:47 on dist-test-slave-dist-test-slave-f27p" I0312 20:11:47.329984 3693 fs_manager.cc:596] Generated new instance metadata in path /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1520885401735786-212/raft_consensus-itest-cluster/master-0/wal/instance: uuid: "017653fcf9844579bc3695e824a2daa2" format_stamp: "Formatted at 2018-03-12 20:11:47 on dist-test-slave-dist-test-slave-f27p" I0312 20:11:47.338233 3693 fs_manager.cc:496] Time spent creating directory manager: real 0.008s user 0.004s sys 0.004s I0312 20:11:47.339344 3693 env_posix.cc:1620] Not raising this process' open files per process limit of 65536; it is already as high as it can go I0312 20:11:47.339596 3693 file_cache.cc:470] Constructed file cache lbm with capacity 26214 I0312 20:11:47.346017 3693 fs_manager.cc:418] Time spent opening block manager: real 0.002s user 0.000s sys 0.004s I0312 20:11:47.346134 3693 fs_manager.cc:429] Opened local filesystem: /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1520885401735786-212/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1520885401735786-212/raft_consensus-itest-cluster/master-0/wal uuid: "017653fcf9844579bc3695e824a2daa2" format_stamp: "Formatted at 2018-03-12 20:11:47 on dist-test-slave-dist-test-slave-f27p" I0312 20:11:47.346320 3693 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1520885401735786-212/raft_consensus-itest-cluster/master-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0312 20:11:47.523330 3693 env_posix.cc:1620] Not raising this process' running threads per effective uid limit of 60284; it is already as high as it can go I0312 20:11:47.523931 3693 master_main.cc:76] Starting Master server... I0312 20:11:47.525147 3710 process_memory.cc:182] Process hard memory limit is 11.777631 GB I0312 20:11:47.525287 3710 process_memory.cc:184] Process soft memory limit is 9.422105 GB I0312 20:11:47.525385 3710 process_memory.cc:187] Process memory pressure threshold is 7.066578 GB I0312 20:11:47.563766 3693 rpc_server.cc:201] RPC server started. Bound to: 127.0.53.62:55229 I0312 20:11:47.589082 3693 webserver.cc:173] Starting webserver on 127.0.53.62:0 I0312 20:11:47.589192 3693 webserver.cc:184] Document root disabled I0312 20:11:47.590410 3693 webserver.cc:311] Webserver started. Bound to: http://127.0.53.62:60659/ I0312 20:11:47.591133 3693 server_base.cc:607] Dumped server information to /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1520885401735786-212/raft_consensus-itest-cluster/master-0/data/info.pb I0312 20:11:47.593818 3754 data_dirs.cc:931] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0312 20:11:47.597465 212 external_mini_cluster.cc:867] Started /tmp/dist-test-task3bwPeE/build/tsan/bin/kudu-master as pid 3693 I0312 20:11:47.598517 212 external_mini_cluster.cc:805] Running /tmp/dist-test-task3bwPeE/build/tsan/bin/kudu-tserver /tmp/dist-test-task3bwPeE/build/tsan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1520885401735786-212/raft_consensus-itest-cluster/ts-0/wal --fs_data_dirs=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1520885401735786-212/raft_consensus-itest-cluster/ts-0/data --block_manager=log --rpc_bind_addresses=127.0.53.1:0 --local_ip_for_outbound_sockets=127.0.53.1 --webserver_interface=127.0.53.1 --webserver_port=0 --tserver_master_addrs=127.0.53.62:55229 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1520885401735786-212/raft_consensus-itest-cluster/ts-0/logs --server_dump_info_path=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1520885401735786-212/raft_consensus-itest-cluster/ts-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --log_cache_size_limit_mb=10 --consensus_rpc_timeout_ms=50 with env {} I0312 20:11:47.607605 3754 tablet_bootstrap.cc:436] T 00000000000000000000000000000000 P 017653fcf9844579bc3695e824a2daa2: Bootstrap starting. I0312 20:11:47.610007 3754 tablet_bootstrap.cc:581] T 00000000000000000000000000000000 P 017653fcf9844579bc3695e824a2daa2: No blocks or log segments found. Creating new log. I0312 20:11:47.611461 3754 log.cc:520] T 00000000000000000000000000000000 P 017653fcf9844579bc3695e824a2daa2: Log is configured to *not* fsync() on all Append() calls I0312 20:11:47.613907 3754 tablet_bootstrap.cc:436] T 00000000000000000000000000000000 P 017653fcf9844579bc3695e824a2daa2: No bootstrap required, opened a new log I0312 20:11:47.625763 3754 raft_consensus.cc:304] T 00000000000000000000000000000000 P 017653fcf9844579bc3695e824a2daa2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "017653fcf9844579bc3695e824a2daa2" member_type: VOTER } I0312 20:11:47.626087 3754 raft_consensus.cc:330] T 00000000000000000000000000000000 P 017653fcf9844579bc3695e824a2daa2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0312 20:11:47.626231 3754 raft_consensus.cc:605] T 00000000000000000000000000000000 P 017653fcf9844579bc3695e824a2daa2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 017653fcf9844579bc3695e824a2daa2, State: Initialized, Role: FOLLOWER I0312 20:11:47.626715 3754 consensus_queue.cc:219] T 00000000000000000000000000000000 P 017653fcf9844579bc3695e824a2daa2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "017653fcf9844579bc3695e824a2daa2" member_type: VOTER } I0312 20:11:47.627080 3754 raft_consensus.cc:344] T 00000000000000000000000000000000 P 017653fcf9844579bc3695e824a2daa2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0312 20:11:47.627239 3754 raft_consensus.cc:436] T 00000000000000000000000000000000 P 017653fcf9844579bc3695e824a2daa2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0312 20:11:47.627394 3754 raft_consensus.cc:2700] T 00000000000000000000000000000000 P 017653fcf9844579bc3695e824a2daa2: Snoozing failure detection for 2.097s (starting election) I0312 20:11:47.627549 3754 raft_consensus.cc:2752] T 00000000000000000000000000000000 P 017653fcf9844579bc3695e824a2daa2 [term 0 FOLLOWER]: Advancing to term 1 I0312 20:11:47.630041 3754 raft_consensus.cc:458] T 00000000000000000000000000000000 P 017653fcf9844579bc3695e824a2daa2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "017653fcf9844579bc3695e824a2daa2" member_type: VOTER } I0312 20:11:47.630467 3754 leader_election.cc:258] T 00000000000000000000000000000000 P 017653fcf9844579bc3695e824a2daa2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0312 20:11:47.632086 3757 raft_consensus.cc:2700] T 00000000000000000000000000000000 P 017653fcf9844579bc3695e824a2daa2: Snoozing failure detection for 2.000s (election complete) I0312 20:11:47.632334 3757 raft_consensus.cc:2507] T 00000000000000000000000000000000 P 017653fcf9844579bc3695e824a2daa2 [term 1 FOLLOWER]: Leader election won for term 1 I0312 20:11:47.633721 3757 raft_consensus.cc:572] T 00000000000000000000000000000000 P 017653fcf9844579bc3695e824a2daa2 [term 1 LEADER]: Becoming Leader. State: Replica: 017653fcf9844579bc3695e824a2daa2, State: Running, Role: LEADER I0312 20:11:47.634140 3757 consensus_queue.cc:196] T 00000000000000000000000000000000 P 017653fcf9844579bc3695e824a2daa2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "017653fcf9844579bc3695e824a2daa2" member_type: VOTER } I0312 20:11:47.642416 3754 sys_catalog.cc:335] T 00000000000000000000000000000000 P 017653fcf9844579bc3695e824a2daa2 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 1 leader_uuid: "017653fcf9844579bc3695e824a2daa2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "017653fcf9844579bc3695e824a2daa2" member_type: VOTER } } I0312 20:11:47.642875 3754 sys_catalog.cc:338] T 00000000000000000000000000000000 P 017653fcf9844579bc3695e824a2daa2 [sys.catalog]: This master's current role is: LEADER I0312 20:11:47.643039 3758 sys_catalog.cc:335] T 00000000000000000000000000000000 P 017653fcf9844579bc3695e824a2daa2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 017653fcf9844579bc3695e824a2daa2. Latest consensus state: current_term: 1 leader_uuid: "017653fcf9844579bc3695e824a2daa2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "017653fcf9844579bc3695e824a2daa2" member_type: VOTER } } I0312 20:11:47.643370 3758 sys_catalog.cc:338] T 00000000000000000000000000000000 P 017653fcf9844579bc3695e824a2daa2 [sys.catalog]: This master's current role is: LEADER I0312 20:11:47.645563 3759 sys_catalog.cc:335] T 00000000000000000000000000000000 P 017653fcf9844579bc3695e824a2daa2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "017653fcf9844579bc3695e824a2daa2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "017653fcf9844579bc3695e824a2daa2" member_type: VOTER } } I0312 20:11:47.645851 3759 sys_catalog.cc:338] T 00000000000000000000000000000000 P 017653fcf9844579bc3695e824a2daa2 [sys.catalog]: This master's current role is: LEADER I0312 20:11:47.654532 3754 sys_catalog.cc:422] T 00000000000000000000000000000000 P 017653fcf9844579bc3695e824a2daa2 [sys.catalog]: configured and running, proceeding with master startup. I0312 20:11:47.654717 3761 catalog_manager.cc:972] Loading table and tablet metadata into memory... I0312 20:11:47.656342 3693 master_main.cc:79] Master server successfully started. I0312 20:11:47.660143 3761 catalog_manager.cc:983] Initializing Kudu internal certificate authority... I0312 20:11:47.711020 3761 catalog_manager.cc:870] Generated new certificate authority record I0312 20:11:47.713346 3761 catalog_manager.cc:992] Loading token signing keys... WARNING: Logging before InitGoogleLogging() is written to STDERR W0312 20:11:47.727759 3755 flags.cc:412] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0312 20:11:47.728204 3755 flags.cc:412] Enabled unsafe flag: --never_fsync=true I0312 20:11:47.729463 3761 catalog_manager.cc:3868] T 00000000000000000000000000000000 P 017653fcf9844579bc3695e824a2daa2: Generated new TSK 0 W0312 20:11:47.744010 3755 flags.cc:412] Enabled experimental flag: --ipki_server_key_size=1024 W0312 20:11:47.744303 3755 flags.cc:412] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.53.1 I0312 20:11:47.763583 3755 tablet_server_main.cc:72] Tablet server non-default flags: --consensus_rpc_timeout_ms=50 --log_cache_size_limit_mb=10 --fs_data_dirs=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1520885401735786-212/raft_consensus-itest-cluster/ts-0/data --fs_wal_dir=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1520885401735786-212/raft_consensus-itest-cluster/ts-0/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.0.53.1:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1520885401735786-212/raft_consensus-itest-cluster/ts-0/data/info.pb --webserver_interface=127.0.53.1 --webserver_port=0 --tserver_master_addrs=127.0.53.62:55229 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.0.53.1 --log_dir=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1520885401735786-212/raft_consensus-itest-cluster/ts-0/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.7.0-SNAPSHOT revision 13e10c63c46de30ac85f34b543cdd6fb44881f61 build type FASTDEBUG built by None at 12 Mar 2018 20:06:40 UTC on jenkins-slave-vn0g build id 12388 TSAN enabled I0312 20:11:47.773735 3755 tablet_server_main.cc:79] Initializing tablet server... I0312 20:11:47.775466 3755 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 1120776us I0312 20:11:47.775882 3755 fs_manager.cc:260] Metadata directory not provided I0312 20:11:47.776001 3755 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0312 20:11:47.776207 3755 server_base.cc:434] Could not load existing FS layout: Not found: /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1520885401735786-212/raft_consensus-itest-cluster/ts-0/data/instance: No such file or directory (error 2) I0312 20:11:47.776320 3755 server_base.cc:435] Attempting to create new FS layout instead I0312 20:11:47.783473 3755 fs_manager.cc:596] Generated new instance metadata in path /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1520885401735786-212/raft_consensus-itest-cluster/ts-0/data/instance: uuid: "98c90dc2c0df4d4e8d8940c8d597e365" format_stamp: "Formatted at 2018-03-12 20:11:47 on dist-test-slave-dist-test-slave-f27p" I0312 20:11:47.784165 3755 fs_manager.cc:596] Generated new instance metadata in path /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1520885401735786-212/raft_consensus-itest-cluster/ts-0/wal/instance: uuid: "98c90dc2c0df4d4e8d8940c8d597e365" format_stamp: "Formatted at 2018-03-12 20:11:47 on dist-test-slave-dist-test-slave-f27p" I0312 20:11:47.792603 3755 fs_manager.cc:496] Time spent creating directory manager: real 0.008s user 0.004s sys 0.004s I0312 20:11:47.793730 3755 env_posix.cc:1620] Not raising this process' open files per process limit of 65536; it is already as high as it can go I0312 20:11:47.793993 3755 file_cache.cc:470] Constructed file cache lbm with capacity 26214 I0312 20:11:47.800426 3755 fs_manager.cc:418] Time spent opening block manager: real 0.002s user 0.004s sys 0.000s I0312 20:11:47.800554 3755 fs_manager.cc:429] Opened local filesystem: /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1520885401735786-212/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1520885401735786-212/raft_consensus-itest-cluster/ts-0/wal uuid: "98c90dc2c0df4d4e8d8940c8d597e365" format_stamp: "Formatted at 2018-03-12 20:11:47 on dist-test-slave-dist-test-slave-f27p" I0312 20:11:47.800747 3755 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1520885401735786-212/raft_consensus-itest-cluster/ts-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0312 20:11:48.004690 3755 env_posix.cc:1620] Not raising this process' running threads per effective uid limit of 60284; it is already as high as it can go I0312 20:11:48.006018 3755 ts_tablet_manager.cc:349] Loaded tablet metadata (0 live tablets) I0312 20:11:48.007169 3755 tablet_server_main.cc:84] Starting tablet server... I0312 20:11:48.112947 3755 rpc_server.cc:201] RPC server started. Bound to: 127.0.53.1:46125 I0312 20:11:48.114032 3755 webserver.cc:173] Starting webserver on 127.0.53.1:0 I0312 20:11:48.114146 3755 webserver.cc:184] Document root disabled I0312 20:11:48.115597 3755 webserver.cc:311] Webserver started. Bound to: http://127.0.53.1:41924/ I0312 20:11:48.116318 3755 server_base.cc:607] Dumped server information to /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1520885401735786-212/raft_consensus-itest-cluster/ts-0/data/info.pb I0312 20:11:48.119138 3755 tablet_server_main.cc:87] Tablet server successfully started. I0312 20:11:48.123657 212 external_mini_cluster.cc:867] Started /tmp/dist-test-task3bwPeE/build/tsan/bin/kudu-tserver as pid 3755 I0312 20:11:48.121587 3887 process_memory.cc:182] Process hard memory limit is 11.777631 GB I0312 20:11:48.144701 3887 process_memory.cc:184] Process soft memory limit is 9.422105 GB I0312 20:11:48.144866 3887 process_memory.cc:187] Process memory pressure threshold is 7.066578 GB I0312 20:11:48.164741 3886 heartbeater.cc:346] Connected to a master server at 127.0.53.62:55229 I0312 20:11:48.165052 3886 heartbeater.cc:426] Registering TS with master... I0312 20:11:48.165647 3886 heartbeater.cc:475] Master 127.0.53.62:55229 requested a full tablet report, sending... I0312 20:11:48.167372 3721 ts_manager.cc:81] Registered new tserver with Master: 98c90dc2c0df4d4e8d8940c8d597e365 (127.0.53.1:46125) I0312 20:11:48.169484 212 external_mini_cluster.cc:498] 1 TS(s) registered with all masters I0312 20:11:48.169817 3721 master_service.cc:243] Signed X509 certificate for tserver {username='slave'} at 127.0.53.1:55583 I0312 20:11:48.189738 3721 catalog_manager.cc:1294] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47470: name: "TestTable" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 1 split_rows_range_bounds { } partition_schema { range_schema { columns { name: "key" } } } I0312 20:11:48.222535 3823 tablet_service.cc:755] Processing CreateTablet for tablet 913291d869f740589ed56c11c463767b (table=TestTable [id=a77b448f95344e13a70124fc659de488]), partition=RANGE (key) PARTITION UNBOUNDED I0312 20:11:48.223448 3823 data_dirs.cc:931] Could only allocate 1 dirs of requested 3 for tablet 913291d869f740589ed56c11c463767b. 1 dirs total, 0 dirs full, 0 dirs failed I0312 20:11:48.239895 3823 ts_tablet_manager.cc:1163] T 913291d869f740589ed56c11c463767b P 98c90dc2c0df4d4e8d8940c8d597e365: Registered tablet (data state: TABLET_DATA_READY) I0312 20:11:48.241647 3899 ts_tablet_manager.cc:1018] T 913291d869f740589ed56c11c463767b P 98c90dc2c0df4d4e8d8940c8d597e365: Bootstrapping tablet I0312 20:11:48.242770 3899 tablet_bootstrap.cc:436] T 913291d869f740589ed56c11c463767b P 98c90dc2c0df4d4e8d8940c8d597e365: Bootstrap starting. I0312 20:11:48.245033 3899 tablet_bootstrap.cc:581] T 913291d869f740589ed56c11c463767b P 98c90dc2c0df4d4e8d8940c8d597e365: No blocks or log segments found. Creating new log. I0312 20:11:48.246258 3899 log.cc:520] T 913291d869f740589ed56c11c463767b P 98c90dc2c0df4d4e8d8940c8d597e365: Log is configured to *not* fsync() on all Append() calls I0312 20:11:48.248658 3899 tablet_bootstrap.cc:436] T 913291d869f740589ed56c11c463767b P 98c90dc2c0df4d4e8d8940c8d597e365: No bootstrap required, opened a new log I0312 20:11:48.248948 3899 ts_tablet_manager.cc:1035] T 913291d869f740589ed56c11c463767b P 98c90dc2c0df4d4e8d8940c8d597e365: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s I0312 20:11:48.259279 3899 raft_consensus.cc:304] T 913291d869f740589ed56c11c463767b P 98c90dc2c0df4d4e8d8940c8d597e365 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98c90dc2c0df4d4e8d8940c8d597e365" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 46125 } } I0312 20:11:48.259716 3899 raft_consensus.cc:330] T 913291d869f740589ed56c11c463767b P 98c90dc2c0df4d4e8d8940c8d597e365 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0312 20:11:48.259876 3899 raft_consensus.cc:605] T 913291d869f740589ed56c11c463767b P 98c90dc2c0df4d4e8d8940c8d597e365 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98c90dc2c0df4d4e8d8940c8d597e365, State: Initialized, Role: FOLLOWER I0312 20:11:48.260332 3899 consensus_queue.cc:219] T 913291d869f740589ed56c11c463767b P 98c90dc2c0df4d4e8d8940c8d597e365 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98c90dc2c0df4d4e8d8940c8d597e365" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 46125 } } I0312 20:11:48.260747 3899 raft_consensus.cc:344] T 913291d869f740589ed56c11c463767b P 98c90dc2c0df4d4e8d8940c8d597e365 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0312 20:11:48.260915 3899 raft_consensus.cc:436] T 913291d869f740589ed56c11c463767b P 98c90dc2c0df4d4e8d8940c8d597e365 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0312 20:11:48.261137 3899 raft_consensus.cc:2700] T 913291d869f740589ed56c11c463767b P 98c90dc2c0df4d4e8d8940c8d597e365: Snoozing failure detection for 1.720s (starting election) I0312 20:11:48.261363 3899 raft_consensus.cc:2752] T 913291d869f740589ed56c11c463767b P 98c90dc2c0df4d4e8d8940c8d597e365 [term 0 FOLLOWER]: Advancing to term 1 I0312 20:11:48.264302 3899 raft_consensus.cc:458] T 913291d869f740589ed56c11c463767b P 98c90dc2c0df4d4e8d8940c8d597e365 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98c90dc2c0df4d4e8d8940c8d597e365" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 46125 } } I0312 20:11:48.264751 3899 leader_election.cc:258] T 913291d869f740589ed56c11c463767b P 98c90dc2c0df4d4e8d8940c8d597e365 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0312 20:11:48.266398 3901 raft_consensus.cc:2700] T 913291d869f740589ed56c11c463767b P 98c90dc2c0df4d4e8d8940c8d597e365: Snoozing failure detection for 1.851s (election complete) I0312 20:11:48.266675 3901 raft_consensus.cc:2507] T 913291d869f740589ed56c11c463767b P 98c90dc2c0df4d4e8d8940c8d597e365 [term 1 FOLLOWER]: Leader election won for term 1 I0312 20:11:48.268100 3899 ts_tablet_manager.cc:1063] T 913291d869f740589ed56c11c463767b P 98c90dc2c0df4d4e8d8940c8d597e365: Time spent starting tablet: real 0.019s user 0.016s sys 0.004s I0312 20:11:48.268523 3886 heartbeater.cc:467] Master 127.0.53.62:55229 was elected leader, sending a full tablet report... I0312 20:11:48.269949 3901 raft_consensus.cc:572] T 913291d869f740589ed56c11c463767b P 98c90dc2c0df4d4e8d8940c8d597e365 [term 1 LEADER]: Becoming Leader. State: Replica: 98c90dc2c0df4d4e8d8940c8d597e365, State: Running, Role: LEADER I0312 20:11:48.270352 3901 consensus_queue.cc:196] T 913291d869f740589ed56c11c463767b P 98c90dc2c0df4d4e8d8940c8d597e365 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98c90dc2c0df4d4e8d8940c8d597e365" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 46125 } } I0312 20:11:48.275440 3721 catalog_manager.cc:3592] T 913291d869f740589ed56c11c463767b P 98c90dc2c0df4d4e8d8940c8d597e365 reported cstate change: term changed from 0 to 1, leader changed from to 98c90dc2c0df4d4e8d8940c8d597e365 (127.0.53.1). New cstate: current_term: 1 leader_uuid: "98c90dc2c0df4d4e8d8940c8d597e365" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98c90dc2c0df4d4e8d8940c8d597e365" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 46125 } health_report { overall_health: HEALTHY } } } I0312 20:11:48.295833 212 external_mini_cluster.cc:498] 1 TS(s) registered with all masters I0312 20:11:48.297794 212 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 98c90dc2c0df4d4e8d8940c8d597e365 to finish bootstrapping I0312 20:11:48.318598 212 external_mini_cluster.cc:997] Killing /tmp/dist-test-task3bwPeE/build/tsan/bin/kudu-tserver with pid 3755 W0312 20:11:48.339051 3702 connection.cc:513] client connection to 127.0.53.1:46125 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0312 20:11:48.339974 3706 connection.cc:513] server connection from 127.0.53.1:55583 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0312 20:11:48.340636 3692 connection.cc:513] client connection to 127.0.53.1:46125 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0312 20:11:48.341512 3688 connection.cc:513] client connection to 127.0.53.1:46125 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) I0312 20:11:48.342473 212 external_mini_cluster.cc:997] Killing /tmp/dist-test-task3bwPeE/build/tsan/bin/kudu-master with pid 3693 W0312 20:11:48.364799 3692 connection.cc:513] client connection to 127.0.53.62:55229 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) [ OK ] RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica (1227 ms) [ RUN ] RaftConsensusElectionITest.LeaderStepDown I0312 20:11:48.376499 212 test_util.cc:192] Using random seed: -1065842793 I0312 20:11:48.380126 212 ts_itest-base.cc:113] Starting cluster with: I0312 20:11:48.380260 212 ts_itest-base.cc:114] -------------- I0312 20:11:48.380375 212 ts_itest-base.cc:115] 3 tablet servers I0312 20:11:48.380486 212 ts_itest-base.cc:116] 3 replicas per TS I0312 20:11:48.380595 212 ts_itest-base.cc:117] -------------- I0312 20:11:48.382750 212 external_mini_cluster.cc:805] Running /tmp/dist-test-task3bwPeE/build/tsan/bin/kudu-master /tmp/dist-test-task3bwPeE/build/tsan/bin/kudu-master --fs_wal_dir=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.LeaderStepDown.1520885401735786-212/raft_consensus-itest-cluster/master-0/wal --fs_data_dirs=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.LeaderStepDown.1520885401735786-212/raft_consensus-itest-cluster/master-0/data --block_manager=log --webserver_interface=localhost --ipki_ca_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.0.53.62:0 --webserver_interface=127.0.53.62 --webserver_port=0 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.LeaderStepDown.1520885401735786-212/raft_consensus-itest-cluster/master-0/logs --server_dump_info_path=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.LeaderStepDown.1520885401735786-212/raft_consensus-itest-cluster/master-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --catalog_manager_wait_for_new_tablets_to_elect_leader=false with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W0312 20:11:48.501312 3911 flags.cc:412] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0312 20:11:48.501613 3911 flags.cc:412] Enabled unsafe flag: --never_fsync=true W0312 20:11:48.517112 3911 flags.cc:412] Enabled experimental flag: --ipki_ca_key_size=1024 W0312 20:11:48.517236 3911 flags.cc:412] Enabled experimental flag: --ipki_server_key_size=1024 W0312 20:11:48.517333 3911 flags.cc:412] Enabled experimental flag: --tsk_num_rsa_bits=512 I0312 20:11:48.537761 3911 master_main.cc:66] Master server non-default flags: --fs_data_dirs=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.LeaderStepDown.1520885401735786-212/raft_consensus-itest-cluster/master-0/data --fs_wal_dir=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.LeaderStepDown.1520885401735786-212/raft_consensus-itest-cluster/master-0/wal --catalog_manager_wait_for_new_tablets_to_elect_leader=false --ipki_ca_key_size=1024 --ipki_server_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.0.53.62:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.LeaderStepDown.1520885401735786-212/raft_consensus-itest-cluster/master-0/data/info.pb --webserver_interface=127.0.53.62 --webserver_port=0 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --log_dir=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.LeaderStepDown.1520885401735786-212/raft_consensus-itest-cluster/master-0/logs --logbuflevel=-1 --logtostderr=true Master server version: kudu 1.7.0-SNAPSHOT revision 13e10c63c46de30ac85f34b543cdd6fb44881f61 build type FASTDEBUG built by None at 12 Mar 2018 20:06:40 UTC on jenkins-slave-vn0g build id 12388 TSAN enabled I0312 20:11:48.546682 3911 master_main.cc:73] Initializing master server... I0312 20:11:48.548166 3911 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 1121276us I0312 20:11:48.548919 3911 fs_manager.cc:260] Metadata directory not provided I0312 20:11:48.549083 3911 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0312 20:11:48.549291 3911 server_base.cc:434] Could not load existing FS layout: Not found: /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.LeaderStepDown.1520885401735786-212/raft_consensus-itest-cluster/master-0/data/instance: No such file or directory (error 2) I0312 20:11:48.549403 3911 server_base.cc:435] Attempting to create new FS layout instead I0312 20:11:48.556571 3911 fs_manager.cc:596] Generated new instance metadata in path /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.LeaderStepDown.1520885401735786-212/raft_consensus-itest-cluster/master-0/data/instance: uuid: "c3cf2cdb0d9147e8b9e4c2117909808e" format_stamp: "Formatted at 2018-03-12 20:11:48 on dist-test-slave-dist-test-slave-f27p" I0312 20:11:48.557271 3911 fs_manager.cc:596] Generated new instance metadata in path /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.LeaderStepDown.1520885401735786-212/raft_consensus-itest-cluster/master-0/wal/instance: uuid: "c3cf2cdb0d9147e8b9e4c2117909808e" format_stamp: "Formatted at 2018-03-12 20:11:48 on dist-test-slave-dist-test-slave-f27p" I0312 20:11:48.565538 3911 fs_manager.cc:496] Time spent creating directory manager: real 0.008s user 0.004s sys 0.004s I0312 20:11:48.566665 3911 env_posix.cc:1620] Not raising this process' open files per process limit of 65536; it is already as high as it can go I0312 20:11:48.566936 3911 file_cache.cc:470] Constructed file cache lbm with capacity 26214 I0312 20:11:48.573362 3911 fs_manager.cc:418] Time spent opening block manager: real 0.002s user 0.004s sys 0.000s I0312 20:11:48.573478 3911 fs_manager.cc:429] Opened local filesystem: /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.LeaderStepDown.1520885401735786-212/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.LeaderStepDown.1520885401735786-212/raft_consensus-itest-cluster/master-0/wal uuid: "c3cf2cdb0d9147e8b9e4c2117909808e" format_stamp: "Formatted at 2018-03-12 20:11:48 on dist-test-slave-dist-test-slave-f27p" I0312 20:11:48.573680 3911 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.LeaderStepDown.1520885401735786-212/raft_consensus-itest-cluster/master-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0312 20:11:48.800901 3911 env_posix.cc:1620] Not raising this process' running threads per effective uid limit of 60284; it is already as high as it can go I0312 20:11:48.801530 3911 master_main.cc:76] Starting Master server... I0312 20:11:48.802760 3928 process_memory.cc:182] Process hard memory limit is 11.777631 GB I0312 20:11:48.802899 3928 process_memory.cc:184] Process soft memory limit is 9.422105 GB I0312 20:11:48.802999 3928 process_memory.cc:187] Process memory pressure threshold is 7.066578 GB I0312 20:11:48.842502 3911 rpc_server.cc:201] RPC server started. Bound to: 127.0.53.62:54833 I0312 20:11:48.867821 3911 webserver.cc:173] Starting webserver on 127.0.53.62:0 I0312 20:11:48.867935 3911 webserver.cc:184] Document root disabled I0312 20:11:48.869208 3911 webserver.cc:311] Webserver started. Bound to: http://127.0.53.62:35047/ I0312 20:11:48.869972 3911 server_base.cc:607] Dumped server information to /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.LeaderStepDown.1520885401735786-212/raft_consensus-itest-cluster/master-0/data/info.pb I0312 20:11:48.872629 3972 data_dirs.cc:931] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0312 20:11:48.875927 212 external_mini_cluster.cc:867] Started /tmp/dist-test-task3bwPeE/build/tsan/bin/kudu-master as pid 3911 I0312 20:11:48.876927 212 external_mini_cluster.cc:805] Running /tmp/dist-test-task3bwPeE/build/tsan/bin/kudu-tserver /tmp/dist-test-task3bwPeE/build/tsan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.LeaderStepDown.1520885401735786-212/raft_consensus-itest-cluster/ts-0/wal --fs_data_dirs=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.LeaderStepDown.1520885401735786-212/raft_consensus-itest-cluster/ts-0/data --block_manager=log --rpc_bind_addresses=127.0.53.1:0 --local_ip_for_outbound_sockets=127.0.53.1 --webserver_interface=127.0.53.1 --webserver_port=0 --tserver_master_addrs=127.0.53.62:54833 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.LeaderStepDown.1520885401735786-212/raft_consensus-itest-cluster/ts-0/logs --server_dump_info_path=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.LeaderStepDown.1520885401735786-212/raft_consensus-itest-cluster/ts-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --enable_leader_failure_detection=false with env {} I0312 20:11:48.886567 3972 tablet_bootstrap.cc:436] T 00000000000000000000000000000000 P c3cf2cdb0d9147e8b9e4c2117909808e: Bootstrap starting. I0312 20:11:48.889034 3972 tablet_bootstrap.cc:581] T 00000000000000000000000000000000 P c3cf2cdb0d9147e8b9e4c2117909808e: No blocks or log segments found. Creating new log. I0312 20:11:48.890239 3972 log.cc:520] T 00000000000000000000000000000000 P c3cf2cdb0d9147e8b9e4c2117909808e: Log is configured to *not* fsync() on all Append() calls I0312 20:11:48.892626 3972 tablet_bootstrap.cc:436] T 00000000000000000000000000000000 P c3cf2cdb0d9147e8b9e4c2117909808e: No bootstrap required, opened a new log I0312 20:11:48.904585 3972 raft_consensus.cc:304] T 00000000000000000000000000000000 P c3cf2cdb0d9147e8b9e4c2117909808e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3cf2cdb0d9147e8b9e4c2117909808e" member_type: VOTER } I0312 20:11:48.904896 3972 raft_consensus.cc:330] T 00000000000000000000000000000000 P c3cf2cdb0d9147e8b9e4c2117909808e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0312 20:11:48.905022 3972 raft_consensus.cc:605] T 00000000000000000000000000000000 P c3cf2cdb0d9147e8b9e4c2117909808e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3cf2cdb0d9147e8b9e4c2117909808e, State: Initialized, Role: FOLLOWER I0312 20:11:48.905490 3972 consensus_queue.cc:219] T 00000000000000000000000000000000 P c3cf2cdb0d9147e8b9e4c2117909808e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3cf2cdb0d9147e8b9e4c2117909808e" member_type: VOTER } I0312 20:11:48.905843 3972 raft_consensus.cc:344] T 00000000000000000000000000000000 P c3cf2cdb0d9147e8b9e4c2117909808e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0312 20:11:48.905984 3972 raft_consensus.cc:436] T 00000000000000000000000000000000 P c3cf2cdb0d9147e8b9e4c2117909808e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0312 20:11:48.906122 3972 raft_consensus.cc:2700] T 00000000000000000000000000000000 P c3cf2cdb0d9147e8b9e4c2117909808e: Snoozing failure detection for 2.142s (starting election) I0312 20:11:48.906258 3972 raft_consensus.cc:2752] T 00000000000000000000000000000000 P c3cf2cdb0d9147e8b9e4c2117909808e [term 0 FOLLOWER]: Advancing to term 1 I0312 20:11:48.908706 3972 raft_consensus.cc:458] T 00000000000000000000000000000000 P c3cf2cdb0d9147e8b9e4c2117909808e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3cf2cdb0d9147e8b9e4c2117909808e" member_type: VOTER } I0312 20:11:48.909140 3972 leader_election.cc:258] T 00000000000000000000000000000000 P c3cf2cdb0d9147e8b9e4c2117909808e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0312 20:11:48.910725 3975 raft_consensus.cc:2700] T 00000000000000000000000000000000 P c3cf2cdb0d9147e8b9e4c2117909808e: Snoozing failure detection for 2.104s (election complete) I0312 20:11:48.911000 3975 raft_consensus.cc:2507] T 00000000000000000000000000000000 P c3cf2cdb0d9147e8b9e4c2117909808e [term 1 FOLLOWER]: Leader election won for term 1 I0312 20:11:48.921607 3975 raft_consensus.cc:572] T 00000000000000000000000000000000 P c3cf2cdb0d9147e8b9e4c2117909808e [term 1 LEADER]: Becoming Leader. State: Replica: c3cf2cdb0d9147e8b9e4c2117909808e, State: Running, Role: LEADER I0312 20:11:48.922013 3975 consensus_queue.cc:196] T 00000000000000000000000000000000 P c3cf2cdb0d9147e8b9e4c2117909808e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3cf2cdb0d9147e8b9e4c2117909808e" member_type: VOTER } I0312 20:11:48.925266 3972 sys_catalog.cc:335] T 00000000000000000000000000000000 P c3cf2cdb0d9147e8b9e4c2117909808e [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 1 leader_uuid: "c3cf2cdb0d9147e8b9e4c2117909808e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3cf2cdb0d9147e8b9e4c2117909808e" member_type: VOTER } } I0312 20:11:48.925649 3972 sys_catalog.cc:338] T 00000000000000000000000000000000 P c3cf2cdb0d9147e8b9e4c2117909808e [sys.catalog]: This master's current role is: LEADER I0312 20:11:48.927595 3972 sys_catalog.cc:422] T 00000000000000000000000000000000 P c3cf2cdb0d9147e8b9e4c2117909808e [sys.catalog]: configured and running, proceeding with master startup. I0312 20:11:48.928278 3976 sys_catalog.cc:335] T 00000000000000000000000000000000 P c3cf2cdb0d9147e8b9e4c2117909808e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c3cf2cdb0d9147e8b9e4c2117909808e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3cf2cdb0d9147e8b9e4c2117909808e" member_type: VOTER } } I0312 20:11:48.927857 3980 catalog_manager.cc:972] Loading table and tablet metadata into memory... I0312 20:11:48.930591 3976 sys_catalog.cc:338] T 00000000000000000000000000000000 P c3cf2cdb0d9147e8b9e4c2117909808e [sys.catalog]: This master's current role is: LEADER I0312 20:11:48.931538 3911 master_main.cc:79] Master server successfully started. I0312 20:11:48.931035 3977 sys_catalog.cc:335] T 00000000000000000000000000000000 P c3cf2cdb0d9147e8b9e4c2117909808e [sys.catalog]: SysCatalogTable state changed. Reason: New leader c3cf2cdb0d9147e8b9e4c2117909808e. Latest consensus state: current_term: 1 leader_uuid: "c3cf2cdb0d9147e8b9e4c2117909808e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3cf2cdb0d9147e8b9e4c2117909808e" member_type: VOTER } } I0312 20:11:48.931810 3977 sys_catalog.cc:338] T 00000000000000000000000000000000 P c3cf2cdb0d9147e8b9e4c2117909808e [sys.catalog]: This master's current role is: LEADER I0312 20:11:48.934931 3980 catalog_manager.cc:983] Initializing Kudu internal certificate authority... I0312 20:11:49.029706 3980 catalog_manager.cc:870] Generated new certificate authority record I0312 20:11:49.031193 3980 catalog_manager.cc:992] Loading token signing keys... I0312 20:11:49.039489 3980 catalog_manager.cc:3868] T 00000000000000000000000000000000 P c3cf2cdb0d9147e8b9e4c2117909808e: Generated new TSK 0 WARNING: Logging before InitGoogleLogging() is written to STDERR W0312 20:11:49.042026 3973 flags.cc:412] Enabled unsafe flag: --enable_leader_failure_detection=false W0312 20:11:49.042518 3973 flags.cc:412] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0312 20:11:49.042800 3973 flags.cc:412] Enabled unsafe flag: --never_fsync=true W0312 20:11:49.066833 3973 flags.cc:412] Enabled experimental flag: --ipki_server_key_size=1024 W0312 20:11:49.067325 3973 flags.cc:412] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.53.1 I0312 20:11:49.097858 3973 tablet_server_main.cc:72] Tablet server non-default flags: --enable_leader_failure_detection=false --fs_data_dirs=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.LeaderStepDown.1520885401735786-212/raft_consensus-itest-cluster/ts-0/data --fs_wal_dir=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.LeaderStepDown.1520885401735786-212/raft_consensus-itest-cluster/ts-0/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.0.53.1:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.LeaderStepDown.1520885401735786-212/raft_consensus-itest-cluster/ts-0/data/info.pb --webserver_interface=127.0.53.1 --webserver_port=0 --tserver_master_addrs=127.0.53.62:54833 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.0.53.1 --log_dir=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.LeaderStepDown.1520885401735786-212/raft_consensus-itest-cluster/ts-0/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.7.0-SNAPSHOT revision 13e10c63c46de30ac85f34b543cdd6fb44881f61 build type FASTDEBUG built by None at 12 Mar 2018 20:06:40 UTC on jenkins-slave-vn0g build id 12388 TSAN enabled I0312 20:11:49.113296 3973 tablet_server_main.cc:79] Initializing tablet server... I0312 20:11:49.115661 3973 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 1121776us I0312 20:11:49.116276 3973 fs_manager.cc:260] Metadata directory not provided I0312 20:11:49.116441 3973 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0312 20:11:49.116708 3973 server_base.cc:434] Could not load existing FS layout: Not found: /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.LeaderStepDown.1520885401735786-212/raft_consensus-itest-cluster/ts-0/data/instance: No such file or directory (error 2) I0312 20:11:49.116864 3973 server_base.cc:435] Attempting to create new FS layout instead I0312 20:11:49.129416 3973 fs_manager.cc:596] Generated new instance metadata in path /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.LeaderStepDown.1520885401735786-212/raft_consensus-itest-cluster/ts-0/data/instance: uuid: "e404af0289d5411ca5649c9d0d381012" format_stamp: "Formatted at 2018-03-12 20:11:49 on dist-test-slave-dist-test-slave-f27p" I0312 20:11:49.130429 3973 fs_manager.cc:596] Generated new instance metadata in path /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.LeaderStepDown.1520885401735786-212/raft_consensus-itest-cluster/ts-0/wal/instance: uuid: "e404af0289d5411ca5649c9d0d381012" format_stamp: "Formatted at 2018-03-12 20:11:49 on dist-test-slave-dist-test-slave-f27p" I0312 20:11:49.143496 3973 fs_manager.cc:496] Time spent creating directory manager: real 0.013s user 0.012s sys 0.004s I0312 20:11:49.145268 3973 env_posix.cc:1620] Not raising this process' open files per process limit of 65536; it is already as high as it can go I0312 20:11:49.145645 3973 file_cache.cc:470] Constructed file cache lbm with capacity 26214 I0312 20:11:49.156380 3973 fs_manager.cc:418] Time spent opening block manager: real 0.002s user 0.000s sys 0.000s I0312 20:11:49.156548 3973 fs_manager.cc:429] Opened local filesystem: /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.LeaderStepDown.1520885401735786-212/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.LeaderStepDown.1520885401735786-212/raft_consensus-itest-cluster/ts-0/wal uuid: "e404af0289d5411ca5649c9d0d381012" format_stamp: "Formatted at 2018-03-12 20:11:49 on dist-test-slave-dist-test-slave-f27p" I0312 20:11:49.156839 3973 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.LeaderStepDown.1520885401735786-212/raft_consensus-itest-cluster/ts-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0312 20:11:49.369992 3973 env_posix.cc:1620] Not raising this process' running threads per effective uid limit of 60284; it is already as high as it can go I0312 20:11:49.371276 3973 ts_tablet_manager.cc:349] Loaded tablet metadata (0 live tablets) I0312 20:11:49.381389 3973 tablet_server_main.cc:84] Starting tablet server... I0312 20:11:50.071732 3973 rpc_server.cc:201] RPC server started. Bound to: 127.0.53.1:33421 I0312 20:11:50.072636 3973 webserver.cc:173] Starting webserver on 127.0.53.1:0 I0312 20:11:50.072748 3973 webserver.cc:184] Document root disabled I0312 20:11:50.074193 3973 webserver.cc:311] Webserver started. Bound to: http://127.0.53.1:57228/ I0312 20:11:50.074992 3973 server_base.cc:607] Dumped server information to /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.LeaderStepDown.1520885401735786-212/raft_consensus-itest-cluster/ts-0/data/info.pb I0312 20:11:50.079561 212 external_mini_cluster.cc:867] Started /tmp/dist-test-task3bwPeE/build/tsan/bin/kudu-tserver as pid 3973 I0312 20:11:50.080775 212 external_mini_cluster.cc:805] Running /tmp/dist-test-task3bwPeE/build/tsan/bin/kudu-tserver /tmp/dist-test-task3bwPeE/build/tsan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.LeaderStepDown.1520885401735786-212/raft_consensus-itest-cluster/ts-1/wal --fs_data_dirs=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.LeaderStepDown.1520885401735786-212/raft_consensus-itest-cluster/ts-1/data --block_manager=log --rpc_bind_addresses=127.0.53.2:0 --local_ip_for_outbound_sockets=127.0.53.2 --webserver_interface=127.0.53.2 --webserver_port=0 --tserver_master_addrs=127.0.53.62:54833 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.LeaderStepDown.1520885401735786-212/raft_consensus-itest-cluster/ts-1/logs --server_dump_info_path=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.LeaderStepDown.1520885401735786-212/raft_consensus-itest-cluster/ts-1/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --enable_leader_failure_detection=false with env {} I0312 20:11:50.088933 3973 tablet_server_main.cc:87] Tablet server successfully started. I0312 20:11:50.110618 4104 heartbeater.cc:346] Connected to a master server at 127.0.53.62:54833 I0312 20:11:50.110913 4104 heartbeater.cc:426] Registering TS with master... I0312 20:11:50.111469 4104 heartbeater.cc:475] Master 127.0.53.62:54833 requested a full tablet report, sending... I0312 20:11:50.113539 3939 ts_manager.cc:81] Registered new tserver with Master: e404af0289d5411ca5649c9d0d381012 (127.0.53.1:33421) I0312 20:11:50.089251 4105 process_memory.cc:182] Process hard memory limit is 11.777631 GB I0312 20:11:50.115813 4105 process_memory.cc:184] Process soft memory limit is 9.422105 GB I0312 20:11:50.115921 4105 process_memory.cc:187] Process memory pressure threshold is 7.066578 GB I0312 20:11:50.116195 3939 master_service.cc:243] Signed X509 certificate for tserver {username='slave'} at 127.0.53.1:46449 WARNING: Logging before InitGoogleLogging() is written to STDERR W0312 20:11:50.217430 4108 flags.cc:412] Enabled unsafe flag: --enable_leader_failure_detection=false W0312 20:11:50.217777 4108 flags.cc:412] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0312 20:11:50.217967 4108 flags.cc:412] Enabled unsafe flag: --never_fsync=true W0312 20:11:50.232216 4108 flags.cc:412] Enabled experimental flag: --ipki_server_key_size=1024 W0312 20:11:50.232539 4108 flags.cc:412] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.53.2 I0312 20:11:50.251130 4108 tablet_server_main.cc:72] Tablet server non-default flags: --enable_leader_failure_detection=false --fs_data_dirs=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.LeaderStepDown.1520885401735786-212/raft_consensus-itest-cluster/ts-1/data --fs_wal_dir=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.LeaderStepDown.1520885401735786-212/raft_consensus-itest-cluster/ts-1/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.0.53.2:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.LeaderStepDown.1520885401735786-212/raft_consensus-itest-cluster/ts-1/data/info.pb --webserver_interface=127.0.53.2 --webserver_port=0 --tserver_master_addrs=127.0.53.62:54833 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.0.53.2 --log_dir=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.LeaderStepDown.1520885401735786-212/raft_consensus-itest-cluster/ts-1/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.7.0-SNAPSHOT revision 13e10c63c46de30ac85f34b543cdd6fb44881f61 build type FASTDEBUG built by None at 12 Mar 2018 20:06:40 UTC on jenkins-slave-vn0g build id 12388 TSAN enabled I0312 20:11:50.260799 4108 tablet_server_main.cc:79] Initializing tablet server... I0312 20:11:50.262503 4108 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 1122276us I0312 20:11:50.262923 4108 fs_manager.cc:260] Metadata directory not provided I0312 20:11:50.263041 4108 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0312 20:11:50.263247 4108 server_base.cc:434] Could not load existing FS layout: Not found: /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.LeaderStepDown.1520885401735786-212/raft_consensus-itest-cluster/ts-1/data/instance: No such file or directory (error 2) I0312 20:11:50.263368 4108 server_base.cc:435] Attempting to create new FS layout instead I0312 20:11:50.270486 4108 fs_manager.cc:596] Generated new instance metadata in path /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.LeaderStepDown.1520885401735786-212/raft_consensus-itest-cluster/ts-1/data/instance: uuid: "0217dac1b3c64fcc87cc67801d606966" format_stamp: "Formatted at 2018-03-12 20:11:50 on dist-test-slave-dist-test-slave-f27p" I0312 20:11:50.271198 4108 fs_manager.cc:596] Generated new instance metadata in path /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.LeaderStepDown.1520885401735786-212/raft_consensus-itest-cluster/ts-1/wal/instance: uuid: "0217dac1b3c64fcc87cc67801d606966" format_stamp: "Formatted at 2018-03-12 20:11:50 on dist-test-slave-dist-test-slave-f27p" I0312 20:11:50.279517 4108 fs_manager.cc:496] Time spent creating directory manager: real 0.008s user 0.004s sys 0.004s I0312 20:11:50.280635 4108 env_posix.cc:1620] Not raising this process' open files per process limit of 65536; it is already as high as it can go I0312 20:11:50.280903 4108 file_cache.cc:470] Constructed file cache lbm with capacity 26214 I0312 20:11:50.287341 4108 fs_manager.cc:418] Time spent opening block manager: real 0.002s user 0.000s sys 0.000s I0312 20:11:50.287461 4108 fs_manager.cc:429] Opened local filesystem: /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.LeaderStepDown.1520885401735786-212/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.LeaderStepDown.1520885401735786-212/raft_consensus-itest-cluster/ts-1/wal uuid: "0217dac1b3c64fcc87cc67801d606966" format_stamp: "Formatted at 2018-03-12 20:11:50 on dist-test-slave-dist-test-slave-f27p" I0312 20:11:50.287658 4108 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.LeaderStepDown.1520885401735786-212/raft_consensus-itest-cluster/ts-1/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0312 20:11:50.518877 4108 env_posix.cc:1620] Not raising this process' running threads per effective uid limit of 60284; it is already as high as it can go I0312 20:11:50.520130 4108 ts_tablet_manager.cc:349] Loaded tablet metadata (0 live tablets) I0312 20:11:50.521206 4108 tablet_server_main.cc:84] Starting tablet server... I0312 20:11:50.608134 4108 rpc_server.cc:201] RPC server started. Bound to: 127.0.53.2:43406 I0312 20:11:50.609006 4108 webserver.cc:173] Starting webserver on 127.0.53.2:0 I0312 20:11:50.609133 4108 webserver.cc:184] Document root disabled I0312 20:11:50.610605 4108 webserver.cc:311] Webserver started. Bound to: http://127.0.53.2:33252/ I0312 20:11:50.611333 4108 server_base.cc:607] Dumped server information to /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.LeaderStepDown.1520885401735786-212/raft_consensus-itest-cluster/ts-1/data/info.pb I0312 20:11:50.612227 212 external_mini_cluster.cc:867] Started /tmp/dist-test-task3bwPeE/build/tsan/bin/kudu-tserver as pid 4108 I0312 20:11:50.613526 212 external_mini_cluster.cc:805] Running /tmp/dist-test-task3bwPeE/build/tsan/bin/kudu-tserver /tmp/dist-test-task3bwPeE/build/tsan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.LeaderStepDown.1520885401735786-212/raft_consensus-itest-cluster/ts-2/wal --fs_data_dirs=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.LeaderStepDown.1520885401735786-212/raft_consensus-itest-cluster/ts-2/data --block_manager=log --rpc_bind_addresses=127.0.53.3:0 --local_ip_for_outbound_sockets=127.0.53.3 --webserver_interface=127.0.53.3 --webserver_port=0 --tserver_master_addrs=127.0.53.62:54833 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.LeaderStepDown.1520885401735786-212/raft_consensus-itest-cluster/ts-2/logs --server_dump_info_path=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.LeaderStepDown.1520885401735786-212/raft_consensus-itest-cluster/ts-2/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --enable_leader_failure_detection=false with env {} I0312 20:11:50.614102 4108 tablet_server_main.cc:87] Tablet server successfully started. I0312 20:11:50.614159 4230 process_memory.cc:182] Process hard memory limit is 11.777631 GB I0312 20:11:50.642282 4230 process_memory.cc:184] Process soft memory limit is 9.422105 GB I0312 20:11:50.642416 4230 process_memory.cc:187] Process memory pressure threshold is 7.066578 GB I0312 20:11:50.658218 4229 heartbeater.cc:346] Connected to a master server at 127.0.53.62:54833 I0312 20:11:50.658448 4229 heartbeater.cc:426] Registering TS with master... I0312 20:11:50.658864 4229 heartbeater.cc:475] Master 127.0.53.62:54833 requested a full tablet report, sending... I0312 20:11:50.660260 3939 ts_manager.cc:81] Registered new tserver with Master: 0217dac1b3c64fcc87cc67801d606966 (127.0.53.2:43406) I0312 20:11:50.661455 3939 master_service.cc:243] Signed X509 certificate for tserver {username='slave'} at 127.0.53.2:53377 WARNING: Logging before InitGoogleLogging() is written to STDERR W0312 20:11:50.737440 4233 flags.cc:412] Enabled unsafe flag: --enable_leader_failure_detection=false W0312 20:11:50.737776 4233 flags.cc:412] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0312 20:11:50.737962 4233 flags.cc:412] Enabled unsafe flag: --never_fsync=true W0312 20:11:50.752106 4233 flags.cc:412] Enabled experimental flag: --ipki_server_key_size=1024 W0312 20:11:50.752409 4233 flags.cc:412] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.53.3 I0312 20:11:50.770871 4233 tablet_server_main.cc:72] Tablet server non-default flags: --enable_leader_failure_detection=false --fs_data_dirs=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.LeaderStepDown.1520885401735786-212/raft_consensus-itest-cluster/ts-2/data --fs_wal_dir=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.LeaderStepDown.1520885401735786-212/raft_consensus-itest-cluster/ts-2/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.0.53.3:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.LeaderStepDown.1520885401735786-212/raft_consensus-itest-cluster/ts-2/data/info.pb --webserver_interface=127.0.53.3 --webserver_port=0 --tserver_master_addrs=127.0.53.62:54833 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.0.53.3 --log_dir=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.LeaderStepDown.1520885401735786-212/raft_consensus-itest-cluster/ts-2/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.7.0-SNAPSHOT revision 13e10c63c46de30ac85f34b543cdd6fb44881f61 build type FASTDEBUG built by None at 12 Mar 2018 20:06:40 UTC on jenkins-slave-vn0g build id 12388 TSAN enabled I0312 20:11:50.780890 4233 tablet_server_main.cc:79] Initializing tablet server... I0312 20:11:50.782624 4233 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 1122276us I0312 20:11:50.783044 4233 fs_manager.cc:260] Metadata directory not provided I0312 20:11:50.783159 4233 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0312 20:11:50.783336 4233 server_base.cc:434] Could not load existing FS layout: Not found: /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.LeaderStepDown.1520885401735786-212/raft_consensus-itest-cluster/ts-2/data/instance: No such file or directory (error 2) I0312 20:11:50.783437 4233 server_base.cc:435] Attempting to create new FS layout instead I0312 20:11:50.790696 4233 fs_manager.cc:596] Generated new instance metadata in path /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.LeaderStepDown.1520885401735786-212/raft_consensus-itest-cluster/ts-2/data/instance: uuid: "3120fde077bd405e9d1160a540b7b910" format_stamp: "Formatted at 2018-03-12 20:11:50 on dist-test-slave-dist-test-slave-f27p" I0312 20:11:50.791375 4233 fs_manager.cc:596] Generated new instance metadata in path /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.LeaderStepDown.1520885401735786-212/raft_consensus-itest-cluster/ts-2/wal/instance: uuid: "3120fde077bd405e9d1160a540b7b910" format_stamp: "Formatted at 2018-03-12 20:11:50 on dist-test-slave-dist-test-slave-f27p" I0312 20:11:50.799369 4233 fs_manager.cc:496] Time spent creating directory manager: real 0.008s user 0.008s sys 0.000s I0312 20:11:50.800604 4233 env_posix.cc:1620] Not raising this process' open files per process limit of 65536; it is already as high as it can go I0312 20:11:50.800894 4233 file_cache.cc:470] Constructed file cache lbm with capacity 26214 I0312 20:11:50.807767 4233 fs_manager.cc:418] Time spent opening block manager: real 0.002s user 0.000s sys 0.000s I0312 20:11:50.807879 4233 fs_manager.cc:429] Opened local filesystem: /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.LeaderStepDown.1520885401735786-212/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.LeaderStepDown.1520885401735786-212/raft_consensus-itest-cluster/ts-2/wal uuid: "3120fde077bd405e9d1160a540b7b910" format_stamp: "Formatted at 2018-03-12 20:11:50 on dist-test-slave-dist-test-slave-f27p" I0312 20:11:50.808101 4233 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.LeaderStepDown.1520885401735786-212/raft_consensus-itest-cluster/ts-2/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0312 20:11:50.988257 4233 env_posix.cc:1620] Not raising this process' running threads per effective uid limit of 60284; it is already as high as it can go I0312 20:11:50.989531 4233 ts_tablet_manager.cc:349] Loaded tablet metadata (0 live tablets) I0312 20:11:50.990465 4233 tablet_server_main.cc:84] Starting tablet server... I0312 20:11:51.077569 4233 rpc_server.cc:201] RPC server started. Bound to: 127.0.53.3:38566 I0312 20:11:51.078483 4233 webserver.cc:173] Starting webserver on 127.0.53.3:0 I0312 20:11:51.078594 4233 webserver.cc:184] Document root disabled I0312 20:11:51.079795 4233 webserver.cc:311] Webserver started. Bound to: http://127.0.53.3:60450/ I0312 20:11:51.080519 4233 server_base.cc:607] Dumped server information to /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.LeaderStepDown.1520885401735786-212/raft_consensus-itest-cluster/ts-2/data/info.pb I0312 20:11:51.083083 4233 tablet_server_main.cc:87] Tablet server successfully started. I0312 20:11:51.084362 212 external_mini_cluster.cc:867] Started /tmp/dist-test-task3bwPeE/build/tsan/bin/kudu-tserver as pid 4233 I0312 20:11:51.083122 4355 process_memory.cc:182] Process hard memory limit is 11.777631 GB I0312 20:11:51.111064 4355 process_memory.cc:184] Process soft memory limit is 9.422105 GB I0312 20:11:51.111182 4355 process_memory.cc:187] Process memory pressure threshold is 7.066578 GB I0312 20:11:51.119529 4104 heartbeater.cc:467] Master 127.0.53.62:54833 was elected leader, sending a full tablet report... I0312 20:11:51.129343 4354 heartbeater.cc:346] Connected to a master server at 127.0.53.62:54833 I0312 20:11:51.129597 4354 heartbeater.cc:426] Registering TS with master... I0312 20:11:51.130013 4354 heartbeater.cc:475] Master 127.0.53.62:54833 requested a full tablet report, sending... I0312 20:11:51.131299 3938 ts_manager.cc:81] Registered new tserver with Master: 3120fde077bd405e9d1160a540b7b910 (127.0.53.3:38566) I0312 20:11:51.132293 212 external_mini_cluster.cc:498] 3 TS(s) registered with all masters I0312 20:11:51.132513 3938 master_service.cc:243] Signed X509 certificate for tserver {username='slave'} at 127.0.53.3:33354 I0312 20:11:51.149418 3938 catalog_manager.cc:1294] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51647: name: "TestTable" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 3 split_rows_range_bounds { } partition_schema { range_schema { columns { name: "key" } } } W0312 20:11:51.157904 3938 catalog_manager.cc:1450] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a replica failure: 4 tablet servers are needed, 3 are alive. Consider bringing up additional tablet server(s) or running both the masters and all tablet servers with --raft_prepare_replacement_before_eviction=false flag (not recommended). I0312 20:11:51.198876 4166 tablet_service.cc:755] Processing CreateTablet for tablet 320f1456e73d44938845f48206e64c26 (table=TestTable [id=f2a85f861a4e4e639473289d97659d5c]), partition=RANGE (key) PARTITION UNBOUNDED I0312 20:11:51.199795 4166 data_dirs.cc:931] Could only allocate 1 dirs of requested 3 for tablet 320f1456e73d44938845f48206e64c26. 1 dirs total, 0 dirs full, 0 dirs failed I0312 20:11:51.200716 4291 tablet_service.cc:755] Processing CreateTablet for tablet 320f1456e73d44938845f48206e64c26 (table=TestTable [id=f2a85f861a4e4e639473289d97659d5c]), partition=RANGE (key) PARTITION UNBOUNDED I0312 20:11:51.201563 4291 data_dirs.cc:931] Could only allocate 1 dirs of requested 3 for tablet 320f1456e73d44938845f48206e64c26. 1 dirs total, 0 dirs full, 0 dirs failed I0312 20:11:51.201946 4041 tablet_service.cc:755] Processing CreateTablet for tablet 320f1456e73d44938845f48206e64c26 (table=TestTable [id=f2a85f861a4e4e639473289d97659d5c]), partition=RANGE (key) PARTITION UNBOUNDED I0312 20:11:51.202607 4041 data_dirs.cc:931] Could only allocate 1 dirs of requested 3 for tablet 320f1456e73d44938845f48206e64c26. 1 dirs total, 0 dirs full, 0 dirs failed I0312 20:11:51.214326 4041 ts_tablet_manager.cc:1163] T 320f1456e73d44938845f48206e64c26 P e404af0289d5411ca5649c9d0d381012: Registered tablet (data state: TABLET_DATA_READY) I0312 20:11:51.215982 4374 ts_tablet_manager.cc:1018] T 320f1456e73d44938845f48206e64c26 P e404af0289d5411ca5649c9d0d381012: Bootstrapping tablet I0312 20:11:51.216456 4374 tablet_bootstrap.cc:436] T 320f1456e73d44938845f48206e64c26 P e404af0289d5411ca5649c9d0d381012: Bootstrap starting. I0312 20:11:51.216429 4166 ts_tablet_manager.cc:1163] T 320f1456e73d44938845f48206e64c26 P 0217dac1b3c64fcc87cc67801d606966: Registered tablet (data state: TABLET_DATA_READY) I0312 20:11:51.218922 4291 ts_tablet_manager.cc:1163] T 320f1456e73d44938845f48206e64c26 P 3120fde077bd405e9d1160a540b7b910: Registered tablet (data state: TABLET_DATA_READY) I0312 20:11:51.219565 4374 tablet_bootstrap.cc:581] T 320f1456e73d44938845f48206e64c26 P e404af0289d5411ca5649c9d0d381012: No blocks or log segments found. Creating new log. I0312 20:11:51.220793 4374 log.cc:520] T 320f1456e73d44938845f48206e64c26 P e404af0289d5411ca5649c9d0d381012: Log is configured to *not* fsync() on all Append() calls I0312 20:11:51.222784 4375 ts_tablet_manager.cc:1018] T 320f1456e73d44938845f48206e64c26 P 0217dac1b3c64fcc87cc67801d606966: Bootstrapping tablet I0312 20:11:51.223428 4374 tablet_bootstrap.cc:436] T 320f1456e73d44938845f48206e64c26 P e404af0289d5411ca5649c9d0d381012: No bootstrap required, opened a new log I0312 20:11:51.223738 4375 tablet_bootstrap.cc:436] T 320f1456e73d44938845f48206e64c26 P 0217dac1b3c64fcc87cc67801d606966: Bootstrap starting. I0312 20:11:51.223851 4374 ts_tablet_manager.cc:1035] T 320f1456e73d44938845f48206e64c26 P e404af0289d5411ca5649c9d0d381012: Time spent bootstrapping tablet: real 0.008s user 0.008s sys 0.000s I0312 20:11:51.226802 4375 tablet_bootstrap.cc:581] T 320f1456e73d44938845f48206e64c26 P 0217dac1b3c64fcc87cc67801d606966: No blocks or log segments found. Creating new log. I0312 20:11:51.227315 4376 ts_tablet_manager.cc:1018] T 320f1456e73d44938845f48206e64c26 P 3120fde077bd405e9d1160a540b7b910: Bootstrapping tablet I0312 20:11:51.227758 4376 tablet_bootstrap.cc:436] T 320f1456e73d44938845f48206e64c26 P 3120fde077bd405e9d1160a540b7b910: Bootstrap starting. I0312 20:11:51.231597 4376 tablet_bootstrap.cc:581] T 320f1456e73d44938845f48206e64c26 P 3120fde077bd405e9d1160a540b7b910: No blocks or log segments found. Creating new log. I0312 20:11:51.233175 4376 log.cc:520] T 320f1456e73d44938845f48206e64c26 P 3120fde077bd405e9d1160a540b7b910: Log is configured to *not* fsync() on all Append() calls I0312 20:11:51.235605 4375 log.cc:520] T 320f1456e73d44938845f48206e64c26 P 0217dac1b3c64fcc87cc67801d606966: Log is configured to *not* fsync() on all Append() calls I0312 20:11:51.237884 4376 tablet_bootstrap.cc:436] T 320f1456e73d44938845f48206e64c26 P 3120fde077bd405e9d1160a540b7b910: No bootstrap required, opened a new log I0312 20:11:51.238283 4376 ts_tablet_manager.cc:1035] T 320f1456e73d44938845f48206e64c26 P 3120fde077bd405e9d1160a540b7b910: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.000s I0312 20:11:51.239333 4374 raft_consensus.cc:304] T 320f1456e73d44938845f48206e64c26 P e404af0289d5411ca5649c9d0d381012 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0217dac1b3c64fcc87cc67801d606966" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 43406 } } peers { permanent_uuid: "e404af0289d5411ca5649c9d0d381012" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 33421 } } peers { permanent_uuid: "3120fde077bd405e9d1160a540b7b910" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 38566 } } I0312 20:11:51.240092 4374 raft_consensus.cc:605] T 320f1456e73d44938845f48206e64c26 P e404af0289d5411ca5649c9d0d381012 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e404af0289d5411ca5649c9d0d381012, State: Initialized, Role: FOLLOWER I0312 20:11:51.240494 4374 consensus_queue.cc:219] T 320f1456e73d44938845f48206e64c26 P e404af0289d5411ca5649c9d0d381012 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0217dac1b3c64fcc87cc67801d606966" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 43406 } } peers { permanent_uuid: "e404af0289d5411ca5649c9d0d381012" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 33421 } } peers { permanent_uuid: "3120fde077bd405e9d1160a540b7b910" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 38566 } } I0312 20:11:51.243537 4374 ts_tablet_manager.cc:1063] T 320f1456e73d44938845f48206e64c26 P e404af0289d5411ca5649c9d0d381012: Time spent starting tablet: real 0.020s user 0.012s sys 0.004s I0312 20:11:51.250368 4375 tablet_bootstrap.cc:436] T 320f1456e73d44938845f48206e64c26 P 0217dac1b3c64fcc87cc67801d606966: No bootstrap required, opened a new log I0312 20:11:51.250753 4375 ts_tablet_manager.cc:1035] T 320f1456e73d44938845f48206e64c26 P 0217dac1b3c64fcc87cc67801d606966: Time spent bootstrapping tablet: real 0.027s user 0.004s sys 0.000s I0312 20:11:51.250732 4376 raft_consensus.cc:304] T 320f1456e73d44938845f48206e64c26 P 3120fde077bd405e9d1160a540b7b910 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0217dac1b3c64fcc87cc67801d606966" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 43406 } } peers { permanent_uuid: "e404af0289d5411ca5649c9d0d381012" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 33421 } } peers { permanent_uuid: "3120fde077bd405e9d1160a540b7b910" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 38566 } } I0312 20:11:51.251488 4376 raft_consensus.cc:605] T 320f1456e73d44938845f48206e64c26 P 3120fde077bd405e9d1160a540b7b910 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3120fde077bd405e9d1160a540b7b910, State: Initialized, Role: FOLLOWER I0312 20:11:51.251895 4376 consensus_queue.cc:219] T 320f1456e73d44938845f48206e64c26 P 3120fde077bd405e9d1160a540b7b910 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0217dac1b3c64fcc87cc67801d606966" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 43406 } } peers { permanent_uuid: "e404af0289d5411ca5649c9d0d381012" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 33421 } } peers { permanent_uuid: "3120fde077bd405e9d1160a540b7b910" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 38566 } } I0312 20:11:51.254446 4354 heartbeater.cc:467] Master 127.0.53.62:54833 was elected leader, sending a full tablet report... I0312 20:11:51.255558 4376 ts_tablet_manager.cc:1063] T 320f1456e73d44938845f48206e64c26 P 3120fde077bd405e9d1160a540b7b910: Time spent starting tablet: real 0.017s user 0.012s sys 0.000s I0312 20:11:51.256836 212 external_mini_cluster.cc:498] 3 TS(s) registered with all masters I0312 20:11:51.259624 212 ts_itest-base.cc:233] Waiting for 1 tablets on tserver e404af0289d5411ca5649c9d0d381012 to finish bootstrapping I0312 20:11:51.264767 4375 raft_consensus.cc:304] T 320f1456e73d44938845f48206e64c26 P 0217dac1b3c64fcc87cc67801d606966 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0217dac1b3c64fcc87cc67801d606966" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 43406 } } peers { permanent_uuid: "e404af0289d5411ca5649c9d0d381012" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 33421 } } peers { permanent_uuid: "3120fde077bd405e9d1160a540b7b910" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 38566 } } I0312 20:11:51.265480 4375 raft_consensus.cc:605] T 320f1456e73d44938845f48206e64c26 P 0217dac1b3c64fcc87cc67801d606966 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0217dac1b3c64fcc87cc67801d606966, State: Initialized, Role: FOLLOWER I0312 20:11:51.265866 4375 consensus_queue.cc:219] T 320f1456e73d44938845f48206e64c26 P 0217dac1b3c64fcc87cc67801d606966 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0217dac1b3c64fcc87cc67801d606966" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 43406 } } peers { permanent_uuid: "e404af0289d5411ca5649c9d0d381012" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 33421 } } peers { permanent_uuid: "3120fde077bd405e9d1160a540b7b910" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 38566 } } I0312 20:11:51.267982 4229 heartbeater.cc:467] Master 127.0.53.62:54833 was elected leader, sending a full tablet report... I0312 20:11:51.268342 4375 ts_tablet_manager.cc:1063] T 320f1456e73d44938845f48206e64c26 P 0217dac1b3c64fcc87cc67801d606966: Time spent starting tablet: real 0.017s user 0.012s sys 0.004s I0312 20:11:51.271011 212 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 0217dac1b3c64fcc87cc67801d606966 to finish bootstrapping I0312 20:11:51.280494 212 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 3120fde077bd405e9d1160a540b7b910 to finish bootstrapping I0312 20:11:51.299031 4061 tablet_service.cc:1105] Received Run Leader Election RPC: tablet_id: "320f1456e73d44938845f48206e64c26" dest_uuid: "e404af0289d5411ca5649c9d0d381012" from {username='slave'} at 127.0.0.1:44725 I0312 20:11:51.299352 4061 raft_consensus.cc:436] T 320f1456e73d44938845f48206e64c26 P e404af0289d5411ca5649c9d0d381012 [term 0 FOLLOWER]: Starting forced leader election (received explicit request) I0312 20:11:51.299525 4061 raft_consensus.cc:2752] T 320f1456e73d44938845f48206e64c26 P e404af0289d5411ca5649c9d0d381012 [term 0 FOLLOWER]: Advancing to term 1 I0312 20:11:51.302428 4061 raft_consensus.cc:458] T 320f1456e73d44938845f48206e64c26 P e404af0289d5411ca5649c9d0d381012 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0217dac1b3c64fcc87cc67801d606966" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 43406 } } peers { permanent_uuid: "e404af0289d5411ca5649c9d0d381012" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 33421 } } peers { permanent_uuid: "3120fde077bd405e9d1160a540b7b910" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 38566 } } I0312 20:11:51.303396 4061 leader_election.cc:231] T 320f1456e73d44938845f48206e64c26 P e404af0289d5411ca5649c9d0d381012 [CANDIDATE]: Term 1 election: Requesting vote from peer 0217dac1b3c64fcc87cc67801d606966 I0312 20:11:51.303778 4061 leader_election.cc:231] T 320f1456e73d44938845f48206e64c26 P e404af0289d5411ca5649c9d0d381012 [CANDIDATE]: Term 1 election: Requesting vote from peer 3120fde077bd405e9d1160a540b7b910 I0312 20:11:51.315167 4186 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "320f1456e73d44938845f48206e64c26" candidate_uuid: "e404af0289d5411ca5649c9d0d381012" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "0217dac1b3c64fcc87cc67801d606966" I0312 20:11:51.315559 4186 raft_consensus.cc:2752] T 320f1456e73d44938845f48206e64c26 P 0217dac1b3c64fcc87cc67801d606966 [term 0 FOLLOWER]: Advancing to term 1 I0312 20:11:51.316838 4311 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "320f1456e73d44938845f48206e64c26" candidate_uuid: "e404af0289d5411ca5649c9d0d381012" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "3120fde077bd405e9d1160a540b7b910" I0312 20:11:51.317266 4311 raft_consensus.cc:2752] T 320f1456e73d44938845f48206e64c26 P 3120fde077bd405e9d1160a540b7b910 [term 0 FOLLOWER]: Advancing to term 1 I0312 20:11:51.318423 4186 raft_consensus.cc:2215] T 320f1456e73d44938845f48206e64c26 P 0217dac1b3c64fcc87cc67801d606966 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e404af0289d5411ca5649c9d0d381012 in term 1. I0312 20:11:51.319264 3996 leader_election.cc:363] T 320f1456e73d44938845f48206e64c26 P e404af0289d5411ca5649c9d0d381012 [CANDIDATE]: Term 1 election: Vote granted by peer 0217dac1b3c64fcc87cc67801d606966 I0312 20:11:51.319430 3996 leader_election.cc:258] T 320f1456e73d44938845f48206e64c26 P e404af0289d5411ca5649c9d0d381012 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0312 20:11:51.319900 4380 raft_consensus.cc:2507] T 320f1456e73d44938845f48206e64c26 P e404af0289d5411ca5649c9d0d381012 [term 1 FOLLOWER]: Leader election won for term 1 I0312 20:11:51.320041 4311 raft_consensus.cc:2215] T 320f1456e73d44938845f48206e64c26 P 3120fde077bd405e9d1160a540b7b910 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e404af0289d5411ca5649c9d0d381012 in term 1. I0312 20:11:51.321013 3993 leader_election.cc:363] T 320f1456e73d44938845f48206e64c26 P e404af0289d5411ca5649c9d0d381012 [CANDIDATE]: Term 1 election: Vote granted by peer 3120fde077bd405e9d1160a540b7b910 I0312 20:11:51.322199 4380 raft_consensus.cc:572] T 320f1456e73d44938845f48206e64c26 P e404af0289d5411ca5649c9d0d381012 [term 1 LEADER]: Becoming Leader. State: Replica: e404af0289d5411ca5649c9d0d381012, State: Running, Role: LEADER I0312 20:11:51.322643 4380 consensus_queue.cc:196] T 320f1456e73d44938845f48206e64c26 P e404af0289d5411ca5649c9d0d381012 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0217dac1b3c64fcc87cc67801d606966" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 43406 } } peers { permanent_uuid: "e404af0289d5411ca5649c9d0d381012" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 33421 } } peers { permanent_uuid: "3120fde077bd405e9d1160a540b7b910" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 38566 } } I0312 20:11:51.327739 3938 catalog_manager.cc:3592] T 320f1456e73d44938845f48206e64c26 P e404af0289d5411ca5649c9d0d381012 reported cstate change: term changed from 0 to 1, leader changed from to e404af0289d5411ca5649c9d0d381012 (127.0.53.1). New cstate: current_term: 1 leader_uuid: "e404af0289d5411ca5649c9d0d381012" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0217dac1b3c64fcc87cc67801d606966" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 43406 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e404af0289d5411ca5649c9d0d381012" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 33421 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3120fde077bd405e9d1160a540b7b910" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 38566 } health_report { overall_health: UNKNOWN } } } I0312 20:11:51.335149 4311 raft_consensus.cc:1058] T 320f1456e73d44938845f48206e64c26 P 3120fde077bd405e9d1160a540b7b910 [term 1 FOLLOWER]: Refusing update from remote peer e404af0289d5411ca5649c9d0d381012: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0312 20:11:51.335325 4186 raft_consensus.cc:1058] T 320f1456e73d44938845f48206e64c26 P 0217dac1b3c64fcc87cc67801d606966 [term 1 FOLLOWER]: Refusing update from remote peer e404af0289d5411ca5649c9d0d381012: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0312 20:11:51.336248 4386 consensus_queue.cc:1017] T 320f1456e73d44938845f48206e64c26 P e404af0289d5411ca5649c9d0d381012 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3120fde077bd405e9d1160a540b7b910" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 38566 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0312 20:11:51.336612 4380 consensus_queue.cc:1017] T 320f1456e73d44938845f48206e64c26 P e404af0289d5411ca5649c9d0d381012 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0217dac1b3c64fcc87cc67801d606966" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 43406 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0312 20:11:51.390367 4061 tablet_service.cc:1133] Received LeaderStepDown RPC: tablet_id: "320f1456e73d44938845f48206e64c26" dest_uuid: "e404af0289d5411ca5649c9d0d381012" from {username='slave'} at 127.0.0.1:44725 I0312 20:11:51.390653 4061 raft_consensus.cc:531] T 320f1456e73d44938845f48206e64c26 P e404af0289d5411ca5649c9d0d381012 [term 1 LEADER]: Received request to step down I0312 20:11:51.390776 4061 raft_consensus.cc:605] T 320f1456e73d44938845f48206e64c26 P e404af0289d5411ca5649c9d0d381012 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e404af0289d5411ca5649c9d0d381012, State: Running, Role: LEADER I0312 20:11:51.391007 4061 consensus_queue.cc:219] T 320f1456e73d44938845f48206e64c26 P e404af0289d5411ca5649c9d0d381012 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0217dac1b3c64fcc87cc67801d606966" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 43406 } } peers { permanent_uuid: "e404af0289d5411ca5649c9d0d381012" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 33421 } } peers { permanent_uuid: "3120fde077bd405e9d1160a540b7b910" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 38566 } } I0312 20:11:51.391427 4061 consensus_peers.cc:458] T 320f1456e73d44938845f48206e64c26 P e404af0289d5411ca5649c9d0d381012 -> Peer 3120fde077bd405e9d1160a540b7b910 (127.0.53.3:38566): Closing peer: 3120fde077bd405e9d1160a540b7b910 I0312 20:11:51.391554 4061 consensus_peers.cc:458] T 320f1456e73d44938845f48206e64c26 P e404af0289d5411ca5649c9d0d381012 -> Peer 0217dac1b3c64fcc87cc67801d606966 (127.0.53.2:43406): Closing peer: 0217dac1b3c64fcc87cc67801d606966 I0312 20:11:51.393075 4061 tablet_service.cc:1133] Received LeaderStepDown RPC: tablet_id: "320f1456e73d44938845f48206e64c26" dest_uuid: "e404af0289d5411ca5649c9d0d381012" from {username='slave'} at 127.0.0.1:44725 I0312 20:11:51.393335 4061 raft_consensus.cc:524] T 320f1456e73d44938845f48206e64c26 P e404af0289d5411ca5649c9d0d381012 [term 1 FOLLOWER]: Rejecting request to step down while not leader I0312 20:11:51.397047 212 external_mini_cluster.cc:997] Killing /tmp/dist-test-task3bwPeE/build/tsan/bin/kudu-tserver with pid 3973 W0312 20:11:51.417832 3922 connection.cc:513] client connection to 127.0.53.1:33421 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0312 20:11:51.417893 3910 connection.cc:513] client connection to 127.0.53.1:33421 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0312 20:11:51.418773 3924 connection.cc:513] server connection from 127.0.53.1:46449 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0312 20:11:51.418848 4244 connection.cc:513] server connection from 127.0.53.1:52773 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0312 20:11:51.418838 4118 connection.cc:513] server connection from 127.0.53.1:34649 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0312 20:11:51.419688 3908 connection.cc:513] client connection to 127.0.53.1:33421 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) I0312 20:11:51.420492 212 external_mini_cluster.cc:997] Killing /tmp/dist-test-task3bwPeE/build/tsan/bin/kudu-tserver with pid 4108 W0312 20:11:51.438594 3924 connection.cc:513] server connection from 127.0.53.2:53377 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0312 20:11:51.438576 3910 connection.cc:513] client connection to 127.0.53.2:43406 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0312 20:11:51.439105 3924 connection.cc:513] client connection to 127.0.53.2:43406 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0312 20:11:51.439436 3909 connection.cc:513] client connection to 127.0.53.2:43406 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) I0312 20:11:51.440323 212 external_mini_cluster.cc:997] Killing /tmp/dist-test-task3bwPeE/build/tsan/bin/kudu-tserver with pid 4233 I0312 20:11:51.458552 212 external_mini_cluster.cc:997] Killing /tmp/dist-test-task3bwPeE/build/tsan/bin/kudu-master with pid 3911 W0312 20:11:51.458525 3921 connection.cc:513] server connection from 127.0.53.3:33354 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0312 20:11:51.458990 3907 connection.cc:513] client connection to 127.0.53.3:38566 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0312 20:11:51.459697 3910 connection.cc:513] client connection to 127.0.53.3:38566 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0312 20:11:51.483218 3910 connection.cc:513] client connection to 127.0.53.62:54833 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) [ OK ] RaftConsensusElectionITest.LeaderStepDown (3119 ms) [ RUN ] RaftConsensusElectionITest.StepDownWithSlowFollower I0312 20:11:51.494961 212 test_util.cc:192] Using random seed: -1062724327 I0312 20:11:51.497942 212 ts_itest-base.cc:113] Starting cluster with: I0312 20:11:51.498046 212 ts_itest-base.cc:114] -------------- I0312 20:11:51.498131 212 ts_itest-base.cc:115] 3 tablet servers I0312 20:11:51.498211 212 ts_itest-base.cc:116] 3 replicas per TS I0312 20:11:51.498296 212 ts_itest-base.cc:117] -------------- I0312 20:11:51.500367 212 external_mini_cluster.cc:805] Running /tmp/dist-test-task3bwPeE/build/tsan/bin/kudu-master /tmp/dist-test-task3bwPeE/build/tsan/bin/kudu-master --fs_wal_dir=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.StepDownWithSlowFollower.1520885401735786-212/raft_consensus-itest-cluster/master-0/wal --fs_data_dirs=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.StepDownWithSlowFollower.1520885401735786-212/raft_consensus-itest-cluster/master-0/data --block_manager=log --webserver_interface=localhost --ipki_ca_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.0.53.62:0 --webserver_interface=127.0.53.62 --webserver_port=0 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.StepDownWithSlowFollower.1520885401735786-212/raft_consensus-itest-cluster/master-0/logs --server_dump_info_path=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.StepDownWithSlowFollower.1520885401735786-212/raft_consensus-itest-cluster/master-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --catalog_manager_wait_for_new_tablets_to_elect_leader=false with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W0312 20:11:51.620652 4402 flags.cc:412] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0312 20:11:51.620992 4402 flags.cc:412] Enabled unsafe flag: --never_fsync=true W0312 20:11:51.636447 4402 flags.cc:412] Enabled experimental flag: --ipki_ca_key_size=1024 W0312 20:11:51.636586 4402 flags.cc:412] Enabled experimental flag: --ipki_server_key_size=1024 W0312 20:11:51.636700 4402 flags.cc:412] Enabled experimental flag: --tsk_num_rsa_bits=512 I0312 20:11:51.657207 4402 master_main.cc:66] Master server non-default flags: --fs_data_dirs=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.StepDownWithSlowFollower.1520885401735786-212/raft_consensus-itest-cluster/master-0/data --fs_wal_dir=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.StepDownWithSlowFollower.1520885401735786-212/raft_consensus-itest-cluster/master-0/wal --catalog_manager_wait_for_new_tablets_to_elect_leader=false --ipki_ca_key_size=1024 --ipki_server_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.0.53.62:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.StepDownWithSlowFollower.1520885401735786-212/raft_consensus-itest-cluster/master-0/data/info.pb --webserver_interface=127.0.53.62 --webserver_port=0 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --log_dir=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.StepDownWithSlowFollower.1520885401735786-212/raft_consensus-itest-cluster/master-0/logs --logbuflevel=-1 --logtostderr=true Master server version: kudu 1.7.0-SNAPSHOT revision 13e10c63c46de30ac85f34b543cdd6fb44881f61 build type FASTDEBUG built by None at 12 Mar 2018 20:06:40 UTC on jenkins-slave-vn0g build id 12388 TSAN enabled I0312 20:11:51.666226 4402 master_main.cc:73] Initializing master server... I0312 20:11:51.667726 4402 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 1122776us I0312 20:11:51.668454 4402 fs_manager.cc:260] Metadata directory not provided I0312 20:11:51.668589 4402 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0312 20:11:51.668797 4402 server_base.cc:434] Could not load existing FS layout: Not found: /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.StepDownWithSlowFollower.1520885401735786-212/raft_consensus-itest-cluster/master-0/data/instance: No such file or directory (error 2) I0312 20:11:51.668915 4402 server_base.cc:435] Attempting to create new FS layout instead I0312 20:11:51.676240 4402 fs_manager.cc:596] Generated new instance metadata in path /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.StepDownWithSlowFollower.1520885401735786-212/raft_consensus-itest-cluster/master-0/data/instance: uuid: "b03e4ab95e654b488f5ef2d3387c8404" format_stamp: "Formatted at 2018-03-12 20:11:51 on dist-test-slave-dist-test-slave-f27p" I0312 20:11:51.676937 4402 fs_manager.cc:596] Generated new instance metadata in path /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.StepDownWithSlowFollower.1520885401735786-212/raft_consensus-itest-cluster/master-0/wal/instance: uuid: "b03e4ab95e654b488f5ef2d3387c8404" format_stamp: "Formatted at 2018-03-12 20:11:51 on dist-test-slave-dist-test-slave-f27p" I0312 20:11:51.685204 4402 fs_manager.cc:496] Time spent creating directory manager: real 0.008s user 0.004s sys 0.000s I0312 20:11:51.686363 4402 env_posix.cc:1620] Not raising this process' open files per process limit of 65536; it is already as high as it can go I0312 20:11:51.686636 4402 file_cache.cc:470] Constructed file cache lbm with capacity 26214 I0312 20:11:51.692956 4402 fs_manager.cc:418] Time spent opening block manager: real 0.001s user 0.004s sys 0.000s I0312 20:11:51.693086 4402 fs_manager.cc:429] Opened local filesystem: /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.StepDownWithSlowFollower.1520885401735786-212/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.StepDownWithSlowFollower.1520885401735786-212/raft_consensus-itest-cluster/master-0/wal uuid: "b03e4ab95e654b488f5ef2d3387c8404" format_stamp: "Formatted at 2018-03-12 20:11:51 on dist-test-slave-dist-test-slave-f27p" I0312 20:11:51.693307 4402 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.StepDownWithSlowFollower.1520885401735786-212/raft_consensus-itest-cluster/master-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0312 20:11:51.907374 4402 env_posix.cc:1620] Not raising this process' running threads per effective uid limit of 60284; it is already as high as it can go I0312 20:11:51.907987 4402 master_main.cc:76] Starting Master server... I0312 20:11:51.909235 4419 process_memory.cc:182] Process hard memory limit is 11.777631 GB I0312 20:11:51.909373 4419 process_memory.cc:184] Process soft memory limit is 9.422105 GB I0312 20:11:51.909468 4419 process_memory.cc:187] Process memory pressure threshold is 7.066578 GB I0312 20:11:51.947790 4402 rpc_server.cc:201] RPC server started. Bound to: 127.0.53.62:50068 I0312 20:11:51.973412 4402 webserver.cc:173] Starting webserver on 127.0.53.62:0 I0312 20:11:51.973515 4402 webserver.cc:184] Document root disabled I0312 20:11:51.974735 4402 webserver.cc:311] Webserver started. Bound to: http://127.0.53.62:55554/ I0312 20:11:51.975468 4402 server_base.cc:607] Dumped server information to /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.StepDownWithSlowFollower.1520885401735786-212/raft_consensus-itest-cluster/master-0/data/info.pb I0312 20:11:51.978121 4463 data_dirs.cc:931] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0312 20:11:51.983443 212 external_mini_cluster.cc:867] Started /tmp/dist-test-task3bwPeE/build/tsan/bin/kudu-master as pid 4402 I0312 20:11:51.984885 212 external_mini_cluster.cc:805] Running /tmp/dist-test-task3bwPeE/build/tsan/bin/kudu-tserver /tmp/dist-test-task3bwPeE/build/tsan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.StepDownWithSlowFollower.1520885401735786-212/raft_consensus-itest-cluster/ts-0/wal --fs_data_dirs=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.StepDownWithSlowFollower.1520885401735786-212/raft_consensus-itest-cluster/ts-0/data --block_manager=log --rpc_bind_addresses=127.0.53.1:0 --local_ip_for_outbound_sockets=127.0.53.1 --webserver_interface=127.0.53.1 --webserver_port=0 --tserver_master_addrs=127.0.53.62:50068 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.StepDownWithSlowFollower.1520885401735786-212/raft_consensus-itest-cluster/ts-0/logs --server_dump_info_path=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.StepDownWithSlowFollower.1520885401735786-212/raft_consensus-itest-cluster/ts-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --enable_leader_failure_detection=false --consensus_rpc_timeout_ms=15000 with env {} I0312 20:11:51.994329 4463 tablet_bootstrap.cc:436] T 00000000000000000000000000000000 P b03e4ab95e654b488f5ef2d3387c8404: Bootstrap starting. I0312 20:11:51.996719 4463 tablet_bootstrap.cc:581] T 00000000000000000000000000000000 P b03e4ab95e654b488f5ef2d3387c8404: No blocks or log segments found. Creating new log. I0312 20:11:51.998028 4463 log.cc:520] T 00000000000000000000000000000000 P b03e4ab95e654b488f5ef2d3387c8404: Log is configured to *not* fsync() on all Append() calls I0312 20:11:52.000483 4463 tablet_bootstrap.cc:436] T 00000000000000000000000000000000 P b03e4ab95e654b488f5ef2d3387c8404: No bootstrap required, opened a new log I0312 20:11:52.012635 4463 raft_consensus.cc:304] T 00000000000000000000000000000000 P b03e4ab95e654b488f5ef2d3387c8404 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b03e4ab95e654b488f5ef2d3387c8404" member_type: VOTER } I0312 20:11:52.012953 4463 raft_consensus.cc:330] T 00000000000000000000000000000000 P b03e4ab95e654b488f5ef2d3387c8404 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0312 20:11:52.013090 4463 raft_consensus.cc:605] T 00000000000000000000000000000000 P b03e4ab95e654b488f5ef2d3387c8404 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b03e4ab95e654b488f5ef2d3387c8404, State: Initialized, Role: FOLLOWER I0312 20:11:52.013530 4463 consensus_queue.cc:219] T 00000000000000000000000000000000 P b03e4ab95e654b488f5ef2d3387c8404 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b03e4ab95e654b488f5ef2d3387c8404" member_type: VOTER } I0312 20:11:52.013885 4463 raft_consensus.cc:344] T 00000000000000000000000000000000 P b03e4ab95e654b488f5ef2d3387c8404 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0312 20:11:52.014025 4463 raft_consensus.cc:436] T 00000000000000000000000000000000 P b03e4ab95e654b488f5ef2d3387c8404 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0312 20:11:52.014164 4463 raft_consensus.cc:2700] T 00000000000000000000000000000000 P b03e4ab95e654b488f5ef2d3387c8404: Snoozing failure detection for 1.692s (starting election) I0312 20:11:52.014300 4463 raft_consensus.cc:2752] T 00000000000000000000000000000000 P b03e4ab95e654b488f5ef2d3387c8404 [term 0 FOLLOWER]: Advancing to term 1 I0312 20:11:52.016880 4463 raft_consensus.cc:458] T 00000000000000000000000000000000 P b03e4ab95e654b488f5ef2d3387c8404 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b03e4ab95e654b488f5ef2d3387c8404" member_type: VOTER } I0312 20:11:52.017294 4463 leader_election.cc:258] T 00000000000000000000000000000000 P b03e4ab95e654b488f5ef2d3387c8404 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0312 20:11:52.018949 4466 raft_consensus.cc:2700] T 00000000000000000000000000000000 P b03e4ab95e654b488f5ef2d3387c8404: Snoozing failure detection for 2.073s (election complete) I0312 20:11:52.019217 4466 raft_consensus.cc:2507] T 00000000000000000000000000000000 P b03e4ab95e654b488f5ef2d3387c8404 [term 1 FOLLOWER]: Leader election won for term 1 I0312 20:11:52.025699 4466 raft_consensus.cc:572] T 00000000000000000000000000000000 P b03e4ab95e654b488f5ef2d3387c8404 [term 1 LEADER]: Becoming Leader. State: Replica: b03e4ab95e654b488f5ef2d3387c8404, State: Running, Role: LEADER I0312 20:11:52.026131 4466 consensus_queue.cc:196] T 00000000000000000000000000000000 P b03e4ab95e654b488f5ef2d3387c8404 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b03e4ab95e654b488f5ef2d3387c8404" member_type: VOTER } I0312 20:11:52.038275 4463 sys_catalog.cc:335] T 00000000000000000000000000000000 P b03e4ab95e654b488f5ef2d3387c8404 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 1 leader_uuid: "b03e4ab95e654b488f5ef2d3387c8404" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b03e4ab95e654b488f5ef2d3387c8404" member_type: VOTER } } I0312 20:11:52.038576 4463 sys_catalog.cc:338] T 00000000000000000000000000000000 P b03e4ab95e654b488f5ef2d3387c8404 [sys.catalog]: This master's current role is: LEADER I0312 20:11:52.040503 4468 sys_catalog.cc:335] T 00000000000000000000000000000000 P b03e4ab95e654b488f5ef2d3387c8404 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b03e4ab95e654b488f5ef2d3387c8404. Latest consensus state: current_term: 1 leader_uuid: "b03e4ab95e654b488f5ef2d3387c8404" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b03e4ab95e654b488f5ef2d3387c8404" member_type: VOTER } } I0312 20:11:52.040460 4467 sys_catalog.cc:335] T 00000000000000000000000000000000 P b03e4ab95e654b488f5ef2d3387c8404 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b03e4ab95e654b488f5ef2d3387c8404" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b03e4ab95e654b488f5ef2d3387c8404" member_type: VOTER } } I0312 20:11:52.040956 4468 sys_catalog.cc:338] T 00000000000000000000000000000000 P b03e4ab95e654b488f5ef2d3387c8404 [sys.catalog]: This master's current role is: LEADER I0312 20:11:52.041098 4467 sys_catalog.cc:338] T 00000000000000000000000000000000 P b03e4ab95e654b488f5ef2d3387c8404 [sys.catalog]: This master's current role is: LEADER I0312 20:11:52.041224 4463 sys_catalog.cc:422] T 00000000000000000000000000000000 P b03e4ab95e654b488f5ef2d3387c8404 [sys.catalog]: configured and running, proceeding with master startup. I0312 20:11:52.041646 4472 catalog_manager.cc:972] Loading table and tablet metadata into memory... I0312 20:11:52.061342 4402 master_main.cc:79] Master server successfully started. I0312 20:11:52.064637 4472 catalog_manager.cc:983] Initializing Kudu internal certificate authority... I0312 20:11:52.093037 4472 catalog_manager.cc:870] Generated new certificate authority record I0312 20:11:52.095351 4472 catalog_manager.cc:992] Loading token signing keys... I0312 20:11:52.105167 4472 catalog_manager.cc:3868] T 00000000000000000000000000000000 P b03e4ab95e654b488f5ef2d3387c8404: Generated new TSK 0 WARNING: Logging before InitGoogleLogging() is written to STDERR W0312 20:11:52.132814 4464 flags.cc:412] Enabled unsafe flag: --enable_leader_failure_detection=false W0312 20:11:52.133177 4464 flags.cc:412] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0312 20:11:52.133376 4464 flags.cc:412] Enabled unsafe flag: --never_fsync=true W0312 20:11:52.147636 4464 flags.cc:412] Enabled experimental flag: --ipki_server_key_size=1024 W0312 20:11:52.147940 4464 flags.cc:412] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.53.1 I0312 20:11:52.171653 4464 tablet_server_main.cc:72] Tablet server non-default flags: --consensus_rpc_timeout_ms=15000 --enable_leader_failure_detection=false --fs_data_dirs=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.StepDownWithSlowFollower.1520885401735786-212/raft_consensus-itest-cluster/ts-0/data --fs_wal_dir=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.StepDownWithSlowFollower.1520885401735786-212/raft_consensus-itest-cluster/ts-0/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.0.53.1:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.StepDownWithSlowFollower.1520885401735786-212/raft_consensus-itest-cluster/ts-0/data/info.pb --webserver_interface=127.0.53.1 --webserver_port=0 --tserver_master_addrs=127.0.53.62:50068 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.0.53.1 --log_dir=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.StepDownWithSlowFollower.1520885401735786-212/raft_consensus-itest-cluster/ts-0/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.7.0-SNAPSHOT revision 13e10c63c46de30ac85f34b543cdd6fb44881f61 build type FASTDEBUG built by None at 12 Mar 2018 20:06:40 UTC on jenkins-slave-vn0g build id 12388 TSAN enabled I0312 20:11:52.201416 4464 tablet_server_main.cc:79] Initializing tablet server... I0312 20:11:52.203218 4464 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 1123276us I0312 20:11:52.203682 4464 fs_manager.cc:260] Metadata directory not provided I0312 20:11:52.203811 4464 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0312 20:11:52.204035 4464 server_base.cc:434] Could not load existing FS layout: Not found: /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.StepDownWithSlowFollower.1520885401735786-212/raft_consensus-itest-cluster/ts-0/data/instance: No such file or directory (error 2) I0312 20:11:52.204147 4464 server_base.cc:435] Attempting to create new FS layout instead I0312 20:11:52.211572 4464 fs_manager.cc:596] Generated new instance metadata in path /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.StepDownWithSlowFollower.1520885401735786-212/raft_consensus-itest-cluster/ts-0/data/instance: uuid: "d7e37bd29573493b9da8c0483ea5f28f" format_stamp: "Formatted at 2018-03-12 20:11:52 on dist-test-slave-dist-test-slave-f27p" I0312 20:11:52.212333 4464 fs_manager.cc:596] Generated new instance metadata in path /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.StepDownWithSlowFollower.1520885401735786-212/raft_consensus-itest-cluster/ts-0/wal/instance: uuid: "d7e37bd29573493b9da8c0483ea5f28f" format_stamp: "Formatted at 2018-03-12 20:11:52 on dist-test-slave-dist-test-slave-f27p" I0312 20:11:52.230229 4464 fs_manager.cc:496] Time spent creating directory manager: real 0.018s user 0.004s sys 0.016s I0312 20:11:52.231391 4464 env_posix.cc:1620] Not raising this process' open files per process limit of 65536; it is already as high as it can go I0312 20:11:52.231683 4464 file_cache.cc:470] Constructed file cache lbm with capacity 26214 I0312 20:11:52.242267 4464 fs_manager.cc:418] Time spent opening block manager: real 0.006s user 0.000s sys 0.008s I0312 20:11:52.242394 4464 fs_manager.cc:429] Opened local filesystem: /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.StepDownWithSlowFollower.1520885401735786-212/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.StepDownWithSlowFollower.1520885401735786-212/raft_consensus-itest-cluster/ts-0/wal uuid: "d7e37bd29573493b9da8c0483ea5f28f" format_stamp: "Formatted at 2018-03-12 20:11:52 on dist-test-slave-dist-test-slave-f27p" I0312 20:11:52.242625 4464 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.StepDownWithSlowFollower.1520885401735786-212/raft_consensus-itest-cluster/ts-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0312 20:11:52.496309 4464 env_posix.cc:1620] Not raising this process' running threads per effective uid limit of 60284; it is already as high as it can go I0312 20:11:52.497637 4464 ts_tablet_manager.cc:349] Loaded tablet metadata (0 live tablets) I0312 20:11:52.509409 4464 tablet_server_main.cc:84] Starting tablet server... I0312 20:11:52.957348 4464 rpc_server.cc:201] RPC server started. Bound to: 127.0.53.1:40547 I0312 20:11:52.958247 4464 webserver.cc:173] Starting webserver on 127.0.53.1:0 I0312 20:11:52.958358 4464 webserver.cc:184] Document root disabled I0312 20:11:52.959820 4464 webserver.cc:311] Webserver started. Bound to: http://127.0.53.1:55895/ I0312 20:11:52.960566 4464 server_base.cc:607] Dumped server information to /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.StepDownWithSlowFollower.1520885401735786-212/raft_consensus-itest-cluster/ts-0/data/info.pb I0312 20:11:52.967576 4597 process_memory.cc:182] Process hard memory limit is 11.777631 GB I0312 20:11:52.967917 4597 process_memory.cc:184] Process soft memory limit is 9.422105 GB I0312 20:11:52.968065 4597 process_memory.cc:187] Process memory pressure threshold is 7.066578 GB I0312 20:11:52.968653 212 external_mini_cluster.cc:867] Started /tmp/dist-test-task3bwPeE/build/tsan/bin/kudu-tserver as pid 4464 I0312 20:11:52.970260 212 external_mini_cluster.cc:805] Running /tmp/dist-test-task3bwPeE/build/tsan/bin/kudu-tserver /tmp/dist-test-task3bwPeE/build/tsan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.StepDownWithSlowFollower.1520885401735786-212/raft_consensus-itest-cluster/ts-1/wal --fs_data_dirs=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.StepDownWithSlowFollower.1520885401735786-212/raft_consensus-itest-cluster/ts-1/data --block_manager=log --rpc_bind_addresses=127.0.53.2:0 --local_ip_for_outbound_sockets=127.0.53.2 --webserver_interface=127.0.53.2 --webserver_port=0 --tserver_master_addrs=127.0.53.62:50068 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.StepDownWithSlowFollower.1520885401735786-212/raft_consensus-itest-cluster/ts-1/logs --server_dump_info_path=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.StepDownWithSlowFollower.1520885401735786-212/raft_consensus-itest-cluster/ts-1/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --enable_leader_failure_detection=false --consensus_rpc_timeout_ms=15000 with env {} I0312 20:11:52.967836 4464 tablet_server_main.cc:87] Tablet server successfully started. I0312 20:11:52.995180 4596 heartbeater.cc:346] Connected to a master server at 127.0.53.62:50068 I0312 20:11:52.995436 4596 heartbeater.cc:426] Registering TS with master... I0312 20:11:52.995873 4596 heartbeater.cc:475] Master 127.0.53.62:50068 requested a full tablet report, sending... I0312 20:11:52.997525 4430 ts_manager.cc:81] Registered new tserver with Master: d7e37bd29573493b9da8c0483ea5f28f (127.0.53.1:40547) I0312 20:11:52.999169 4430 master_service.cc:243] Signed X509 certificate for tserver {username='slave'} at 127.0.53.1:60554 WARNING: Logging before InitGoogleLogging() is written to STDERR W0312 20:11:53.152529 4600 flags.cc:412] Enabled unsafe flag: --enable_leader_failure_detection=false W0312 20:11:53.152972 4600 flags.cc:412] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0312 20:11:53.153264 4600 flags.cc:412] Enabled unsafe flag: --never_fsync=true W0312 20:11:53.176162 4600 flags.cc:412] Enabled experimental flag: --ipki_server_key_size=1024 W0312 20:11:53.176611 4600 flags.cc:412] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.53.2 I0312 20:11:53.205759 4600 tablet_server_main.cc:72] Tablet server non-default flags: --consensus_rpc_timeout_ms=15000 --enable_leader_failure_detection=false --fs_data_dirs=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.StepDownWithSlowFollower.1520885401735786-212/raft_consensus-itest-cluster/ts-1/data --fs_wal_dir=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.StepDownWithSlowFollower.1520885401735786-212/raft_consensus-itest-cluster/ts-1/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.0.53.2:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.StepDownWithSlowFollower.1520885401735786-212/raft_consensus-itest-cluster/ts-1/data/info.pb --webserver_interface=127.0.53.2 --webserver_port=0 --tserver_master_addrs=127.0.53.62:50068 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.0.53.2 --log_dir=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.StepDownWithSlowFollower.1520885401735786-212/raft_consensus-itest-cluster/ts-1/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.7.0-SNAPSHOT revision 13e10c63c46de30ac85f34b543cdd6fb44881f61 build type FASTDEBUG built by None at 12 Mar 2018 20:06:40 UTC on jenkins-slave-vn0g build id 12388 TSAN enabled I0312 20:11:53.220058 4600 tablet_server_main.cc:79] Initializing tablet server... I0312 20:11:53.222355 4600 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 1123776us I0312 20:11:53.222954 4600 fs_manager.cc:260] Metadata directory not provided I0312 20:11:53.223116 4600 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0312 20:11:53.223373 4600 server_base.cc:434] Could not load existing FS layout: Not found: /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.StepDownWithSlowFollower.1520885401735786-212/raft_consensus-itest-cluster/ts-1/data/instance: No such file or directory (error 2) I0312 20:11:53.223520 4600 server_base.cc:435] Attempting to create new FS layout instead I0312 20:11:53.235782 4600 fs_manager.cc:596] Generated new instance metadata in path /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.StepDownWithSlowFollower.1520885401735786-212/raft_consensus-itest-cluster/ts-1/data/instance: uuid: "eb6f9b014d344497bdbeb7e336ea5185" format_stamp: "Formatted at 2018-03-12 20:11:53 on dist-test-slave-dist-test-slave-f27p" I0312 20:11:53.236712 4600 fs_manager.cc:596] Generated new instance metadata in path /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.StepDownWithSlowFollower.1520885401735786-212/raft_consensus-itest-cluster/ts-1/wal/instance: uuid: "eb6f9b014d344497bdbeb7e336ea5185" format_stamp: "Formatted at 2018-03-12 20:11:53 on dist-test-slave-dist-test-slave-f27p" I0312 20:11:53.249606 4600 fs_manager.cc:496] Time spent creating directory manager: real 0.012s user 0.012s sys 0.000s I0312 20:11:53.251217 4600 env_posix.cc:1620] Not raising this process' open files per process limit of 65536; it is already as high as it can go I0312 20:11:53.251591 4600 file_cache.cc:470] Constructed file cache lbm with capacity 26214 I0312 20:11:53.261865 4600 fs_manager.cc:418] Time spent opening block manager: real 0.002s user 0.000s sys 0.004s I0312 20:11:53.262022 4600 fs_manager.cc:429] Opened local filesystem: /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.StepDownWithSlowFollower.1520885401735786-212/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.StepDownWithSlowFollower.1520885401735786-212/raft_consensus-itest-cluster/ts-1/wal uuid: "eb6f9b014d344497bdbeb7e336ea5185" format_stamp: "Formatted at 2018-03-12 20:11:53 on dist-test-slave-dist-test-slave-f27p" I0312 20:11:53.262289 4600 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.StepDownWithSlowFollower.1520885401735786-212/raft_consensus-itest-cluster/ts-1/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0312 20:11:53.441687 4600 env_posix.cc:1620] Not raising this process' running threads per effective uid limit of 60284; it is already as high as it can go I0312 20:11:53.442941 4600 ts_tablet_manager.cc:349] Loaded tablet metadata (0 live tablets) I0312 20:11:53.443922 4600 tablet_server_main.cc:84] Starting tablet server... I0312 20:11:53.532025 4600 rpc_server.cc:201] RPC server started. Bound to: 127.0.53.2:38269 I0312 20:11:53.532920 4600 webserver.cc:173] Starting webserver on 127.0.53.2:0 I0312 20:11:53.533033 4600 webserver.cc:184] Document root disabled I0312 20:11:53.534262 4600 webserver.cc:311] Webserver started. Bound to: http://127.0.53.2:50190/ I0312 20:11:53.534977 4600 server_base.cc:607] Dumped server information to /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.StepDownWithSlowFollower.1520885401735786-212/raft_consensus-itest-cluster/ts-1/data/info.pb I0312 20:11:53.537518 4600 tablet_server_main.cc:87] Tablet server successfully started. I0312 20:11:53.537744 212 external_mini_cluster.cc:867] Started /tmp/dist-test-task3bwPeE/build/tsan/bin/kudu-tserver as pid 4600 I0312 20:11:53.539007 212 external_mini_cluster.cc:805] Running /tmp/dist-test-task3bwPeE/build/tsan/bin/kudu-tserver /tmp/dist-test-task3bwPeE/build/tsan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.StepDownWithSlowFollower.1520885401735786-212/raft_consensus-itest-cluster/ts-2/wal --fs_data_dirs=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.StepDownWithSlowFollower.1520885401735786-212/raft_consensus-itest-cluster/ts-2/data --block_manager=log --rpc_bind_addresses=127.0.53.3:0 --local_ip_for_outbound_sockets=127.0.53.3 --webserver_interface=127.0.53.3 --webserver_port=0 --tserver_master_addrs=127.0.53.62:50068 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.StepDownWithSlowFollower.1520885401735786-212/raft_consensus-itest-cluster/ts-2/logs --server_dump_info_path=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.StepDownWithSlowFollower.1520885401735786-212/raft_consensus-itest-cluster/ts-2/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --enable_leader_failure_detection=false --consensus_rpc_timeout_ms=15000 with env {} I0312 20:11:53.557078 4721 heartbeater.cc:346] Connected to a master server at 127.0.53.62:50068 I0312 20:11:53.557332 4721 heartbeater.cc:426] Registering TS with master... I0312 20:11:53.557777 4721 heartbeater.cc:475] Master 127.0.53.62:50068 requested a full tablet report, sending... I0312 20:11:53.559104 4430 ts_manager.cc:81] Registered new tserver with Master: eb6f9b014d344497bdbeb7e336ea5185 (127.0.53.2:38269) I0312 20:11:53.560232 4430 master_service.cc:243] Signed X509 certificate for tserver {username='slave'} at 127.0.53.2:51629 I0312 20:11:53.540655 4722 process_memory.cc:182] Process hard memory limit is 11.777631 GB I0312 20:11:53.565881 4722 process_memory.cc:184] Process soft memory limit is 9.422105 GB I0312 20:11:53.565973 4722 process_memory.cc:187] Process memory pressure threshold is 7.066578 GB WARNING: Logging before InitGoogleLogging() is written to STDERR W0312 20:11:53.663004 4725 flags.cc:412] Enabled unsafe flag: --enable_leader_failure_detection=false W0312 20:11:53.663327 4725 flags.cc:412] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0312 20:11:53.663508 4725 flags.cc:412] Enabled unsafe flag: --never_fsync=true W0312 20:11:53.677608 4725 flags.cc:412] Enabled experimental flag: --ipki_server_key_size=1024 W0312 20:11:53.677891 4725 flags.cc:412] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.53.3 I0312 20:11:53.696600 4725 tablet_server_main.cc:72] Tablet server non-default flags: --consensus_rpc_timeout_ms=15000 --enable_leader_failure_detection=false --fs_data_dirs=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.StepDownWithSlowFollower.1520885401735786-212/raft_consensus-itest-cluster/ts-2/data --fs_wal_dir=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.StepDownWithSlowFollower.1520885401735786-212/raft_consensus-itest-cluster/ts-2/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.0.53.3:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.StepDownWithSlowFollower.1520885401735786-212/raft_consensus-itest-cluster/ts-2/data/info.pb --webserver_interface=127.0.53.3 --webserver_port=0 --tserver_master_addrs=127.0.53.62:50068 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.0.53.3 --log_dir=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.StepDownWithSlowFollower.1520885401735786-212/raft_consensus-itest-cluster/ts-2/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.7.0-SNAPSHOT revision 13e10c63c46de30ac85f34b543cdd6fb44881f61 build type FASTDEBUG built by None at 12 Mar 2018 20:06:40 UTC on jenkins-slave-vn0g build id 12388 TSAN enabled I0312 20:11:53.706269 4725 tablet_server_main.cc:79] Initializing tablet server... I0312 20:11:53.708009 4725 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 1123776us I0312 20:11:53.708433 4725 fs_manager.cc:260] Metadata directory not provided I0312 20:11:53.708552 4725 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0312 20:11:53.708746 4725 server_base.cc:434] Could not load existing FS layout: Not found: /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.StepDownWithSlowFollower.1520885401735786-212/raft_consensus-itest-cluster/ts-2/data/instance: No such file or directory (error 2) I0312 20:11:53.708858 4725 server_base.cc:435] Attempting to create new FS layout instead I0312 20:11:53.715968 4725 fs_manager.cc:596] Generated new instance metadata in path /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.StepDownWithSlowFollower.1520885401735786-212/raft_consensus-itest-cluster/ts-2/data/instance: uuid: "63a9ad19ea5747a8a2bf8e82823284cb" format_stamp: "Formatted at 2018-03-12 20:11:53 on dist-test-slave-dist-test-slave-f27p" I0312 20:11:53.716655 4725 fs_manager.cc:596] Generated new instance metadata in path /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.StepDownWithSlowFollower.1520885401735786-212/raft_consensus-itest-cluster/ts-2/wal/instance: uuid: "63a9ad19ea5747a8a2bf8e82823284cb" format_stamp: "Formatted at 2018-03-12 20:11:53 on dist-test-slave-dist-test-slave-f27p" I0312 20:11:53.724736 4725 fs_manager.cc:496] Time spent creating directory manager: real 0.008s user 0.004s sys 0.004s I0312 20:11:53.725896 4725 env_posix.cc:1620] Not raising this process' open files per process limit of 65536; it is already as high as it can go I0312 20:11:53.726181 4725 file_cache.cc:470] Constructed file cache lbm with capacity 26214 I0312 20:11:53.732437 4725 fs_manager.cc:418] Time spent opening block manager: real 0.001s user 0.004s sys 0.000s I0312 20:11:53.732553 4725 fs_manager.cc:429] Opened local filesystem: /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.StepDownWithSlowFollower.1520885401735786-212/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.StepDownWithSlowFollower.1520885401735786-212/raft_consensus-itest-cluster/ts-2/wal uuid: "63a9ad19ea5747a8a2bf8e82823284cb" format_stamp: "Formatted at 2018-03-12 20:11:53 on dist-test-slave-dist-test-slave-f27p" I0312 20:11:53.732753 4725 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.StepDownWithSlowFollower.1520885401735786-212/raft_consensus-itest-cluster/ts-2/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0312 20:11:53.897567 4725 env_posix.cc:1620] Not raising this process' running threads per effective uid limit of 60284; it is already as high as it can go I0312 20:11:53.898859 4725 ts_tablet_manager.cc:349] Loaded tablet metadata (0 live tablets) I0312 20:11:53.899821 4725 tablet_server_main.cc:84] Starting tablet server... I0312 20:11:53.989923 4725 rpc_server.cc:201] RPC server started. Bound to: 127.0.53.3:59549 I0312 20:11:53.990805 4725 webserver.cc:173] Starting webserver on 127.0.53.3:0 I0312 20:11:53.990903 4725 webserver.cc:184] Document root disabled I0312 20:11:53.992103 4725 webserver.cc:311] Webserver started. Bound to: http://127.0.53.3:51212/ I0312 20:11:53.992816 4725 server_base.cc:607] Dumped server information to /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.StepDownWithSlowFollower.1520885401735786-212/raft_consensus-itest-cluster/ts-2/data/info.pb I0312 20:11:53.997792 4847 process_memory.cc:182] Process hard memory limit is 11.777631 GB I0312 20:11:53.998210 4847 process_memory.cc:184] Process soft memory limit is 9.422105 GB I0312 20:11:53.998340 4847 process_memory.cc:187] Process memory pressure threshold is 7.066578 GB I0312 20:11:53.998940 212 external_mini_cluster.cc:867] Started /tmp/dist-test-task3bwPeE/build/tsan/bin/kudu-tserver as pid 4725 I0312 20:11:54.001574 4596 heartbeater.cc:467] Master 127.0.53.62:50068 was elected leader, sending a full tablet report... I0312 20:11:54.021533 4846 heartbeater.cc:346] Connected to a master server at 127.0.53.62:50068 I0312 20:11:54.021782 4846 heartbeater.cc:426] Registering TS with master... I0312 20:11:54.022212 4846 heartbeater.cc:475] Master 127.0.53.62:50068 requested a full tablet report, sending... I0312 20:11:54.023511 4430 ts_manager.cc:81] Registered new tserver with Master: 63a9ad19ea5747a8a2bf8e82823284cb (127.0.53.3:59549) I0312 20:11:54.024633 4430 master_service.cc:243] Signed X509 certificate for tserver {username='slave'} at 127.0.53.3:50348 I0312 20:11:53.998162 4725 tablet_server_main.cc:87] Tablet server successfully started. I0312 20:11:54.026798 212 external_mini_cluster.cc:498] 3 TS(s) registered with all masters I0312 20:11:54.044100 4430 catalog_manager.cc:1294] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42734: name: "TestTable" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 3 split_rows_range_bounds { } partition_schema { range_schema { columns { name: "key" } } } W0312 20:11:54.052538 4430 catalog_manager.cc:1450] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a replica failure: 4 tablet servers are needed, 3 are alive. Consider bringing up additional tablet server(s) or running both the masters and all tablet servers with --raft_prepare_replacement_before_eviction=false flag (not recommended). I0312 20:11:54.090987 4533 tablet_service.cc:755] Processing CreateTablet for tablet 6ba89589faf649d6bc84738d4904ab46 (table=TestTable [id=15027ef964f84064b2174152e83bcbd3]), partition=RANGE (key) PARTITION UNBOUNDED I0312 20:11:54.091665 4533 data_dirs.cc:931] Could only allocate 1 dirs of requested 3 for tablet 6ba89589faf649d6bc84738d4904ab46. 1 dirs total, 0 dirs full, 0 dirs failed I0312 20:11:54.092056 4783 tablet_service.cc:755] Processing CreateTablet for tablet 6ba89589faf649d6bc84738d4904ab46 (table=TestTable [id=15027ef964f84064b2174152e83bcbd3]), partition=RANGE (key) PARTITION UNBOUNDED I0312 20:11:54.092111 4658 tablet_service.cc:755] Processing CreateTablet for tablet 6ba89589faf649d6bc84738d4904ab46 (table=TestTable [id=15027ef964f84064b2174152e83bcbd3]), partition=RANGE (key) PARTITION UNBOUNDED I0312 20:11:54.092864 4783 data_dirs.cc:931] Could only allocate 1 dirs of requested 3 for tablet 6ba89589faf649d6bc84738d4904ab46. 1 dirs total, 0 dirs full, 0 dirs failed I0312 20:11:54.097250 4658 data_dirs.cc:931] Could only allocate 1 dirs of requested 3 for tablet 6ba89589faf649d6bc84738d4904ab46. 1 dirs total, 0 dirs full, 0 dirs failed I0312 20:11:54.103148 4533 ts_tablet_manager.cc:1163] T 6ba89589faf649d6bc84738d4904ab46 P d7e37bd29573493b9da8c0483ea5f28f: Registered tablet (data state: TABLET_DATA_READY) I0312 20:11:54.111450 4783 ts_tablet_manager.cc:1163] T 6ba89589faf649d6bc84738d4904ab46 P 63a9ad19ea5747a8a2bf8e82823284cb: Registered tablet (data state: TABLET_DATA_READY) I0312 20:11:54.114013 4867 ts_tablet_manager.cc:1018] T 6ba89589faf649d6bc84738d4904ab46 P 63a9ad19ea5747a8a2bf8e82823284cb: Bootstrapping tablet I0312 20:11:54.114487 4867 tablet_bootstrap.cc:436] T 6ba89589faf649d6bc84738d4904ab46 P 63a9ad19ea5747a8a2bf8e82823284cb: Bootstrap starting. I0312 20:11:54.115217 4658 ts_tablet_manager.cc:1163] T 6ba89589faf649d6bc84738d4904ab46 P eb6f9b014d344497bdbeb7e336ea5185: Registered tablet (data state: TABLET_DATA_READY) I0312 20:11:54.116820 4867 tablet_bootstrap.cc:581] T 6ba89589faf649d6bc84738d4904ab46 P 63a9ad19ea5747a8a2bf8e82823284cb: No blocks or log segments found. Creating new log. I0312 20:11:54.118428 4866 ts_tablet_manager.cc:1018] T 6ba89589faf649d6bc84738d4904ab46 P d7e37bd29573493b9da8c0483ea5f28f: Bootstrapping tablet I0312 20:11:54.118790 4866 tablet_bootstrap.cc:436] T 6ba89589faf649d6bc84738d4904ab46 P d7e37bd29573493b9da8c0483ea5f28f: Bootstrap starting. I0312 20:11:54.119374 4868 ts_tablet_manager.cc:1018] T 6ba89589faf649d6bc84738d4904ab46 P eb6f9b014d344497bdbeb7e336ea5185: Bootstrapping tablet I0312 20:11:54.119757 4868 tablet_bootstrap.cc:436] T 6ba89589faf649d6bc84738d4904ab46 P eb6f9b014d344497bdbeb7e336ea5185: Bootstrap starting. I0312 20:11:54.121021 4866 tablet_bootstrap.cc:581] T 6ba89589faf649d6bc84738d4904ab46 P d7e37bd29573493b9da8c0483ea5f28f: No blocks or log segments found. Creating new log. I0312 20:11:54.122231 4866 log.cc:520] T 6ba89589faf649d6bc84738d4904ab46 P d7e37bd29573493b9da8c0483ea5f28f: Log is configured to *not* fsync() on all Append() calls I0312 20:11:54.122625 4868 tablet_bootstrap.cc:581] T 6ba89589faf649d6bc84738d4904ab46 P eb6f9b014d344497bdbeb7e336ea5185: No blocks or log segments found. Creating new log. I0312 20:11:54.124002 4868 log.cc:520] T 6ba89589faf649d6bc84738d4904ab46 P eb6f9b014d344497bdbeb7e336ea5185: Log is configured to *not* fsync() on all Append() calls I0312 20:11:54.125821 4867 log.cc:520] T 6ba89589faf649d6bc84738d4904ab46 P 63a9ad19ea5747a8a2bf8e82823284cb: Log is configured to *not* fsync() on all Append() calls I0312 20:11:54.127826 4868 tablet_bootstrap.cc:436] T 6ba89589faf649d6bc84738d4904ab46 P eb6f9b014d344497bdbeb7e336ea5185: No bootstrap required, opened a new log I0312 20:11:54.128212 4868 ts_tablet_manager.cc:1035] T 6ba89589faf649d6bc84738d4904ab46 P eb6f9b014d344497bdbeb7e336ea5185: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s I0312 20:11:54.129390 4867 tablet_bootstrap.cc:436] T 6ba89589faf649d6bc84738d4904ab46 P 63a9ad19ea5747a8a2bf8e82823284cb: No bootstrap required, opened a new log I0312 20:11:54.129799 4867 ts_tablet_manager.cc:1035] T 6ba89589faf649d6bc84738d4904ab46 P 63a9ad19ea5747a8a2bf8e82823284cb: Time spent bootstrapping tablet: real 0.016s user 0.008s sys 0.000s I0312 20:11:54.130035 4866 tablet_bootstrap.cc:436] T 6ba89589faf649d6bc84738d4904ab46 P d7e37bd29573493b9da8c0483ea5f28f: No bootstrap required, opened a new log I0312 20:11:54.130427 4866 ts_tablet_manager.cc:1035] T 6ba89589faf649d6bc84738d4904ab46 P d7e37bd29573493b9da8c0483ea5f28f: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.004s I0312 20:11:54.138943 4868 raft_consensus.cc:304] T 6ba89589faf649d6bc84738d4904ab46 P eb6f9b014d344497bdbeb7e336ea5185 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6f9b014d344497bdbeb7e336ea5185" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 38269 } } peers { permanent_uuid: "63a9ad19ea5747a8a2bf8e82823284cb" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 59549 } } peers { permanent_uuid: "d7e37bd29573493b9da8c0483ea5f28f" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 40547 } } I0312 20:11:54.139497 4868 raft_consensus.cc:605] T 6ba89589faf649d6bc84738d4904ab46 P eb6f9b014d344497bdbeb7e336ea5185 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb6f9b014d344497bdbeb7e336ea5185, State: Initialized, Role: FOLLOWER I0312 20:11:54.139837 4868 consensus_queue.cc:219] T 6ba89589faf649d6bc84738d4904ab46 P eb6f9b014d344497bdbeb7e336ea5185 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6f9b014d344497bdbeb7e336ea5185" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 38269 } } peers { permanent_uuid: "63a9ad19ea5747a8a2bf8e82823284cb" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 59549 } } peers { permanent_uuid: "d7e37bd29573493b9da8c0483ea5f28f" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 40547 } } I0312 20:11:54.145095 4867 raft_consensus.cc:304] T 6ba89589faf649d6bc84738d4904ab46 P 63a9ad19ea5747a8a2bf8e82823284cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6f9b014d344497bdbeb7e336ea5185" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 38269 } } peers { permanent_uuid: "63a9ad19ea5747a8a2bf8e82823284cb" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 59549 } } peers { permanent_uuid: "d7e37bd29573493b9da8c0483ea5f28f" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 40547 } } I0312 20:11:54.145952 4867 raft_consensus.cc:605] T 6ba89589faf649d6bc84738d4904ab46 P 63a9ad19ea5747a8a2bf8e82823284cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63a9ad19ea5747a8a2bf8e82823284cb, State: Initialized, Role: FOLLOWER I0312 20:11:54.145828 4866 raft_consensus.cc:304] T 6ba89589faf649d6bc84738d4904ab46 P d7e37bd29573493b9da8c0483ea5f28f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6f9b014d344497bdbeb7e336ea5185" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 38269 } } peers { permanent_uuid: "63a9ad19ea5747a8a2bf8e82823284cb" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 59549 } } peers { permanent_uuid: "d7e37bd29573493b9da8c0483ea5f28f" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 40547 } } I0312 20:11:54.146559 4866 raft_consensus.cc:605] T 6ba89589faf649d6bc84738d4904ab46 P d7e37bd29573493b9da8c0483ea5f28f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7e37bd29573493b9da8c0483ea5f28f, State: Initialized, Role: FOLLOWER I0312 20:11:54.146344 4867 consensus_queue.cc:219] T 6ba89589faf649d6bc84738d4904ab46 P 63a9ad19ea5747a8a2bf8e82823284cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6f9b014d344497bdbeb7e336ea5185" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 38269 } } peers { permanent_uuid: "63a9ad19ea5747a8a2bf8e82823284cb" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 59549 } } peers { permanent_uuid: "d7e37bd29573493b9da8c0483ea5f28f" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 40547 } } I0312 20:11:54.146946 4866 consensus_queue.cc:219] T 6ba89589faf649d6bc84738d4904ab46 P d7e37bd29573493b9da8c0483ea5f28f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6f9b014d344497bdbeb7e336ea5185" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 38269 } } peers { permanent_uuid: "63a9ad19ea5747a8a2bf8e82823284cb" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 59549 } } peers { permanent_uuid: "d7e37bd29573493b9da8c0483ea5f28f" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 40547 } } I0312 20:11:54.148298 4721 heartbeater.cc:467] Master 127.0.53.62:50068 was elected leader, sending a full tablet report... I0312 20:11:54.148511 4846 heartbeater.cc:467] Master 127.0.53.62:50068 was elected leader, sending a full tablet report... I0312 20:11:54.150893 4868 ts_tablet_manager.cc:1063] T 6ba89589faf649d6bc84738d4904ab46 P eb6f9b014d344497bdbeb7e336ea5185: Time spent starting tablet: real 0.022s user 0.016s sys 0.004s I0312 20:11:54.151988 4867 ts_tablet_manager.cc:1063] T 6ba89589faf649d6bc84738d4904ab46 P 63a9ad19ea5747a8a2bf8e82823284cb: Time spent starting tablet: real 0.022s user 0.016s sys 0.004s I0312 20:11:54.153159 4866 ts_tablet_manager.cc:1063] T 6ba89589faf649d6bc84738d4904ab46 P d7e37bd29573493b9da8c0483ea5f28f: Time spent starting tablet: real 0.023s user 0.020s sys 0.000s I0312 20:11:54.161713 212 external_mini_cluster.cc:498] 3 TS(s) registered with all masters I0312 20:11:54.163792 212 ts_itest-base.cc:233] Waiting for 1 tablets on tserver d7e37bd29573493b9da8c0483ea5f28f to finish bootstrapping I0312 20:11:54.172546 212 ts_itest-base.cc:233] Waiting for 1 tablets on tserver eb6f9b014d344497bdbeb7e336ea5185 to finish bootstrapping I0312 20:11:54.180953 212 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 63a9ad19ea5747a8a2bf8e82823284cb to finish bootstrapping I0312 20:11:54.198335 4553 tablet_service.cc:1105] Received Run Leader Election RPC: tablet_id: "6ba89589faf649d6bc84738d4904ab46" dest_uuid: "d7e37bd29573493b9da8c0483ea5f28f" from {username='slave'} at 127.0.0.1:42537 I0312 20:11:54.198657 4553 raft_consensus.cc:436] T 6ba89589faf649d6bc84738d4904ab46 P d7e37bd29573493b9da8c0483ea5f28f [term 0 FOLLOWER]: Starting forced leader election (received explicit request) I0312 20:11:54.198827 4553 raft_consensus.cc:2752] T 6ba89589faf649d6bc84738d4904ab46 P d7e37bd29573493b9da8c0483ea5f28f [term 0 FOLLOWER]: Advancing to term 1 I0312 20:11:54.201606 4553 raft_consensus.cc:458] T 6ba89589faf649d6bc84738d4904ab46 P d7e37bd29573493b9da8c0483ea5f28f [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6f9b014d344497bdbeb7e336ea5185" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 38269 } } peers { permanent_uuid: "63a9ad19ea5747a8a2bf8e82823284cb" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 59549 } } peers { permanent_uuid: "d7e37bd29573493b9da8c0483ea5f28f" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 40547 } } I0312 20:11:54.202592 4553 leader_election.cc:231] T 6ba89589faf649d6bc84738d4904ab46 P d7e37bd29573493b9da8c0483ea5f28f [CANDIDATE]: Term 1 election: Requesting vote from peer eb6f9b014d344497bdbeb7e336ea5185 I0312 20:11:54.202991 4553 leader_election.cc:231] T 6ba89589faf649d6bc84738d4904ab46 P d7e37bd29573493b9da8c0483ea5f28f [CANDIDATE]: Term 1 election: Requesting vote from peer 63a9ad19ea5747a8a2bf8e82823284cb I0312 20:11:54.215430 4803 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "6ba89589faf649d6bc84738d4904ab46" candidate_uuid: "d7e37bd29573493b9da8c0483ea5f28f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "63a9ad19ea5747a8a2bf8e82823284cb" I0312 20:11:54.215534 4678 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "6ba89589faf649d6bc84738d4904ab46" candidate_uuid: "d7e37bd29573493b9da8c0483ea5f28f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "eb6f9b014d344497bdbeb7e336ea5185" I0312 20:11:54.215818 4803 raft_consensus.cc:2752] T 6ba89589faf649d6bc84738d4904ab46 P 63a9ad19ea5747a8a2bf8e82823284cb [term 0 FOLLOWER]: Advancing to term 1 I0312 20:11:54.215917 4678 raft_consensus.cc:2752] T 6ba89589faf649d6bc84738d4904ab46 P eb6f9b014d344497bdbeb7e336ea5185 [term 0 FOLLOWER]: Advancing to term 1 I0312 20:11:54.218621 4803 raft_consensus.cc:2215] T 6ba89589faf649d6bc84738d4904ab46 P 63a9ad19ea5747a8a2bf8e82823284cb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7e37bd29573493b9da8c0483ea5f28f in term 1. W0312 20:11:54.218920 4597 tablet.cc:1679] T 6ba89589faf649d6bc84738d4904ab46 P d7e37bd29573493b9da8c0483ea5f28f: Can't schedule compaction. Clean time has not been advanced past its initial value. I0312 20:11:54.218968 4678 raft_consensus.cc:2215] T 6ba89589faf649d6bc84738d4904ab46 P eb6f9b014d344497bdbeb7e336ea5185 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7e37bd29573493b9da8c0483ea5f28f in term 1. I0312 20:11:54.219682 4484 leader_election.cc:363] T 6ba89589faf649d6bc84738d4904ab46 P d7e37bd29573493b9da8c0483ea5f28f [CANDIDATE]: Term 1 election: Vote granted by peer 63a9ad19ea5747a8a2bf8e82823284cb I0312 20:11:54.219928 4484 leader_election.cc:258] T 6ba89589faf649d6bc84738d4904ab46 P d7e37bd29573493b9da8c0483ea5f28f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0312 20:11:54.220212 4488 leader_election.cc:363] T 6ba89589faf649d6bc84738d4904ab46 P d7e37bd29573493b9da8c0483ea5f28f [CANDIDATE]: Term 1 election: Vote granted by peer eb6f9b014d344497bdbeb7e336ea5185 I0312 20:11:54.220470 4874 raft_consensus.cc:2507] T 6ba89589faf649d6bc84738d4904ab46 P d7e37bd29573493b9da8c0483ea5f28f [term 1 FOLLOWER]: Leader election won for term 1 I0312 20:11:54.221675 4874 raft_consensus.cc:572] T 6ba89589faf649d6bc84738d4904ab46 P d7e37bd29573493b9da8c0483ea5f28f [term 1 LEADER]: Becoming Leader. State: Replica: d7e37bd29573493b9da8c0483ea5f28f, State: Running, Role: LEADER I0312 20:11:54.222060 4874 consensus_queue.cc:196] T 6ba89589faf649d6bc84738d4904ab46 P d7e37bd29573493b9da8c0483ea5f28f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6f9b014d344497bdbeb7e336ea5185" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 38269 } } peers { permanent_uuid: "63a9ad19ea5747a8a2bf8e82823284cb" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 59549 } } peers { permanent_uuid: "d7e37bd29573493b9da8c0483ea5f28f" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 40547 } } I0312 20:11:54.227110 4430 catalog_manager.cc:3592] T 6ba89589faf649d6bc84738d4904ab46 P d7e37bd29573493b9da8c0483ea5f28f reported cstate change: term changed from 0 to 1, leader changed from to d7e37bd29573493b9da8c0483ea5f28f (127.0.53.1). New cstate: current_term: 1 leader_uuid: "d7e37bd29573493b9da8c0483ea5f28f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6f9b014d344497bdbeb7e336ea5185" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 38269 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "63a9ad19ea5747a8a2bf8e82823284cb" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 59549 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7e37bd29573493b9da8c0483ea5f28f" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 40547 } health_report { overall_health: HEALTHY } } } I0312 20:11:55.252864 4553 tablet_service.cc:1133] Received LeaderStepDown RPC: tablet_id: "6ba89589faf649d6bc84738d4904ab46" dest_uuid: "d7e37bd29573493b9da8c0483ea5f28f" from {username='slave'} at 127.0.0.1:42537 I0312 20:11:55.253197 4553 raft_consensus.cc:531] T 6ba89589faf649d6bc84738d4904ab46 P d7e37bd29573493b9da8c0483ea5f28f [term 1 LEADER]: Received request to step down I0312 20:11:55.253334 4553 raft_consensus.cc:605] T 6ba89589faf649d6bc84738d4904ab46 P d7e37bd29573493b9da8c0483ea5f28f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d7e37bd29573493b9da8c0483ea5f28f, State: Running, Role: LEADER I0312 20:11:55.253582 4553 consensus_queue.cc:219] T 6ba89589faf649d6bc84738d4904ab46 P d7e37bd29573493b9da8c0483ea5f28f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6f9b014d344497bdbeb7e336ea5185" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 38269 } } peers { permanent_uuid: "63a9ad19ea5747a8a2bf8e82823284cb" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 59549 } } peers { permanent_uuid: "d7e37bd29573493b9da8c0483ea5f28f" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 40547 } } I0312 20:11:55.253988 4553 consensus_peers.cc:458] T 6ba89589faf649d6bc84738d4904ab46 P d7e37bd29573493b9da8c0483ea5f28f -> Peer 63a9ad19ea5747a8a2bf8e82823284cb (127.0.53.3:59549): Closing peer: 63a9ad19ea5747a8a2bf8e82823284cb I0312 20:11:55.254122 4553 consensus_peers.cc:458] T 6ba89589faf649d6bc84738d4904ab46 P d7e37bd29573493b9da8c0483ea5f28f -> Peer eb6f9b014d344497bdbeb7e336ea5185 (127.0.53.2:38269): Closing peer: eb6f9b014d344497bdbeb7e336ea5185 I0312 20:11:55.255411 212 external_mini_cluster.cc:997] Killing /tmp/dist-test-task3bwPeE/build/tsan/bin/kudu-tserver with pid 4464 W0312 20:11:55.275357 4411 connection.cc:513] server connection from 127.0.53.1:60554 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0312 20:11:55.275380 4401 connection.cc:513] client connection to 127.0.53.1:40547 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0312 20:11:55.275338 4412 connection.cc:513] client connection to 127.0.53.1:40547 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0312 20:11:55.276234 4398 connection.cc:513] client connection to 127.0.53.1:40547 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) I0312 20:11:55.277287 212 external_mini_cluster.cc:997] Killing /tmp/dist-test-task3bwPeE/build/tsan/bin/kudu-tserver with pid 4600 W0312 20:11:55.294665 4415 connection.cc:513] client connection to 127.0.53.2:38269 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) I0312 20:11:55.294837 212 external_mini_cluster.cc:997] Killing /tmp/dist-test-task3bwPeE/build/tsan/bin/kudu-tserver with pid 4725 W0312 20:11:55.295434 4413 connection.cc:513] server connection from 127.0.53.2:51629 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0312 20:11:55.296010 4401 connection.cc:513] client connection to 127.0.53.2:38269 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0312 20:11:55.311990 4401 connection.cc:513] client connection to 127.0.53.3:59549 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0312 20:11:55.312070 4415 connection.cc:513] server connection from 127.0.53.3:50348 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0312 20:11:55.312093 4411 connection.cc:513] client connection to 127.0.53.3:59549 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) I0312 20:11:55.313014 212 external_mini_cluster.cc:997] Killing /tmp/dist-test-task3bwPeE/build/tsan/bin/kudu-master with pid 4402 W0312 20:11:55.338749 4401 connection.cc:513] client connection to 127.0.53.62:50068 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) [ OK ] RaftConsensusElectionITest.StepDownWithSlowFollower (3857 ms) [ RUN ] RaftConsensusElectionITest.ElectPendingVoter I0312 20:11:55.352612 212 test_util.cc:192] Using random seed: -1058866680 I0312 20:11:55.356086 212 ts_itest-base.cc:113] Starting cluster with: I0312 20:11:55.356184 212 ts_itest-base.cc:114] -------------- I0312 20:11:55.356264 212 ts_itest-base.cc:115] 5 tablet servers I0312 20:11:55.356341 212 ts_itest-base.cc:116] 5 replicas per TS I0312 20:11:55.356413 212 ts_itest-base.cc:117] -------------- I0312 20:11:55.358563 212 external_mini_cluster.cc:805] Running /tmp/dist-test-task3bwPeE/build/tsan/bin/kudu-master /tmp/dist-test-task3bwPeE/build/tsan/bin/kudu-master --fs_wal_dir=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ElectPendingVoter.1520885401735786-212/raft_consensus-itest-cluster/master-0/wal --fs_data_dirs=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ElectPendingVoter.1520885401735786-212/raft_consensus-itest-cluster/master-0/data --block_manager=log --webserver_interface=localhost --ipki_ca_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.0.53.62:0 --webserver_interface=127.0.53.62 --webserver_port=0 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ElectPendingVoter.1520885401735786-212/raft_consensus-itest-cluster/master-0/logs --server_dump_info_path=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ElectPendingVoter.1520885401735786-212/raft_consensus-itest-cluster/master-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --catalog_manager_wait_for_new_tablets_to_elect_leader=false with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W0312 20:11:55.477737 4886 flags.cc:412] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0312 20:11:55.478041 4886 flags.cc:412] Enabled unsafe flag: --never_fsync=true W0312 20:11:55.493438 4886 flags.cc:412] Enabled experimental flag: --ipki_ca_key_size=1024 W0312 20:11:55.493562 4886 flags.cc:412] Enabled experimental flag: --ipki_server_key_size=1024 W0312 20:11:55.493659 4886 flags.cc:412] Enabled experimental flag: --tsk_num_rsa_bits=512 I0312 20:11:55.514231 4886 master_main.cc:66] Master server non-default flags: --fs_data_dirs=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ElectPendingVoter.1520885401735786-212/raft_consensus-itest-cluster/master-0/data --fs_wal_dir=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ElectPendingVoter.1520885401735786-212/raft_consensus-itest-cluster/master-0/wal --catalog_manager_wait_for_new_tablets_to_elect_leader=false --ipki_ca_key_size=1024 --ipki_server_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.0.53.62:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ElectPendingVoter.1520885401735786-212/raft_consensus-itest-cluster/master-0/data/info.pb --webserver_interface=127.0.53.62 --webserver_port=0 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --log_dir=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ElectPendingVoter.1520885401735786-212/raft_consensus-itest-cluster/master-0/logs --logbuflevel=-1 --logtostderr=true Master server version: kudu 1.7.0-SNAPSHOT revision 13e10c63c46de30ac85f34b543cdd6fb44881f61 build type FASTDEBUG built by None at 12 Mar 2018 20:06:40 UTC on jenkins-slave-vn0g build id 12388 TSAN enabled I0312 20:11:55.523247 4886 master_main.cc:73] Initializing master server... I0312 20:11:55.524752 4886 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 1124776us I0312 20:11:55.525463 4886 fs_manager.cc:260] Metadata directory not provided I0312 20:11:55.525598 4886 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0312 20:11:55.525806 4886 server_base.cc:434] Could not load existing FS layout: Not found: /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ElectPendingVoter.1520885401735786-212/raft_consensus-itest-cluster/master-0/data/instance: No such file or directory (error 2) I0312 20:11:55.525918 4886 server_base.cc:435] Attempting to create new FS layout instead I0312 20:11:55.533139 4886 fs_manager.cc:596] Generated new instance metadata in path /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ElectPendingVoter.1520885401735786-212/raft_consensus-itest-cluster/master-0/data/instance: uuid: "12171b7294364fbf989df8f52b83108b" format_stamp: "Formatted at 2018-03-12 20:11:55 on dist-test-slave-dist-test-slave-f27p" I0312 20:11:55.533828 4886 fs_manager.cc:596] Generated new instance metadata in path /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ElectPendingVoter.1520885401735786-212/raft_consensus-itest-cluster/master-0/wal/instance: uuid: "12171b7294364fbf989df8f52b83108b" format_stamp: "Formatted at 2018-03-12 20:11:55 on dist-test-slave-dist-test-slave-f27p" I0312 20:11:55.542035 4886 fs_manager.cc:496] Time spent creating directory manager: real 0.008s user 0.004s sys 0.004s I0312 20:11:55.543169 4886 env_posix.cc:1620] Not raising this process' open files per process limit of 65536; it is already as high as it can go I0312 20:11:55.543440 4886 file_cache.cc:470] Constructed file cache lbm with capacity 26214 I0312 20:11:55.549829 4886 fs_manager.cc:418] Time spent opening block manager: real 0.002s user 0.000s sys 0.004s I0312 20:11:55.549942 4886 fs_manager.cc:429] Opened local filesystem: /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ElectPendingVoter.1520885401735786-212/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ElectPendingVoter.1520885401735786-212/raft_consensus-itest-cluster/master-0/wal uuid: "12171b7294364fbf989df8f52b83108b" format_stamp: "Formatted at 2018-03-12 20:11:55 on dist-test-slave-dist-test-slave-f27p" I0312 20:11:55.550139 4886 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ElectPendingVoter.1520885401735786-212/raft_consensus-itest-cluster/master-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0312 20:11:55.722945 4886 env_posix.cc:1620] Not raising this process' running threads per effective uid limit of 60284; it is already as high as it can go I0312 20:11:55.723551 4886 master_main.cc:76] Starting Master server... I0312 20:11:55.724783 4903 process_memory.cc:182] Process hard memory limit is 11.777631 GB I0312 20:11:55.724913 4903 process_memory.cc:184] Process soft memory limit is 9.422105 GB I0312 20:11:55.725013 4903 process_memory.cc:187] Process memory pressure threshold is 7.066578 GB I0312 20:11:55.763068 4886 rpc_server.cc:201] RPC server started. Bound to: 127.0.53.62:41398 I0312 20:11:55.788501 4886 webserver.cc:173] Starting webserver on 127.0.53.62:0 I0312 20:11:55.788607 4886 webserver.cc:184] Document root disabled I0312 20:11:55.789855 4886 webserver.cc:311] Webserver started. Bound to: http://127.0.53.62:42743/ I0312 20:11:55.790598 4886 server_base.cc:607] Dumped server information to /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ElectPendingVoter.1520885401735786-212/raft_consensus-itest-cluster/master-0/data/info.pb I0312 20:11:55.791067 212 external_mini_cluster.cc:867] Started /tmp/dist-test-task3bwPeE/build/tsan/bin/kudu-master as pid 4886 I0312 20:11:55.792369 212 external_mini_cluster.cc:805] Running /tmp/dist-test-task3bwPeE/build/tsan/bin/kudu-tserver /tmp/dist-test-task3bwPeE/build/tsan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ElectPendingVoter.1520885401735786-212/raft_consensus-itest-cluster/ts-0/wal --fs_data_dirs=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ElectPendingVoter.1520885401735786-212/raft_consensus-itest-cluster/ts-0/data --block_manager=log --rpc_bind_addresses=127.0.53.1:0 --local_ip_for_outbound_sockets=127.0.53.1 --webserver_interface=127.0.53.1 --webserver_port=0 --tserver_master_addrs=127.0.53.62:41398 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ElectPendingVoter.1520885401735786-212/raft_consensus-itest-cluster/ts-0/logs --server_dump_info_path=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ElectPendingVoter.1520885401735786-212/raft_consensus-itest-cluster/ts-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --enable_leader_failure_detection=false with env {} I0312 20:11:55.793913 4947 data_dirs.cc:931] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0312 20:11:55.808809 4947 tablet_bootstrap.cc:436] T 00000000000000000000000000000000 P 12171b7294364fbf989df8f52b83108b: Bootstrap starting. I0312 20:11:55.811074 4947 tablet_bootstrap.cc:581] T 00000000000000000000000000000000 P 12171b7294364fbf989df8f52b83108b: No blocks or log segments found. Creating new log. I0312 20:11:55.812249 4947 log.cc:520] T 00000000000000000000000000000000 P 12171b7294364fbf989df8f52b83108b: Log is configured to *not* fsync() on all Append() calls I0312 20:11:55.814661 4947 tablet_bootstrap.cc:436] T 00000000000000000000000000000000 P 12171b7294364fbf989df8f52b83108b: No bootstrap required, opened a new log I0312 20:11:55.826534 4947 raft_consensus.cc:304] T 00000000000000000000000000000000 P 12171b7294364fbf989df8f52b83108b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12171b7294364fbf989df8f52b83108b" member_type: VOTER } I0312 20:11:55.826848 4947 raft_consensus.cc:330] T 00000000000000000000000000000000 P 12171b7294364fbf989df8f52b83108b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0312 20:11:55.826972 4947 raft_consensus.cc:605] T 00000000000000000000000000000000 P 12171b7294364fbf989df8f52b83108b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12171b7294364fbf989df8f52b83108b, State: Initialized, Role: FOLLOWER I0312 20:11:55.827404 4947 consensus_queue.cc:219] T 00000000000000000000000000000000 P 12171b7294364fbf989df8f52b83108b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12171b7294364fbf989df8f52b83108b" member_type: VOTER } I0312 20:11:55.827759 4947 raft_consensus.cc:344] T 00000000000000000000000000000000 P 12171b7294364fbf989df8f52b83108b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0312 20:11:55.827903 4947 raft_consensus.cc:436] T 00000000000000000000000000000000 P 12171b7294364fbf989df8f52b83108b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0312 20:11:55.828043 4947 raft_consensus.cc:2700] T 00000000000000000000000000000000 P 12171b7294364fbf989df8f52b83108b: Snoozing failure detection for 2.108s (starting election) I0312 20:11:55.828179 4947 raft_consensus.cc:2752] T 00000000000000000000000000000000 P 12171b7294364fbf989df8f52b83108b [term 0 FOLLOWER]: Advancing to term 1 I0312 20:11:55.830597 4947 raft_consensus.cc:458] T 00000000000000000000000000000000 P 12171b7294364fbf989df8f52b83108b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12171b7294364fbf989df8f52b83108b" member_type: VOTER } I0312 20:11:55.831017 4947 leader_election.cc:258] T 00000000000000000000000000000000 P 12171b7294364fbf989df8f52b83108b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0312 20:11:55.832595 4950 raft_consensus.cc:2700] T 00000000000000000000000000000000 P 12171b7294364fbf989df8f52b83108b: Snoozing failure detection for 1.547s (election complete) I0312 20:11:55.832893 4950 raft_consensus.cc:2507] T 00000000000000000000000000000000 P 12171b7294364fbf989df8f52b83108b [term 1 FOLLOWER]: Leader election won for term 1 I0312 20:11:55.841557 4950 raft_consensus.cc:572] T 00000000000000000000000000000000 P 12171b7294364fbf989df8f52b83108b [term 1 LEADER]: Becoming Leader. State: Replica: 12171b7294364fbf989df8f52b83108b, State: Running, Role: LEADER I0312 20:11:55.841949 4950 consensus_queue.cc:196] T 00000000000000000000000000000000 P 12171b7294364fbf989df8f52b83108b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12171b7294364fbf989df8f52b83108b" member_type: VOTER } I0312 20:11:55.844954 4947 sys_catalog.cc:335] T 00000000000000000000000000000000 P 12171b7294364fbf989df8f52b83108b [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 1 leader_uuid: "12171b7294364fbf989df8f52b83108b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12171b7294364fbf989df8f52b83108b" member_type: VOTER } } I0312 20:11:55.845432 4947 sys_catalog.cc:338] T 00000000000000000000000000000000 P 12171b7294364fbf989df8f52b83108b [sys.catalog]: This master's current role is: LEADER I0312 20:11:55.848175 4951 sys_catalog.cc:335] T 00000000000000000000000000000000 P 12171b7294364fbf989df8f52b83108b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "12171b7294364fbf989df8f52b83108b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12171b7294364fbf989df8f52b83108b" member_type: VOTER } } I0312 20:11:55.848632 4951 sys_catalog.cc:338] T 00000000000000000000000000000000 P 12171b7294364fbf989df8f52b83108b [sys.catalog]: This master's current role is: LEADER I0312 20:11:55.848179 4952 sys_catalog.cc:335] T 00000000000000000000000000000000 P 12171b7294364fbf989df8f52b83108b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 12171b7294364fbf989df8f52b83108b. Latest consensus state: current_term: 1 leader_uuid: "12171b7294364fbf989df8f52b83108b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12171b7294364fbf989df8f52b83108b" member_type: VOTER } } I0312 20:11:55.848837 4952 sys_catalog.cc:338] T 00000000000000000000000000000000 P 12171b7294364fbf989df8f52b83108b [sys.catalog]: This master's current role is: LEADER I0312 20:11:55.849133 4947 sys_catalog.cc:422] T 00000000000000000000000000000000 P 12171b7294364fbf989df8f52b83108b [sys.catalog]: configured and running, proceeding with master startup. I0312 20:11:55.849364 4955 catalog_manager.cc:972] Loading table and tablet metadata into memory... I0312 20:11:55.850939 4886 master_main.cc:79] Master server successfully started. I0312 20:11:55.854277 4955 catalog_manager.cc:983] Initializing Kudu internal certificate authority... I0312 20:11:55.887574 4955 catalog_manager.cc:870] Generated new certificate authority record I0312 20:11:55.889798 4955 catalog_manager.cc:992] Loading token signing keys... I0312 20:11:55.903069 4955 catalog_manager.cc:3868] T 00000000000000000000000000000000 P 12171b7294364fbf989df8f52b83108b: Generated new TSK 0 WARNING: Logging before InitGoogleLogging() is written to STDERR W0312 20:11:55.933475 4948 flags.cc:412] Enabled unsafe flag: --enable_leader_failure_detection=false W0312 20:11:55.933802 4948 flags.cc:412] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0312 20:11:55.933975 4948 flags.cc:412] Enabled unsafe flag: --never_fsync=true W0312 20:11:55.948024 4948 flags.cc:412] Enabled experimental flag: --ipki_server_key_size=1024 W0312 20:11:55.948315 4948 flags.cc:412] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.53.1 I0312 20:11:55.967288 4948 tablet_server_main.cc:72] Tablet server non-default flags: --enable_leader_failure_detection=false --fs_data_dirs=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ElectPendingVoter.1520885401735786-212/raft_consensus-itest-cluster/ts-0/data --fs_wal_dir=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ElectPendingVoter.1520885401735786-212/raft_consensus-itest-cluster/ts-0/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.0.53.1:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ElectPendingVoter.1520885401735786-212/raft_consensus-itest-cluster/ts-0/data/info.pb --webserver_interface=127.0.53.1 --webserver_port=0 --tserver_master_addrs=127.0.53.62:41398 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.0.53.1 --log_dir=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ElectPendingVoter.1520885401735786-212/raft_consensus-itest-cluster/ts-0/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.7.0-SNAPSHOT revision 13e10c63c46de30ac85f34b543cdd6fb44881f61 build type FASTDEBUG built by None at 12 Mar 2018 20:06:40 UTC on jenkins-slave-vn0g build id 12388 TSAN enabled I0312 20:11:55.977289 4948 tablet_server_main.cc:79] Initializing tablet server... I0312 20:11:55.979009 4948 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 1124776us I0312 20:11:55.979439 4948 fs_manager.cc:260] Metadata directory not provided I0312 20:11:55.979558 4948 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0312 20:11:55.979758 4948 server_base.cc:434] Could not load existing FS layout: Not found: /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ElectPendingVoter.1520885401735786-212/raft_consensus-itest-cluster/ts-0/data/instance: No such file or directory (error 2) I0312 20:11:55.979868 4948 server_base.cc:435] Attempting to create new FS layout instead I0312 20:11:55.987015 4948 fs_manager.cc:596] Generated new instance metadata in path /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ElectPendingVoter.1520885401735786-212/raft_consensus-itest-cluster/ts-0/data/instance: uuid: "704c3a8d0e764c66879b2e52d812ffdf" format_stamp: "Formatted at 2018-03-12 20:11:55 on dist-test-slave-dist-test-slave-f27p" I0312 20:11:55.987705 4948 fs_manager.cc:596] Generated new instance metadata in path /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ElectPendingVoter.1520885401735786-212/raft_consensus-itest-cluster/ts-0/wal/instance: uuid: "704c3a8d0e764c66879b2e52d812ffdf" format_stamp: "Formatted at 2018-03-12 20:11:55 on dist-test-slave-dist-test-slave-f27p" I0312 20:11:55.996145 4948 fs_manager.cc:496] Time spent creating directory manager: real 0.008s user 0.008s sys 0.004s I0312 20:11:55.997259 4948 env_posix.cc:1620] Not raising this process' open files per process limit of 65536; it is already as high as it can go I0312 20:11:55.997524 4948 file_cache.cc:470] Constructed file cache lbm with capacity 26214 I0312 20:11:56.003947 4948 fs_manager.cc:418] Time spent opening block manager: real 0.002s user 0.000s sys 0.000s I0312 20:11:56.004072 4948 fs_manager.cc:429] Opened local filesystem: /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ElectPendingVoter.1520885401735786-212/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ElectPendingVoter.1520885401735786-212/raft_consensus-itest-cluster/ts-0/wal uuid: "704c3a8d0e764c66879b2e52d812ffdf" format_stamp: "Formatted at 2018-03-12 20:11:55 on dist-test-slave-dist-test-slave-f27p" I0312 20:11:56.004271 4948 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ElectPendingVoter.1520885401735786-212/raft_consensus-itest-cluster/ts-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0312 20:11:56.200487 4948 env_posix.cc:1620] Not raising this process' running threads per effective uid limit of 60284; it is already as high as it can go I0312 20:11:56.201771 4948 ts_tablet_manager.cc:349] Loaded tablet metadata (0 live tablets) I0312 20:11:56.202968 4948 tablet_server_main.cc:84] Starting tablet server... I0312 20:11:56.307152 4948 rpc_server.cc:201] RPC server started. Bound to: 127.0.53.1:34688 I0312 20:11:56.308040 4948 webserver.cc:173] Starting webserver on 127.0.53.1:0 I0312 20:11:56.308153 4948 webserver.cc:184] Document root disabled I0312 20:11:56.309598 4948 webserver.cc:311] Webserver started. Bound to: http://127.0.53.1:43810/ I0312 20:11:56.310333 4948 server_base.cc:607] Dumped server information to /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ElectPendingVoter.1520885401735786-212/raft_consensus-itest-cluster/ts-0/data/info.pb I0312 20:11:56.318895 212 external_mini_cluster.cc:867] Started /tmp/dist-test-task3bwPeE/build/tsan/bin/kudu-tserver as pid 4948 I0312 20:11:56.318687 5080 process_memory.cc:182] Process hard memory limit is 11.777631 GB I0312 20:11:56.319780 5080 process_memory.cc:184] Process soft memory limit is 9.422105 GB I0312 20:11:56.319955 5080 process_memory.cc:187] Process memory pressure threshold is 7.066578 GB I0312 20:11:56.320183 212 external_mini_cluster.cc:805] Running /tmp/dist-test-task3bwPeE/build/tsan/bin/kudu-tserver /tmp/dist-test-task3bwPeE/build/tsan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ElectPendingVoter.1520885401735786-212/raft_consensus-itest-cluster/ts-1/wal --fs_data_dirs=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ElectPendingVoter.1520885401735786-212/raft_consensus-itest-cluster/ts-1/data --block_manager=log --rpc_bind_addresses=127.0.53.2:0 --local_ip_for_outbound_sockets=127.0.53.2 --webserver_interface=127.0.53.2 --webserver_port=0 --tserver_master_addrs=127.0.53.62:41398 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ElectPendingVoter.1520885401735786-212/raft_consensus-itest-cluster/ts-1/logs --server_dump_info_path=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ElectPendingVoter.1520885401735786-212/raft_consensus-itest-cluster/ts-1/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --enable_leader_failure_detection=false with env {} I0312 20:11:56.346848 5079 heartbeater.cc:346] Connected to a master server at 127.0.53.62:41398 I0312 20:11:56.347133 5079 heartbeater.cc:426] Registering TS with master... I0312 20:11:56.347667 5079 heartbeater.cc:475] Master 127.0.53.62:41398 requested a full tablet report, sending... I0312 20:11:56.320178 4948 tablet_server_main.cc:87] Tablet server successfully started. I0312 20:11:56.349592 4914 ts_manager.cc:81] Registered new tserver with Master: 704c3a8d0e764c66879b2e52d812ffdf (127.0.53.1:34688) I0312 20:11:56.351248 4914 master_service.cc:243] Signed X509 certificate for tserver {username='slave'} at 127.0.53.1:40727 WARNING: Logging before InitGoogleLogging() is written to STDERR W0312 20:11:56.504238 5083 flags.cc:412] Enabled unsafe flag: --enable_leader_failure_detection=false W0312 20:11:56.504688 5083 flags.cc:412] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0312 20:11:56.504964 5083 flags.cc:412] Enabled unsafe flag: --never_fsync=true W0312 20:11:56.528904 5083 flags.cc:412] Enabled experimental flag: --ipki_server_key_size=1024 W0312 20:11:56.529368 5083 flags.cc:412] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.53.2 I0312 20:11:56.557112 5083 tablet_server_main.cc:72] Tablet server non-default flags: --enable_leader_failure_detection=false --fs_data_dirs=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ElectPendingVoter.1520885401735786-212/raft_consensus-itest-cluster/ts-1/data --fs_wal_dir=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ElectPendingVoter.1520885401735786-212/raft_consensus-itest-cluster/ts-1/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.0.53.2:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ElectPendingVoter.1520885401735786-212/raft_consensus-itest-cluster/ts-1/data/info.pb --webserver_interface=127.0.53.2 --webserver_port=0 --tserver_master_addrs=127.0.53.62:41398 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.0.53.2 --log_dir=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ElectPendingVoter.1520885401735786-212/raft_consensus-itest-cluster/ts-1/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.7.0-SNAPSHOT revision 13e10c63c46de30ac85f34b543cdd6fb44881f61 build type FASTDEBUG built by None at 12 Mar 2018 20:06:40 UTC on jenkins-slave-vn0g build id 12388 TSAN enabled I0312 20:11:56.572139 5083 tablet_server_main.cc:79] Initializing tablet server... I0312 20:11:56.574538 5083 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 1125276us I0312 20:11:56.575111 5083 fs_manager.cc:260] Metadata directory not provided I0312 20:11:56.575284 5083 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0312 20:11:56.575562 5083 server_base.cc:434] Could not load existing FS layout: Not found: /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ElectPendingVoter.1520885401735786-212/raft_consensus-itest-cluster/ts-1/data/instance: No such file or directory (error 2) I0312 20:11:56.575723 5083 server_base.cc:435] Attempting to create new FS layout instead I0312 20:11:56.587841 5083 fs_manager.cc:596] Generated new instance metadata in path /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ElectPendingVoter.1520885401735786-212/raft_consensus-itest-cluster/ts-1/data/instance: uuid: "eff8b6e74b4044c982cb22453e464f58" format_stamp: "Formatted at 2018-03-12 20:11:56 on dist-test-slave-dist-test-slave-f27p" I0312 20:11:56.588812 5083 fs_manager.cc:596] Generated new instance metadata in path /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ElectPendingVoter.1520885401735786-212/raft_consensus-itest-cluster/ts-1/wal/instance: uuid: "eff8b6e74b4044c982cb22453e464f58" format_stamp: "Formatted at 2018-03-12 20:11:56 on dist-test-slave-dist-test-slave-f27p" I0312 20:11:56.601655 5083 fs_manager.cc:496] Time spent creating directory manager: real 0.012s user 0.012s sys 0.000s I0312 20:11:56.603360 5083 env_posix.cc:1620] Not raising this process' open files per process limit of 65536; it is already as high as it can go I0312 20:11:56.603757 5083 file_cache.cc:470] Constructed file cache lbm with capacity 26214 I0312 20:11:56.615813 5083 fs_manager.cc:418] Time spent opening block manager: real 0.004s user 0.004s sys 0.004s I0312 20:11:56.615952 5083 fs_manager.cc:429] Opened local filesystem: /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ElectPendingVoter.1520885401735786-212/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ElectPendingVoter.1520885401735786-212/raft_consensus-itest-cluster/ts-1/wal uuid: "eff8b6e74b4044c982cb22453e464f58" format_stamp: "Formatted at 2018-03-12 20:11:56 on dist-test-slave-dist-test-slave-f27p" I0312 20:11:56.616189 5083 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ElectPendingVoter.1520885401735786-212/raft_consensus-itest-cluster/ts-1/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0312 20:11:56.830950 5083 env_posix.cc:1620] Not raising this process' running threads per effective uid limit of 60284; it is already as high as it can go I0312 20:11:56.832239 5083 ts_tablet_manager.cc:349] Loaded tablet metadata (0 live tablets) I0312 20:11:56.833411 5083 tablet_server_main.cc:84] Starting tablet server... I0312 20:11:56.938323 5083 rpc_server.cc:201] RPC server started. Bound to: 127.0.53.2:50867 I0312 20:11:56.939240 5083 webserver.cc:173] Starting webserver on 127.0.53.2:0 I0312 20:11:56.939353 5083 webserver.cc:184] Document root disabled I0312 20:11:56.940718 5083 webserver.cc:311] Webserver started. Bound to: http://127.0.53.2:50632/ I0312 20:11:56.941483 5083 server_base.cc:607] Dumped server information to /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ElectPendingVoter.1520885401735786-212/raft_consensus-itest-cluster/ts-1/data/info.pb I0312 20:11:56.948824 212 external_mini_cluster.cc:867] Started /tmp/dist-test-task3bwPeE/build/tsan/bin/kudu-tserver as pid 5083 I0312 20:11:56.949774 212 external_mini_cluster.cc:805] Running /tmp/dist-test-task3bwPeE/build/tsan/bin/kudu-tserver /tmp/dist-test-task3bwPeE/build/tsan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ElectPendingVoter.1520885401735786-212/raft_consensus-itest-cluster/ts-2/wal --fs_data_dirs=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ElectPendingVoter.1520885401735786-212/raft_consensus-itest-cluster/ts-2/data --block_manager=log --rpc_bind_addresses=127.0.53.3:0 --local_ip_for_outbound_sockets=127.0.53.3 --webserver_interface=127.0.53.3 --webserver_port=0 --tserver_master_addrs=127.0.53.62:41398 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ElectPendingVoter.1520885401735786-212/raft_consensus-itest-cluster/ts-2/logs --server_dump_info_path=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ElectPendingVoter.1520885401735786-212/raft_consensus-itest-cluster/ts-2/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --enable_leader_failure_detection=false with env {} I0312 20:11:56.955327 5083 tablet_server_main.cc:87] Tablet server successfully started. I0312 20:11:56.995208 5204 heartbeater.cc:346] Connected to a master server at 127.0.53.62:41398 I0312 20:11:56.995529 5204 heartbeater.cc:426] Registering TS with master... I0312 20:11:56.996112 5204 heartbeater.cc:475] Master 127.0.53.62:41398 requested a full tablet report, sending... I0312 20:11:56.997892 4914 ts_manager.cc:81] Registered new tserver with Master: eff8b6e74b4044c982cb22453e464f58 (127.0.53.2:50867) I0312 20:11:56.999475 4914 master_service.cc:243] Signed X509 certificate for tserver {username='slave'} at 127.0.53.2:56349 I0312 20:11:56.969537 5205 process_memory.cc:182] Process hard memory limit is 11.777631 GB I0312 20:11:57.005882 5205 process_memory.cc:184] Process soft memory limit is 9.422105 GB I0312 20:11:57.005995 5205 process_memory.cc:187] Process memory pressure threshold is 7.066578 GB WARNING: Logging before InitGoogleLogging() is written to STDERR W0312 20:11:57.083803 5208 flags.cc:412] Enabled unsafe flag: --enable_leader_failure_detection=false W0312 20:11:57.084156 5208 flags.cc:412] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0312 20:11:57.084353 5208 flags.cc:412] Enabled unsafe flag: --never_fsync=true W0312 20:11:57.098577 5208 flags.cc:412] Enabled experimental flag: --ipki_server_key_size=1024 W0312 20:11:57.098888 5208 flags.cc:412] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.53.3 I0312 20:11:57.118598 5208 tablet_server_main.cc:72] Tablet server non-default flags: --enable_leader_failure_detection=false --fs_data_dirs=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ElectPendingVoter.1520885401735786-212/raft_consensus-itest-cluster/ts-2/data --fs_wal_dir=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ElectPendingVoter.1520885401735786-212/raft_consensus-itest-cluster/ts-2/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.0.53.3:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ElectPendingVoter.1520885401735786-212/raft_consensus-itest-cluster/ts-2/data/info.pb --webserver_interface=127.0.53.3 --webserver_port=0 --tserver_master_addrs=127.0.53.62:41398 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.0.53.3 --log_dir=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ElectPendingVoter.1520885401735786-212/raft_consensus-itest-cluster/ts-2/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.7.0-SNAPSHOT revision 13e10c63c46de30ac85f34b543cdd6fb44881f61 build type FASTDEBUG built by None at 12 Mar 2018 20:06:40 UTC on jenkins-slave-vn0g build id 12388 TSAN enabled I0312 20:11:57.129379 5208 tablet_server_main.cc:79] Initializing tablet server... I0312 20:11:57.131152 5208 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 1125776us I0312 20:11:57.131582 5208 fs_manager.cc:260] Metadata directory not provided I0312 20:11:57.131700 5208 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0312 20:11:57.131912 5208 server_base.cc:434] Could not load existing FS layout: Not found: /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ElectPendingVoter.1520885401735786-212/raft_consensus-itest-cluster/ts-2/data/instance: No such file or directory (error 2) I0312 20:11:57.132025 5208 server_base.cc:435] Attempting to create new FS layout instead I0312 20:11:57.139358 5208 fs_manager.cc:596] Generated new instance metadata in path /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ElectPendingVoter.1520885401735786-212/raft_consensus-itest-cluster/ts-2/data/instance: uuid: "876c901434c04ed4ab4bfbf51b338b96" format_stamp: "Formatted at 2018-03-12 20:11:57 on dist-test-slave-dist-test-slave-f27p" I0312 20:11:57.140046 5208 fs_manager.cc:596] Generated new instance metadata in path /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ElectPendingVoter.1520885401735786-212/raft_consensus-itest-cluster/ts-2/wal/instance: uuid: "876c901434c04ed4ab4bfbf51b338b96" format_stamp: "Formatted at 2018-03-12 20:11:57 on dist-test-slave-dist-test-slave-f27p" I0312 20:11:57.148667 5208 fs_manager.cc:496] Time spent creating directory manager: real 0.008s user 0.004s sys 0.008s I0312 20:11:57.149899 5208 env_posix.cc:1620] Not raising this process' open files per process limit of 65536; it is already as high as it can go I0312 20:11:57.150292 5208 file_cache.cc:470] Constructed file cache lbm with capacity 26214 I0312 20:11:57.157043 5208 fs_manager.cc:418] Time spent opening block manager: real 0.002s user 0.000s sys 0.000s I0312 20:11:57.157186 5208 fs_manager.cc:429] Opened local filesystem: /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ElectPendingVoter.1520885401735786-212/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ElectPendingVoter.1520885401735786-212/raft_consensus-itest-cluster/ts-2/wal uuid: "876c901434c04ed4ab4bfbf51b338b96" format_stamp: "Formatted at 2018-03-12 20:11:57 on dist-test-slave-dist-test-slave-f27p" I0312 20:11:57.157400 5208 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ElectPendingVoter.1520885401735786-212/raft_consensus-itest-cluster/ts-2/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0312 20:11:57.353672 5079 heartbeater.cc:467] Master 127.0.53.62:41398 was elected leader, sending a full tablet report... I0312 20:11:57.411140 5208 env_posix.cc:1620] Not raising this process' running threads per effective uid limit of 60284; it is already as high as it can go I0312 20:11:57.412590 5208 ts_tablet_manager.cc:349] Loaded tablet metadata (0 live tablets) I0312 20:11:57.413579 5208 tablet_server_main.cc:84] Starting tablet server... I0312 20:11:57.499433 5208 rpc_server.cc:201] RPC server started. Bound to: 127.0.53.3:58832 I0312 20:11:57.500341 5208 webserver.cc:173] Starting webserver on 127.0.53.3:0 I0312 20:11:57.500449 5208 webserver.cc:184] Document root disabled I0312 20:11:57.501665 5208 webserver.cc:311] Webserver started. Bound to: http://127.0.53.3:43487/ I0312 20:11:57.502411 5208 server_base.cc:607] Dumped server information to /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ElectPendingVoter.1520885401735786-212/raft_consensus-itest-cluster/ts-2/data/info.pb I0312 20:11:57.505004 5208 tablet_server_main.cc:87] Tablet server successfully started. I0312 20:11:57.512585 212 external_mini_cluster.cc:867] Started /tmp/dist-test-task3bwPeE/build/tsan/bin/kudu-tserver as pid 5208 I0312 20:11:57.513888 212 external_mini_cluster.cc:805] Running /tmp/dist-test-task3bwPeE/build/tsan/bin/kudu-tserver /tmp/dist-test-task3bwPeE/build/tsan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ElectPendingVoter.1520885401735786-212/raft_consensus-itest-cluster/ts-3/wal --fs_data_dirs=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ElectPendingVoter.1520885401735786-212/raft_consensus-itest-cluster/ts-3/data --block_manager=log --rpc_bind_addresses=127.0.53.4:0 --local_ip_for_outbound_sockets=127.0.53.4 --webserver_interface=127.0.53.4 --webserver_port=0 --tserver_master_addrs=127.0.53.62:41398 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ElectPendingVoter.1520885401735786-212/raft_consensus-itest-cluster/ts-3/logs --server_dump_info_path=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ElectPendingVoter.1520885401735786-212/raft_consensus-itest-cluster/ts-3/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --enable_leader_failure_detection=false with env {} I0312 20:11:57.524204 5329 heartbeater.cc:346] Connected to a master server at 127.0.53.62:41398 I0312 20:11:57.524461 5329 heartbeater.cc:426] Registering TS with master... I0312 20:11:57.524905 5329 heartbeater.cc:475] Master 127.0.53.62:41398 requested a full tablet report, sending... I0312 20:11:57.526247 4914 ts_manager.cc:81] Registered new tserver with Master: 876c901434c04ed4ab4bfbf51b338b96 (127.0.53.3:58832) I0312 20:11:57.527348 4914 master_service.cc:243] Signed X509 certificate for tserver {username='slave'} at 127.0.53.3:51066 I0312 20:11:57.506364 5330 process_memory.cc:182] Process hard memory limit is 11.777631 GB I0312 20:11:57.531615 5330 process_memory.cc:184] Process soft memory limit is 9.422105 GB I0312 20:11:57.531713 5330 process_memory.cc:187] Process memory pressure threshold is 7.066578 GB WARNING: Logging before InitGoogleLogging() is written to STDERR W0312 20:11:57.633349 5333 flags.cc:412] Enabled unsafe flag: --enable_leader_failure_detection=false W0312 20:11:57.633716 5333 flags.cc:412] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0312 20:11:57.633895 5333 flags.cc:412] Enabled unsafe flag: --never_fsync=true W0312 20:11:57.648020 5333 flags.cc:412] Enabled experimental flag: --ipki_server_key_size=1024 W0312 20:11:57.648326 5333 flags.cc:412] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.53.4 I0312 20:11:57.667271 5333 tablet_server_main.cc:72] Tablet server non-default flags: --enable_leader_failure_detection=false --fs_data_dirs=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ElectPendingVoter.1520885401735786-212/raft_consensus-itest-cluster/ts-3/data --fs_wal_dir=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ElectPendingVoter.1520885401735786-212/raft_consensus-itest-cluster/ts-3/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.0.53.4:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ElectPendingVoter.1520885401735786-212/raft_consensus-itest-cluster/ts-3/data/info.pb --webserver_interface=127.0.53.4 --webserver_port=0 --tserver_master_addrs=127.0.53.62:41398 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.0.53.4 --log_dir=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ElectPendingVoter.1520885401735786-212/raft_consensus-itest-cluster/ts-3/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.7.0-SNAPSHOT revision 13e10c63c46de30ac85f34b543cdd6fb44881f61 build type FASTDEBUG built by None at 12 Mar 2018 20:06:40 UTC on jenkins-slave-vn0g build id 12388 TSAN enabled I0312 20:11:57.676991 5333 tablet_server_main.cc:79] Initializing tablet server... I0312 20:11:57.678685 5333 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 1125776us I0312 20:11:57.679133 5333 fs_manager.cc:260] Metadata directory not provided I0312 20:11:57.679255 5333 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0312 20:11:57.679455 5333 server_base.cc:434] Could not load existing FS layout: Not found: /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ElectPendingVoter.1520885401735786-212/raft_consensus-itest-cluster/ts-3/data/instance: No such file or directory (error 2) I0312 20:11:57.679564 5333 server_base.cc:435] Attempting to create new FS layout instead I0312 20:11:57.686703 5333 fs_manager.cc:596] Generated new instance metadata in path /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ElectPendingVoter.1520885401735786-212/raft_consensus-itest-cluster/ts-3/data/instance: uuid: "6cb3d7538445412e92d6469ad2a62d90" format_stamp: "Formatted at 2018-03-12 20:11:57 on dist-test-slave-dist-test-slave-f27p" I0312 20:11:57.687397 5333 fs_manager.cc:596] Generated new instance metadata in path /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ElectPendingVoter.1520885401735786-212/raft_consensus-itest-cluster/ts-3/wal/instance: uuid: "6cb3d7538445412e92d6469ad2a62d90" format_stamp: "Formatted at 2018-03-12 20:11:57 on dist-test-slave-dist-test-slave-f27p" I0312 20:11:57.695511 5333 fs_manager.cc:496] Time spent creating directory manager: real 0.008s user 0.008s sys 0.000s I0312 20:11:57.696609 5333 env_posix.cc:1620] Not raising this process' open files per process limit of 65536; it is already as high as it can go I0312 20:11:57.696874 5333 file_cache.cc:470] Constructed file cache lbm with capacity 26214 I0312 20:11:57.703172 5333 fs_manager.cc:418] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s I0312 20:11:57.703289 5333 fs_manager.cc:429] Opened local filesystem: /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ElectPendingVoter.1520885401735786-212/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ElectPendingVoter.1520885401735786-212/raft_consensus-itest-cluster/ts-3/wal uuid: "6cb3d7538445412e92d6469ad2a62d90" format_stamp: "Formatted at 2018-03-12 20:11:57 on dist-test-slave-dist-test-slave-f27p" I0312 20:11:57.703480 5333 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ElectPendingVoter.1520885401735786-212/raft_consensus-itest-cluster/ts-3/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0312 20:11:57.909476 5333 env_posix.cc:1620] Not raising this process' running threads per effective uid limit of 60284; it is already as high as it can go I0312 20:11:57.910734 5333 ts_tablet_manager.cc:349] Loaded tablet metadata (0 live tablets) I0312 20:11:57.911748 5333 tablet_server_main.cc:84] Starting tablet server... I0312 20:11:57.999054 5333 rpc_server.cc:201] RPC server started. Bound to: 127.0.53.4:33153 I0312 20:11:57.999954 5333 webserver.cc:173] Starting webserver on 127.0.53.4:0 I0312 20:11:58.000074 5333 webserver.cc:184] Document root disabled I0312 20:11:58.001308 5333 webserver.cc:311] Webserver started. Bound to: http://127.0.53.4:50799/ I0312 20:11:58.002050 5333 server_base.cc:607] Dumped server information to /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ElectPendingVoter.1520885401735786-212/raft_consensus-itest-cluster/ts-3/data/info.pb I0312 20:11:58.002810 5204 heartbeater.cc:467] Master 127.0.53.62:41398 was elected leader, sending a full tablet report... I0312 20:11:58.004757 5333 tablet_server_main.cc:87] Tablet server successfully started. I0312 20:11:58.010562 212 external_mini_cluster.cc:867] Started /tmp/dist-test-task3bwPeE/build/tsan/bin/kudu-tserver as pid 5333 I0312 20:11:58.011864 212 external_mini_cluster.cc:805] Running /tmp/dist-test-task3bwPeE/build/tsan/bin/kudu-tserver /tmp/dist-test-task3bwPeE/build/tsan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ElectPendingVoter.1520885401735786-212/raft_consensus-itest-cluster/ts-4/wal --fs_data_dirs=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ElectPendingVoter.1520885401735786-212/raft_consensus-itest-cluster/ts-4/data --block_manager=log --rpc_bind_addresses=127.0.53.5:0 --local_ip_for_outbound_sockets=127.0.53.5 --webserver_interface=127.0.53.5 --webserver_port=0 --tserver_master_addrs=127.0.53.62:41398 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ElectPendingVoter.1520885401735786-212/raft_consensus-itest-cluster/ts-4/logs --server_dump_info_path=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ElectPendingVoter.1520885401735786-212/raft_consensus-itest-cluster/ts-4/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --enable_leader_failure_detection=false with env {} I0312 20:11:58.006291 5455 process_memory.cc:182] Process hard memory limit is 11.777631 GB I0312 20:11:58.026268 5455 process_memory.cc:184] Process soft memory limit is 9.422105 GB I0312 20:11:58.026383 5455 process_memory.cc:187] Process memory pressure threshold is 7.066578 GB I0312 20:11:58.028173 5454 heartbeater.cc:346] Connected to a master server at 127.0.53.62:41398 I0312 20:11:58.028410 5454 heartbeater.cc:426] Registering TS with master... I0312 20:11:58.028842 5454 heartbeater.cc:475] Master 127.0.53.62:41398 requested a full tablet report, sending... I0312 20:11:58.030230 4914 ts_manager.cc:81] Registered new tserver with Master: 6cb3d7538445412e92d6469ad2a62d90 (127.0.53.4:33153) I0312 20:11:58.031365 4914 master_service.cc:243] Signed X509 certificate for tserver {username='slave'} at 127.0.53.4:60478 WARNING: Logging before InitGoogleLogging() is written to STDERR W0312 20:11:58.131533 5457 flags.cc:412] Enabled unsafe flag: --enable_leader_failure_detection=false W0312 20:11:58.131844 5457 flags.cc:412] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0312 20:11:58.132024 5457 flags.cc:412] Enabled unsafe flag: --never_fsync=true W0312 20:11:58.146185 5457 flags.cc:412] Enabled experimental flag: --ipki_server_key_size=1024 W0312 20:11:58.146469 5457 flags.cc:412] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.53.5 I0312 20:11:58.165047 5457 tablet_server_main.cc:72] Tablet server non-default flags: --enable_leader_failure_detection=false --fs_data_dirs=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ElectPendingVoter.1520885401735786-212/raft_consensus-itest-cluster/ts-4/data --fs_wal_dir=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ElectPendingVoter.1520885401735786-212/raft_consensus-itest-cluster/ts-4/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.0.53.5:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ElectPendingVoter.1520885401735786-212/raft_consensus-itest-cluster/ts-4/data/info.pb --webserver_interface=127.0.53.5 --webserver_port=0 --tserver_master_addrs=127.0.53.62:41398 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.0.53.5 --log_dir=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ElectPendingVoter.1520885401735786-212/raft_consensus-itest-cluster/ts-4/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.7.0-SNAPSHOT revision 13e10c63c46de30ac85f34b543cdd6fb44881f61 build type FASTDEBUG built by None at 12 Mar 2018 20:06:40 UTC on jenkins-slave-vn0g build id 12388 TSAN enabled I0312 20:11:58.174666 5457 tablet_server_main.cc:79] Initializing tablet server... I0312 20:11:58.176349 5457 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 1126276us I0312 20:11:58.176759 5457 fs_manager.cc:260] Metadata directory not provided I0312 20:11:58.176875 5457 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0312 20:11:58.177083 5457 server_base.cc:434] Could not load existing FS layout: Not found: /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ElectPendingVoter.1520885401735786-212/raft_consensus-itest-cluster/ts-4/data/instance: No such file or directory (error 2) I0312 20:11:58.177191 5457 server_base.cc:435] Attempting to create new FS layout instead I0312 20:11:58.184289 5457 fs_manager.cc:596] Generated new instance metadata in path /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ElectPendingVoter.1520885401735786-212/raft_consensus-itest-cluster/ts-4/data/instance: uuid: "35336ac513e14662ac43dcb2acd85121" format_stamp: "Formatted at 2018-03-12 20:11:58 on dist-test-slave-dist-test-slave-f27p" I0312 20:11:58.185211 5457 fs_manager.cc:596] Generated new instance metadata in path /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ElectPendingVoter.1520885401735786-212/raft_consensus-itest-cluster/ts-4/wal/instance: uuid: "35336ac513e14662ac43dcb2acd85121" format_stamp: "Formatted at 2018-03-12 20:11:58 on dist-test-slave-dist-test-slave-f27p" I0312 20:11:58.193781 5457 fs_manager.cc:496] Time spent creating directory manager: real 0.008s user 0.004s sys 0.004s I0312 20:11:58.194842 5457 env_posix.cc:1620] Not raising this process' open files per process limit of 65536; it is already as high as it can go I0312 20:11:58.195093 5457 file_cache.cc:470] Constructed file cache lbm with capacity 26214 I0312 20:11:58.202193 5457 fs_manager.cc:418] Time spent opening block manager: real 0.002s user 0.004s sys 0.004s I0312 20:11:58.202301 5457 fs_manager.cc:429] Opened local filesystem: /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ElectPendingVoter.1520885401735786-212/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ElectPendingVoter.1520885401735786-212/raft_consensus-itest-cluster/ts-4/wal uuid: "35336ac513e14662ac43dcb2acd85121" format_stamp: "Formatted at 2018-03-12 20:11:58 on dist-test-slave-dist-test-slave-f27p" I0312 20:11:58.202476 5457 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ElectPendingVoter.1520885401735786-212/raft_consensus-itest-cluster/ts-4/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0312 20:11:58.382608 5457 env_posix.cc:1620] Not raising this process' running threads per effective uid limit of 60284; it is already as high as it can go I0312 20:11:58.383832 5457 ts_tablet_manager.cc:349] Loaded tablet metadata (0 live tablets) I0312 20:11:58.384809 5457 tablet_server_main.cc:84] Starting tablet server... I0312 20:11:58.477468 5457 rpc_server.cc:201] RPC server started. Bound to: 127.0.53.5:46654 I0312 20:11:58.478693 5457 webserver.cc:173] Starting webserver on 127.0.53.5:0 I0312 20:11:58.478834 5457 webserver.cc:184] Document root disabled I0312 20:11:58.480286 5457 webserver.cc:311] Webserver started. Bound to: http://127.0.53.5:56467/ I0312 20:11:58.481137 5457 server_base.cc:607] Dumped server information to /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.ElectPendingVoter.1520885401735786-212/raft_consensus-itest-cluster/ts-4/data/info.pb I0312 20:11:58.485618 212 external_mini_cluster.cc:867] Started /tmp/dist-test-task3bwPeE/build/tsan/bin/kudu-tserver as pid 5457 I0312 20:11:58.486807 5457 tablet_server_main.cc:87] Tablet server successfully started. I0312 20:11:58.487586 5579 process_memory.cc:182] Process hard memory limit is 11.777631 GB I0312 20:11:58.517820 5579 process_memory.cc:184] Process soft memory limit is 9.422105 GB I0312 20:11:58.517933 5579 process_memory.cc:187] Process memory pressure threshold is 7.066578 GB I0312 20:11:58.518323 5578 heartbeater.cc:346] Connected to a master server at 127.0.53.62:41398 I0312 20:11:58.518630 5578 heartbeater.cc:426] Registering TS with master... I0312 20:11:58.519186 5578 heartbeater.cc:475] Master 127.0.53.62:41398 requested a full tablet report, sending... I0312 20:11:58.520576 4914 ts_manager.cc:81] Registered new tserver with Master: 35336ac513e14662ac43dcb2acd85121 (127.0.53.5:46654) I0312 20:11:58.521670 4914 master_service.cc:243] Signed X509 certificate for tserver {username='slave'} at 127.0.53.5:40974 I0312 20:11:58.523792 212 external_mini_cluster.cc:498] 5 TS(s) registered with all masters I0312 20:11:58.529642 5329 heartbeater.cc:467] Master 127.0.53.62:41398 was elected leader, sending a full tablet report... I0312 20:11:58.542115 4914 catalog_manager.cc:1294] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46103: name: "TestTable" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 5 split_rows_range_bounds { } partition_schema { range_schema { columns { name: "key" } } } W0312 20:11:58.550503 4914 catalog_manager.cc:1450] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a replica failure: 6 tablet servers are needed, 5 are alive. Consider bringing up additional tablet server(s) or running both the masters and all tablet servers with --raft_prepare_replacement_before_eviction=false flag (not recommended). I0312 20:11:58.611135 5016 tablet_service.cc:755] Processing CreateTablet for tablet 14cfe357f81b40f48e9a6ff8221ae3ed (table=TestTable [id=2c2236a8cb8045509a26326f6145f167]), partition=RANGE (key) PARTITION UNBOUNDED I0312 20:11:58.612067 5016 data_dirs.cc:931] Could only allocate 1 dirs of requested 3 for tablet 14cfe357f81b40f48e9a6ff8221ae3ed. 1 dirs total, 0 dirs full, 0 dirs failed I0312 20:11:58.615773 5391 tablet_service.cc:755] Processing CreateTablet for tablet 14cfe357f81b40f48e9a6ff8221ae3ed (table=TestTable [id=2c2236a8cb8045509a26326f6145f167]), partition=RANGE (key) PARTITION UNBOUNDED I0312 20:11:58.616647 5391 data_dirs.cc:931] Could only allocate 1 dirs of requested 3 for tablet 14cfe357f81b40f48e9a6ff8221ae3ed. 1 dirs total, 0 dirs full, 0 dirs failed I0312 20:11:58.624600 5266 tablet_service.cc:755] Processing CreateTablet for tablet 14cfe357f81b40f48e9a6ff8221ae3ed (table=TestTable [id=2c2236a8cb8045509a26326f6145f167]), partition=RANGE (key) PARTITION UNBOUNDED I0312 20:11:58.625457 5266 data_dirs.cc:931] Could only allocate 1 dirs of requested 3 for tablet 14cfe357f81b40f48e9a6ff8221ae3ed. 1 dirs total, 0 dirs full, 0 dirs failed I0312 20:11:58.625473 5141 tablet_service.cc:755] Processing CreateTablet for tablet 14cfe357f81b40f48e9a6ff8221ae3ed (table=TestTable [id=2c2236a8cb8045509a26326f6145f167]), partition=RANGE (key) PARTITION UNBOUNDED I0312 20:11:58.626307 5141 data_dirs.cc:931] Could only allocate 1 dirs of requested 3 for tablet 14cfe357f81b40f48e9a6ff8221ae3ed. 1 dirs total, 0 dirs full, 0 dirs failed I0312 20:11:58.627084 5515 tablet_service.cc:755] Processing CreateTablet for tablet 14cfe357f81b40f48e9a6ff8221ae3ed (table=TestTable [id=2c2236a8cb8045509a26326f6145f167]), partition=RANGE (key) PARTITION UNBOUNDED I0312 20:11:58.627933 5515 data_dirs.cc:931] Could only allocate 1 dirs of requested 3 for tablet 14cfe357f81b40f48e9a6ff8221ae3ed. 1 dirs total, 0 dirs full, 0 dirs failed I0312 20:11:58.628682 5016 ts_tablet_manager.cc:1163] T 14cfe357f81b40f48e9a6ff8221ae3ed P 704c3a8d0e764c66879b2e52d812ffdf: Registered tablet (data state: TABLET_DATA_READY) I0312 20:11:58.641330 5266 ts_tablet_manager.cc:1163] T 14cfe357f81b40f48e9a6ff8221ae3ed P 876c901434c04ed4ab4bfbf51b338b96: Registered tablet (data state: TABLET_DATA_READY) I0312 20:11:58.643082 5141 ts_tablet_manager.cc:1163] T 14cfe357f81b40f48e9a6ff8221ae3ed P eff8b6e74b4044c982cb22453e464f58: Registered tablet (data state: TABLET_DATA_READY) I0312 20:11:58.651859 5604 ts_tablet_manager.cc:1018] T 14cfe357f81b40f48e9a6ff8221ae3ed P eff8b6e74b4044c982cb22453e464f58: Bootstrapping tablet I0312 20:11:58.652376 5604 tablet_bootstrap.cc:436] T 14cfe357f81b40f48e9a6ff8221ae3ed P eff8b6e74b4044c982cb22453e464f58: Bootstrap starting. I0312 20:11:58.655730 5604 tablet_bootstrap.cc:581] T 14cfe357f81b40f48e9a6ff8221ae3ed P eff8b6e74b4044c982cb22453e464f58: No blocks or log segments found. Creating new log. I0312 20:11:58.655830 5391 ts_tablet_manager.cc:1163] T 14cfe357f81b40f48e9a6ff8221ae3ed P 6cb3d7538445412e92d6469ad2a62d90: Registered tablet (data state: TABLET_DATA_READY) I0312 20:11:58.657027 5515 ts_tablet_manager.cc:1163] T 14cfe357f81b40f48e9a6ff8221ae3ed P 35336ac513e14662ac43dcb2acd85121: Registered tablet (data state: TABLET_DATA_READY) I0312 20:11:58.657507 5604 log.cc:520] T 14cfe357f81b40f48e9a6ff8221ae3ed P eff8b6e74b4044c982cb22453e464f58: Log is configured to *not* fsync() on all Append() calls I0312 20:11:58.660845 5604 tablet_bootstrap.cc:436] T 14cfe357f81b40f48e9a6ff8221ae3ed P eff8b6e74b4044c982cb22453e464f58: No bootstrap required, opened a new log I0312 20:11:58.660907 5606 ts_tablet_manager.cc:1018] T 14cfe357f81b40f48e9a6ff8221ae3ed P 6cb3d7538445412e92d6469ad2a62d90: Bootstrapping tablet I0312 20:11:58.660938 5603 ts_tablet_manager.cc:1018] T 14cfe357f81b40f48e9a6ff8221ae3ed P 876c901434c04ed4ab4bfbf51b338b96: Bootstrapping tablet I0312 20:11:58.661293 5604 ts_tablet_manager.cc:1035] T 14cfe357f81b40f48e9a6ff8221ae3ed P eff8b6e74b4044c982cb22453e464f58: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s I0312 20:11:58.661480 5606 tablet_bootstrap.cc:436] T 14cfe357f81b40f48e9a6ff8221ae3ed P 6cb3d7538445412e92d6469ad2a62d90: Bootstrap starting. I0312 20:11:58.662302 5602 ts_tablet_manager.cc:1018] T 14cfe357f81b40f48e9a6ff8221ae3ed P 704c3a8d0e764c66879b2e52d812ffdf: Bootstrapping tablet I0312 20:11:58.662829 5602 tablet_bootstrap.cc:436] T 14cfe357f81b40f48e9a6ff8221ae3ed P 704c3a8d0e764c66879b2e52d812ffdf: Bootstrap starting. I0312 20:11:58.664036 5606 tablet_bootstrap.cc:581] T 14cfe357f81b40f48e9a6ff8221ae3ed P 6cb3d7538445412e92d6469ad2a62d90: No blocks or log segments found. Creating new log. I0312 20:11:58.665237 5606 log.cc:520] T 14cfe357f81b40f48e9a6ff8221ae3ed P 6cb3d7538445412e92d6469ad2a62d90: Log is configured to *not* fsync() on all Append() calls I0312 20:11:58.665596 5603 tablet_bootstrap.cc:436] T 14cfe357f81b40f48e9a6ff8221ae3ed P 876c901434c04ed4ab4bfbf51b338b96: Bootstrap starting. I0312 20:11:58.666502 5607 ts_tablet_manager.cc:1018] T 14cfe357f81b40f48e9a6ff8221ae3ed P 35336ac513e14662ac43dcb2acd85121: Bootstrapping tablet I0312 20:11:58.666877 5607 tablet_bootstrap.cc:436] T 14cfe357f81b40f48e9a6ff8221ae3ed P 35336ac513e14662ac43dcb2acd85121: Bootstrap starting. I0312 20:11:58.668737 5603 tablet_bootstrap.cc:581] T 14cfe357f81b40f48e9a6ff8221ae3ed P 876c901434c04ed4ab4bfbf51b338b96: No blocks or log segments found. Creating new log. I0312 20:11:58.669143 5607 tablet_bootstrap.cc:581] T 14cfe357f81b40f48e9a6ff8221ae3ed P 35336ac513e14662ac43dcb2acd85121: No blocks or log segments found. Creating new log. I0312 20:11:58.670339 5607 log.cc:520] T 14cfe357f81b40f48e9a6ff8221ae3ed P 35336ac513e14662ac43dcb2acd85121: Log is configured to *not* fsync() on all Append() calls I0312 20:11:58.675019 5602 tablet_bootstrap.cc:581] T 14cfe357f81b40f48e9a6ff8221ae3ed P 704c3a8d0e764c66879b2e52d812ffdf: No blocks or log segments found. Creating new log. I0312 20:11:58.676589 5602 log.cc:520] T 14cfe357f81b40f48e9a6ff8221ae3ed P 704c3a8d0e764c66879b2e52d812ffdf: Log is configured to *not* fsync() on all Append() calls I0312 20:11:58.676610 5604 raft_consensus.cc:304] T 14cfe357f81b40f48e9a6ff8221ae3ed P eff8b6e74b4044c982cb22453e464f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "876c901434c04ed4ab4bfbf51b338b96" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 58832 } } peers { permanent_uuid: "6cb3d7538445412e92d6469ad2a62d90" member_type: VOTER last_known_addr { host: "127.0.53.4" port: 33153 } } peers { permanent_uuid: "704c3a8d0e764c66879b2e52d812ffdf" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 34688 } } peers { permanent_uuid: "35336ac513e14662ac43dcb2acd85121" member_type: VOTER last_known_addr { host: "127.0.53.5" port: 46654 } } peers { permanent_uuid: "eff8b6e74b4044c982cb22453e464f58" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 50867 } } I0312 20:11:58.677650 5604 raft_consensus.cc:605] T 14cfe357f81b40f48e9a6ff8221ae3ed P eff8b6e74b4044c982cb22453e464f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eff8b6e74b4044c982cb22453e464f58, State: Initialized, Role: FOLLOWER I0312 20:11:58.678076 5604 consensus_queue.cc:219] T 14cfe357f81b40f48e9a6ff8221ae3ed P eff8b6e74b4044c982cb22453e464f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "876c901434c04ed4ab4bfbf51b338b96" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 58832 } } peers { permanent_uuid: "6cb3d7538445412e92d6469ad2a62d90" member_type: VOTER last_known_addr { host: "127.0.53.4" port: 33153 } } peers { permanent_uuid: "704c3a8d0e764c66879b2e52d812ffdf" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 34688 } } peers { permanent_uuid: "35336ac513e14662ac43dcb2acd85121" member_type: VOTER last_known_addr { host: "127.0.53.5" port: 46654 } } peers { permanent_uuid: "eff8b6e74b4044c982cb22453e464f58" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 50867 } } I0312 20:11:58.681550 5603 log.cc:520] T 14cfe357f81b40f48e9a6ff8221ae3ed P 876c901434c04ed4ab4bfbf51b338b96: Log is configured to *not* fsync() on all Append() calls I0312 20:11:58.685211 5603 tablet_bootstrap.cc:436] T 14cfe357f81b40f48e9a6ff8221ae3ed P 876c901434c04ed4ab4bfbf51b338b96: No bootstrap required, opened a new log I0312 20:11:58.685621 5603 ts_tablet_manager.cc:1035] T 14cfe357f81b40f48e9a6ff8221ae3ed P 876c901434c04ed4ab4bfbf51b338b96: Time spent bootstrapping tablet: real 0.020s user 0.008s sys 0.000s I0312 20:11:58.685715 5606 tablet_bootstrap.cc:436] T 14cfe357f81b40f48e9a6ff8221ae3ed P 6cb3d7538445412e92d6469ad2a62d90: No bootstrap required, opened a new log I0312 20:11:58.686112 5606 ts_tablet_manager.cc:1035] T 14cfe357f81b40f48e9a6ff8221ae3ed P 6cb3d7538445412e92d6469ad2a62d90: Time spent bootstrapping tablet: real 0.025s user 0.012s sys 0.000s I0312 20:11:58.686264 5607 tablet_bootstrap.cc:436] T 14cfe357f81b40f48e9a6ff8221ae3ed P 35336ac513e14662ac43dcb2acd85121: No bootstrap required, opened a new log I0312 20:11:58.686658 5607 ts_tablet_manager.cc:1035] T 14cfe357f81b40f48e9a6ff8221ae3ed P 35336ac513e14662ac43dcb2acd85121: Time spent bootstrapping tablet: real 0.020s user 0.000s sys 0.008s I0312 20:11:58.698344 5602 tablet_bootstrap.cc:436] T 14cfe357f81b40f48e9a6ff8221ae3ed P 704c3a8d0e764c66879b2e52d812ffdf: No bootstrap required, opened a new log I0312 20:11:58.700428 5604 ts_tablet_manager.cc:1063] T 14cfe357f81b40f48e9a6ff8221ae3ed P eff8b6e74b4044c982cb22453e464f58: Time spent starting tablet: real 0.039s user 0.016s sys 0.020s I0312 20:11:58.701050 5603 raft_consensus.cc:304] T 14cfe357f81b40f48e9a6ff8221ae3ed P 876c901434c04ed4ab4bfbf51b338b96 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "876c901434c04ed4ab4bfbf51b338b96" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 58832 } } peers { permanent_uuid: "6cb3d7538445412e92d6469ad2a62d90" member_type: VOTER last_known_addr { host: "127.0.53.4" port: 33153 } } peers { permanent_uuid: "704c3a8d0e764c66879b2e52d812ffdf" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 34688 } } peers { permanent_uuid: "35336ac513e14662ac43dcb2acd85121" member_type: VOTER last_known_addr { host: "127.0.53.5" port: 46654 } } peers { permanent_uuid: "eff8b6e74b4044c982cb22453e464f58" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 50867 } } I0312 20:11:58.701880 5602 ts_tablet_manager.cc:1035] T 14cfe357f81b40f48e9a6ff8221ae3ed P 704c3a8d0e764c66879b2e52d812ffdf: Time spent bootstrapping tablet: real 0.036s user 0.004s sys 0.004s I0312 20:11:58.702056 5603 raft_consensus.cc:605] T 14cfe357f81b40f48e9a6ff8221ae3ed P 876c901434c04ed4ab4bfbf51b338b96 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 876c901434c04ed4ab4bfbf51b338b96, State: Initialized, Role: FOLLOWER I0312 20:11:58.702905 5603 consensus_queue.cc:219] T 14cfe357f81b40f48e9a6ff8221ae3ed P 876c901434c04ed4ab4bfbf51b338b96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "876c901434c04ed4ab4bfbf51b338b96" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 58832 } } peers { permanent_uuid: "6cb3d7538445412e92d6469ad2a62d90" member_type: VOTER last_known_addr { host: "127.0.53.4" port: 33153 } } peers { permanent_uuid: "704c3a8d0e764c66879b2e52d812ffdf" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 34688 } } peers { permanent_uuid: "35336ac513e14662ac43dcb2acd85121" member_type: VOTER last_known_addr { host: "127.0.53.5" port: 46654 } } peers { permanent_uuid: "eff8b6e74b4044c982cb22453e464f58" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 50867 } } I0312 20:11:58.702833 5607 raft_consensus.cc:304] T 14cfe357f81b40f48e9a6ff8221ae3ed P 35336ac513e14662ac43dcb2acd85121 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "876c901434c04ed4ab4bfbf51b338b96" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 58832 } } peers { permanent_uuid: "6cb3d7538445412e92d6469ad2a62d90" member_type: VOTER last_known_addr { host: "127.0.53.4" port: 33153 } } peers { permanent_uuid: "704c3a8d0e764c66879b2e52d812ffdf" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 34688 } } peers { permanent_uuid: "35336ac513e14662ac43dcb2acd85121" member_type: VOTER last_known_addr { host: "127.0.53.5" port: 46654 } } peers { permanent_uuid: "eff8b6e74b4044c982cb22453e464f58" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 50867 } } I0312 20:11:58.703719 5607 raft_consensus.cc:605] T 14cfe357f81b40f48e9a6ff8221ae3ed P 35336ac513e14662ac43dcb2acd85121 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35336ac513e14662ac43dcb2acd85121, State: Initialized, Role: FOLLOWER I0312 20:11:58.704149 5607 consensus_queue.cc:219] T 14cfe357f81b40f48e9a6ff8221ae3ed P 35336ac513e14662ac43dcb2acd85121 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "876c901434c04ed4ab4bfbf51b338b96" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 58832 } } peers { permanent_uuid: "6cb3d7538445412e92d6469ad2a62d90" member_type: VOTER last_known_addr { host: "127.0.53.4" port: 33153 } } peers { permanent_uuid: "704c3a8d0e764c66879b2e52d812ffdf" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 34688 } } peers { permanent_uuid: "35336ac513e14662ac43dcb2acd85121" member_type: VOTER last_known_addr { host: "127.0.53.5" port: 46654 } } peers { permanent_uuid: "eff8b6e74b4044c982cb22453e464f58" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 50867 } } I0312 20:11:58.716531 5603 ts_tablet_manager.cc:1063] T 14cfe357f81b40f48e9a6ff8221ae3ed P 876c901434c04ed4ab4bfbf51b338b96: Time spent starting tablet: real 0.031s user 0.020s sys 0.008s I0312 20:11:58.719951 212 external_mini_cluster.cc:498] 5 TS(s) registered with all masters I0312 20:11:58.720948 5606 raft_consensus.cc:304] T 14cfe357f81b40f48e9a6ff8221ae3ed P 6cb3d7538445412e92d6469ad2a62d90 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "876c901434c04ed4ab4bfbf51b338b96" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 58832 } } peers { permanent_uuid: "6cb3d7538445412e92d6469ad2a62d90" member_type: VOTER last_known_addr { host: "127.0.53.4" port: 33153 } } peers { permanent_uuid: "704c3a8d0e764c66879b2e52d812ffdf" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 34688 } } peers { permanent_uuid: "35336ac513e14662ac43dcb2acd85121" member_type: VOTER last_known_addr { host: "127.0.53.5" port: 46654 } } peers { permanent_uuid: "eff8b6e74b4044c982cb22453e464f58" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 50867 } } I0312 20:11:58.721879 5606 raft_consensus.cc:605] T 14cfe357f81b40f48e9a6ff8221ae3ed P 6cb3d7538445412e92d6469ad2a62d90 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cb3d7538445412e92d6469ad2a62d90, State: Initialized, Role: FOLLOWER I0312 20:11:58.722234 5606 consensus_queue.cc:219] T 14cfe357f81b40f48e9a6ff8221ae3ed P 6cb3d7538445412e92d6469ad2a62d90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "876c901434c04ed4ab4bfbf51b338b96" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 58832 } } peers { permanent_uuid: "6cb3d7538445412e92d6469ad2a62d90" member_type: VOTER last_known_addr { host: "127.0.53.4" port: 33153 } } peers { permanent_uuid: "704c3a8d0e764c66879b2e52d812ffdf" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 34688 } } peers { permanent_uuid: "35336ac513e14662ac43dcb2acd85121" member_type: VOTER last_known_addr { host: "127.0.53.5" port: 46654 } } peers { permanent_uuid: "eff8b6e74b4044c982cb22453e464f58" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 50867 } } I0312 20:11:58.724675 5602 raft_consensus.cc:304] T 14cfe357f81b40f48e9a6ff8221ae3ed P 704c3a8d0e764c66879b2e52d812ffdf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "876c901434c04ed4ab4bfbf51b338b96" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 58832 } } peers { permanent_uuid: "6cb3d7538445412e92d6469ad2a62d90" member_type: VOTER last_known_addr { host: "127.0.53.4" port: 33153 } } peers { permanent_uuid: "704c3a8d0e764c66879b2e52d812ffdf" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 34688 } } peers { permanent_uuid: "35336ac513e14662ac43dcb2acd85121" member_type: VOTER last_known_addr { host: "127.0.53.5" port: 46654 } } peers { permanent_uuid: "eff8b6e74b4044c982cb22453e464f58" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 50867 } } I0312 20:11:58.725378 5602 raft_consensus.cc:605] T 14cfe357f81b40f48e9a6ff8221ae3ed P 704c3a8d0e764c66879b2e52d812ffdf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 704c3a8d0e764c66879b2e52d812ffdf, State: Initialized, Role: FOLLOWER I0312 20:11:58.725680 5602 consensus_queue.cc:219] T 14cfe357f81b40f48e9a6ff8221ae3ed P 704c3a8d0e764c66879b2e52d812ffdf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "876c901434c04ed4ab4bfbf51b338b96" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 58832 } } peers { permanent_uuid: "6cb3d7538445412e92d6469ad2a62d90" member_type: VOTER last_known_addr { host: "127.0.53.4" port: 33153 } } peers { permanent_uuid: "704c3a8d0e764c66879b2e52d812ffdf" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 34688 } } peers { permanent_uuid: "35336ac513e14662ac43dcb2acd85121" member_type: VOTER last_known_addr { host: "127.0.53.5" port: 46654 } } peers { permanent_uuid: "eff8b6e74b4044c982cb22453e464f58" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 50867 } } I0312 20:11:58.726649 5454 heartbeater.cc:467] Master 127.0.53.62:41398 was elected leader, sending a full tablet report... I0312 20:11:58.727505 5578 heartbeater.cc:467] Master 127.0.53.62:41398 was elected leader, sending a full tablet report... I0312 20:11:58.728118 5607 ts_tablet_manager.cc:1063] T 14cfe357f81b40f48e9a6ff8221ae3ed P 35336ac513e14662ac43dcb2acd85121: Time spent starting tablet: real 0.041s user 0.020s sys 0.016s I0312 20:11:58.729531 5606 ts_tablet_manager.cc:1063] T 14cfe357f81b40f48e9a6ff8221ae3ed P 6cb3d7538445412e92d6469ad2a62d90: Time spent starting tablet: real 0.043s user 0.012s sys 0.004s I0312 20:11:58.730329 5602 ts_tablet_manager.cc:1063] T 14cfe357f81b40f48e9a6ff8221ae3ed P 704c3a8d0e764c66879b2e52d812ffdf: Time spent starting tablet: real 0.028s user 0.008s sys 0.008s I0312 20:11:58.734902 212 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 704c3a8d0e764c66879b2e52d812ffdf to finish bootstrapping I0312 20:11:58.748353 212 ts_itest-base.cc:233] Waiting for 1 tablets on tserver eff8b6e74b4044c982cb22453e464f58 to finish bootstrapping W0312 20:11:58.757086 5205 tablet.cc:1679] T 14cfe357f81b40f48e9a6ff8221ae3ed P eff8b6e74b4044c982cb22453e464f58: Can't schedule compaction. Clean time has not been advanced past its initial value. I0312 20:11:58.759196 212 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 876c901434c04ed4ab4bfbf51b338b96 to finish bootstrapping W0312 20:11:58.768277 5579 tablet.cc:1679] T 14cfe357f81b40f48e9a6ff8221ae3ed P 35336ac513e14662ac43dcb2acd85121: Can't schedule compaction. Clean time has not been advanced past its initial value. I0312 20:11:58.768931 212 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 6cb3d7538445412e92d6469ad2a62d90 to finish bootstrapping W0312 20:11:58.776947 5455 tablet.cc:1679] T 14cfe357f81b40f48e9a6ff8221ae3ed P 6cb3d7538445412e92d6469ad2a62d90: Can't schedule compaction. Clean time has not been advanced past its initial value. I0312 20:11:58.777655 212 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 35336ac513e14662ac43dcb2acd85121 to finish bootstrapping W0312 20:11:58.782557 5330 tablet.cc:1679] T 14cfe357f81b40f48e9a6ff8221ae3ed P 876c901434c04ed4ab4bfbf51b338b96: Can't schedule compaction. Clean time has not been advanced past its initial value. I0312 20:11:58.794144 5036 tablet_service.cc:1105] Received Run Leader Election RPC: tablet_id: "14cfe357f81b40f48e9a6ff8221ae3ed" dest_uuid: "704c3a8d0e764c66879b2e52d812ffdf" from {username='slave'} at 127.0.0.1:54347 I0312 20:11:58.794492 5036 raft_consensus.cc:436] T 14cfe357f81b40f48e9a6ff8221ae3ed P 704c3a8d0e764c66879b2e52d812ffdf [term 0 FOLLOWER]: Starting forced leader election (received explicit request) I0312 20:11:58.794672 5036 raft_consensus.cc:2752] T 14cfe357f81b40f48e9a6ff8221ae3ed P 704c3a8d0e764c66879b2e52d812ffdf [term 0 FOLLOWER]: Advancing to term 1 I0312 20:11:58.797583 5036 raft_consensus.cc:458] T 14cfe357f81b40f48e9a6ff8221ae3ed P 704c3a8d0e764c66879b2e52d812ffdf [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "876c901434c04ed4ab4bfbf51b338b96" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 58832 } } peers { permanent_uuid: "6cb3d7538445412e92d6469ad2a62d90" member_type: VOTER last_known_addr { host: "127.0.53.4" port: 33153 } } peers { permanent_uuid: "704c3a8d0e764c66879b2e52d812ffdf" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 34688 } } peers { permanent_uuid: "35336ac513e14662ac43dcb2acd85121" member_type: VOTER last_known_addr { host: "127.0.53.5" port: 46654 } } peers { permanent_uuid: "eff8b6e74b4044c982cb22453e464f58" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 50867 } } I0312 20:11:58.798883 5036 leader_election.cc:231] T 14cfe357f81b40f48e9a6ff8221ae3ed P 704c3a8d0e764c66879b2e52d812ffdf [CANDIDATE]: Term 1 election: Requesting vote from peer 876c901434c04ed4ab4bfbf51b338b96 I0312 20:11:58.799300 5036 leader_election.cc:231] T 14cfe357f81b40f48e9a6ff8221ae3ed P 704c3a8d0e764c66879b2e52d812ffdf [CANDIDATE]: Term 1 election: Requesting vote from peer 6cb3d7538445412e92d6469ad2a62d90 I0312 20:11:58.800175 5036 leader_election.cc:231] T 14cfe357f81b40f48e9a6ff8221ae3ed P 704c3a8d0e764c66879b2e52d812ffdf [CANDIDATE]: Term 1 election: Requesting vote from peer 35336ac513e14662ac43dcb2acd85121 I0312 20:11:58.800545 5036 leader_election.cc:231] T 14cfe357f81b40f48e9a6ff8221ae3ed P 704c3a8d0e764c66879b2e52d812ffdf [CANDIDATE]: Term 1 election: Requesting vote from peer eff8b6e74b4044c982cb22453e464f58 I0312 20:11:58.818002 5286 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "14cfe357f81b40f48e9a6ff8221ae3ed" candidate_uuid: "704c3a8d0e764c66879b2e52d812ffdf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "876c901434c04ed4ab4bfbf51b338b96" I0312 20:11:58.818532 5286 raft_consensus.cc:2752] T 14cfe357f81b40f48e9a6ff8221ae3ed P 876c901434c04ed4ab4bfbf51b338b96 [term 0 FOLLOWER]: Advancing to term 1 W0312 20:11:58.821424 5080 tablet.cc:1679] T 14cfe357f81b40f48e9a6ff8221ae3ed P 704c3a8d0e764c66879b2e52d812ffdf: Can't schedule compaction. Clean time has not been advanced past its initial value. I0312 20:11:58.822083 5411 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "14cfe357f81b40f48e9a6ff8221ae3ed" candidate_uuid: "704c3a8d0e764c66879b2e52d812ffdf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "6cb3d7538445412e92d6469ad2a62d90" I0312 20:11:58.822491 5411 raft_consensus.cc:2752] T 14cfe357f81b40f48e9a6ff8221ae3ed P 6cb3d7538445412e92d6469ad2a62d90 [term 0 FOLLOWER]: Advancing to term 1 I0312 20:11:58.823235 5286 raft_consensus.cc:2215] T 14cfe357f81b40f48e9a6ff8221ae3ed P 876c901434c04ed4ab4bfbf51b338b96 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 704c3a8d0e764c66879b2e52d812ffdf in term 1. I0312 20:11:58.824299 4967 leader_election.cc:363] T 14cfe357f81b40f48e9a6ff8221ae3ed P 704c3a8d0e764c66879b2e52d812ffdf [CANDIDATE]: Term 1 election: Vote granted by peer 876c901434c04ed4ab4bfbf51b338b96 I0312 20:11:58.824579 5535 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "14cfe357f81b40f48e9a6ff8221ae3ed" candidate_uuid: "704c3a8d0e764c66879b2e52d812ffdf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "35336ac513e14662ac43dcb2acd85121" I0312 20:11:58.825145 5535 raft_consensus.cc:2752] T 14cfe357f81b40f48e9a6ff8221ae3ed P 35336ac513e14662ac43dcb2acd85121 [term 0 FOLLOWER]: Advancing to term 1 I0312 20:11:58.825460 5411 raft_consensus.cc:2215] T 14cfe357f81b40f48e9a6ff8221ae3ed P 6cb3d7538445412e92d6469ad2a62d90 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 704c3a8d0e764c66879b2e52d812ffdf in term 1. I0312 20:11:58.825768 5161 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "14cfe357f81b40f48e9a6ff8221ae3ed" candidate_uuid: "704c3a8d0e764c66879b2e52d812ffdf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "eff8b6e74b4044c982cb22453e464f58" I0312 20:11:58.826309 5161 raft_consensus.cc:2752] T 14cfe357f81b40f48e9a6ff8221ae3ed P eff8b6e74b4044c982cb22453e464f58 [term 0 FOLLOWER]: Advancing to term 1 I0312 20:11:58.829476 4971 leader_election.cc:363] T 14cfe357f81b40f48e9a6ff8221ae3ed P 704c3a8d0e764c66879b2e52d812ffdf [CANDIDATE]: Term 1 election: Vote granted by peer 6cb3d7538445412e92d6469ad2a62d90 I0312 20:11:58.829704 4971 leader_election.cc:258] T 14cfe357f81b40f48e9a6ff8221ae3ed P 704c3a8d0e764c66879b2e52d812ffdf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0312 20:11:58.830305 5616 raft_consensus.cc:2507] T 14cfe357f81b40f48e9a6ff8221ae3ed P 704c3a8d0e764c66879b2e52d812ffdf [term 1 FOLLOWER]: Leader election won for term 1 I0312 20:11:58.830636 5161 raft_consensus.cc:2215] T 14cfe357f81b40f48e9a6ff8221ae3ed P eff8b6e74b4044c982cb22453e464f58 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 704c3a8d0e764c66879b2e52d812ffdf in term 1. I0312 20:11:58.831704 5616 raft_consensus.cc:572] T 14cfe357f81b40f48e9a6ff8221ae3ed P 704c3a8d0e764c66879b2e52d812ffdf [term 1 LEADER]: Becoming Leader. State: Replica: 704c3a8d0e764c66879b2e52d812ffdf, State: Running, Role: LEADER I0312 20:11:58.831763 4968 leader_election.cc:363] T 14cfe357f81b40f48e9a6ff8221ae3ed P 704c3a8d0e764c66879b2e52d812ffdf [CANDIDATE]: Term 1 election: Vote granted by peer eff8b6e74b4044c982cb22453e464f58 I0312 20:11:58.831867 5535 raft_consensus.cc:2215] T 14cfe357f81b40f48e9a6ff8221ae3ed P 35336ac513e14662ac43dcb2acd85121 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 704c3a8d0e764c66879b2e52d812ffdf in term 1. I0312 20:11:58.832705 4971 leader_election.cc:363] T 14cfe357f81b40f48e9a6ff8221ae3ed P 704c3a8d0e764c66879b2e52d812ffdf [CANDIDATE]: Term 1 election: Vote granted by peer 35336ac513e14662ac43dcb2acd85121 I0312 20:11:58.832252 5616 consensus_queue.cc:196] T 14cfe357f81b40f48e9a6ff8221ae3ed P 704c3a8d0e764c66879b2e52d812ffdf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "876c901434c04ed4ab4bfbf51b338b96" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 58832 } } peers { permanent_uuid: "6cb3d7538445412e92d6469ad2a62d90" member_type: VOTER last_known_addr { host: "127.0.53.4" port: 33153 } } peers { permanent_uuid: "704c3a8d0e764c66879b2e52d812ffdf" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 34688 } } peers { permanent_uuid: "35336ac513e14662ac43dcb2acd85121" member_type: VOTER last_known_addr { host: "127.0.53.5" port: 46654 } } peers { permanent_uuid: "eff8b6e74b4044c982cb22453e464f58" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 50867 } } I0312 20:11:58.839932 4913 catalog_manager.cc:3592] T 14cfe357f81b40f48e9a6ff8221ae3ed P 704c3a8d0e764c66879b2e52d812ffdf reported cstate change: term changed from 0 to 1, leader changed from to 704c3a8d0e764c66879b2e52d812ffdf (127.0.53.1). New cstate: current_term: 1 leader_uuid: "704c3a8d0e764c66879b2e52d812ffdf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "876c901434c04ed4ab4bfbf51b338b96" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 58832 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6cb3d7538445412e92d6469ad2a62d90" member_type: VOTER last_known_addr { host: "127.0.53.4" port: 33153 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "704c3a8d0e764c66879b2e52d812ffdf" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 34688 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "35336ac513e14662ac43dcb2acd85121" member_type: VOTER last_known_addr { host: "127.0.53.5" port: 46654 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eff8b6e74b4044c982cb22453e464f58" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 50867 } health_report { overall_health: UNKNOWN } } } I0312 20:11:59.224766 5535 raft_consensus.cc:1058] T 14cfe357f81b40f48e9a6ff8221ae3ed P 35336ac513e14662ac43dcb2acd85121 [term 1 FOLLOWER]: Refusing update from remote peer 704c3a8d0e764c66879b2e52d812ffdf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch) I0312 20:11:59.226294 5622 consensus_queue.cc:1017] T 14cfe357f81b40f48e9a6ff8221ae3ed P 704c3a8d0e764c66879b2e52d812ffdf [LEADER]: Connected to new peer: Peer: permanent_uuid: "35336ac513e14662ac43dcb2acd85121" member_type: VOTER last_known_addr { host: "127.0.53.5" port: 46654 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0312 20:11:59.278430 5161 raft_consensus.cc:1058] T 14cfe357f81b40f48e9a6ff8221ae3ed P eff8b6e74b4044c982cb22453e464f58 [term 1 FOLLOWER]: Refusing update from remote peer 704c3a8d0e764c66879b2e52d812ffdf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch) I0312 20:11:59.278587 5286 raft_consensus.cc:1058] T 14cfe357f81b40f48e9a6ff8221ae3ed P 876c901434c04ed4ab4bfbf51b338b96 [term 1 FOLLOWER]: Refusing update from remote peer 704c3a8d0e764c66879b2e52d812ffdf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch) I0312 20:11:59.279629 5622 consensus_queue.cc:1017] T 14cfe357f81b40f48e9a6ff8221ae3ed P 704c3a8d0e764c66879b2e52d812ffdf [LEADER]: Connected to new peer: Peer: permanent_uuid: "eff8b6e74b4044c982cb22453e464f58" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 50867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0312 20:11:59.280130 5616 consensus_queue.cc:1017] T 14cfe357f81b40f48e9a6ff8221ae3ed P 704c3a8d0e764c66879b2e52d812ffdf [LEADER]: Connected to new peer: Peer: permanent_uuid: "876c901434c04ed4ab4bfbf51b338b96" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 58832 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0312 20:11:59.296885 5411 raft_consensus.cc:1058] T 14cfe357f81b40f48e9a6ff8221ae3ed P 6cb3d7538445412e92d6469ad2a62d90 [term 1 FOLLOWER]: Refusing update from remote peer 704c3a8d0e764c66879b2e52d812ffdf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch) I0312 20:11:59.298557 5629 consensus_queue.cc:1017] T 14cfe357f81b40f48e9a6ff8221ae3ed P 704c3a8d0e764c66879b2e52d812ffdf [LEADER]: Connected to new peer: Peer: permanent_uuid: "6cb3d7538445412e92d6469ad2a62d90" member_type: VOTER last_known_addr { host: "127.0.53.4" port: 33153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s I0312 20:11:59.300568 212 external_mini_cluster.cc:997] Killing /tmp/dist-test-task3bwPeE/build/tsan/bin/kudu-master with pid 4886 W0312 20:11:59.325908 5094 connection.cc:513] client connection to 127.0.53.62:41398 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0312 20:11:59.325911 4969 connection.cc:513] client connection to 127.0.53.62:41398 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0312 20:11:59.327101 5219 connection.cc:513] client connection to 127.0.53.62:41398 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0312 20:11:59.328012 5344 connection.cc:513] client connection to 127.0.53.62:41398 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0312 20:11:59.328032 5468 connection.cc:513] client connection to 127.0.53.62:41398 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0312 20:11:59.328505 5454 heartbeater.cc:591] Failed to heartbeat to 127.0.53.62:41398: Network error: Failed to send heartbeat to master: failed to read from TLS socket: Connection reset by peer (error 104) W0312 20:11:59.328799 5344 connection.cc:186] Error closing socket: Network error: TlsSocket::Close: Broken pipe W0312 20:11:59.329128 5219 connection.cc:513] server connection from 127.0.0.1:57580 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0312 20:11:59.329156 4885 connection.cc:513] client connection to 127.0.53.62:41398 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0312 20:11:59.329821 5585 connection.cc:513] client connection to 127.0.53.62:41398 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0312 20:11:59.330572 4971 connection.cc:513] server connection from 127.0.0.1:54339 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0312 20:11:59.330659 5342 connection.cc:513] server connection from 127.0.0.1:50597 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0312 20:11:59.331197 5093 connection.cc:513] server connection from 127.0.0.1:52118 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0312 20:11:59.331975 5466 connection.cc:513] server connection from 127.0.0.1:50255 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) I0312 20:11:59.332171 212 raft_consensus_election-itest.cc:406] Removing tserver with uuid 35336ac513e14662ac43dcb2acd85121 I0312 20:11:59.334185 5036 consensus_peers.cc:458] T 14cfe357f81b40f48e9a6ff8221ae3ed P 704c3a8d0e764c66879b2e52d812ffdf -> Peer 35336ac513e14662ac43dcb2acd85121 (127.0.53.5:46654): Closing peer: 35336ac513e14662ac43dcb2acd85121 I0312 20:11:59.334416 5036 consensus_peers.cc:458] T 14cfe357f81b40f48e9a6ff8221ae3ed P 704c3a8d0e764c66879b2e52d812ffdf -> Peer eff8b6e74b4044c982cb22453e464f58 (127.0.53.2:50867): Closing peer: eff8b6e74b4044c982cb22453e464f58 I0312 20:11:59.334583 5036 consensus_peers.cc:458] T 14cfe357f81b40f48e9a6ff8221ae3ed P 704c3a8d0e764c66879b2e52d812ffdf -> Peer 6cb3d7538445412e92d6469ad2a62d90 (127.0.53.4:33153): Closing peer: 6cb3d7538445412e92d6469ad2a62d90 I0312 20:11:59.334754 5036 consensus_peers.cc:458] T 14cfe357f81b40f48e9a6ff8221ae3ed P 704c3a8d0e764c66879b2e52d812ffdf -> Peer 876c901434c04ed4ab4bfbf51b338b96 (127.0.53.3:58832): Closing peer: 876c901434c04ed4ab4bfbf51b338b96 I0312 20:11:59.335535 5036 consensus_queue.cc:196] T 14cfe357f81b40f48e9a6ff8221ae3ed P 704c3a8d0e764c66879b2e52d812ffdf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "876c901434c04ed4ab4bfbf51b338b96" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 58832 } } peers { permanent_uuid: "6cb3d7538445412e92d6469ad2a62d90" member_type: VOTER last_known_addr { host: "127.0.53.4" port: 33153 } } peers { permanent_uuid: "704c3a8d0e764c66879b2e52d812ffdf" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 34688 } } peers { permanent_uuid: "eff8b6e74b4044c982cb22453e464f58" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 50867 } } I0312 20:11:59.339664 5161 raft_consensus.cc:1058] T 14cfe357f81b40f48e9a6ff8221ae3ed P eff8b6e74b4044c982cb22453e464f58 [term 1 FOLLOWER]: Refusing update from remote peer 704c3a8d0e764c66879b2e52d812ffdf: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0312 20:11:59.340068 5411 raft_consensus.cc:1058] T 14cfe357f81b40f48e9a6ff8221ae3ed P 6cb3d7538445412e92d6469ad2a62d90 [term 1 FOLLOWER]: Refusing update from remote peer 704c3a8d0e764c66879b2e52d812ffdf: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0312 20:11:59.340739 5286 raft_consensus.cc:1058] T 14cfe357f81b40f48e9a6ff8221ae3ed P 876c901434c04ed4ab4bfbf51b338b96 [term 1 FOLLOWER]: Refusing update from remote peer 704c3a8d0e764c66879b2e52d812ffdf: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0312 20:11:59.341055 5622 consensus_queue.cc:1017] T 14cfe357f81b40f48e9a6ff8221ae3ed P 704c3a8d0e764c66879b2e52d812ffdf [LEADER]: Connected to new peer: Peer: permanent_uuid: "6cb3d7538445412e92d6469ad2a62d90" member_type: VOTER last_known_addr { host: "127.0.53.4" port: 33153 }, Status: LMP_MISMATCH, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s I0312 20:11:59.341850 5622 consensus_queue.cc:1017] T 14cfe357f81b40f48e9a6ff8221ae3ed P 704c3a8d0e764c66879b2e52d812ffdf [LEADER]: Connected to new peer: Peer: permanent_uuid: "eff8b6e74b4044c982cb22453e464f58" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 50867 }, Status: LMP_MISMATCH, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s I0312 20:11:59.342350 5616 consensus_queue.cc:1017] T 14cfe357f81b40f48e9a6ff8221ae3ed P 704c3a8d0e764c66879b2e52d812ffdf [LEADER]: Connected to new peer: Peer: permanent_uuid: "876c901434c04ed4ab4bfbf51b338b96" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 58832 }, Status: LMP_MISMATCH, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s W0312 20:11:59.343999 5454 heartbeater.cc:591] Failed to heartbeat to 127.0.53.62:41398: Network error: Failed to ping master at 127.0.53.62:41398: Client connection negotiation failed: client connection to 127.0.53.62:41398: connect: Connection refused (error 111) I0312 20:11:59.346593 5616 raft_consensus.cc:2654] T 14cfe357f81b40f48e9a6ff8221ae3ed P 704c3a8d0e764c66879b2e52d812ffdf [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 35336ac513e14662ac43dcb2acd85121 (127.0.53.5) evicted. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "876c901434c04ed4ab4bfbf51b338b96" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 58832 } } peers { permanent_uuid: "6cb3d7538445412e92d6469ad2a62d90" member_type: VOTER last_known_addr { host: "127.0.53.4" port: 33153 } } peers { permanent_uuid: "704c3a8d0e764c66879b2e52d812ffdf" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 34688 } } peers { permanent_uuid: "eff8b6e74b4044c982cb22453e464f58" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 50867 } } } I0312 20:11:59.347846 5286 raft_consensus.cc:2654] T 14cfe357f81b40f48e9a6ff8221ae3ed P 876c901434c04ed4ab4bfbf51b338b96 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 35336ac513e14662ac43dcb2acd85121 (127.0.53.5) evicted. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "876c901434c04ed4ab4bfbf51b338b96" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 58832 } } peers { permanent_uuid: "6cb3d7538445412e92d6469ad2a62d90" member_type: VOTER last_known_addr { host: "127.0.53.4" port: 33153 } } peers { permanent_uuid: "704c3a8d0e764c66879b2e52d812ffdf" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 34688 } } peers { permanent_uuid: "eff8b6e74b4044c982cb22453e464f58" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 50867 } } } I0312 20:11:59.348953 5161 raft_consensus.cc:2654] T 14cfe357f81b40f48e9a6ff8221ae3ed P eff8b6e74b4044c982cb22453e464f58 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 35336ac513e14662ac43dcb2acd85121 (127.0.53.5) evicted. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "876c901434c04ed4ab4bfbf51b338b96" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 58832 } } peers { permanent_uuid: "6cb3d7538445412e92d6469ad2a62d90" member_type: VOTER last_known_addr { host: "127.0.53.4" port: 33153 } } peers { permanent_uuid: "704c3a8d0e764c66879b2e52d812ffdf" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 34688 } } peers { permanent_uuid: "eff8b6e74b4044c982cb22453e464f58" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 50867 } } } I0312 20:11:59.355690 5411 raft_consensus.cc:2654] T 14cfe357f81b40f48e9a6ff8221ae3ed P 6cb3d7538445412e92d6469ad2a62d90 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 35336ac513e14662ac43dcb2acd85121 (127.0.53.5) evicted. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "876c901434c04ed4ab4bfbf51b338b96" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 58832 } } peers { permanent_uuid: "6cb3d7538445412e92d6469ad2a62d90" member_type: VOTER last_known_addr { host: "127.0.53.4" port: 33153 } } peers { permanent_uuid: "704c3a8d0e764c66879b2e52d812ffdf" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 34688 } } peers { permanent_uuid: "eff8b6e74b4044c982cb22453e464f58" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 50867 } } } W0312 20:11:59.363711 5204 heartbeater.cc:591] Failed to heartbeat to 127.0.53.62:41398: Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.53.62:41398: connect: Connection refused (error 111) W0312 20:11:59.364512 5329 heartbeater.cc:591] Failed to heartbeat to 127.0.53.62:41398: Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.53.62:41398: connect: Connection refused (error 111) W0312 20:11:59.367135 5454 heartbeater.cc:591] Failed to heartbeat to 127.0.53.62:41398: Network error: Failed to ping master at 127.0.53.62:41398: Client connection negotiation failed: client connection to 127.0.53.62:41398: connect: Connection refused (error 111) W0312 20:11:59.368791 5079 heartbeater.cc:591] Failed to heartbeat to 127.0.53.62:41398: Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.53.62:41398: connect: Connection refused (error 111) W0312 20:11:59.370369 5079 heartbeater.cc:591] Failed to heartbeat to 127.0.53.62:41398: Network error: Failed to ping master at 127.0.53.62:41398: Client connection negotiation failed: client connection to 127.0.53.62:41398: connect: Connection refused (error 111) I0312 20:11:59.410008 212 raft_consensus_election-itest.cc:416] Pausing 3 replicas... I0312 20:11:59.441867 212 raft_consensus_election-itest.cc:424] Adding back Peer 35336ac513e14662ac43dcb2acd85121 and expecting timeout... I0312 20:11:59.443367 5036 consensus_peers.cc:458] T 14cfe357f81b40f48e9a6ff8221ae3ed P 704c3a8d0e764c66879b2e52d812ffdf -> Peer eff8b6e74b4044c982cb22453e464f58 (127.0.53.2:50867): Closing peer: eff8b6e74b4044c982cb22453e464f58 I0312 20:11:59.443609 5036 consensus_peers.cc:458] T 14cfe357f81b40f48e9a6ff8221ae3ed P 704c3a8d0e764c66879b2e52d812ffdf -> Peer 6cb3d7538445412e92d6469ad2a62d90 (127.0.53.4:33153): Closing peer: 6cb3d7538445412e92d6469ad2a62d90 I0312 20:11:59.443758 5036 consensus_peers.cc:458] T 14cfe357f81b40f48e9a6ff8221ae3ed P 704c3a8d0e764c66879b2e52d812ffdf -> Peer 876c901434c04ed4ab4bfbf51b338b96 (127.0.53.3:58832): Closing peer: 876c901434c04ed4ab4bfbf51b338b96 I0312 20:11:59.444223 5036 consensus_queue.cc:196] T 14cfe357f81b40f48e9a6ff8221ae3ed P 704c3a8d0e764c66879b2e52d812ffdf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "876c901434c04ed4ab4bfbf51b338b96" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 58832 } } peers { permanent_uuid: "6cb3d7538445412e92d6469ad2a62d90" member_type: VOTER last_known_addr { host: "127.0.53.4" port: 33153 } } peers { permanent_uuid: "704c3a8d0e764c66879b2e52d812ffdf" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 34688 } } peers { permanent_uuid: "eff8b6e74b4044c982cb22453e464f58" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 50867 } } peers { permanent_uuid: "35336ac513e14662ac43dcb2acd85121" member_type: VOTER last_known_addr { host: "127.0.53.5" port: 46654 } attrs { } } I0312 20:11:59.448042 5535 raft_consensus.cc:1058] T 14cfe357f81b40f48e9a6ff8221ae3ed P 35336ac513e14662ac43dcb2acd85121 [term 1 FOLLOWER]: Refusing update from remote peer 704c3a8d0e764c66879b2e52d812ffdf: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 3. (index mismatch) I0312 20:11:59.449049 5640 consensus_queue.cc:1017] T 14cfe357f81b40f48e9a6ff8221ae3ed P 704c3a8d0e764c66879b2e52d812ffdf [LEADER]: Connected to new peer: Peer: permanent_uuid: "35336ac513e14662ac43dcb2acd85121" member_type: VOTER last_known_addr { host: "127.0.53.5" port: 46654 } attrs { }, Status: LMP_MISMATCH, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s I0312 20:11:59.452656 5535 raft_consensus.cc:684] T 14cfe357f81b40f48e9a6ff8221ae3ed P 35336ac513e14662ac43dcb2acd85121 [term 1 NON_PARTICIPANT]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time. Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "876c901434c04ed4ab4bfbf51b338b96" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 58832 } } peers { permanent_uuid: "6cb3d7538445412e92d6469ad2a62d90" member_type: VOTER last_known_addr { host: "127.0.53.4" port: 33153 } } peers { permanent_uuid: "704c3a8d0e764c66879b2e52d812ffdf" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 34688 } } peers { permanent_uuid: "35336ac513e14662ac43dcb2acd85121" member_type: VOTER last_known_addr { host: "127.0.53.5" port: 46654 } } peers { permanent_uuid: "eff8b6e74b4044c982cb22453e464f58" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 50867 } }. Pending config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "876c901434c04ed4ab4bfbf51b338b96" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 58832 } } peers { permanent_uuid: "6cb3d7538445412e92d6469ad2a62d90" member_type: VOTER last_known_addr { host: "127.0.53.4" port: 33153 } } peers { permanent_uuid: "704c3a8d0e764c66879b2e52d812ffdf" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 34688 } } peers { permanent_uuid: "eff8b6e74b4044c982cb22453e464f58" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 50867 } }: New config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "876c901434c04ed4ab4bfbf51b338b96" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 58832 } } peers { permanent_uuid: "6cb3d7538445412e92d6469ad2a62d90" member_type: VOTER last_known_addr { host: "127.0.53.4" port: 33153 } } peers { permanent_uuid: "704c3a8d0e764c66879b2e52d812ffdf" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 34688 } } peers { permanent_uuid: "eff8b6e74b4044c982cb22453e464f58" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 50867 } } peers { permanent_uuid: "35336ac513e14662ac43dcb2acd85121" member_type: VOTER last_known_addr { host: "127.0.53.5" port: 46654 } attrs { } } W0312 20:11:59.452880 5535 raft_consensus.cc:1373] T 14cfe357f81b40f48e9a6ff8221ae3ed P 35336ac513e14662ac43dcb2acd85121 [term 1 NON_PARTICIPANT]: Could not prepare transaction for op 1.3 and following 1 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time. Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "876c901434c04ed4ab4bfbf51b338b96" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 58832 } } peers { permanent_uuid: "6cb3d7538445412e92d6469ad2a62d90" member_type: VOTER last_known_addr { host: "127.0.53.4" port: 33153 } } peers { permanent_uuid: "704c3a8d0e764c66879b2e52d812ffdf" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 34688 } } peers { permanent_uuid: "35336ac513e14662ac43dcb2acd85121" member_type: VOTER last_known_addr { host: "127.0.53.5" port: 46654 } } peers { permanent_uuid: "eff8b6e74b4044c982cb22453e464f58" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 50867 } }. Pending config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "876c901434c04ed4ab4bfbf51b338b96" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 58832 } } peers { permanent_uuid: "6cb3d7538445412e92d6469ad2a62d90" member_type: VOTER last_known_addr { host: "127.0.53.4" port: 33153 } } peers { permanent_uuid: "704c3a8d0e764c66879b2e52d812ffdf" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 34688 } } peers { permanent_uuid: "eff8b6e74b4044c982cb22453e464f58" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 50867 } }: New config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "876c901434c04ed4ab4bfbf51b338b96" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 58832 } } peers { permanent_uuid: "6cb3d7538445412e92d6469ad2a62d90" member_type: VOTER last_known_addr { host: "127.0.53.4" port: 33153 } } peers { permanent_uuid: "704c3a8d0e764c66879b2e52d812ffdf" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 34688 } } peers { permanent_uuid: "eff8b6e74b4044c982cb22453e464f58" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 50867 } } peers { permanent_uuid: "35336ac513e14662ac43dcb2acd85121" member_type: VOTER last_known_addr { host: "127.0.53.5" port: 46654 } attrs { } } I0312 20:11:59.453524 5535 raft_consensus.cc:2654] T 14cfe357f81b40f48e9a6ff8221ae3ed P 35336ac513e14662ac43dcb2acd85121 [term 1 NON_PARTICIPANT]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 35336ac513e14662ac43dcb2acd85121 (127.0.53.5) evicted. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "876c901434c04ed4ab4bfbf51b338b96" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 58832 } } peers { permanent_uuid: "6cb3d7538445412e92d6469ad2a62d90" member_type: VOTER last_known_addr { host: "127.0.53.4" port: 33153 } } peers { permanent_uuid: "704c3a8d0e764c66879b2e52d812ffdf" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 34688 } } peers { permanent_uuid: "eff8b6e74b4044c982cb22453e464f58" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 50867 } } } W0312 20:11:59.459452 5079 heartbeater.cc:591] Failed to heartbeat to 127.0.53.62:41398: Network error: Failed to ping master at 127.0.53.62:41398: Client connection negotiation failed: client connection to 127.0.53.62:41398: connect: Connection refused (error 111) W0312 20:11:59.459516 5578 heartbeater.cc:591] Failed to heartbeat to 127.0.53.62:41398: Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.53.62:41398: connect: Connection refused (error 111) I0312 20:11:59.460356 5535 raft_consensus.cc:1278] T 14cfe357f81b40f48e9a6ff8221ae3ed P 35336ac513e14662ac43dcb2acd85121 [term 1 NON_PARTICIPANT]: Allowing update even though not a member of the config I0312 20:11:59.543243 212 raft_consensus_election-itest.cc:428] Timeout achieved. I0312 20:11:59.553661 212 raft_consensus_election-itest.cc:439] Forcing Peer 704c3a8d0e764c66879b2e52d812ffdf to step down... I0312 20:11:59.554502 5036 tablet_service.cc:1133] Received LeaderStepDown RPC: tablet_id: "14cfe357f81b40f48e9a6ff8221ae3ed" dest_uuid: "704c3a8d0e764c66879b2e52d812ffdf" from {username='slave'} at 127.0.0.1:54347 I0312 20:11:59.554849 5036 raft_consensus.cc:531] T 14cfe357f81b40f48e9a6ff8221ae3ed P 704c3a8d0e764c66879b2e52d812ffdf [term 1 LEADER]: Received request to step down I0312 20:11:59.554968 5036 raft_consensus.cc:605] T 14cfe357f81b40f48e9a6ff8221ae3ed P 704c3a8d0e764c66879b2e52d812ffdf [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 704c3a8d0e764c66879b2e52d812ffdf, State: Running, Role: LEADER I0312 20:11:59.555222 5036 consensus_queue.cc:219] T 14cfe357f81b40f48e9a6ff8221ae3ed P 704c3a8d0e764c66879b2e52d812ffdf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "876c901434c04ed4ab4bfbf51b338b96" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 58832 } } peers { permanent_uuid: "6cb3d7538445412e92d6469ad2a62d90" member_type: VOTER last_known_addr { host: "127.0.53.4" port: 33153 } } peers { permanent_uuid: "704c3a8d0e764c66879b2e52d812ffdf" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 34688 } } peers { permanent_uuid: "eff8b6e74b4044c982cb22453e464f58" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 50867 } } peers { permanent_uuid: "35336ac513e14662ac43dcb2acd85121" member_type: VOTER last_known_addr { host: "127.0.53.5" port: 46654 } attrs { } } I0312 20:11:59.555721 5036 consensus_peers.cc:458] T 14cfe357f81b40f48e9a6ff8221ae3ed P 704c3a8d0e764c66879b2e52d812ffdf -> Peer 35336ac513e14662ac43dcb2acd85121 (127.0.53.5:46654): Closing peer: 35336ac513e14662ac43dcb2acd85121 I0312 20:11:59.555845 5036 consensus_peers.cc:458] T 14cfe357f81b40f48e9a6ff8221ae3ed P 704c3a8d0e764c66879b2e52d812ffdf -> Peer eff8b6e74b4044c982cb22453e464f58 (127.0.53.2:50867): Closing peer: eff8b6e74b4044c982cb22453e464f58 I0312 20:11:59.555951 5036 consensus_peers.cc:458] T 14cfe357f81b40f48e9a6ff8221ae3ed P 704c3a8d0e764c66879b2e52d812ffdf -> Peer 6cb3d7538445412e92d6469ad2a62d90 (127.0.53.4:33153): Closing peer: 6cb3d7538445412e92d6469ad2a62d90 I0312 20:11:59.556053 5036 consensus_peers.cc:458] T 14cfe357f81b40f48e9a6ff8221ae3ed P 704c3a8d0e764c66879b2e52d812ffdf -> Peer 876c901434c04ed4ab4bfbf51b338b96 (127.0.53.3:58832): Closing peer: 876c901434c04ed4ab4bfbf51b338b96 I0312 20:11:59.556836 212 raft_consensus_election-itest.cc:443] Resuming Peer eff8b6e74b4044c982cb22453e464f58 ... I0312 20:11:59.558112 212 raft_consensus_election-itest.cc:448] Trying to elect Peer 35336ac513e14662ac43dcb2acd85121 ... I0312 20:11:59.558791 5161 raft_consensus.cc:1058] T 14cfe357f81b40f48e9a6ff8221ae3ed P eff8b6e74b4044c982cb22453e464f58 [term 1 FOLLOWER]: Refusing update from remote peer 704c3a8d0e764c66879b2e52d812ffdf: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch) I0312 20:11:59.558990 5535 tablet_service.cc:1105] Received Run Leader Election RPC: tablet_id: "14cfe357f81b40f48e9a6ff8221ae3ed" dest_uuid: "35336ac513e14662ac43dcb2acd85121" from {username='slave'} at 127.0.0.1:50265 I0312 20:11:59.559376 5535 raft_consensus.cc:436] T 14cfe357f81b40f48e9a6ff8221ae3ed P 35336ac513e14662ac43dcb2acd85121 [term 1 FOLLOWER]: Starting forced leader election (received explicit request) I0312 20:11:59.559538 5535 raft_consensus.cc:2752] T 14cfe357f81b40f48e9a6ff8221ae3ed P 35336ac513e14662ac43dcb2acd85121 [term 1 FOLLOWER]: Advancing to term 2 I0312 20:11:59.562173 5535 raft_consensus.cc:458] T 14cfe357f81b40f48e9a6ff8221ae3ed P 35336ac513e14662ac43dcb2acd85121 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "876c901434c04ed4ab4bfbf51b338b96" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 58832 } } peers { permanent_uuid: "6cb3d7538445412e92d6469ad2a62d90" member_type: VOTER last_known_addr { host: "127.0.53.4" port: 33153 } } peers { permanent_uuid: "704c3a8d0e764c66879b2e52d812ffdf" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 34688 } } peers { permanent_uuid: "eff8b6e74b4044c982cb22453e464f58" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 50867 } } peers { permanent_uuid: "35336ac513e14662ac43dcb2acd85121" member_type: VOTER last_known_addr { host: "127.0.53.5" port: 46654 } attrs { } } I0312 20:11:59.563339 5535 leader_election.cc:231] T 14cfe357f81b40f48e9a6ff8221ae3ed P 35336ac513e14662ac43dcb2acd85121 [CANDIDATE]: Term 2 election: Requesting vote from peer 876c901434c04ed4ab4bfbf51b338b96 I0312 20:11:59.563697 5535 leader_election.cc:231] T 14cfe357f81b40f48e9a6ff8221ae3ed P 35336ac513e14662ac43dcb2acd85121 [CANDIDATE]: Term 2 election: Requesting vote from peer 6cb3d7538445412e92d6469ad2a62d90 I0312 20:11:59.563948 5535 leader_election.cc:231] T 14cfe357f81b40f48e9a6ff8221ae3ed P 35336ac513e14662ac43dcb2acd85121 [CANDIDATE]: Term 2 election: Requesting vote from peer 704c3a8d0e764c66879b2e52d812ffdf I0312 20:11:59.564760 5535 leader_election.cc:231] T 14cfe357f81b40f48e9a6ff8221ae3ed P 35336ac513e14662ac43dcb2acd85121 [CANDIDATE]: Term 2 election: Requesting vote from peer eff8b6e74b4044c982cb22453e464f58 I0312 20:11:59.576936 212 cluster_itest_util.cc:245] Not converged past 4 yet: 1.3 1.2 1.3 I0312 20:11:59.581822 5036 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "14cfe357f81b40f48e9a6ff8221ae3ed" candidate_uuid: "35336ac513e14662ac43dcb2acd85121" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "704c3a8d0e764c66879b2e52d812ffdf" I0312 20:11:59.582173 5036 raft_consensus.cc:2752] T 14cfe357f81b40f48e9a6ff8221ae3ed P 704c3a8d0e764c66879b2e52d812ffdf [term 1 FOLLOWER]: Advancing to term 2 I0312 20:11:59.584830 5036 raft_consensus.cc:2215] T 14cfe357f81b40f48e9a6ff8221ae3ed P 704c3a8d0e764c66879b2e52d812ffdf [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35336ac513e14662ac43dcb2acd85121 in term 2. I0312 20:11:59.585669 5467 leader_election.cc:363] T 14cfe357f81b40f48e9a6ff8221ae3ed P 35336ac513e14662ac43dcb2acd85121 [CANDIDATE]: Term 2 election: Vote granted by peer 704c3a8d0e764c66879b2e52d812ffdf I0312 20:11:59.586769 5161 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "14cfe357f81b40f48e9a6ff8221ae3ed" candidate_uuid: "35336ac513e14662ac43dcb2acd85121" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "eff8b6e74b4044c982cb22453e464f58" I0312 20:11:59.587080 5161 raft_consensus.cc:1571] T 14cfe357f81b40f48e9a6ff8221ae3ed P eff8b6e74b4044c982cb22453e464f58 [term 1 FOLLOWER]: Handling vote request from an unknown peer 35336ac513e14662ac43dcb2acd85121 I0312 20:11:59.587219 5161 raft_consensus.cc:2752] T 14cfe357f81b40f48e9a6ff8221ae3ed P eff8b6e74b4044c982cb22453e464f58 [term 1 FOLLOWER]: Advancing to term 2 I0312 20:11:59.589817 5161 raft_consensus.cc:2215] T 14cfe357f81b40f48e9a6ff8221ae3ed P eff8b6e74b4044c982cb22453e464f58 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35336ac513e14662ac43dcb2acd85121 in term 2. I0312 20:11:59.590430 5467 leader_election.cc:363] T 14cfe357f81b40f48e9a6ff8221ae3ed P 35336ac513e14662ac43dcb2acd85121 [CANDIDATE]: Term 2 election: Vote granted by peer eff8b6e74b4044c982cb22453e464f58 I0312 20:11:59.590601 5467 leader_election.cc:258] T 14cfe357f81b40f48e9a6ff8221ae3ed P 35336ac513e14662ac43dcb2acd85121 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. I0312 20:11:59.591161 5614 raft_consensus.cc:2507] T 14cfe357f81b40f48e9a6ff8221ae3ed P 35336ac513e14662ac43dcb2acd85121 [term 2 FOLLOWER]: Leader election won for term 2 I0312 20:11:59.592371 5614 raft_consensus.cc:572] T 14cfe357f81b40f48e9a6ff8221ae3ed P 35336ac513e14662ac43dcb2acd85121 [term 2 LEADER]: Becoming Leader. State: Replica: 35336ac513e14662ac43dcb2acd85121, State: Running, Role: LEADER I0312 20:11:59.592854 5614 consensus_queue.cc:196] T 14cfe357f81b40f48e9a6ff8221ae3ed P 35336ac513e14662ac43dcb2acd85121 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "876c901434c04ed4ab4bfbf51b338b96" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 58832 } } peers { permanent_uuid: "6cb3d7538445412e92d6469ad2a62d90" member_type: VOTER last_known_addr { host: "127.0.53.4" port: 33153 } } peers { permanent_uuid: "704c3a8d0e764c66879b2e52d812ffdf" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 34688 } } peers { permanent_uuid: "eff8b6e74b4044c982cb22453e464f58" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 50867 } } peers { permanent_uuid: "35336ac513e14662ac43dcb2acd85121" member_type: VOTER last_known_addr { host: "127.0.53.5" port: 46654 } attrs { } } W0312 20:11:59.595731 5578 heartbeater.cc:591] Failed to heartbeat to 127.0.53.62:41398: Network error: Failed to ping master at 127.0.53.62:41398: Client connection negotiation failed: client connection to 127.0.53.62:41398: connect: Connection refused (error 111) I0312 20:11:59.681077 212 cluster_itest_util.cc:245] Not converged past 4 yet: 2.4 1.2 1.3 I0312 20:11:59.885241 212 cluster_itest_util.cc:245] Not converged past 4 yet: 2.4 1.2 1.3 I0312 20:11:59.981530 5036 raft_consensus.cc:1058] T 14cfe357f81b40f48e9a6ff8221ae3ed P 704c3a8d0e764c66879b2e52d812ffdf [term 2 FOLLOWER]: Refusing update from remote peer 35336ac513e14662ac43dcb2acd85121: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch) I0312 20:11:59.982694 5614 consensus_queue.cc:1017] T 14cfe357f81b40f48e9a6ff8221ae3ed P 35336ac513e14662ac43dcb2acd85121 [LEADER]: Connected to new peer: Peer: permanent_uuid: "704c3a8d0e764c66879b2e52d812ffdf" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 34688 }, Status: LMP_MISMATCH, Last received: 1.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s W0312 20:11:59.986919 5079 heartbeater.cc:591] Failed to heartbeat to 127.0.53.62:41398: Network error: Failed to ping master at 127.0.53.62:41398: Client connection negotiation failed: client connection to 127.0.53.62:41398: connect: Connection refused (error 111) I0312 20:12:00.032699 5161 raft_consensus.cc:1058] T 14cfe357f81b40f48e9a6ff8221ae3ed P eff8b6e74b4044c982cb22453e464f58 [term 2 FOLLOWER]: Refusing update from remote peer 35336ac513e14662ac43dcb2acd85121: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 4. (index mismatch) I0312 20:12:00.033604 5614 consensus_queue.cc:1017] T 14cfe357f81b40f48e9a6ff8221ae3ed P 35336ac513e14662ac43dcb2acd85121 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eff8b6e74b4044c982cb22453e464f58" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 50867 }, Status: LMP_MISMATCH, Last received: 1.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s W0312 20:12:00.038377 5204 heartbeater.cc:591] Failed to heartbeat to 127.0.53.62:41398: Network error: Failed to ping master at 127.0.53.62:41398: Client connection negotiation failed: client connection to 127.0.53.62:41398: connect: Connection refused (error 111) I0312 20:12:00.038524 5647 raft_consensus.cc:2654] T 14cfe357f81b40f48e9a6ff8221ae3ed P 35336ac513e14662ac43dcb2acd85121 [term 2 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 35336ac513e14662ac43dcb2acd85121 (127.0.53.5) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "876c901434c04ed4ab4bfbf51b338b96" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 58832 } } peers { permanent_uuid: "6cb3d7538445412e92d6469ad2a62d90" member_type: VOTER last_known_addr { host: "127.0.53.4" port: 33153 } } peers { permanent_uuid: "704c3a8d0e764c66879b2e52d812ffdf" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 34688 } } peers { permanent_uuid: "eff8b6e74b4044c982cb22453e464f58" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 50867 } } peers { permanent_uuid: "35336ac513e14662ac43dcb2acd85121" member_type: VOTER last_known_addr { host: "127.0.53.5" port: 46654 } attrs { } } } I0312 20:12:00.039470 5161 raft_consensus.cc:2654] T 14cfe357f81b40f48e9a6ff8221ae3ed P eff8b6e74b4044c982cb22453e464f58 [term 2 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 35336ac513e14662ac43dcb2acd85121 (127.0.53.5) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "876c901434c04ed4ab4bfbf51b338b96" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 58832 } } peers { permanent_uuid: "6cb3d7538445412e92d6469ad2a62d90" member_type: VOTER last_known_addr { host: "127.0.53.4" port: 33153 } } peers { permanent_uuid: "704c3a8d0e764c66879b2e52d812ffdf" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 34688 } } peers { permanent_uuid: "eff8b6e74b4044c982cb22453e464f58" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 50867 } } peers { permanent_uuid: "35336ac513e14662ac43dcb2acd85121" member_type: VOTER last_known_addr { host: "127.0.53.5" port: 46654 } attrs { } } } W0312 20:12:00.044215 5578 heartbeater.cc:591] Failed to heartbeat to 127.0.53.62:41398: Network error: Failed to ping master at 127.0.53.62:41398: Client connection negotiation failed: client connection to 127.0.53.62:41398: connect: Connection refused (error 111) W0312 20:12:00.046288 5204 heartbeater.cc:591] Failed to heartbeat to 127.0.53.62:41398: Network error: Failed to ping master at 127.0.53.62:41398: Client connection negotiation failed: client connection to 127.0.53.62:41398: connect: Connection refused (error 111) I0312 20:12:00.046835 5036 raft_consensus.cc:2654] T 14cfe357f81b40f48e9a6ff8221ae3ed P 704c3a8d0e764c66879b2e52d812ffdf [term 2 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 35336ac513e14662ac43dcb2acd85121 (127.0.53.5) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "876c901434c04ed4ab4bfbf51b338b96" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 58832 } } peers { permanent_uuid: "6cb3d7538445412e92d6469ad2a62d90" member_type: VOTER last_known_addr { host: "127.0.53.4" port: 33153 } } peers { permanent_uuid: "704c3a8d0e764c66879b2e52d812ffdf" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 34688 } } peers { permanent_uuid: "eff8b6e74b4044c982cb22453e464f58" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 50867 } } peers { permanent_uuid: "35336ac513e14662ac43dcb2acd85121" member_type: VOTER last_known_addr { host: "127.0.53.5" port: 46654 } attrs { } } } W0312 20:12:00.048080 5204 heartbeater.cc:591] Failed to heartbeat to 127.0.53.62:41398: Network error: Failed to ping master at 127.0.53.62:41398: Client connection negotiation failed: client connection to 127.0.53.62:41398: connect: Connection refused (error 111) W0312 20:12:00.048739 5578 heartbeater.cc:591] Failed to heartbeat to 127.0.53.62:41398: Network error: Failed to ping master at 127.0.53.62:41398: Client connection negotiation failed: client connection to 127.0.53.62:41398: connect: Connection refused (error 111) W0312 20:12:00.050298 5578 heartbeater.cc:591] Failed to heartbeat to 127.0.53.62:41398: Network error: Failed to ping master at 127.0.53.62:41398: Client connection negotiation failed: client connection to 127.0.53.62:41398: connect: Connection refused (error 111) W0312 20:12:00.051831 5036 rpcz_store.cc:251] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:54347 (request call id 48) took 609ms (client timeout 100). W0312 20:12:00.052834 5036 rpcz_store.cc:255] Trace: 0312 20:11:59.442718 (+ 0us) service_pool.cc:163] Inserting onto call queue 0312 20:11:59.442806 (+ 88us) service_pool.cc:222] Handling call 0312 20:11:59.446251 (+ 3445us) log.cc:582] Serialized 582 byte log entry 0312 20:12:00.051774 (+605523us) inbound_call.cc:157] Queueing success response Metrics: {"dns_us":139,"mutex_wait_us":37,"raft.queue_time_us":1028,"raft.run_cpu_time_us":1070,"raft.run_wall_time_us":2436,"spinlock_wait_cycles":1935360,"thread_start_us":1015,"threads_started":1} W0312 20:12:00.053026 5079 heartbeater.cc:591] Failed to heartbeat to 127.0.53.62:41398: Network error: Failed to ping master at 127.0.53.62:41398: Client connection negotiation failed: client connection to 127.0.53.62:41398: connect: Connection refused (error 111) W0312 20:12:00.055301 5079 heartbeater.cc:591] Failed to heartbeat to 127.0.53.62:41398: Network error: Failed to ping master at 127.0.53.62:41398: Client connection negotiation failed: client connection to 127.0.53.62:41398: connect: Connection refused (error 111) I0312 20:12:00.189630 212 raft_consensus_election-itest.cc:454] Resuming remaining nodes... I0312 20:12:00.192230 5286 raft_consensus.cc:1058] T 14cfe357f81b40f48e9a6ff8221ae3ed P 876c901434c04ed4ab4bfbf51b338b96 [term 1 FOLLOWER]: Refusing update from remote peer 704c3a8d0e764c66879b2e52d812ffdf: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch) I0312 20:12:00.194244 5411 raft_consensus.cc:1058] T 14cfe357f81b40f48e9a6ff8221ae3ed P 6cb3d7538445412e92d6469ad2a62d90 [term 1 FOLLOWER]: Refusing update from remote peer 704c3a8d0e764c66879b2e52d812ffdf: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch) I0312 20:12:00.224303 5411 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "14cfe357f81b40f48e9a6ff8221ae3ed" candidate_uuid: "35336ac513e14662ac43dcb2acd85121" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "6cb3d7538445412e92d6469ad2a62d90" I0312 20:12:00.224711 5411 raft_consensus.cc:1571] T 14cfe357f81b40f48e9a6ff8221ae3ed P 6cb3d7538445412e92d6469ad2a62d90 [term 1 FOLLOWER]: Handling vote request from an unknown peer 35336ac513e14662ac43dcb2acd85121 I0312 20:12:00.224895 5411 raft_consensus.cc:2752] T 14cfe357f81b40f48e9a6ff8221ae3ed P 6cb3d7538445412e92d6469ad2a62d90 [term 1 FOLLOWER]: Advancing to term 2 I0312 20:12:00.229495 5411 raft_consensus.cc:2215] T 14cfe357f81b40f48e9a6ff8221ae3ed P 6cb3d7538445412e92d6469ad2a62d90 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35336ac513e14662ac43dcb2acd85121 in term 2. I0312 20:12:00.230159 5410 raft_consensus.cc:1058] T 14cfe357f81b40f48e9a6ff8221ae3ed P 6cb3d7538445412e92d6469ad2a62d90 [term 2 FOLLOWER]: Refusing update from remote peer 35336ac513e14662ac43dcb2acd85121: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 4. (index mismatch) I0312 20:12:00.231804 5614 consensus_queue.cc:1017] T 14cfe357f81b40f48e9a6ff8221ae3ed P 35336ac513e14662ac43dcb2acd85121 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6cb3d7538445412e92d6469ad2a62d90" member_type: VOTER last_known_addr { host: "127.0.53.4" port: 33153 }, Status: LMP_MISMATCH, Last received: 1.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s I0312 20:12:00.235338 5470 leader_election.cc:363] T 14cfe357f81b40f48e9a6ff8221ae3ed P 35336ac513e14662ac43dcb2acd85121 [CANDIDATE]: Term 2 election: Vote granted by peer 6cb3d7538445412e92d6469ad2a62d90 W0312 20:12:00.250497 5454 heartbeater.cc:591] Failed to heartbeat to 127.0.53.62:41398: Network error: Failed to ping master at 127.0.53.62:41398: Client connection negotiation failed: client connection to 127.0.53.62:41398: connect: Connection refused (error 111) I0312 20:12:00.249897 5410 raft_consensus.cc:2654] T 14cfe357f81b40f48e9a6ff8221ae3ed P 6cb3d7538445412e92d6469ad2a62d90 [term 2 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 35336ac513e14662ac43dcb2acd85121 (127.0.53.5) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "876c901434c04ed4ab4bfbf51b338b96" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 58832 } } peers { permanent_uuid: "6cb3d7538445412e92d6469ad2a62d90" member_type: VOTER last_known_addr { host: "127.0.53.4" port: 33153 } } peers { permanent_uuid: "704c3a8d0e764c66879b2e52d812ffdf" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 34688 } } peers { permanent_uuid: "eff8b6e74b4044c982cb22453e464f58" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 50867 } } peers { permanent_uuid: "35336ac513e14662ac43dcb2acd85121" member_type: VOTER last_known_addr { host: "127.0.53.5" port: 46654 } attrs { } } } W0312 20:12:00.258738 5454 heartbeater.cc:591] Failed to heartbeat to 127.0.53.62:41398: Network error: Failed to ping master at 127.0.53.62:41398: Client connection negotiation failed: client connection to 127.0.53.62:41398: connect: Connection refused (error 111) W0312 20:12:00.260336 5454 heartbeater.cc:591] Failed to heartbeat to 127.0.53.62:41398: Network error: Failed to ping master at 127.0.53.62:41398: Client connection negotiation failed: client connection to 127.0.53.62:41398: connect: Connection refused (error 111) I0312 20:12:00.264636 5286 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "14cfe357f81b40f48e9a6ff8221ae3ed" candidate_uuid: "35336ac513e14662ac43dcb2acd85121" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "876c901434c04ed4ab4bfbf51b338b96" I0312 20:12:00.265203 5285 raft_consensus.cc:2752] T 14cfe357f81b40f48e9a6ff8221ae3ed P 876c901434c04ed4ab4bfbf51b338b96 [term 1 FOLLOWER]: Advancing to term 2 I0312 20:12:00.268206 5285 raft_consensus.cc:1058] T 14cfe357f81b40f48e9a6ff8221ae3ed P 876c901434c04ed4ab4bfbf51b338b96 [term 2 FOLLOWER]: Refusing update from remote peer 35336ac513e14662ac43dcb2acd85121: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 4. (index mismatch) I0312 20:12:00.268489 5286 raft_consensus.cc:1571] T 14cfe357f81b40f48e9a6ff8221ae3ed P 876c901434c04ed4ab4bfbf51b338b96 [term 2 FOLLOWER]: Handling vote request from an unknown peer 35336ac513e14662ac43dcb2acd85121 W0312 20:12:00.269495 5578 heartbeater.cc:591] Failed to heartbeat to 127.0.53.62:41398: Network error: Failed to ping master at 127.0.53.62:41398: Client connection negotiation failed: client connection to 127.0.53.62:41398: connect: Connection refused (error 111) I0312 20:12:00.272289 5286 raft_consensus.cc:2215] T 14cfe357f81b40f48e9a6ff8221ae3ed P 876c901434c04ed4ab4bfbf51b338b96 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35336ac513e14662ac43dcb2acd85121 in term 2. I0312 20:12:00.272886 5647 consensus_queue.cc:1017] T 14cfe357f81b40f48e9a6ff8221ae3ed P 35336ac513e14662ac43dcb2acd85121 [LEADER]: Connected to new peer: Peer: permanent_uuid: "876c901434c04ed4ab4bfbf51b338b96" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 58832 }, Status: LMP_MISMATCH, Last received: 1.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s I0312 20:12:00.273507 5466 leader_election.cc:363] T 14cfe357f81b40f48e9a6ff8221ae3ed P 35336ac513e14662ac43dcb2acd85121 [CANDIDATE]: Term 2 election: Vote granted by peer 876c901434c04ed4ab4bfbf51b338b96 W0312 20:12:00.290866 5329 heartbeater.cc:591] Failed to heartbeat to 127.0.53.62:41398: Network error: Failed to ping master at 127.0.53.62:41398: Client connection negotiation failed: client connection to 127.0.53.62:41398: connect: Connection refused (error 111) I0312 20:12:00.293114 5286 raft_consensus.cc:2654] T 14cfe357f81b40f48e9a6ff8221ae3ed P 876c901434c04ed4ab4bfbf51b338b96 [term 2 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 35336ac513e14662ac43dcb2acd85121 (127.0.53.5) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "876c901434c04ed4ab4bfbf51b338b96" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 58832 } } peers { permanent_uuid: "6cb3d7538445412e92d6469ad2a62d90" member_type: VOTER last_known_addr { host: "127.0.53.4" port: 33153 } } peers { permanent_uuid: "704c3a8d0e764c66879b2e52d812ffdf" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 34688 } } peers { permanent_uuid: "eff8b6e74b4044c982cb22453e464f58" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 50867 } } peers { permanent_uuid: "35336ac513e14662ac43dcb2acd85121" member_type: VOTER last_known_addr { host: "127.0.53.5" port: 46654 } attrs { } } } W0312 20:12:00.298205 5329 heartbeater.cc:591] Failed to heartbeat to 127.0.53.62:41398: Network error: Failed to ping master at 127.0.53.62:41398: Client connection negotiation failed: client connection to 127.0.53.62:41398: connect: Connection refused (error 111) W0312 20:12:00.300016 5329 heartbeater.cc:591] Failed to heartbeat to 127.0.53.62:41398: Network error: Failed to ping master at 127.0.53.62:41398: Client connection negotiation failed: client connection to 127.0.53.62:41398: connect: Connection refused (error 111) I0312 20:12:00.300354 212 external_mini_cluster.cc:997] Killing /tmp/dist-test-task3bwPeE/build/tsan/bin/kudu-tserver with pid 4948 W0312 20:12:00.301082 5578 heartbeater.cc:591] Failed to heartbeat to 127.0.53.62:41398: Network error: Failed to ping master at 127.0.53.62:41398: Client connection negotiation failed: client connection to 127.0.53.62:41398: connect: Connection refused (error 111) W0312 20:12:00.324698 4885 connection.cc:513] client connection to 127.0.53.1:34688 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0312 20:12:00.324728 5467 connection.cc:513] client connection to 127.0.53.1:34688 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0312 20:12:00.324759 5219 connection.cc:513] server connection from 127.0.53.1:56544 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0312 20:12:00.325533 4882 connection.cc:513] client connection to 127.0.53.1:34688 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0312 20:12:00.326197 5093 connection.cc:513] server connection from 127.0.53.1:38583 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0312 20:12:00.326352 5343 connection.cc:513] server connection from 127.0.53.1:43742 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0312 20:12:00.327064 5470 connection.cc:513] server connection from 127.0.53.1:35407 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) I0312 20:12:00.327834 212 external_mini_cluster.cc:997] Killing /tmp/dist-test-task3bwPeE/build/tsan/bin/kudu-tserver with pid 5083 W0312 20:12:00.355239 4885 connection.cc:513] client connection to 127.0.53.2:50867 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0312 20:12:00.355315 5467 connection.cc:513] client connection to 127.0.53.2:50867 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0312 20:12:00.356117 4882 connection.cc:513] client connection to 127.0.53.2:50867 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) I0312 20:12:00.356344 212 external_mini_cluster.cc:997] Killing /tmp/dist-test-task3bwPeE/build/tsan/bin/kudu-tserver with pid 5208 W0312 20:12:00.384469 5466 connection.cc:513] client connection to 127.0.53.3:58832 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0312 20:12:00.384469 4885 connection.cc:513] client connection to 127.0.53.3:58832 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0312 20:12:00.385320 4881 connection.cc:513] client connection to 127.0.53.3:58832 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) I0312 20:12:00.386122 212 external_mini_cluster.cc:997] Killing /tmp/dist-test-task3bwPeE/build/tsan/bin/kudu-tserver with pid 5333 W0312 20:12:00.414170 4885 connection.cc:513] client connection to 127.0.53.4:33153 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) I0312 20:12:00.414362 212 external_mini_cluster.cc:997] Killing /tmp/dist-test-task3bwPeE/build/tsan/bin/kudu-tserver with pid 5457 W0312 20:12:00.417331 4884 connection.cc:513] client connection to 127.0.53.4:33153 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0312 20:12:00.437188 4884 connection.cc:513] client connection to 127.0.53.5:46654 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0312 20:12:00.437247 4885 connection.cc:513] client connection to 127.0.53.5:46654 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) [ OK ] RaftConsensusElectionITest.ElectPendingVoter (5099 ms) [ RUN ] RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy I0312 20:12:00.450675 212 test_util.cc:192] Using random seed: -1053768611 I0312 20:12:00.453670 212 ts_itest-base.cc:113] Starting cluster with: I0312 20:12:00.453773 212 ts_itest-base.cc:114] -------------- I0312 20:12:00.453860 212 ts_itest-base.cc:115] 3 tablet servers I0312 20:12:00.453946 212 ts_itest-base.cc:116] 3 replicas per TS I0312 20:12:00.454033 212 ts_itest-base.cc:117] -------------- I0312 20:12:00.456128 212 external_mini_cluster.cc:805] Running /tmp/dist-test-task3bwPeE/build/tsan/bin/kudu-master /tmp/dist-test-task3bwPeE/build/tsan/bin/kudu-master --fs_wal_dir=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1520885401735786-212/raft_consensus-itest-cluster/master-0/wal --fs_data_dirs=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1520885401735786-212/raft_consensus-itest-cluster/master-0/data --block_manager=log --webserver_interface=localhost --ipki_ca_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.0.53.62:0 --webserver_interface=127.0.53.62 --webserver_port=0 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1520885401735786-212/raft_consensus-itest-cluster/master-0/logs --server_dump_info_path=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1520885401735786-212/raft_consensus-itest-cluster/master-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --master_add_server_when_underreplicated=false --raft_prepare_replacement_before_eviction=false with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W0312 20:12:00.575541 5677 flags.cc:412] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0312 20:12:00.575876 5677 flags.cc:412] Enabled unsafe flag: --never_fsync=true W0312 20:12:00.591289 5677 flags.cc:412] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false W0312 20:12:00.591483 5677 flags.cc:412] Enabled experimental flag: --ipki_ca_key_size=1024 W0312 20:12:00.591599 5677 flags.cc:412] Enabled experimental flag: --ipki_server_key_size=1024 W0312 20:12:00.591711 5677 flags.cc:412] Enabled experimental flag: --tsk_num_rsa_bits=512 I0312 20:12:00.612419 5677 master_main.cc:66] Master server non-default flags: --raft_prepare_replacement_before_eviction=false --fs_data_dirs=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1520885401735786-212/raft_consensus-itest-cluster/master-0/data --fs_wal_dir=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1520885401735786-212/raft_consensus-itest-cluster/master-0/wal --master_add_server_when_underreplicated=false --ipki_ca_key_size=1024 --ipki_server_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.0.53.62:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1520885401735786-212/raft_consensus-itest-cluster/master-0/data/info.pb --webserver_interface=127.0.53.62 --webserver_port=0 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --log_dir=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1520885401735786-212/raft_consensus-itest-cluster/master-0/logs --logbuflevel=-1 --logtostderr=true Master server version: kudu 1.7.0-SNAPSHOT revision 13e10c63c46de30ac85f34b543cdd6fb44881f61 build type FASTDEBUG built by None at 12 Mar 2018 20:06:40 UTC on jenkins-slave-vn0g build id 12388 TSAN enabled I0312 20:12:00.621457 5677 master_main.cc:73] Initializing master server... I0312 20:12:00.622974 5677 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 1127276us I0312 20:12:00.623687 5677 fs_manager.cc:260] Metadata directory not provided I0312 20:12:00.623827 5677 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0312 20:12:00.624042 5677 server_base.cc:434] Could not load existing FS layout: Not found: /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1520885401735786-212/raft_consensus-itest-cluster/master-0/data/instance: No such file or directory (error 2) I0312 20:12:00.624152 5677 server_base.cc:435] Attempting to create new FS layout instead I0312 20:12:00.631518 5677 fs_manager.cc:596] Generated new instance metadata in path /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1520885401735786-212/raft_consensus-itest-cluster/master-0/data/instance: uuid: "edc74cc9e16f4934b5ef99cafc75d0b9" format_stamp: "Formatted at 2018-03-12 20:12:00 on dist-test-slave-dist-test-slave-f27p" I0312 20:12:00.632208 5677 fs_manager.cc:596] Generated new instance metadata in path /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1520885401735786-212/raft_consensus-itest-cluster/master-0/wal/instance: uuid: "edc74cc9e16f4934b5ef99cafc75d0b9" format_stamp: "Formatted at 2018-03-12 20:12:00 on dist-test-slave-dist-test-slave-f27p" I0312 20:12:00.640383 5677 fs_manager.cc:496] Time spent creating directory manager: real 0.008s user 0.004s sys 0.004s I0312 20:12:00.641549 5677 env_posix.cc:1620] Not raising this process' open files per process limit of 65536; it is already as high as it can go I0312 20:12:00.641829 5677 file_cache.cc:470] Constructed file cache lbm with capacity 26214 I0312 20:12:00.648200 5677 fs_manager.cc:418] Time spent opening block manager: real 0.001s user 0.004s sys 0.000s I0312 20:12:00.648313 5677 fs_manager.cc:429] Opened local filesystem: /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1520885401735786-212/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1520885401735786-212/raft_consensus-itest-cluster/master-0/wal uuid: "edc74cc9e16f4934b5ef99cafc75d0b9" format_stamp: "Formatted at 2018-03-12 20:12:00 on dist-test-slave-dist-test-slave-f27p" I0312 20:12:00.648519 5677 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1520885401735786-212/raft_consensus-itest-cluster/master-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0312 20:12:00.833407 5677 env_posix.cc:1620] Not raising this process' running threads per effective uid limit of 60284; it is already as high as it can go I0312 20:12:00.834049 5677 master_main.cc:76] Starting Master server... I0312 20:12:00.835279 5694 process_memory.cc:182] Process hard memory limit is 11.777631 GB I0312 20:12:00.835422 5694 process_memory.cc:184] Process soft memory limit is 9.422105 GB I0312 20:12:00.835515 5694 process_memory.cc:187] Process memory pressure threshold is 7.066578 GB I0312 20:12:00.873545 5677 rpc_server.cc:201] RPC server started. Bound to: 127.0.53.62:53264 I0312 20:12:00.898838 5677 webserver.cc:173] Starting webserver on 127.0.53.62:0 I0312 20:12:00.898948 5677 webserver.cc:184] Document root disabled I0312 20:12:00.900194 5677 webserver.cc:311] Webserver started. Bound to: http://127.0.53.62:36897/ I0312 20:12:00.900955 5677 server_base.cc:607] Dumped server information to /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1520885401735786-212/raft_consensus-itest-cluster/master-0/data/info.pb I0312 20:12:00.903712 5738 data_dirs.cc:931] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0312 20:12:00.909966 212 external_mini_cluster.cc:867] Started /tmp/dist-test-task3bwPeE/build/tsan/bin/kudu-master as pid 5677 I0312 20:12:00.911598 212 external_mini_cluster.cc:805] Running /tmp/dist-test-task3bwPeE/build/tsan/bin/kudu-tserver /tmp/dist-test-task3bwPeE/build/tsan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1520885401735786-212/raft_consensus-itest-cluster/ts-0/wal --fs_data_dirs=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1520885401735786-212/raft_consensus-itest-cluster/ts-0/data --block_manager=log --rpc_bind_addresses=127.0.53.1:0 --local_ip_for_outbound_sockets=127.0.53.1 --webserver_interface=127.0.53.1 --webserver_port=0 --tserver_master_addrs=127.0.53.62:53264 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1520885401735786-212/raft_consensus-itest-cluster/ts-0/logs --server_dump_info_path=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1520885401735786-212/raft_consensus-itest-cluster/ts-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=false --log_compression_codec=none --log_cache_size_limit_mb=1 --log_segment_size_mb=1 --log_async_preallocate_segments=false --log_min_segments_to_retain=1 --log_max_segments_to_retain=3 --maintenance_manager_polling_interval_ms=100 --log_target_replay_size_mb=1 --max_cell_size_bytes=1000000 with env {} I0312 20:12:00.920017 5738 tablet_bootstrap.cc:436] T 00000000000000000000000000000000 P edc74cc9e16f4934b5ef99cafc75d0b9: Bootstrap starting. I0312 20:12:00.922336 5738 tablet_bootstrap.cc:581] T 00000000000000000000000000000000 P edc74cc9e16f4934b5ef99cafc75d0b9: No blocks or log segments found. Creating new log. I0312 20:12:00.923552 5738 log.cc:520] T 00000000000000000000000000000000 P edc74cc9e16f4934b5ef99cafc75d0b9: Log is configured to *not* fsync() on all Append() calls I0312 20:12:00.925971 5738 tablet_bootstrap.cc:436] T 00000000000000000000000000000000 P edc74cc9e16f4934b5ef99cafc75d0b9: No bootstrap required, opened a new log I0312 20:12:00.937888 5738 raft_consensus.cc:304] T 00000000000000000000000000000000 P edc74cc9e16f4934b5ef99cafc75d0b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edc74cc9e16f4934b5ef99cafc75d0b9" member_type: VOTER } I0312 20:12:00.938201 5738 raft_consensus.cc:330] T 00000000000000000000000000000000 P edc74cc9e16f4934b5ef99cafc75d0b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0312 20:12:00.938325 5738 raft_consensus.cc:605] T 00000000000000000000000000000000 P edc74cc9e16f4934b5ef99cafc75d0b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: edc74cc9e16f4934b5ef99cafc75d0b9, State: Initialized, Role: FOLLOWER I0312 20:12:00.938760 5738 consensus_queue.cc:219] T 00000000000000000000000000000000 P edc74cc9e16f4934b5ef99cafc75d0b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edc74cc9e16f4934b5ef99cafc75d0b9" member_type: VOTER } I0312 20:12:00.939110 5738 raft_consensus.cc:344] T 00000000000000000000000000000000 P edc74cc9e16f4934b5ef99cafc75d0b9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0312 20:12:00.939260 5738 raft_consensus.cc:436] T 00000000000000000000000000000000 P edc74cc9e16f4934b5ef99cafc75d0b9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0312 20:12:00.939402 5738 raft_consensus.cc:2700] T 00000000000000000000000000000000 P edc74cc9e16f4934b5ef99cafc75d0b9: Snoozing failure detection for 1.501s (starting election) I0312 20:12:00.939538 5738 raft_consensus.cc:2752] T 00000000000000000000000000000000 P edc74cc9e16f4934b5ef99cafc75d0b9 [term 0 FOLLOWER]: Advancing to term 1 I0312 20:12:00.941958 5738 raft_consensus.cc:458] T 00000000000000000000000000000000 P edc74cc9e16f4934b5ef99cafc75d0b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edc74cc9e16f4934b5ef99cafc75d0b9" member_type: VOTER } I0312 20:12:00.942378 5738 leader_election.cc:258] T 00000000000000000000000000000000 P edc74cc9e16f4934b5ef99cafc75d0b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0312 20:12:00.943976 5741 raft_consensus.cc:2700] T 00000000000000000000000000000000 P edc74cc9e16f4934b5ef99cafc75d0b9: Snoozing failure detection for 1.641s (election complete) I0312 20:12:00.944278 5741 raft_consensus.cc:2507] T 00000000000000000000000000000000 P edc74cc9e16f4934b5ef99cafc75d0b9 [term 1 FOLLOWER]: Leader election won for term 1 I0312 20:12:00.949563 5741 raft_consensus.cc:572] T 00000000000000000000000000000000 P edc74cc9e16f4934b5ef99cafc75d0b9 [term 1 LEADER]: Becoming Leader. State: Replica: edc74cc9e16f4934b5ef99cafc75d0b9, State: Running, Role: LEADER I0312 20:12:00.950281 5741 consensus_queue.cc:196] T 00000000000000000000000000000000 P edc74cc9e16f4934b5ef99cafc75d0b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edc74cc9e16f4934b5ef99cafc75d0b9" member_type: VOTER } I0312 20:12:00.966538 5738 sys_catalog.cc:335] T 00000000000000000000000000000000 P edc74cc9e16f4934b5ef99cafc75d0b9 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 1 leader_uuid: "edc74cc9e16f4934b5ef99cafc75d0b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edc74cc9e16f4934b5ef99cafc75d0b9" member_type: VOTER } } I0312 20:12:00.966951 5738 sys_catalog.cc:338] T 00000000000000000000000000000000 P edc74cc9e16f4934b5ef99cafc75d0b9 [sys.catalog]: This master's current role is: LEADER I0312 20:12:00.967464 5743 sys_catalog.cc:335] T 00000000000000000000000000000000 P edc74cc9e16f4934b5ef99cafc75d0b9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "edc74cc9e16f4934b5ef99cafc75d0b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edc74cc9e16f4934b5ef99cafc75d0b9" member_type: VOTER } } I0312 20:12:00.967829 5743 sys_catalog.cc:338] T 00000000000000000000000000000000 P edc74cc9e16f4934b5ef99cafc75d0b9 [sys.catalog]: This master's current role is: LEADER I0312 20:12:00.968446 5742 sys_catalog.cc:335] T 00000000000000000000000000000000 P edc74cc9e16f4934b5ef99cafc75d0b9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader edc74cc9e16f4934b5ef99cafc75d0b9. Latest consensus state: current_term: 1 leader_uuid: "edc74cc9e16f4934b5ef99cafc75d0b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edc74cc9e16f4934b5ef99cafc75d0b9" member_type: VOTER } } I0312 20:12:00.968907 5742 sys_catalog.cc:338] T 00000000000000000000000000000000 P edc74cc9e16f4934b5ef99cafc75d0b9 [sys.catalog]: This master's current role is: LEADER I0312 20:12:00.969882 5738 sys_catalog.cc:422] T 00000000000000000000000000000000 P edc74cc9e16f4934b5ef99cafc75d0b9 [sys.catalog]: configured and running, proceeding with master startup. I0312 20:12:00.970181 5746 catalog_manager.cc:972] Loading table and tablet metadata into memory... I0312 20:12:00.971642 5677 master_main.cc:79] Master server successfully started. I0312 20:12:00.975659 5746 catalog_manager.cc:983] Initializing Kudu internal certificate authority... I0312 20:12:01.014997 5746 catalog_manager.cc:870] Generated new certificate authority record I0312 20:12:01.017678 5746 catalog_manager.cc:992] Loading token signing keys... I0312 20:12:01.028385 5746 catalog_manager.cc:3868] T 00000000000000000000000000000000 P edc74cc9e16f4934b5ef99cafc75d0b9: Generated new TSK 0 WARNING: Logging before InitGoogleLogging() is written to STDERR W0312 20:12:01.053784 5739 flags.cc:412] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0312 20:12:01.054116 5739 flags.cc:412] Enabled unsafe flag: --max_cell_size_bytes=1000000 W0312 20:12:01.054242 5739 flags.cc:412] Enabled unsafe flag: --never_fsync=true W0312 20:12:01.068248 5739 flags.cc:412] Enabled experimental flag: --log_compression_codec=none W0312 20:12:01.068387 5739 flags.cc:412] Enabled experimental flag: --log_max_segments_to_retain=3 W0312 20:12:01.068558 5739 flags.cc:412] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false W0312 20:12:01.068712 5739 flags.cc:412] Enabled experimental flag: --ipki_server_key_size=1024 W0312 20:12:01.068996 5739 flags.cc:412] Enabled experimental flag: --log_target_replay_size_mb=1 W0312 20:12:01.069140 5739 flags.cc:412] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.53.1 I0312 20:12:01.088886 5739 tablet_server_main.cc:72] Tablet server non-default flags: --log_compression_codec=none --log_max_segments_to_retain=3 --log_cache_size_limit_mb=1 --log_async_preallocate_segments=false --log_segment_size_mb=1 --raft_prepare_replacement_before_eviction=false --fs_data_dirs=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1520885401735786-212/raft_consensus-itest-cluster/ts-0/data --fs_wal_dir=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1520885401735786-212/raft_consensus-itest-cluster/ts-0/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.0.53.1:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1520885401735786-212/raft_consensus-itest-cluster/ts-0/data/info.pb --webserver_interface=127.0.53.1 --webserver_port=0 --max_cell_size_bytes=1000000 --tserver_master_addrs=127.0.53.62:53264 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --log_target_replay_size_mb=1 --maintenance_manager_polling_interval_ms=100 --enable_minidumps=false --local_ip_for_outbound_sockets=127.0.53.1 --log_dir=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1520885401735786-212/raft_consensus-itest-cluster/ts-0/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.7.0-SNAPSHOT revision 13e10c63c46de30ac85f34b543cdd6fb44881f61 build type FASTDEBUG built by None at 12 Mar 2018 20:06:40 UTC on jenkins-slave-vn0g build id 12388 TSAN enabled I0312 20:12:01.098865 5739 tablet_server_main.cc:79] Initializing tablet server... I0312 20:12:01.100606 5739 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 1127776us I0312 20:12:01.101028 5739 fs_manager.cc:260] Metadata directory not provided I0312 20:12:01.101163 5739 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0312 20:12:01.101367 5739 server_base.cc:434] Could not load existing FS layout: Not found: /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1520885401735786-212/raft_consensus-itest-cluster/ts-0/data/instance: No such file or directory (error 2) I0312 20:12:01.101481 5739 server_base.cc:435] Attempting to create new FS layout instead I0312 20:12:01.108708 5739 fs_manager.cc:596] Generated new instance metadata in path /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1520885401735786-212/raft_consensus-itest-cluster/ts-0/data/instance: uuid: "ac1592be0a6240378933768820f57ed3" format_stamp: "Formatted at 2018-03-12 20:12:01 on dist-test-slave-dist-test-slave-f27p" I0312 20:12:01.109398 5739 fs_manager.cc:596] Generated new instance metadata in path /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1520885401735786-212/raft_consensus-itest-cluster/ts-0/wal/instance: uuid: "ac1592be0a6240378933768820f57ed3" format_stamp: "Formatted at 2018-03-12 20:12:01 on dist-test-slave-dist-test-slave-f27p" I0312 20:12:01.117768 5739 fs_manager.cc:496] Time spent creating directory manager: real 0.008s user 0.004s sys 0.004s I0312 20:12:01.118878 5739 env_posix.cc:1620] Not raising this process' open files per process limit of 65536; it is already as high as it can go I0312 20:12:01.119154 5739 file_cache.cc:470] Constructed file cache lbm with capacity 26214 I0312 20:12:01.125787 5739 fs_manager.cc:418] Time spent opening block manager: real 0.002s user 0.004s sys 0.000s I0312 20:12:01.125902 5739 fs_manager.cc:429] Opened local filesystem: /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1520885401735786-212/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1520885401735786-212/raft_consensus-itest-cluster/ts-0/wal uuid: "ac1592be0a6240378933768820f57ed3" format_stamp: "Formatted at 2018-03-12 20:12:01 on dist-test-slave-dist-test-slave-f27p" I0312 20:12:01.126106 5739 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1520885401735786-212/raft_consensus-itest-cluster/ts-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0312 20:12:01.372442 5739 env_posix.cc:1620] Not raising this process' running threads per effective uid limit of 60284; it is already as high as it can go I0312 20:12:01.374295 5739 ts_tablet_manager.cc:349] Loaded tablet metadata (0 live tablets) I0312 20:12:01.375438 5739 tablet_server_main.cc:84] Starting tablet server... I0312 20:12:01.518048 5739 rpc_server.cc:201] RPC server started. Bound to: 127.0.53.1:59968 I0312 20:12:01.519376 5739 webserver.cc:173] Starting webserver on 127.0.53.1:0 I0312 20:12:01.519531 5739 webserver.cc:184] Document root disabled I0312 20:12:01.521034 5739 webserver.cc:311] Webserver started. Bound to: http://127.0.53.1:46305/ I0312 20:12:01.522004 5739 server_base.cc:607] Dumped server information to /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1520885401735786-212/raft_consensus-itest-cluster/ts-0/data/info.pb I0312 20:12:01.529635 212 external_mini_cluster.cc:867] Started /tmp/dist-test-task3bwPeE/build/tsan/bin/kudu-tserver as pid 5739 I0312 20:12:01.531029 212 external_mini_cluster.cc:805] Running /tmp/dist-test-task3bwPeE/build/tsan/bin/kudu-tserver /tmp/dist-test-task3bwPeE/build/tsan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1520885401735786-212/raft_consensus-itest-cluster/ts-1/wal --fs_data_dirs=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1520885401735786-212/raft_consensus-itest-cluster/ts-1/data --block_manager=log --rpc_bind_addresses=127.0.53.2:0 --local_ip_for_outbound_sockets=127.0.53.2 --webserver_interface=127.0.53.2 --webserver_port=0 --tserver_master_addrs=127.0.53.62:53264 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1520885401735786-212/raft_consensus-itest-cluster/ts-1/logs --server_dump_info_path=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1520885401735786-212/raft_consensus-itest-cluster/ts-1/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=false --log_compression_codec=none --log_cache_size_limit_mb=1 --log_segment_size_mb=1 --log_async_preallocate_segments=false --log_min_segments_to_retain=1 --log_max_segments_to_retain=3 --maintenance_manager_polling_interval_ms=100 --log_target_replay_size_mb=1 --max_cell_size_bytes=1000000 with env {} I0312 20:12:01.533890 5871 process_memory.cc:182] Process hard memory limit is 11.777631 GB I0312 20:12:01.534380 5871 process_memory.cc:184] Process soft memory limit is 9.422105 GB I0312 20:12:01.534512 5871 process_memory.cc:187] Process memory pressure threshold is 7.066578 GB I0312 20:12:01.534183 5739 tablet_server_main.cc:87] Tablet server successfully started. I0312 20:12:01.583808 5870 heartbeater.cc:346] Connected to a master server at 127.0.53.62:53264 I0312 20:12:01.584048 5870 heartbeater.cc:426] Registering TS with master... I0312 20:12:01.584451 5870 heartbeater.cc:475] Master 127.0.53.62:53264 requested a full tablet report, sending... I0312 20:12:01.586112 5705 ts_manager.cc:81] Registered new tserver with Master: ac1592be0a6240378933768820f57ed3 (127.0.53.1:59968) I0312 20:12:01.588541 5705 master_service.cc:243] Signed X509 certificate for tserver {username='slave'} at 127.0.53.1:38459 WARNING: Logging before InitGoogleLogging() is written to STDERR W0312 20:12:01.731539 5874 flags.cc:412] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0312 20:12:01.731885 5874 flags.cc:412] Enabled unsafe flag: --max_cell_size_bytes=1000000 W0312 20:12:01.732012 5874 flags.cc:412] Enabled unsafe flag: --never_fsync=true W0312 20:12:01.746232 5874 flags.cc:412] Enabled experimental flag: --log_compression_codec=none W0312 20:12:01.746374 5874 flags.cc:412] Enabled experimental flag: --log_max_segments_to_retain=3 W0312 20:12:01.746546 5874 flags.cc:412] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false W0312 20:12:01.746692 5874 flags.cc:412] Enabled experimental flag: --ipki_server_key_size=1024 W0312 20:12:01.746978 5874 flags.cc:412] Enabled experimental flag: --log_target_replay_size_mb=1 W0312 20:12:01.747107 5874 flags.cc:412] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.53.2 I0312 20:12:01.767051 5874 tablet_server_main.cc:72] Tablet server non-default flags: --log_compression_codec=none --log_max_segments_to_retain=3 --log_cache_size_limit_mb=1 --log_async_preallocate_segments=false --log_segment_size_mb=1 --raft_prepare_replacement_before_eviction=false --fs_data_dirs=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1520885401735786-212/raft_consensus-itest-cluster/ts-1/data --fs_wal_dir=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1520885401735786-212/raft_consensus-itest-cluster/ts-1/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.0.53.2:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1520885401735786-212/raft_consensus-itest-cluster/ts-1/data/info.pb --webserver_interface=127.0.53.2 --webserver_port=0 --max_cell_size_bytes=1000000 --tserver_master_addrs=127.0.53.62:53264 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --log_target_replay_size_mb=1 --maintenance_manager_polling_interval_ms=100 --enable_minidumps=false --local_ip_for_outbound_sockets=127.0.53.2 --log_dir=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1520885401735786-212/raft_consensus-itest-cluster/ts-1/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.7.0-SNAPSHOT revision 13e10c63c46de30ac85f34b543cdd6fb44881f61 build type FASTDEBUG built by None at 12 Mar 2018 20:06:40 UTC on jenkins-slave-vn0g build id 12388 TSAN enabled I0312 20:12:01.777415 5874 tablet_server_main.cc:79] Initializing tablet server... I0312 20:12:01.779175 5874 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 1127776us I0312 20:12:01.779624 5874 fs_manager.cc:260] Metadata directory not provided I0312 20:12:01.779762 5874 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0312 20:12:01.779965 5874 server_base.cc:434] Could not load existing FS layout: Not found: /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1520885401735786-212/raft_consensus-itest-cluster/ts-1/data/instance: No such file or directory (error 2) I0312 20:12:01.780079 5874 server_base.cc:435] Attempting to create new FS layout instead I0312 20:12:01.787261 5874 fs_manager.cc:596] Generated new instance metadata in path /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1520885401735786-212/raft_consensus-itest-cluster/ts-1/data/instance: uuid: "1183c4bf4e494b75a57827848998cacc" format_stamp: "Formatted at 2018-03-12 20:12:01 on dist-test-slave-dist-test-slave-f27p" I0312 20:12:01.787976 5874 fs_manager.cc:596] Generated new instance metadata in path /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1520885401735786-212/raft_consensus-itest-cluster/ts-1/wal/instance: uuid: "1183c4bf4e494b75a57827848998cacc" format_stamp: "Formatted at 2018-03-12 20:12:01 on dist-test-slave-dist-test-slave-f27p" I0312 20:12:01.796490 5874 fs_manager.cc:496] Time spent creating directory manager: real 0.008s user 0.012s sys 0.000s I0312 20:12:01.797646 5874 env_posix.cc:1620] Not raising this process' open files per process limit of 65536; it is already as high as it can go I0312 20:12:01.797919 5874 file_cache.cc:470] Constructed file cache lbm with capacity 26214 I0312 20:12:01.804390 5874 fs_manager.cc:418] Time spent opening block manager: real 0.002s user 0.004s sys 0.000s I0312 20:12:01.804507 5874 fs_manager.cc:429] Opened local filesystem: /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1520885401735786-212/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1520885401735786-212/raft_consensus-itest-cluster/ts-1/wal uuid: "1183c4bf4e494b75a57827848998cacc" format_stamp: "Formatted at 2018-03-12 20:12:01 on dist-test-slave-dist-test-slave-f27p" I0312 20:12:01.804713 5874 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1520885401735786-212/raft_consensus-itest-cluster/ts-1/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0312 20:12:02.026767 5874 env_posix.cc:1620] Not raising this process' running threads per effective uid limit of 60284; it is already as high as it can go I0312 20:12:02.028079 5874 ts_tablet_manager.cc:349] Loaded tablet metadata (0 live tablets) I0312 20:12:02.029228 5874 tablet_server_main.cc:84] Starting tablet server... I0312 20:12:02.135820 5874 rpc_server.cc:201] RPC server started. Bound to: 127.0.53.2:47310 I0312 20:12:02.136718 5874 webserver.cc:173] Starting webserver on 127.0.53.2:0 I0312 20:12:02.136833 5874 webserver.cc:184] Document root disabled I0312 20:12:02.138375 5874 webserver.cc:311] Webserver started. Bound to: http://127.0.53.2:50074/ I0312 20:12:02.139128 5874 server_base.cc:607] Dumped server information to /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1520885401735786-212/raft_consensus-itest-cluster/ts-1/data/info.pb I0312 20:12:02.142077 5874 tablet_server_main.cc:87] Tablet server successfully started. I0312 20:12:02.145653 212 external_mini_cluster.cc:867] Started /tmp/dist-test-task3bwPeE/build/tsan/bin/kudu-tserver as pid 5874 I0312 20:12:02.147346 212 external_mini_cluster.cc:805] Running /tmp/dist-test-task3bwPeE/build/tsan/bin/kudu-tserver /tmp/dist-test-task3bwPeE/build/tsan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1520885401735786-212/raft_consensus-itest-cluster/ts-2/wal --fs_data_dirs=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1520885401735786-212/raft_consensus-itest-cluster/ts-2/data --block_manager=log --rpc_bind_addresses=127.0.53.3:0 --local_ip_for_outbound_sockets=127.0.53.3 --webserver_interface=127.0.53.3 --webserver_port=0 --tserver_master_addrs=127.0.53.62:53264 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1520885401735786-212/raft_consensus-itest-cluster/ts-2/logs --server_dump_info_path=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1520885401735786-212/raft_consensus-itest-cluster/ts-2/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=false --log_compression_codec=none --log_cache_size_limit_mb=1 --log_segment_size_mb=1 --log_async_preallocate_segments=false --log_min_segments_to_retain=1 --log_max_segments_to_retain=3 --maintenance_manager_polling_interval_ms=100 --log_target_replay_size_mb=1 --max_cell_size_bytes=1000000 with env {} I0312 20:12:02.144961 5997 process_memory.cc:182] Process hard memory limit is 11.777631 GB I0312 20:12:02.172202 5997 process_memory.cc:184] Process soft memory limit is 9.422105 GB I0312 20:12:02.172325 5997 process_memory.cc:187] Process memory pressure threshold is 7.066578 GB I0312 20:12:02.195641 5996 heartbeater.cc:346] Connected to a master server at 127.0.53.62:53264 I0312 20:12:02.195953 5996 heartbeater.cc:426] Registering TS with master... I0312 20:12:02.196511 5996 heartbeater.cc:475] Master 127.0.53.62:53264 requested a full tablet report, sending... I0312 20:12:02.198248 5705 ts_manager.cc:81] Registered new tserver with Master: 1183c4bf4e494b75a57827848998cacc (127.0.53.2:47310) I0312 20:12:02.200556 5705 master_service.cc:243] Signed X509 certificate for tserver {username='slave'} at 127.0.53.2:36086 WARNING: Logging before InitGoogleLogging() is written to STDERR W0312 20:12:02.282639 6000 flags.cc:412] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0312 20:12:02.282975 6000 flags.cc:412] Enabled unsafe flag: --max_cell_size_bytes=1000000 W0312 20:12:02.283100 6000 flags.cc:412] Enabled unsafe flag: --never_fsync=true W0312 20:12:02.297224 6000 flags.cc:412] Enabled experimental flag: --log_compression_codec=none W0312 20:12:02.297361 6000 flags.cc:412] Enabled experimental flag: --log_max_segments_to_retain=3 W0312 20:12:02.297523 6000 flags.cc:412] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false W0312 20:12:02.297665 6000 flags.cc:412] Enabled experimental flag: --ipki_server_key_size=1024 W0312 20:12:02.297948 6000 flags.cc:412] Enabled experimental flag: --log_target_replay_size_mb=1 W0312 20:12:02.298077 6000 flags.cc:412] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.53.3 I0312 20:12:02.318023 6000 tablet_server_main.cc:72] Tablet server non-default flags: --log_compression_codec=none --log_max_segments_to_retain=3 --log_cache_size_limit_mb=1 --log_async_preallocate_segments=false --log_segment_size_mb=1 --raft_prepare_replacement_before_eviction=false --fs_data_dirs=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1520885401735786-212/raft_consensus-itest-cluster/ts-2/data --fs_wal_dir=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1520885401735786-212/raft_consensus-itest-cluster/ts-2/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.0.53.3:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1520885401735786-212/raft_consensus-itest-cluster/ts-2/data/info.pb --webserver_interface=127.0.53.3 --webserver_port=0 --max_cell_size_bytes=1000000 --tserver_master_addrs=127.0.53.62:53264 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --log_target_replay_size_mb=1 --maintenance_manager_polling_interval_ms=100 --enable_minidumps=false --local_ip_for_outbound_sockets=127.0.53.3 --log_dir=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1520885401735786-212/raft_consensus-itest-cluster/ts-2/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.7.0-SNAPSHOT revision 13e10c63c46de30ac85f34b543cdd6fb44881f61 build type FASTDEBUG built by None at 12 Mar 2018 20:06:40 UTC on jenkins-slave-vn0g build id 12388 TSAN enabled I0312 20:12:02.327579 6000 tablet_server_main.cc:79] Initializing tablet server... I0312 20:12:02.329272 6000 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 1128276us I0312 20:12:02.329707 6000 fs_manager.cc:260] Metadata directory not provided I0312 20:12:02.329836 6000 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0312 20:12:02.330066 6000 server_base.cc:434] Could not load existing FS layout: Not found: /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1520885401735786-212/raft_consensus-itest-cluster/ts-2/data/instance: No such file or directory (error 2) I0312 20:12:02.330178 6000 server_base.cc:435] Attempting to create new FS layout instead I0312 20:12:02.337347 6000 fs_manager.cc:596] Generated new instance metadata in path /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1520885401735786-212/raft_consensus-itest-cluster/ts-2/data/instance: uuid: "0c2c01e0fc8840bdb67647734ba4c3f0" format_stamp: "Formatted at 2018-03-12 20:12:02 on dist-test-slave-dist-test-slave-f27p" I0312 20:12:02.338059 6000 fs_manager.cc:596] Generated new instance metadata in path /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1520885401735786-212/raft_consensus-itest-cluster/ts-2/wal/instance: uuid: "0c2c01e0fc8840bdb67647734ba4c3f0" format_stamp: "Formatted at 2018-03-12 20:12:02 on dist-test-slave-dist-test-slave-f27p" I0312 20:12:02.346132 6000 fs_manager.cc:496] Time spent creating directory manager: real 0.008s user 0.004s sys 0.004s I0312 20:12:02.347265 6000 env_posix.cc:1620] Not raising this process' open files per process limit of 65536; it is already as high as it can go I0312 20:12:02.347544 6000 file_cache.cc:470] Constructed file cache lbm with capacity 26214 I0312 20:12:02.353922 6000 fs_manager.cc:418] Time spent opening block manager: real 0.001s user 0.004s sys 0.004s I0312 20:12:02.354040 6000 fs_manager.cc:429] Opened local filesystem: /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1520885401735786-212/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1520885401735786-212/raft_consensus-itest-cluster/ts-2/wal uuid: "0c2c01e0fc8840bdb67647734ba4c3f0" format_stamp: "Formatted at 2018-03-12 20:12:02 on dist-test-slave-dist-test-slave-f27p" I0312 20:12:02.354260 6000 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1520885401735786-212/raft_consensus-itest-cluster/ts-2/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0312 20:12:02.521827 6000 env_posix.cc:1620] Not raising this process' running threads per effective uid limit of 60284; it is already as high as it can go I0312 20:12:02.523113 6000 ts_tablet_manager.cc:349] Loaded tablet metadata (0 live tablets) I0312 20:12:02.524055 6000 tablet_server_main.cc:84] Starting tablet server... I0312 20:12:02.591101 5870 heartbeater.cc:467] Master 127.0.53.62:53264 was elected leader, sending a full tablet report... I0312 20:12:02.611448 6000 rpc_server.cc:201] RPC server started. Bound to: 127.0.53.3:60846 I0312 20:12:02.612324 6000 webserver.cc:173] Starting webserver on 127.0.53.3:0 I0312 20:12:02.612437 6000 webserver.cc:184] Document root disabled I0312 20:12:02.613677 6000 webserver.cc:311] Webserver started. Bound to: http://127.0.53.3:44251/ I0312 20:12:02.614380 6000 server_base.cc:607] Dumped server information to /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1520885401735786-212/raft_consensus-itest-cluster/ts-2/data/info.pb I0312 20:12:02.616298 212 external_mini_cluster.cc:867] Started /tmp/dist-test-task3bwPeE/build/tsan/bin/kudu-tserver as pid 6000 I0312 20:12:02.616920 6122 process_memory.cc:182] Process hard memory limit is 11.777631 GB I0312 20:12:02.617120 6122 process_memory.cc:184] Process soft memory limit is 9.422105 GB I0312 20:12:02.617254 6122 process_memory.cc:187] Process memory pressure threshold is 7.066578 GB I0312 20:12:02.617003 6000 tablet_server_main.cc:87] Tablet server successfully started. I0312 20:12:02.662425 6121 heartbeater.cc:346] Connected to a master server at 127.0.53.62:53264 I0312 20:12:02.662665 6121 heartbeater.cc:426] Registering TS with master... I0312 20:12:02.663094 6121 heartbeater.cc:475] Master 127.0.53.62:53264 requested a full tablet report, sending... I0312 20:12:02.664397 5705 ts_manager.cc:81] Registered new tserver with Master: 0c2c01e0fc8840bdb67647734ba4c3f0 (127.0.53.3:60846) I0312 20:12:02.665537 5705 master_service.cc:243] Signed X509 certificate for tserver {username='slave'} at 127.0.53.3:49005 I0312 20:12:02.667281 212 external_mini_cluster.cc:498] 3 TS(s) registered with all masters I0312 20:12:02.684420 5705 catalog_manager.cc:1294] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39049: name: "TestTable" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 3 split_rows_range_bounds { } partition_schema { range_schema { columns { name: "key" } } } I0312 20:12:02.724735 6058 tablet_service.cc:755] Processing CreateTablet for tablet 9a02c73fbfaa40d5bf986d34f5b785f9 (table=TestTable [id=ce3d9a5ec62f446d8f8a4df6c8c754d8]), partition=RANGE (key) PARTITION UNBOUNDED I0312 20:12:02.725525 6058 data_dirs.cc:931] Could only allocate 1 dirs of requested 3 for tablet 9a02c73fbfaa40d5bf986d34f5b785f9. 1 dirs total, 0 dirs full, 0 dirs failed I0312 20:12:02.727663 5807 tablet_service.cc:755] Processing CreateTablet for tablet 9a02c73fbfaa40d5bf986d34f5b785f9 (table=TestTable [id=ce3d9a5ec62f446d8f8a4df6c8c754d8]), partition=RANGE (key) PARTITION UNBOUNDED I0312 20:12:02.728345 5933 tablet_service.cc:755] Processing CreateTablet for tablet 9a02c73fbfaa40d5bf986d34f5b785f9 (table=TestTable [id=ce3d9a5ec62f446d8f8a4df6c8c754d8]), partition=RANGE (key) PARTITION UNBOUNDED I0312 20:12:02.728528 5807 data_dirs.cc:931] Could only allocate 1 dirs of requested 3 for tablet 9a02c73fbfaa40d5bf986d34f5b785f9. 1 dirs total, 0 dirs full, 0 dirs failed I0312 20:12:02.728978 5933 data_dirs.cc:931] Could only allocate 1 dirs of requested 3 for tablet 9a02c73fbfaa40d5bf986d34f5b785f9. 1 dirs total, 0 dirs full, 0 dirs failed I0312 20:12:02.740875 5933 ts_tablet_manager.cc:1163] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 1183c4bf4e494b75a57827848998cacc: Registered tablet (data state: TABLET_DATA_READY) I0312 20:12:02.741822 6058 ts_tablet_manager.cc:1163] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 0c2c01e0fc8840bdb67647734ba4c3f0: Registered tablet (data state: TABLET_DATA_READY) I0312 20:12:02.744786 5807 ts_tablet_manager.cc:1163] T 9a02c73fbfaa40d5bf986d34f5b785f9 P ac1592be0a6240378933768820f57ed3: Registered tablet (data state: TABLET_DATA_READY) I0312 20:12:02.748288 6143 ts_tablet_manager.cc:1018] T 9a02c73fbfaa40d5bf986d34f5b785f9 P ac1592be0a6240378933768820f57ed3: Bootstrapping tablet I0312 20:12:02.748626 6142 ts_tablet_manager.cc:1018] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 0c2c01e0fc8840bdb67647734ba4c3f0: Bootstrapping tablet I0312 20:12:02.749100 6141 ts_tablet_manager.cc:1018] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 1183c4bf4e494b75a57827848998cacc: Bootstrapping tablet I0312 20:12:02.749145 6142 tablet_bootstrap.cc:436] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 0c2c01e0fc8840bdb67647734ba4c3f0: Bootstrap starting. I0312 20:12:02.749449 6143 tablet_bootstrap.cc:436] T 9a02c73fbfaa40d5bf986d34f5b785f9 P ac1592be0a6240378933768820f57ed3: Bootstrap starting. I0312 20:12:02.749608 6141 tablet_bootstrap.cc:436] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 1183c4bf4e494b75a57827848998cacc: Bootstrap starting. I0312 20:12:02.751993 6143 tablet_bootstrap.cc:581] T 9a02c73fbfaa40d5bf986d34f5b785f9 P ac1592be0a6240378933768820f57ed3: No blocks or log segments found. Creating new log. I0312 20:12:02.752218 6142 tablet_bootstrap.cc:581] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 0c2c01e0fc8840bdb67647734ba4c3f0: No blocks or log segments found. Creating new log. I0312 20:12:02.752825 6141 tablet_bootstrap.cc:581] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 1183c4bf4e494b75a57827848998cacc: No blocks or log segments found. Creating new log. I0312 20:12:02.753198 6143 log.cc:520] T 9a02c73fbfaa40d5bf986d34f5b785f9 P ac1592be0a6240378933768820f57ed3: Log is configured to *not* fsync() on all Append() calls I0312 20:12:02.753756 6142 log.cc:520] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 0c2c01e0fc8840bdb67647734ba4c3f0: Log is configured to *not* fsync() on all Append() calls I0312 20:12:02.756186 6141 log.cc:520] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 1183c4bf4e494b75a57827848998cacc: Log is configured to *not* fsync() on all Append() calls I0312 20:12:02.758841 6142 tablet_bootstrap.cc:436] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 0c2c01e0fc8840bdb67647734ba4c3f0: No bootstrap required, opened a new log I0312 20:12:02.759166 6143 tablet_bootstrap.cc:436] T 9a02c73fbfaa40d5bf986d34f5b785f9 P ac1592be0a6240378933768820f57ed3: No bootstrap required, opened a new log I0312 20:12:02.759245 6142 ts_tablet_manager.cc:1035] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 0c2c01e0fc8840bdb67647734ba4c3f0: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s I0312 20:12:02.759469 6143 ts_tablet_manager.cc:1035] T 9a02c73fbfaa40d5bf986d34f5b785f9 P ac1592be0a6240378933768820f57ed3: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s I0312 20:12:02.759917 6141 tablet_bootstrap.cc:436] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 1183c4bf4e494b75a57827848998cacc: No bootstrap required, opened a new log I0312 20:12:02.760272 6141 ts_tablet_manager.cc:1035] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 1183c4bf4e494b75a57827848998cacc: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.004s I0312 20:12:02.770413 6143 raft_consensus.cc:304] T 9a02c73fbfaa40d5bf986d34f5b785f9 P ac1592be0a6240378933768820f57ed3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c2c01e0fc8840bdb67647734ba4c3f0" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 60846 } } peers { permanent_uuid: "ac1592be0a6240378933768820f57ed3" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 59968 } } peers { permanent_uuid: "1183c4bf4e494b75a57827848998cacc" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 47310 } } I0312 20:12:02.770980 6143 raft_consensus.cc:330] T 9a02c73fbfaa40d5bf986d34f5b785f9 P ac1592be0a6240378933768820f57ed3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0312 20:12:02.771140 6143 raft_consensus.cc:605] T 9a02c73fbfaa40d5bf986d34f5b785f9 P ac1592be0a6240378933768820f57ed3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac1592be0a6240378933768820f57ed3, State: Initialized, Role: FOLLOWER I0312 20:12:02.771584 6143 consensus_queue.cc:219] T 9a02c73fbfaa40d5bf986d34f5b785f9 P ac1592be0a6240378933768820f57ed3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c2c01e0fc8840bdb67647734ba4c3f0" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 60846 } } peers { permanent_uuid: "ac1592be0a6240378933768820f57ed3" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 59968 } } peers { permanent_uuid: "1183c4bf4e494b75a57827848998cacc" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 47310 } } I0312 20:12:02.774821 6142 raft_consensus.cc:304] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 0c2c01e0fc8840bdb67647734ba4c3f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c2c01e0fc8840bdb67647734ba4c3f0" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 60846 } } peers { permanent_uuid: "ac1592be0a6240378933768820f57ed3" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 59968 } } peers { permanent_uuid: "1183c4bf4e494b75a57827848998cacc" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 47310 } } I0312 20:12:02.775555 6142 raft_consensus.cc:330] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 0c2c01e0fc8840bdb67647734ba4c3f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0312 20:12:02.775743 6142 raft_consensus.cc:605] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 0c2c01e0fc8840bdb67647734ba4c3f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c2c01e0fc8840bdb67647734ba4c3f0, State: Initialized, Role: FOLLOWER I0312 20:12:02.775647 6141 raft_consensus.cc:304] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 1183c4bf4e494b75a57827848998cacc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c2c01e0fc8840bdb67647734ba4c3f0" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 60846 } } peers { permanent_uuid: "ac1592be0a6240378933768820f57ed3" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 59968 } } peers { permanent_uuid: "1183c4bf4e494b75a57827848998cacc" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 47310 } } I0312 20:12:02.776373 6141 raft_consensus.cc:330] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 1183c4bf4e494b75a57827848998cacc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0312 20:12:02.776557 6141 raft_consensus.cc:605] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 1183c4bf4e494b75a57827848998cacc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1183c4bf4e494b75a57827848998cacc, State: Initialized, Role: FOLLOWER I0312 20:12:02.776360 6142 consensus_queue.cc:219] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 0c2c01e0fc8840bdb67647734ba4c3f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c2c01e0fc8840bdb67647734ba4c3f0" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 60846 } } peers { permanent_uuid: "ac1592be0a6240378933768820f57ed3" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 59968 } } peers { permanent_uuid: "1183c4bf4e494b75a57827848998cacc" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 47310 } } I0312 20:12:02.777076 6141 consensus_queue.cc:219] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 1183c4bf4e494b75a57827848998cacc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c2c01e0fc8840bdb67647734ba4c3f0" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 60846 } } peers { permanent_uuid: "ac1592be0a6240378933768820f57ed3" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 59968 } } peers { permanent_uuid: "1183c4bf4e494b75a57827848998cacc" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 47310 } } I0312 20:12:02.779250 6143 ts_tablet_manager.cc:1063] T 9a02c73fbfaa40d5bf986d34f5b785f9 P ac1592be0a6240378933768820f57ed3: Time spent starting tablet: real 0.020s user 0.012s sys 0.008s I0312 20:12:02.780673 6121 heartbeater.cc:467] Master 127.0.53.62:53264 was elected leader, sending a full tablet report... I0312 20:12:02.780725 5996 heartbeater.cc:467] Master 127.0.53.62:53264 was elected leader, sending a full tablet report... I0312 20:12:02.782292 6141 ts_tablet_manager.cc:1063] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 1183c4bf4e494b75a57827848998cacc: Time spent starting tablet: real 0.022s user 0.020s sys 0.000s I0312 20:12:02.783391 6142 ts_tablet_manager.cc:1063] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 0c2c01e0fc8840bdb67647734ba4c3f0: Time spent starting tablet: real 0.024s user 0.016s sys 0.004s W0312 20:12:02.817760 6122 tablet.cc:1679] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 0c2c01e0fc8840bdb67647734ba4c3f0: Can't schedule compaction. Clean time has not been advanced past its initial value. W0312 20:12:02.836406 5871 tablet.cc:1679] T 9a02c73fbfaa40d5bf986d34f5b785f9 P ac1592be0a6240378933768820f57ed3: Can't schedule compaction. Clean time has not been advanced past its initial value. W0312 20:12:02.873415 5997 tablet.cc:1679] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 1183c4bf4e494b75a57827848998cacc: Can't schedule compaction. Clean time has not been advanced past its initial value. I0312 20:12:02.900753 6149 raft_consensus.cc:436] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 1183c4bf4e494b75a57827848998cacc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0312 20:12:02.900945 6149 raft_consensus.cc:2700] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 1183c4bf4e494b75a57827848998cacc: Snoozing failure detection for 2.246s (starting election) I0312 20:12:02.901154 6149 raft_consensus.cc:458] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 1183c4bf4e494b75a57827848998cacc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c2c01e0fc8840bdb67647734ba4c3f0" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 60846 } } peers { permanent_uuid: "ac1592be0a6240378933768820f57ed3" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 59968 } } peers { permanent_uuid: "1183c4bf4e494b75a57827848998cacc" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 47310 } } I0312 20:12:02.902180 6149 leader_election.cc:231] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 1183c4bf4e494b75a57827848998cacc [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 0c2c01e0fc8840bdb67647734ba4c3f0 I0312 20:12:02.902590 6149 leader_election.cc:231] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 1183c4bf4e494b75a57827848998cacc [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer ac1592be0a6240378933768820f57ed3 I0312 20:12:02.913935 6078 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "9a02c73fbfaa40d5bf986d34f5b785f9" candidate_uuid: "1183c4bf4e494b75a57827848998cacc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c2c01e0fc8840bdb67647734ba4c3f0" is_pre_election: true I0312 20:12:02.914383 6078 raft_consensus.cc:2700] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 0c2c01e0fc8840bdb67647734ba4c3f0: Snoozing failure detection for 1.801s (vote granted) I0312 20:12:02.914572 6078 raft_consensus.cc:2215] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 0c2c01e0fc8840bdb67647734ba4c3f0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1183c4bf4e494b75a57827848998cacc in term 0. I0312 20:12:02.915488 5887 leader_election.cc:363] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 1183c4bf4e494b75a57827848998cacc [CANDIDATE]: Term 1 pre-election: Vote granted by peer 0c2c01e0fc8840bdb67647734ba4c3f0 I0312 20:12:02.915649 5887 leader_election.cc:258] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 1183c4bf4e494b75a57827848998cacc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. I0312 20:12:02.916026 6149 raft_consensus.cc:2700] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 1183c4bf4e494b75a57827848998cacc: Snoozing failure detection for 2.153s (election complete) I0312 20:12:02.916240 6149 raft_consensus.cc:2507] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 1183c4bf4e494b75a57827848998cacc [term 0 FOLLOWER]: Leader pre-election won for term 1 I0312 20:12:02.916405 6149 raft_consensus.cc:436] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 1183c4bf4e494b75a57827848998cacc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0312 20:12:02.916528 6149 raft_consensus.cc:2700] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 1183c4bf4e494b75a57827848998cacc: Snoozing failure detection for 1.893s (starting election) I0312 20:12:02.916671 6149 raft_consensus.cc:2752] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 1183c4bf4e494b75a57827848998cacc [term 0 FOLLOWER]: Advancing to term 1 I0312 20:12:02.916929 5827 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "9a02c73fbfaa40d5bf986d34f5b785f9" candidate_uuid: "1183c4bf4e494b75a57827848998cacc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac1592be0a6240378933768820f57ed3" is_pre_election: true I0312 20:12:02.917429 5827 raft_consensus.cc:2700] T 9a02c73fbfaa40d5bf986d34f5b785f9 P ac1592be0a6240378933768820f57ed3: Snoozing failure detection for 1.685s (vote granted) I0312 20:12:02.917608 5827 raft_consensus.cc:2215] T 9a02c73fbfaa40d5bf986d34f5b785f9 P ac1592be0a6240378933768820f57ed3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1183c4bf4e494b75a57827848998cacc in term 0. I0312 20:12:02.918462 5887 leader_election.cc:363] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 1183c4bf4e494b75a57827848998cacc [CANDIDATE]: Term 1 pre-election: Vote granted by peer ac1592be0a6240378933768820f57ed3 I0312 20:12:02.919958 6149 raft_consensus.cc:458] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 1183c4bf4e494b75a57827848998cacc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c2c01e0fc8840bdb67647734ba4c3f0" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 60846 } } peers { permanent_uuid: "ac1592be0a6240378933768820f57ed3" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 59968 } } peers { permanent_uuid: "1183c4bf4e494b75a57827848998cacc" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 47310 } } I0312 20:12:02.920678 6149 leader_election.cc:231] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 1183c4bf4e494b75a57827848998cacc [CANDIDATE]: Term 1 election: Requesting vote from peer 0c2c01e0fc8840bdb67647734ba4c3f0 I0312 20:12:02.920961 6149 leader_election.cc:231] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 1183c4bf4e494b75a57827848998cacc [CANDIDATE]: Term 1 election: Requesting vote from peer ac1592be0a6240378933768820f57ed3 I0312 20:12:02.921718 6078 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "9a02c73fbfaa40d5bf986d34f5b785f9" candidate_uuid: "1183c4bf4e494b75a57827848998cacc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c2c01e0fc8840bdb67647734ba4c3f0" I0312 20:12:02.922080 6078 raft_consensus.cc:2752] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 0c2c01e0fc8840bdb67647734ba4c3f0 [term 0 FOLLOWER]: Advancing to term 1 I0312 20:12:02.922019 5827 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "9a02c73fbfaa40d5bf986d34f5b785f9" candidate_uuid: "1183c4bf4e494b75a57827848998cacc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac1592be0a6240378933768820f57ed3" I0312 20:12:02.922276 6078 raft_consensus.cc:2700] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 0c2c01e0fc8840bdb67647734ba4c3f0: Snoozing failure detection for 1.500s (vote granted) I0312 20:12:02.922314 5827 raft_consensus.cc:2752] T 9a02c73fbfaa40d5bf986d34f5b785f9 P ac1592be0a6240378933768820f57ed3 [term 0 FOLLOWER]: Advancing to term 1 I0312 20:12:02.922488 5827 raft_consensus.cc:2700] T 9a02c73fbfaa40d5bf986d34f5b785f9 P ac1592be0a6240378933768820f57ed3: Snoozing failure detection for 2.149s (vote granted) I0312 20:12:02.925195 5827 raft_consensus.cc:2215] T 9a02c73fbfaa40d5bf986d34f5b785f9 P ac1592be0a6240378933768820f57ed3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1183c4bf4e494b75a57827848998cacc in term 1. I0312 20:12:02.925215 6078 raft_consensus.cc:2215] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 0c2c01e0fc8840bdb67647734ba4c3f0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1183c4bf4e494b75a57827848998cacc in term 1. I0312 20:12:02.925892 5887 leader_election.cc:363] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 1183c4bf4e494b75a57827848998cacc [CANDIDATE]: Term 1 election: Vote granted by peer ac1592be0a6240378933768820f57ed3 I0312 20:12:02.926033 5887 leader_election.cc:258] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 1183c4bf4e494b75a57827848998cacc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0312 20:12:02.926334 6149 raft_consensus.cc:2700] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 1183c4bf4e494b75a57827848998cacc: Snoozing failure detection for 1.556s (election complete) I0312 20:12:02.926424 5887 leader_election.cc:363] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 1183c4bf4e494b75a57827848998cacc [CANDIDATE]: Term 1 election: Vote granted by peer 0c2c01e0fc8840bdb67647734ba4c3f0 I0312 20:12:02.926545 6149 raft_consensus.cc:2507] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 1183c4bf4e494b75a57827848998cacc [term 1 FOLLOWER]: Leader election won for term 1 I0312 20:12:02.927618 6149 raft_consensus.cc:572] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 1183c4bf4e494b75a57827848998cacc [term 1 LEADER]: Becoming Leader. State: Replica: 1183c4bf4e494b75a57827848998cacc, State: Running, Role: LEADER I0312 20:12:02.927978 6149 consensus_queue.cc:196] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 1183c4bf4e494b75a57827848998cacc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c2c01e0fc8840bdb67647734ba4c3f0" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 60846 } } peers { permanent_uuid: "ac1592be0a6240378933768820f57ed3" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 59968 } } peers { permanent_uuid: "1183c4bf4e494b75a57827848998cacc" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 47310 } } I0312 20:12:02.932610 5704 catalog_manager.cc:3592] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 1183c4bf4e494b75a57827848998cacc reported cstate change: term changed from 0 to 1, leader changed from to 1183c4bf4e494b75a57827848998cacc (127.0.53.2). New cstate: current_term: 1 leader_uuid: "1183c4bf4e494b75a57827848998cacc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c2c01e0fc8840bdb67647734ba4c3f0" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 60846 } } peers { permanent_uuid: "ac1592be0a6240378933768820f57ed3" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 59968 } } peers { permanent_uuid: "1183c4bf4e494b75a57827848998cacc" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 47310 } } } I0312 20:12:02.956605 212 external_mini_cluster.cc:498] 3 TS(s) registered with all masters I0312 20:12:02.958632 212 ts_itest-base.cc:233] Waiting for 1 tablets on tserver ac1592be0a6240378933768820f57ed3 to finish bootstrapping I0312 20:12:02.967644 212 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 1183c4bf4e494b75a57827848998cacc to finish bootstrapping I0312 20:12:02.976333 212 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 0c2c01e0fc8840bdb67647734ba4c3f0 to finish bootstrapping I0312 20:12:03.008781 212 cluster_itest_util.cc:245] Not converged past 1 yet: 0.0 1.1 0.0 I0312 20:12:03.112941 212 cluster_itest_util.cc:245] Not converged past 1 yet: 0.0 1.1 0.0 I0312 20:12:03.317276 212 cluster_itest_util.cc:245] Not converged past 1 yet: 0.0 1.1 0.0 I0312 20:12:03.405005 6078 raft_consensus.cc:1058] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 0c2c01e0fc8840bdb67647734ba4c3f0 [term 1 FOLLOWER]: Refusing update from remote peer 1183c4bf4e494b75a57827848998cacc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch) I0312 20:12:03.406137 6149 consensus_queue.cc:1017] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 1183c4bf4e494b75a57827848998cacc [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c2c01e0fc8840bdb67647734ba4c3f0" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 60846 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0312 20:12:03.414006 5827 raft_consensus.cc:1058] T 9a02c73fbfaa40d5bf986d34f5b785f9 P ac1592be0a6240378933768820f57ed3 [term 1 FOLLOWER]: Refusing update from remote peer 1183c4bf4e494b75a57827848998cacc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch) I0312 20:12:03.415381 6152 consensus_queue.cc:1017] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 1183c4bf4e494b75a57827848998cacc [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac1592be0a6240378933768820f57ed3" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 59968 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0312 20:12:03.636315 212 test_util.cc:192] Using random seed: -1050582977 I0312 20:12:03.652477 212 test_workload.cc:306] TestWorkload: Skipping table creation because table TestTable already exists I0312 20:12:03.656719 212 raft_consensus-itest-base.cc:230] Waiting until we've written at least 4MB... I0312 20:12:03.768575 6153 log.cc:644] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 1183c4bf4e494b75a57827848998cacc: Max segment size reached. Starting new segment allocation I0312 20:12:03.771643 6153 log.cc:571] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 1183c4bf4e494b75a57827848998cacc: Rolled over to a new log segment at /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1520885401735786-212/raft_consensus-itest-cluster/ts-1/wal/wals/9a02c73fbfaa40d5bf986d34f5b785f9/wal-000000002 I0312 20:12:03.791138 6158 log.cc:644] T 9a02c73fbfaa40d5bf986d34f5b785f9 P ac1592be0a6240378933768820f57ed3: Max segment size reached. Starting new segment allocation I0312 20:12:03.793453 6158 log.cc:571] T 9a02c73fbfaa40d5bf986d34f5b785f9 P ac1592be0a6240378933768820f57ed3: Rolled over to a new log segment at /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1520885401735786-212/raft_consensus-itest-cluster/ts-0/wal/wals/9a02c73fbfaa40d5bf986d34f5b785f9/wal-000000002 I0312 20:12:03.865732 6158 log.cc:644] T 9a02c73fbfaa40d5bf986d34f5b785f9 P ac1592be0a6240378933768820f57ed3: Max segment size reached. Starting new segment allocation I0312 20:12:03.866750 6158 log.cc:571] T 9a02c73fbfaa40d5bf986d34f5b785f9 P ac1592be0a6240378933768820f57ed3: Rolled over to a new log segment at /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1520885401735786-212/raft_consensus-itest-cluster/ts-0/wal/wals/9a02c73fbfaa40d5bf986d34f5b785f9/wal-000000003 I0312 20:12:03.891438 6153 log.cc:644] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 1183c4bf4e494b75a57827848998cacc: Max segment size reached. Starting new segment allocation I0312 20:12:03.892948 6153 log.cc:571] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 1183c4bf4e494b75a57827848998cacc: Rolled over to a new log segment at /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1520885401735786-212/raft_consensus-itest-cluster/ts-1/wal/wals/9a02c73fbfaa40d5bf986d34f5b785f9/wal-000000003 I0312 20:12:03.928282 6158 log.cc:644] T 9a02c73fbfaa40d5bf986d34f5b785f9 P ac1592be0a6240378933768820f57ed3: Max segment size reached. Starting new segment allocation I0312 20:12:03.929320 6158 log.cc:571] T 9a02c73fbfaa40d5bf986d34f5b785f9 P ac1592be0a6240378933768820f57ed3: Rolled over to a new log segment at /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1520885401735786-212/raft_consensus-itest-cluster/ts-0/wal/wals/9a02c73fbfaa40d5bf986d34f5b785f9/wal-000000004 I0312 20:12:03.940484 5871 maintenance_manager.cc:302] P ac1592be0a6240378933768820f57ed3: Scheduling FlushMRSOp(9a02c73fbfaa40d5bf986d34f5b785f9): 2101727 bytes log retention I0312 20:12:03.941995 5755 tablet.cc:1153] T 9a02c73fbfaa40d5bf986d34f5b785f9 P ac1592be0a6240378933768820f57ed3: Flush: entering stage 1 (old memrowset already frozen for inserts) I0312 20:12:03.942131 5755 compaction.cc:914] Selected 1 rowsets to compact: I0312 20:12:03.942240 5755 compaction.cc:917] memrowset(current size on disk: ~0 bytes) I0312 20:12:03.942350 5755 tablet.cc:1155] T 9a02c73fbfaa40d5bf986d34f5b785f9 P ac1592be0a6240378933768820f57ed3: Memstore in-memory size: 2998537 bytes I0312 20:12:03.942489 5755 tablet.cc:1444] T 9a02c73fbfaa40d5bf986d34f5b785f9 P ac1592be0a6240378933768820f57ed3: Flush: entering phase 1 (flushing snapshot). Phase 1 snapshot: MvccSnapshot[committed={T|T < 6229547105882443776 or (T in {6229547105882443776})}] I0312 20:12:03.956017 6153 log.cc:644] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 1183c4bf4e494b75a57827848998cacc: Max segment size reached. Starting new segment allocation I0312 20:12:03.957309 6153 log.cc:571] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 1183c4bf4e494b75a57827848998cacc: Rolled over to a new log segment at /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1520885401735786-212/raft_consensus-itest-cluster/ts-1/wal/wals/9a02c73fbfaa40d5bf986d34f5b785f9/wal-000000004 I0312 20:12:03.960386 5755 multi_column_writer.cc:98] Opened CFile writers for 3 column(s) I0312 20:12:03.983927 5997 maintenance_manager.cc:302] P 1183c4bf4e494b75a57827848998cacc: Scheduling FlushMRSOp(9a02c73fbfaa40d5bf986d34f5b785f9): 2758374 bytes log retention I0312 20:12:03.992830 5880 tablet.cc:1153] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 1183c4bf4e494b75a57827848998cacc: Flush: entering stage 1 (old memrowset already frozen for inserts) I0312 20:12:03.993003 5880 compaction.cc:914] Selected 1 rowsets to compact: I0312 20:12:03.993160 5880 compaction.cc:917] memrowset(current size on disk: ~0 bytes) I0312 20:12:03.993270 5880 tablet.cc:1155] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 1183c4bf4e494b75a57827848998cacc: Memstore in-memory size: 4038921 bytes I0312 20:12:03.993652 5880 tablet.cc:1444] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 1183c4bf4e494b75a57827848998cacc: Flush: entering phase 1 (flushing snapshot). Phase 1 snapshot: MvccSnapshot[committed={T|T < 6229547106006933504 or (T in {6229547106006933504})}] I0312 20:12:04.052294 5880 multi_column_writer.cc:98] Opened CFile writers for 3 column(s) I0312 20:12:04.090497 6158 log.cc:644] T 9a02c73fbfaa40d5bf986d34f5b785f9 P ac1592be0a6240378933768820f57ed3: Max segment size reached. Starting new segment allocation I0312 20:12:04.091904 6158 log.cc:571] T 9a02c73fbfaa40d5bf986d34f5b785f9 P ac1592be0a6240378933768820f57ed3: Rolled over to a new log segment at /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1520885401735786-212/raft_consensus-itest-cluster/ts-0/wal/wals/9a02c73fbfaa40d5bf986d34f5b785f9/wal-000000005 I0312 20:12:04.166582 5755 tablet.cc:1529] T 9a02c73fbfaa40d5bf986d34f5b785f9 P ac1592be0a6240378933768820f57ed3: Flush: entering phase 2 (starting to duplicate updates in new rowsets) I0312 20:12:04.167065 5755 tablet.cc:1584] T 9a02c73fbfaa40d5bf986d34f5b785f9 P ac1592be0a6240378933768820f57ed3: Flush Phase 2: carrying over any updates which arrived during Phase 1 I0312 20:12:04.167362 5755 tablet.cc:1586] T 9a02c73fbfaa40d5bf986d34f5b785f9 P ac1592be0a6240378933768820f57ed3: Phase 2 snapshot: MvccSnapshot[committed={T|T < 6229547106358947840 or (T in {6229547106358947840})}] I0312 20:12:04.188665 6153 log.cc:644] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 1183c4bf4e494b75a57827848998cacc: Max segment size reached. Starting new segment allocation I0312 20:12:04.190222 6153 log.cc:571] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 1183c4bf4e494b75a57827848998cacc: Rolled over to a new log segment at /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1520885401735786-212/raft_consensus-itest-cluster/ts-1/wal/wals/9a02c73fbfaa40d5bf986d34f5b785f9/wal-000000005 I0312 20:12:04.194957 5755 tablet.cc:1634] T 9a02c73fbfaa40d5bf986d34f5b785f9 P ac1592be0a6240378933768820f57ed3: Flush successful on 16 rows (135803 bytes) I0312 20:12:04.230248 5755 maintenance_manager.cc:492] P ac1592be0a6240378933768820f57ed3: FlushMRSOp(9a02c73fbfaa40d5bf986d34f5b785f9) complete. Timing: real 0.289s user 0.132s sys 0.092s Metrics: {"cfile_init":1,"compiler_manager_pool.queue_time_us":5096,"data dirs.queue_time_us":50156,"data dirs.run_cpu_time_us":258,"data dirs.run_wall_time_us":1041,"lbm_read_time_us":220,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2131,"lbm_writes_lt_1ms":23,"thread_start_us":54780,"threads_started":2} I0312 20:12:04.235064 5871 maintenance_manager.cc:302] P ac1592be0a6240378933768820f57ed3: Scheduling LogGCOp(9a02c73fbfaa40d5bf986d34f5b785f9): free 1313608 bytes of WAL I0312 20:12:04.235743 5755 log_reader.cc:376] T 9a02c73fbfaa40d5bf986d34f5b785f9: removed 2 log segments from log reader I0312 20:12:04.235976 5755 log.cc:856] T 9a02c73fbfaa40d5bf986d34f5b785f9 P ac1592be0a6240378933768820f57ed3: Deleting log segment in path: /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1520885401735786-212/raft_consensus-itest-cluster/ts-0/wal/wals/9a02c73fbfaa40d5bf986d34f5b785f9/wal-000000001 (ops 1-6) I0312 20:12:04.236268 5755 log.cc:856] T 9a02c73fbfaa40d5bf986d34f5b785f9 P ac1592be0a6240378933768820f57ed3: Deleting log segment in path: /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1520885401735786-212/raft_consensus-itest-cluster/ts-0/wal/wals/9a02c73fbfaa40d5bf986d34f5b785f9/wal-000000002 (ops 7-11) I0312 20:12:04.236933 5755 maintenance_manager.cc:492] P ac1592be0a6240378933768820f57ed3: LogGCOp(9a02c73fbfaa40d5bf986d34f5b785f9) complete. Timing: real 0.001s user 0.004s sys 0.000s Metrics: {} I0312 20:12:04.237638 5871 maintenance_manager.cc:302] P ac1592be0a6240378933768820f57ed3: Scheduling UndoDeltaBlockGCOp(9a02c73fbfaa40d5bf986d34f5b785f9): 316 bytes on disk I0312 20:12:04.238922 5755 maintenance_manager.cc:492] P ac1592be0a6240378933768820f57ed3: UndoDeltaBlockGCOp(9a02c73fbfaa40d5bf986d34f5b785f9) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":256,"lbm_reads_lt_1ms":4} I0312 20:12:04.260195 6158 log.cc:644] T 9a02c73fbfaa40d5bf986d34f5b785f9 P ac1592be0a6240378933768820f57ed3: Max segment size reached. Starting new segment allocation I0312 20:12:04.261669 6158 log.cc:571] T 9a02c73fbfaa40d5bf986d34f5b785f9 P ac1592be0a6240378933768820f57ed3: Rolled over to a new log segment at /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1520885401735786-212/raft_consensus-itest-cluster/ts-0/wal/wals/9a02c73fbfaa40d5bf986d34f5b785f9/wal-000000006 I0312 20:12:04.325500 5880 tablet.cc:1529] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 1183c4bf4e494b75a57827848998cacc: Flush: entering phase 2 (starting to duplicate updates in new rowsets) I0312 20:12:04.326053 5880 tablet.cc:1584] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 1183c4bf4e494b75a57827848998cacc: Flush Phase 2: carrying over any updates which arrived during Phase 1 I0312 20:12:04.326304 5880 tablet.cc:1586] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 1183c4bf4e494b75a57827848998cacc: Phase 2 snapshot: MvccSnapshot[committed={T|T < 6229547107467542528 or (T in {6229547107467542528})}] I0312 20:12:04.341650 5871 maintenance_manager.cc:302] P ac1592be0a6240378933768820f57ed3: Scheduling LogGCOp(9a02c73fbfaa40d5bf986d34f5b785f9): free 788119 bytes of WAL I0312 20:12:04.342281 5755 log_reader.cc:376] T 9a02c73fbfaa40d5bf986d34f5b785f9: removed 1 log segments from log reader I0312 20:12:04.342449 5755 log.cc:856] T 9a02c73fbfaa40d5bf986d34f5b785f9 P ac1592be0a6240378933768820f57ed3: Deleting log segment in path: /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1520885401735786-212/raft_consensus-itest-cluster/ts-0/wal/wals/9a02c73fbfaa40d5bf986d34f5b785f9/wal-000000003 (ops 12-17) I0312 20:12:04.342913 5755 maintenance_manager.cc:492] P ac1592be0a6240378933768820f57ed3: LogGCOp(9a02c73fbfaa40d5bf986d34f5b785f9) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {} I0312 20:12:04.343693 5871 maintenance_manager.cc:302] P ac1592be0a6240378933768820f57ed3: Scheduling FlushMRSOp(9a02c73fbfaa40d5bf986d34f5b785f9): 1707539 bytes log retention I0312 20:12:04.344563 5755 tablet.cc:1153] T 9a02c73fbfaa40d5bf986d34f5b785f9 P ac1592be0a6240378933768820f57ed3: Flush: entering stage 1 (old memrowset already frozen for inserts) I0312 20:12:04.344719 5755 compaction.cc:914] Selected 1 rowsets to compact: I0312 20:12:04.344835 5755 compaction.cc:917] memrowset(current size on disk: ~0 bytes) I0312 20:12:04.344949 5755 tablet.cc:1155] T 9a02c73fbfaa40d5bf986d34f5b785f9 P ac1592be0a6240378933768820f57ed3: Memstore in-memory size: 2998537 bytes I0312 20:12:04.345170 5755 tablet.cc:1444] T 9a02c73fbfaa40d5bf986d34f5b785f9 P ac1592be0a6240378933768820f57ed3: Flush: entering phase 1 (flushing snapshot). Phase 1 snapshot: MvccSnapshot[committed={T|T < 6229547107467542528 or (T in {6229547107467542528})}] I0312 20:12:04.346750 5755 multi_column_writer.cc:98] Opened CFile writers for 3 column(s) I0312 20:12:04.351621 5880 tablet.cc:1634] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 1183c4bf4e494b75a57827848998cacc: Flush successful on 20 rows (135869 bytes) I0312 20:12:04.357187 6153 log.cc:644] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 1183c4bf4e494b75a57827848998cacc: Max segment size reached. Starting new segment allocation I0312 20:12:04.358574 6153 log.cc:571] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 1183c4bf4e494b75a57827848998cacc: Rolled over to a new log segment at /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1520885401735786-212/raft_consensus-itest-cluster/ts-1/wal/wals/9a02c73fbfaa40d5bf986d34f5b785f9/wal-000000006 I0312 20:12:04.399442 5880 maintenance_manager.cc:492] P 1183c4bf4e494b75a57827848998cacc: FlushMRSOp(9a02c73fbfaa40d5bf986d34f5b785f9) complete. Timing: real 0.415s user 0.172s sys 0.036s Metrics: {"cfile_init":1,"compiler_manager_pool.queue_time_us":5831,"data dirs.queue_time_us":1193,"data dirs.run_cpu_time_us":283,"data dirs.run_wall_time_us":12905,"lbm_read_time_us":225,"lbm_reads_lt_1ms":4,"lbm_write_time_us":12430,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":22,"spinlock_wait_cycles":458752,"thread_start_us":13925,"threads_started":2} I0312 20:12:04.401229 5997 maintenance_manager.cc:302] P 1183c4bf4e494b75a57827848998cacc: Scheduling LogGCOp(9a02c73fbfaa40d5bf986d34f5b785f9): free 1838966 bytes of WAL I0312 20:12:04.401908 5880 log_reader.cc:376] T 9a02c73fbfaa40d5bf986d34f5b785f9: removed 2 log segments from log reader I0312 20:12:04.402137 5880 log.cc:856] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 1183c4bf4e494b75a57827848998cacc: Deleting log segment in path: /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1520885401735786-212/raft_consensus-itest-cluster/ts-1/wal/wals/9a02c73fbfaa40d5bf986d34f5b785f9/wal-000000001 (ops 1-8) I0312 20:12:04.402407 5880 log.cc:856] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 1183c4bf4e494b75a57827848998cacc: Deleting log segment in path: /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1520885401735786-212/raft_consensus-itest-cluster/ts-1/wal/wals/9a02c73fbfaa40d5bf986d34f5b785f9/wal-000000002 (ops 9-15) I0312 20:12:04.403156 5880 maintenance_manager.cc:492] P 1183c4bf4e494b75a57827848998cacc: LogGCOp(9a02c73fbfaa40d5bf986d34f5b785f9) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {} I0312 20:12:04.403864 5997 maintenance_manager.cc:302] P 1183c4bf4e494b75a57827848998cacc: Scheduling FlushMRSOp(9a02c73fbfaa40d5bf986d34f5b785f9): 2758376 bytes log retention I0312 20:12:04.404709 5880 tablet.cc:1153] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 1183c4bf4e494b75a57827848998cacc: Flush: entering stage 1 (old memrowset already frozen for inserts) I0312 20:12:04.404896 5880 compaction.cc:914] Selected 1 rowsets to compact: I0312 20:12:04.405032 5880 compaction.cc:917] memrowset(current size on disk: ~0 bytes) I0312 20:12:04.405201 5880 tablet.cc:1155] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 1183c4bf4e494b75a57827848998cacc: Memstore in-memory size: 2998537 bytes I0312 20:12:04.405382 5880 tablet.cc:1444] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 1183c4bf4e494b75a57827848998cacc: Flush: entering phase 1 (flushing snapshot). Phase 1 snapshot: MvccSnapshot[committed={T|T < 6229547107620839424 or (T in {6229547107620839424})}] I0312 20:12:04.440178 212 raft_consensus-itest-base.cc:236] Waiting for log GC on 1183c4bf4e494b75a57827848998cacc I0312 20:12:04.440735 212 raft_consensus-itest-base.cc:241] Log GC complete on 1183c4bf4e494b75a57827848998cacc I0312 20:12:04.442194 5880 multi_column_writer.cc:98] Opened CFile writers for 3 column(s) I0312 20:12:04.518045 5755 tablet.cc:1529] T 9a02c73fbfaa40d5bf986d34f5b785f9 P ac1592be0a6240378933768820f57ed3: Flush: entering phase 2 (starting to duplicate updates in new rowsets) I0312 20:12:04.518352 5755 tablet.cc:1584] T 9a02c73fbfaa40d5bf986d34f5b785f9 P ac1592be0a6240378933768820f57ed3: Flush Phase 2: carrying over any updates which arrived during Phase 1 I0312 20:12:04.518448 5755 tablet.cc:1586] T 9a02c73fbfaa40d5bf986d34f5b785f9 P ac1592be0a6240378933768820f57ed3: Phase 2 snapshot: MvccSnapshot[committed={T|T < 6229547107738787840 or (T in {6229547107738787840})}] I0312 20:12:04.524690 5755 tablet.cc:1634] T 9a02c73fbfaa40d5bf986d34f5b785f9 P ac1592be0a6240378933768820f57ed3: Flush successful on 16 rows (135803 bytes) I0312 20:12:04.540824 5880 tablet.cc:1529] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 1183c4bf4e494b75a57827848998cacc: Flush: entering phase 2 (starting to duplicate updates in new rowsets) I0312 20:12:04.541170 5880 tablet.cc:1584] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 1183c4bf4e494b75a57827848998cacc: Flush Phase 2: carrying over any updates which arrived during Phase 1 I0312 20:12:04.541280 5880 tablet.cc:1586] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 1183c4bf4e494b75a57827848998cacc: Phase 2 snapshot: MvccSnapshot[committed={T|T < 6229547107738787840 or (T in {6229547107738787840})}] I0312 20:12:04.547292 5880 tablet.cc:1634] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 1183c4bf4e494b75a57827848998cacc: Flush successful on 13 rows (135803 bytes) I0312 20:12:04.548522 5755 maintenance_manager.cc:492] P ac1592be0a6240378933768820f57ed3: FlushMRSOp(9a02c73fbfaa40d5bf986d34f5b785f9) complete. Timing: real 0.204s user 0.100s sys 0.012s Metrics: {"cfile_init":1,"data dirs.queue_time_us":88,"data dirs.run_cpu_time_us":282,"data dirs.run_wall_time_us":1373,"lbm_read_time_us":125,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1374,"lbm_writes_lt_1ms":23} I0312 20:12:04.550022 5871 maintenance_manager.cc:302] P ac1592be0a6240378933768820f57ed3: Scheduling CompactRowSetsOp(9a02c73fbfaa40d5bf986d34f5b785f9): perf score=0.966897 I0312 20:12:04.550951 5755 tablet.cc:1664] T 9a02c73fbfaa40d5bf986d34f5b785f9 P ac1592be0a6240378933768820f57ed3: Compaction: stage 1 complete, picked 2 rowsets to compact or flush I0312 20:12:04.551244 5755 compaction.cc:914] Selected 2 rowsets to compact: I0312 20:12:04.551398 5755 compaction.cc:917] RowSet(0)(current size on disk: ~136119 bytes) I0312 20:12:04.551632 5755 compaction.cc:917] RowSet(1)(current size on disk: ~136119 bytes) I0312 20:12:04.551834 5755 tablet.cc:1444] T 9a02c73fbfaa40d5bf986d34f5b785f9 P ac1592be0a6240378933768820f57ed3: Compaction: entering phase 1 (flushing snapshot). Phase 1 snapshot: MvccSnapshot[committed={T|T < 6229547107738787840 or (T in {6229547107738787840})}] I0312 20:12:04.553781 5755 multi_column_writer.cc:98] Opened CFile writers for 3 column(s) I0312 20:12:04.570848 5880 maintenance_manager.cc:492] P 1183c4bf4e494b75a57827848998cacc: FlushMRSOp(9a02c73fbfaa40d5bf986d34f5b785f9) complete. Timing: real 0.167s user 0.080s sys 0.004s Metrics: {"cfile_init":1,"data dirs.queue_time_us":81,"data dirs.run_cpu_time_us":245,"data dirs.run_wall_time_us":1357,"lbm_read_time_us":133,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2248,"lbm_writes_lt_1ms":23} I0312 20:12:04.572264 5997 maintenance_manager.cc:302] P 1183c4bf4e494b75a57827848998cacc: Scheduling LogGCOp(9a02c73fbfaa40d5bf986d34f5b785f9): free 919408 bytes of WAL I0312 20:12:04.572850 5880 log_reader.cc:376] T 9a02c73fbfaa40d5bf986d34f5b785f9: removed 1 log segments from log reader I0312 20:12:04.573035 5880 log.cc:856] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 1183c4bf4e494b75a57827848998cacc: Deleting log segment in path: /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1520885401735786-212/raft_consensus-itest-cluster/ts-1/wal/wals/9a02c73fbfaa40d5bf986d34f5b785f9/wal-000000003 (ops 16-22) I0312 20:12:04.573581 5880 maintenance_manager.cc:492] P 1183c4bf4e494b75a57827848998cacc: LogGCOp(9a02c73fbfaa40d5bf986d34f5b785f9) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {} I0312 20:12:04.574286 5997 maintenance_manager.cc:302] P 1183c4bf4e494b75a57827848998cacc: Scheduling CompactRowSetsOp(9a02c73fbfaa40d5bf986d34f5b785f9): perf score=0.881457 I0312 20:12:04.575101 5880 tablet.cc:1664] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 1183c4bf4e494b75a57827848998cacc: Compaction: stage 1 complete, picked 2 rowsets to compact or flush I0312 20:12:04.575297 5880 compaction.cc:914] Selected 2 rowsets to compact: I0312 20:12:04.575435 5880 compaction.cc:917] RowSet(0)(current size on disk: ~136226 bytes) I0312 20:12:04.575625 5880 compaction.cc:917] RowSet(1)(current size on disk: ~136092 bytes) I0312 20:12:04.575803 5880 tablet.cc:1444] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 1183c4bf4e494b75a57827848998cacc: Compaction: entering phase 1 (flushing snapshot). Phase 1 snapshot: MvccSnapshot[committed={T|T < 6229547107738787840 or (T in {6229547107738787840})}] I0312 20:12:04.577698 5880 multi_column_writer.cc:98] Opened CFile writers for 3 column(s) I0312 20:12:04.897311 5755 tablet.cc:1529] T 9a02c73fbfaa40d5bf986d34f5b785f9 P ac1592be0a6240378933768820f57ed3: Compaction: entering phase 2 (starting to duplicate updates in new rowsets) I0312 20:12:04.897750 5755 tablet.cc:1584] T 9a02c73fbfaa40d5bf986d34f5b785f9 P ac1592be0a6240378933768820f57ed3: Compaction Phase 2: carrying over any updates which arrived during Phase 1 I0312 20:12:04.897930 5755 tablet.cc:1586] T 9a02c73fbfaa40d5bf986d34f5b785f9 P ac1592be0a6240378933768820f57ed3: Phase 2 snapshot: MvccSnapshot[committed={T|T < 6229547107738787840 or (T in {6229547107738787840})}] I0312 20:12:04.926105 5880 tablet.cc:1529] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 1183c4bf4e494b75a57827848998cacc: Compaction: entering phase 2 (starting to duplicate updates in new rowsets) I0312 20:12:04.926475 5880 tablet.cc:1584] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 1183c4bf4e494b75a57827848998cacc: Compaction Phase 2: carrying over any updates which arrived during Phase 1 I0312 20:12:04.926627 5880 tablet.cc:1586] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 1183c4bf4e494b75a57827848998cacc: Phase 2 snapshot: MvccSnapshot[committed={T|T < 6229547107738787840 or (T in {6229547107738787840})}] I0312 20:12:05.031257 5755 tablet.cc:1634] T 9a02c73fbfaa40d5bf986d34f5b785f9 P ac1592be0a6240378933768820f57ed3: Compaction successful on 32 rows (135937 bytes) I0312 20:12:05.032444 5755 maintenance_manager.cc:492] P ac1592be0a6240378933768820f57ed3: CompactRowSetsOp(9a02c73fbfaa40d5bf986d34f5b785f9) complete. Timing: real 0.482s user 0.396s sys 0.068s Metrics: {"cfile_cache_hit":10,"cfile_cache_hit_bytes":262392,"cfile_cache_miss":26,"cfile_cache_miss_bytes":264076,"cfile_init":6,"data dirs.queue_time_us":150,"data dirs.run_cpu_time_us":200,"data dirs.run_wall_time_us":1398,"delta_iterators_relevant":4,"lbm_read_time_us":4456,"lbm_reads_1-10_ms":2,"lbm_reads_lt_1ms":48,"lbm_write_time_us":1882,"lbm_writes_lt_1ms":23} I0312 20:12:05.034353 5871 maintenance_manager.cc:302] P ac1592be0a6240378933768820f57ed3: Scheduling UndoDeltaBlockGCOp(9a02c73fbfaa40d5bf986d34f5b785f9): 491 bytes on disk I0312 20:12:05.035881 5755 maintenance_manager.cc:492] P ac1592be0a6240378933768820f57ed3: UndoDeltaBlockGCOp(9a02c73fbfaa40d5bf986d34f5b785f9) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":188,"lbm_reads_lt_1ms":4} I0312 20:12:05.059834 5880 tablet.cc:1634] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 1183c4bf4e494b75a57827848998cacc: Compaction successful on 33 rows (136001 bytes) I0312 20:12:05.070266 5880 maintenance_manager.cc:492] P 1183c4bf4e494b75a57827848998cacc: CompactRowSetsOp(9a02c73fbfaa40d5bf986d34f5b785f9) complete. Timing: real 0.495s user 0.336s sys 0.092s Metrics: {"cfile_cache_hit":10,"cfile_cache_hit_bytes":262392,"cfile_cache_miss":26,"cfile_cache_miss_bytes":264232,"cfile_init":7,"data dirs.queue_time_us":9283,"data dirs.run_cpu_time_us":236,"data dirs.run_wall_time_us":1261,"delta_iterators_relevant":4,"lbm_read_time_us":4715,"lbm_reads_1-10_ms":2,"lbm_reads_lt_1ms":52,"lbm_write_time_us":1934,"lbm_writes_lt_1ms":23} I0312 20:12:05.071614 5997 maintenance_manager.cc:302] P 1183c4bf4e494b75a57827848998cacc: Scheduling UndoDeltaBlockGCOp(9a02c73fbfaa40d5bf986d34f5b785f9): 506 bytes on disk I0312 20:12:05.072752 5880 maintenance_manager.cc:492] P 1183c4bf4e494b75a57827848998cacc: UndoDeltaBlockGCOp(9a02c73fbfaa40d5bf986d34f5b785f9) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":207,"lbm_reads_lt_1ms":4} I0312 20:12:06.442641 212 raft_consensus-itest-base.cc:257] Servers converged with original term 1 I0312 20:12:06.442759 212 raft_consensus-itest-base.cc:261] Resuming 0c2c01e0fc8840bdb67647734ba4c3f0 I0312 20:12:06.457854 6155 raft_consensus.cc:436] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 0c2c01e0fc8840bdb67647734ba4c3f0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1183c4bf4e494b75a57827848998cacc) I0312 20:12:06.458065 6155 raft_consensus.cc:2700] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 0c2c01e0fc8840bdb67647734ba4c3f0: Snoozing failure detection for 1.537s (starting election) I0312 20:12:06.458276 6155 raft_consensus.cc:458] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 0c2c01e0fc8840bdb67647734ba4c3f0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c2c01e0fc8840bdb67647734ba4c3f0" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 60846 } } peers { permanent_uuid: "ac1592be0a6240378933768820f57ed3" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 59968 } } peers { permanent_uuid: "1183c4bf4e494b75a57827848998cacc" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 47310 } } I0312 20:12:06.459923 6155 leader_election.cc:231] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 0c2c01e0fc8840bdb67647734ba4c3f0 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer ac1592be0a6240378933768820f57ed3 I0312 20:12:06.460458 6155 leader_election.cc:231] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 0c2c01e0fc8840bdb67647734ba4c3f0 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 1183c4bf4e494b75a57827848998cacc I0312 20:12:06.481523 6186 consensus_peers.cc:230] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 1183c4bf4e494b75a57827848998cacc -> Peer 0c2c01e0fc8840bdb67647734ba4c3f0 (127.0.53.3:60846): Could not obtain request from queue for peer: 0c2c01e0fc8840bdb67647734ba4c3f0. Status: Not found: Failed to read ops 3..31: Segment 1 which contained index 3 has been GCed I0312 20:12:06.482215 6186 raft_consensus.cc:815] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 1183c4bf4e494b75a57827848998cacc: Attempting to remove follower 0c2c01e0fc8840bdb67647734ba4c3f0 from the Raft config. Reason: The logs necessary to catch up peer 0c2c01e0fc8840bdb67647734ba4c3f0 have been garbage collected. The replica will never be able to catch up I0312 20:12:06.482843 6186 consensus_peers.cc:458] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 1183c4bf4e494b75a57827848998cacc -> Peer ac1592be0a6240378933768820f57ed3 (127.0.53.1:59968): Closing peer: ac1592be0a6240378933768820f57ed3 I0312 20:12:06.482993 6186 consensus_peers.cc:458] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 1183c4bf4e494b75a57827848998cacc -> Peer 0c2c01e0fc8840bdb67647734ba4c3f0 (127.0.53.3:60846): Closing peer: 0c2c01e0fc8840bdb67647734ba4c3f0 I0312 20:12:06.483558 6186 consensus_queue.cc:196] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 1183c4bf4e494b75a57827848998cacc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 37, Committed index: 37, Last appended: 1.37, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 38 OBSOLETE_local: false peers { permanent_uuid: "ac1592be0a6240378933768820f57ed3" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 59968 } } peers { permanent_uuid: "1183c4bf4e494b75a57827848998cacc" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 47310 } } I0312 20:12:06.485190 5827 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "9a02c73fbfaa40d5bf986d34f5b785f9" candidate_uuid: "0c2c01e0fc8840bdb67647734ba4c3f0" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "ac1592be0a6240378933768820f57ed3" is_pre_election: true I0312 20:12:06.485738 5827 raft_consensus.cc:2176] T 9a02c73fbfaa40d5bf986d34f5b785f9 P ac1592be0a6240378933768820f57ed3 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0c2c01e0fc8840bdb67647734ba4c3f0 for term 2 because replica is either leader or believes a valid leader to be alive. I0312 20:12:06.486393 5953 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "9a02c73fbfaa40d5bf986d34f5b785f9" candidate_uuid: "0c2c01e0fc8840bdb67647734ba4c3f0" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "1183c4bf4e494b75a57827848998cacc" is_pre_election: true I0312 20:12:06.486843 5953 raft_consensus.cc:1571] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 1183c4bf4e494b75a57827848998cacc [term 1 LEADER]: Handling vote request from an unknown peer 0c2c01e0fc8840bdb67647734ba4c3f0 I0312 20:12:06.486843 6013 leader_election.cc:377] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 0c2c01e0fc8840bdb67647734ba4c3f0 [CANDIDATE]: Term 2 pre-election: Vote denied by peer ac1592be0a6240378933768820f57ed3. Message: Invalid argument: T 9a02c73fbfaa40d5bf986d34f5b785f9 P ac1592be0a6240378933768820f57ed3 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0c2c01e0fc8840bdb67647734ba4c3f0 for term 2 because replica is either leader or believes a valid leader to be alive. I0312 20:12:06.487078 5953 raft_consensus.cc:2176] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 1183c4bf4e494b75a57827848998cacc [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate 0c2c01e0fc8840bdb67647734ba4c3f0 for term 2 because replica is either leader or believes a valid leader to be alive. I0312 20:12:06.487872 5827 raft_consensus.cc:1058] T 9a02c73fbfaa40d5bf986d34f5b785f9 P ac1592be0a6240378933768820f57ed3 [term 1 FOLLOWER]: Refusing update from remote peer 1183c4bf4e494b75a57827848998cacc: Log matching property violated. Preceding OpId in replica: term: 1 index: 37. Preceding OpId from leader: term: 1 index: 38. (index mismatch) I0312 20:12:06.487943 6011 leader_election.cc:377] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 0c2c01e0fc8840bdb67647734ba4c3f0 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 1183c4bf4e494b75a57827848998cacc. Message: Invalid argument: T 9a02c73fbfaa40d5bf986d34f5b785f9 P 1183c4bf4e494b75a57827848998cacc [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate 0c2c01e0fc8840bdb67647734ba4c3f0 for term 2 because replica is either leader or believes a valid leader to be alive. I0312 20:12:06.488126 6011 leader_election.cc:258] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 0c2c01e0fc8840bdb67647734ba4c3f0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. I0312 20:12:06.488497 6155 raft_consensus.cc:2700] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 0c2c01e0fc8840bdb67647734ba4c3f0: Snoozing failure detection for 2.113s (election complete) I0312 20:12:06.488692 6155 raft_consensus.cc:2455] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 0c2c01e0fc8840bdb67647734ba4c3f0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority I0312 20:12:06.489490 6192 consensus_queue.cc:1017] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 1183c4bf4e494b75a57827848998cacc [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac1592be0a6240378933768820f57ed3" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 59968 }, Status: LMP_MISMATCH, Last received: 1.37, Next index: 38, Last known committed idx: 37, Time since last communication: 0.000s I0312 20:12:06.494427 6192 raft_consensus.cc:2654] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 1183c4bf4e494b75a57827848998cacc [term 1 LEADER]: Committing config change with OpId 1.38: config changed from index -1 to 38, VOTER 0c2c01e0fc8840bdb67647734ba4c3f0 (127.0.53.3) evicted. New config: { opid_index: 38 OBSOLETE_local: false peers { permanent_uuid: "ac1592be0a6240378933768820f57ed3" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 59968 } } peers { permanent_uuid: "1183c4bf4e494b75a57827848998cacc" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 47310 } } } I0312 20:12:06.495456 5827 raft_consensus.cc:2654] T 9a02c73fbfaa40d5bf986d34f5b785f9 P ac1592be0a6240378933768820f57ed3 [term 1 FOLLOWER]: Committing config change with OpId 1.38: config changed from index -1 to 38, VOTER 0c2c01e0fc8840bdb67647734ba4c3f0 (127.0.53.3) evicted. New config: { opid_index: 38 OBSOLETE_local: false peers { permanent_uuid: "ac1592be0a6240378933768820f57ed3" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 59968 } } peers { permanent_uuid: "1183c4bf4e494b75a57827848998cacc" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 47310 } } } I0312 20:12:06.500293 5705 catalog_manager.cc:3592] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 1183c4bf4e494b75a57827848998cacc reported cstate change: config changed from index -1 to 38, VOTER 0c2c01e0fc8840bdb67647734ba4c3f0 (127.0.53.3) evicted. New cstate: current_term: 1 leader_uuid: "1183c4bf4e494b75a57827848998cacc" committed_config { opid_index: 38 OBSOLETE_local: false peers { permanent_uuid: "ac1592be0a6240378933768820f57ed3" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 59968 } } peers { permanent_uuid: "1183c4bf4e494b75a57827848998cacc" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 47310 } } } I0312 20:12:06.506271 5705 catalog_manager.cc:2919] Sending DeleteTablet(TABLET_DATA_TOMBSTONED) for tablet 9a02c73fbfaa40d5bf986d34f5b785f9 on 0c2c01e0fc8840bdb67647734ba4c3f0 (127.0.53.3:60846) (TS 0c2c01e0fc8840bdb67647734ba4c3f0 not found in new config with opid_index 38) I0312 20:12:06.507510 6058 tablet_service.cc:796] Processing DeleteTablet for tablet 9a02c73fbfaa40d5bf986d34f5b785f9 with delete_type TABLET_DATA_TOMBSTONED (TS 0c2c01e0fc8840bdb67647734ba4c3f0 not found in new config with opid_index 38) from {username='slave'} at 127.0.0.1:39249 I0312 20:12:06.508990 6202 tablet_replica.cc:262] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 0c2c01e0fc8840bdb67647734ba4c3f0: stopping tablet replica I0312 20:12:06.509217 6202 maintenance_manager.cc:235] P 0c2c01e0fc8840bdb67647734ba4c3f0: Unregistered op CompactRowSetsOp(9a02c73fbfaa40d5bf986d34f5b785f9) I0312 20:12:06.509356 6202 maintenance_manager.cc:235] P 0c2c01e0fc8840bdb67647734ba4c3f0: Unregistered op MinorDeltaCompactionOp(9a02c73fbfaa40d5bf986d34f5b785f9) I0312 20:12:06.509475 6202 maintenance_manager.cc:235] P 0c2c01e0fc8840bdb67647734ba4c3f0: Unregistered op MajorDeltaCompactionOp(9a02c73fbfaa40d5bf986d34f5b785f9) I0312 20:12:06.509608 6202 maintenance_manager.cc:235] P 0c2c01e0fc8840bdb67647734ba4c3f0: Unregistered op UndoDeltaBlockGCOp(9a02c73fbfaa40d5bf986d34f5b785f9) I0312 20:12:06.509768 6202 maintenance_manager.cc:235] P 0c2c01e0fc8840bdb67647734ba4c3f0: Unregistered op FlushMRSOp(9a02c73fbfaa40d5bf986d34f5b785f9) I0312 20:12:06.509891 6202 maintenance_manager.cc:235] P 0c2c01e0fc8840bdb67647734ba4c3f0: Unregistered op FlushDeltaMemStoresOp(9a02c73fbfaa40d5bf986d34f5b785f9) I0312 20:12:06.510020 6202 maintenance_manager.cc:235] P 0c2c01e0fc8840bdb67647734ba4c3f0: Unregistered op LogGCOp(9a02c73fbfaa40d5bf986d34f5b785f9) I0312 20:12:06.510180 6202 raft_consensus.cc:2012] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 0c2c01e0fc8840bdb67647734ba4c3f0 [term 1 FOLLOWER]: Raft consensus shutting down. I0312 20:12:06.510399 6202 pending_rounds.cc:63] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 0c2c01e0fc8840bdb67647734ba4c3f0: Trying to abort 1 pending transactions. I0312 20:12:06.510521 6202 pending_rounds.cc:68] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 0c2c01e0fc8840bdb67647734ba4c3f0: Aborting transaction as it isn't in flight: id { term: 1 index: 2 } timestamp: 6229547105037762560 op_type: WRITE_OP write_request { tablet_id: "9a02c73fbfaa40d5bf986d34f5b785f9" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true } } row_operations { rows: "\001\007\000\030\277\240\001\372&\346\261\000\000\000\000\000\000\000\000\000\000\002\000\000\000\000\000" indirect_data: "0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 I0312 20:12:06.531193 6202 raft_consensus.cc:2039] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 0c2c01e0fc8840bdb67647734ba4c3f0 [term 1 FOLLOWER]: Raft consensus is shut down! I0312 20:12:06.533027 6202 ts_tablet_manager.cc:1360] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 0c2c01e0fc8840bdb67647734ba4c3f0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED I0312 20:12:06.539748 6202 ts_tablet_manager.cc:1373] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 0c2c01e0fc8840bdb67647734ba4c3f0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2 I0312 20:12:06.539971 6202 log.cc:981] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 0c2c01e0fc8840bdb67647734ba4c3f0: Deleting WAL directory at /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1520885401735786-212/raft_consensus-itest-cluster/ts-2/wal/wals/9a02c73fbfaa40d5bf986d34f5b785f9 I0312 20:12:06.541237 5690 catalog_manager.cc:2897] TS 0c2c01e0fc8840bdb67647734ba4c3f0 (127.0.53.3:60846): tablet 9a02c73fbfaa40d5bf986d34f5b785f9 (table TestTable [id=ce3d9a5ec62f446d8f8a4df6c8c754d8]) successfully deleted I0312 20:12:06.542404 5871 maintenance_manager.cc:302] P ac1592be0a6240378933768820f57ed3: Scheduling LogGCOp(9a02c73fbfaa40d5bf986d34f5b785f9): free 1707539 bytes of WAL I0312 20:12:06.542497 6118 generic_service.cc:112] {username='slave'} at 127.0.0.1:39254 changed flags via RPC: tablet_copy_fault_crash_on_fetch_all from '0' to '1.0' I0312 20:12:06.542855 5755 log_reader.cc:376] T 9a02c73fbfaa40d5bf986d34f5b785f9: removed 2 log segments from log reader I0312 20:12:06.542994 5755 log.cc:856] T 9a02c73fbfaa40d5bf986d34f5b785f9 P ac1592be0a6240378933768820f57ed3: Deleting log segment in path: /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1520885401735786-212/raft_consensus-itest-cluster/ts-0/wal/wals/9a02c73fbfaa40d5bf986d34f5b785f9/wal-000000004 (ops 18-23) I0312 20:12:06.543150 5755 log.cc:856] T 9a02c73fbfaa40d5bf986d34f5b785f9 P ac1592be0a6240378933768820f57ed3: Deleting log segment in path: /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1520885401735786-212/raft_consensus-itest-cluster/ts-0/wal/wals/9a02c73fbfaa40d5bf986d34f5b785f9/wal-000000005 (ops 24-30) I0312 20:12:06.543663 5755 maintenance_manager.cc:492] P ac1592be0a6240378933768820f57ed3: LogGCOp(9a02c73fbfaa40d5bf986d34f5b785f9) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {} I0312 20:12:06.544658 5953 consensus_peers.cc:458] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 1183c4bf4e494b75a57827848998cacc -> Peer ac1592be0a6240378933768820f57ed3 (127.0.53.1:59968): Closing peer: ac1592be0a6240378933768820f57ed3 I0312 20:12:06.545023 5953 consensus_queue.cc:196] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 1183c4bf4e494b75a57827848998cacc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 38, Committed index: 38, Last appended: 1.38, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 39 OBSOLETE_local: false peers { permanent_uuid: "ac1592be0a6240378933768820f57ed3" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 59968 } } peers { permanent_uuid: "1183c4bf4e494b75a57827848998cacc" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 47310 } } peers { permanent_uuid: "0c2c01e0fc8840bdb67647734ba4c3f0" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 60846 } attrs { } } I0312 20:12:06.547924 5827 raft_consensus.cc:1058] T 9a02c73fbfaa40d5bf986d34f5b785f9 P ac1592be0a6240378933768820f57ed3 [term 1 FOLLOWER]: Refusing update from remote peer 1183c4bf4e494b75a57827848998cacc: Log matching property violated. Preceding OpId in replica: term: 1 index: 38. Preceding OpId from leader: term: 1 index: 39. (index mismatch) W0312 20:12:06.548338 5887 consensus_peers.cc:436] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 1183c4bf4e494b75a57827848998cacc -> Peer 0c2c01e0fc8840bdb67647734ba4c3f0 (127.0.53.3:60846): Couldn't send request to peer 0c2c01e0fc8840bdb67647734ba4c3f0 for tablet 9a02c73fbfaa40d5bf986d34f5b785f9. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. Retrying in the next heartbeat period. Already tried 1 times. I0312 20:12:06.548880 6192 consensus_queue.cc:1017] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 1183c4bf4e494b75a57827848998cacc [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac1592be0a6240378933768820f57ed3" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 59968 }, Status: LMP_MISMATCH, Last received: 1.38, Next index: 39, Last known committed idx: 38, Time since last communication: 0.000s I0312 20:12:06.551811 6186 raft_consensus.cc:2654] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 1183c4bf4e494b75a57827848998cacc [term 1 LEADER]: Committing config change with OpId 1.39: config changed from index 38 to 39, VOTER 0c2c01e0fc8840bdb67647734ba4c3f0 (127.0.53.3) added. New config: { opid_index: 39 OBSOLETE_local: false peers { permanent_uuid: "ac1592be0a6240378933768820f57ed3" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 59968 } } peers { permanent_uuid: "1183c4bf4e494b75a57827848998cacc" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 47310 } } peers { permanent_uuid: "0c2c01e0fc8840bdb67647734ba4c3f0" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 60846 } attrs { } } } I0312 20:12:06.552775 5827 raft_consensus.cc:2654] T 9a02c73fbfaa40d5bf986d34f5b785f9 P ac1592be0a6240378933768820f57ed3 [term 1 FOLLOWER]: Committing config change with OpId 1.39: config changed from index 38 to 39, VOTER 0c2c01e0fc8840bdb67647734ba4c3f0 (127.0.53.3) added. New config: { opid_index: 39 OBSOLETE_local: false peers { permanent_uuid: "ac1592be0a6240378933768820f57ed3" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 59968 } } peers { permanent_uuid: "1183c4bf4e494b75a57827848998cacc" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 47310 } } peers { permanent_uuid: "0c2c01e0fc8840bdb67647734ba4c3f0" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 60846 } attrs { } } } I0312 20:12:06.558099 5704 catalog_manager.cc:3592] T 9a02c73fbfaa40d5bf986d34f5b785f9 P ac1592be0a6240378933768820f57ed3 reported cstate change: config changed from index 38 to 39, VOTER 0c2c01e0fc8840bdb67647734ba4c3f0 (127.0.53.3) added. New cstate: current_term: 1 leader_uuid: "1183c4bf4e494b75a57827848998cacc" committed_config { opid_index: 39 OBSOLETE_local: false peers { permanent_uuid: "ac1592be0a6240378933768820f57ed3" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 59968 } } peers { permanent_uuid: "1183c4bf4e494b75a57827848998cacc" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 47310 } } peers { permanent_uuid: "0c2c01e0fc8840bdb67647734ba4c3f0" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 60846 } attrs { } } } I0312 20:12:07.139634 6204 ts_tablet_manager.cc:662] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 0c2c01e0fc8840bdb67647734ba4c3f0: Initiating tablet copy from peer 1183c4bf4e494b75a57827848998cacc (127.0.53.2:47310) I0312 20:12:07.139912 6204 tablet_copy_client.cc:237] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 0c2c01e0fc8840bdb67647734ba4c3f0: tablet copy: Beginning tablet copy session from remote peer at address 127.0.53.2:47310 I0312 20:12:07.140985 5973 tablet_copy_service.cc:133] P 1183c4bf4e494b75a57827848998cacc: Received BeginTabletCopySession request for tablet 9a02c73fbfaa40d5bf986d34f5b785f9 from peer 0c2c01e0fc8840bdb67647734ba4c3f0 ({username='slave'} at 127.0.53.3:42324) I0312 20:12:07.141278 5973 tablet_copy_service.cc:154] P 1183c4bf4e494b75a57827848998cacc: Beginning new tablet copy session on tablet 9a02c73fbfaa40d5bf986d34f5b785f9 from peer 0c2c01e0fc8840bdb67647734ba4c3f0 at {username='slave'} at 127.0.53.3:42324: session id = 0c2c01e0fc8840bdb67647734ba4c3f0-9a02c73fbfaa40d5bf986d34f5b785f9 I0312 20:12:07.145334 5973 tablet_copy_source_session.cc:202] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 1183c4bf4e494b75a57827848998cacc: Tablet Copy: opened 5 blocks and 3 log segments I0312 20:12:07.147385 6204 ts_tablet_manager.cc:1360] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 0c2c01e0fc8840bdb67647734ba4c3f0: Deleting tablet data with delete state TABLET_DATA_COPYING I0312 20:12:07.153085 6204 ts_tablet_manager.cc:1373] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 0c2c01e0fc8840bdb67647734ba4c3f0: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.2 I0312 20:12:07.153337 6204 data_dirs.cc:931] Could only allocate 1 dirs of requested 3 for tablet 9a02c73fbfaa40d5bf986d34f5b785f9. 1 dirs total, 0 dirs full, 0 dirs failed I0312 20:12:07.155957 6204 ts_tablet_manager.cc:1163] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 0c2c01e0fc8840bdb67647734ba4c3f0: Registered tablet (data state: TABLET_DATA_COPYING) W0312 20:12:07.156282 6204 fault_injection.cc:41] FAULT INJECTION ENABLED! W0312 20:12:07.156383 6204 fault_injection.cc:42] THIS SERVER MAY CRASH! E0312 20:12:07.156486 6204 fault_injection.cc:57] Injecting fault: FLAGS_tablet_copy_fault_crash_on_fetch_all (process will exit) W0312 20:12:07.704979 5887 consensus_peers.cc:436] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 1183c4bf4e494b75a57827848998cacc -> Peer 0c2c01e0fc8840bdb67647734ba4c3f0 (127.0.53.3:60846): Couldn't send request to peer 0c2c01e0fc8840bdb67647734ba4c3f0 for tablet 9a02c73fbfaa40d5bf986d34f5b785f9. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. Retrying in the next heartbeat period. Already tried 2 times. W0312 20:12:08.176311 5690 connection.cc:513] server connection from 127.0.53.3:49005 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0312 20:12:08.176318 5887 connection.cc:513] client connection to 127.0.53.3:60846 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0312 20:12:08.176375 5676 connection.cc:513] client connection to 127.0.53.3:60846 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0312 20:12:08.176916 5690 connection.cc:513] client connection to 127.0.53.3:60846 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0312 20:12:08.177260 5885 connection.cc:513] server connection from 127.0.53.3:42324 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0312 20:12:08.177307 5675 connection.cc:513] client connection to 127.0.53.3:60846 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0312 20:12:08.178087 5760 connection.cc:513] server connection from 127.0.53.3:45063 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) I0312 20:12:08.268752 212 external_mini_cluster.cc:997] Killing /tmp/dist-test-task3bwPeE/build/tsan/bin/kudu-tserver with pid 5739 W0312 20:12:08.298296 5690 connection.cc:513] server connection from 127.0.53.1:38459 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0312 20:12:08.298336 5887 connection.cc:513] client connection to 127.0.53.1:59968 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0312 20:12:08.298604 5887 consensus_peers.cc:436] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 1183c4bf4e494b75a57827848998cacc -> Peer ac1592be0a6240378933768820f57ed3 (127.0.53.1:59968): Couldn't send request to peer ac1592be0a6240378933768820f57ed3 for tablet 9a02c73fbfaa40d5bf986d34f5b785f9. Status: Network error: failed to read from TLS socket: Connection reset by peer (error 104). Retrying in the next heartbeat period. Already tried 1 times. W0312 20:12:08.298741 5690 connection.cc:513] client connection to 127.0.53.1:59968 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0312 20:12:08.299199 5887 connection.cc:186] Error closing socket: Network error: TlsSocket::Close: Broken pipe W0312 20:12:08.299756 5675 connection.cc:513] client connection to 127.0.53.1:59968 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0312 20:12:08.299747 5676 connection.cc:513] client connection to 127.0.53.1:59968 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0312 20:12:08.307869 5887 consensus_peers.cc:436] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 1183c4bf4e494b75a57827848998cacc -> Peer 0c2c01e0fc8840bdb67647734ba4c3f0 (127.0.53.3:60846): Couldn't send request to peer 0c2c01e0fc8840bdb67647734ba4c3f0 for tablet 9a02c73fbfaa40d5bf986d34f5b785f9. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.0.53.3:60846: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 3 times. I0312 20:12:08.309249 212 external_mini_cluster.cc:997] Killing /tmp/dist-test-task3bwPeE/build/tsan/bin/kudu-tserver with pid 5874 W0312 20:12:08.340781 5676 connection.cc:513] client connection to 127.0.53.2:47310 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0312 20:12:08.340775 5688 connection.cc:513] client connection to 127.0.53.2:47310 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0312 20:12:08.341183 5674 connection.cc:513] client connection to 127.0.53.2:47310 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0312 20:12:08.340783 5686 connection.cc:513] server connection from 127.0.53.2:36086 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) I0312 20:12:08.343132 212 external_mini_cluster.cc:997] Killing /tmp/dist-test-task3bwPeE/build/tsan/bin/kudu-master with pid 5677 W0312 20:12:08.365485 5676 connection.cc:513] client connection to 127.0.53.62:53264 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0312 20:12:08.365562 6126 connection.cc:513] client connection to 127.0.53.62:53264 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) I0312 20:12:08.369499 212 external_mini_cluster.cc:805] Running /tmp/dist-test-task3bwPeE/build/tsan/bin/kudu-master /tmp/dist-test-task3bwPeE/build/tsan/bin/kudu-master --fs_wal_dir=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1520885401735786-212/raft_consensus-itest-cluster/master-0/wal --fs_data_dirs=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1520885401735786-212/raft_consensus-itest-cluster/master-0/data --block_manager=log --webserver_interface=localhost --ipki_ca_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.0.53.62:53264 --webserver_interface=127.0.53.62 --webserver_port=36897 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1520885401735786-212/raft_consensus-itest-cluster/master-0/logs --server_dump_info_path=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1520885401735786-212/raft_consensus-itest-cluster/master-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --master_add_server_when_underreplicated=false --raft_prepare_replacement_before_eviction=false with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W0312 20:12:08.489831 6208 flags.cc:412] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0312 20:12:08.490135 6208 flags.cc:412] Enabled unsafe flag: --never_fsync=true W0312 20:12:08.505527 6208 flags.cc:412] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false W0312 20:12:08.505694 6208 flags.cc:412] Enabled experimental flag: --ipki_ca_key_size=1024 W0312 20:12:08.505795 6208 flags.cc:412] Enabled experimental flag: --ipki_server_key_size=1024 W0312 20:12:08.505890 6208 flags.cc:412] Enabled experimental flag: --tsk_num_rsa_bits=512 I0312 20:12:08.526454 6208 master_main.cc:66] Master server non-default flags: --raft_prepare_replacement_before_eviction=false --fs_data_dirs=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1520885401735786-212/raft_consensus-itest-cluster/master-0/data --fs_wal_dir=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1520885401735786-212/raft_consensus-itest-cluster/master-0/wal --master_add_server_when_underreplicated=false --ipki_ca_key_size=1024 --ipki_server_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.0.53.62:53264 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1520885401735786-212/raft_consensus-itest-cluster/master-0/data/info.pb --webserver_interface=127.0.53.62 --webserver_port=36897 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --log_dir=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1520885401735786-212/raft_consensus-itest-cluster/master-0/logs --logbuflevel=-1 --logtostderr=true Master server version: kudu 1.7.0-SNAPSHOT revision 13e10c63c46de30ac85f34b543cdd6fb44881f61 build type FASTDEBUG built by None at 12 Mar 2018 20:06:40 UTC on jenkins-slave-vn0g build id 12388 TSAN enabled I0312 20:12:08.535574 6208 master_main.cc:73] Initializing master server... I0312 20:12:08.537061 6208 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 1131276us I0312 20:12:08.537808 6208 fs_manager.cc:260] Metadata directory not provided I0312 20:12:08.537953 6208 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0312 20:12:08.542657 6208 fs_manager.cc:406] Time spent opening directory manager: real 0.003s user 0.008s sys 0.000s I0312 20:12:08.542910 6208 env_posix.cc:1620] Not raising this process' open files per process limit of 65536; it is already as high as it can go I0312 20:12:08.543175 6208 file_cache.cc:470] Constructed file cache lbm with capacity 26214 I0312 20:12:08.549624 6208 fs_manager.cc:418] Time spent opening block manager: real 0.002s user 0.004s sys 0.000s I0312 20:12:08.549748 6208 fs_manager.cc:429] Opened local filesystem: /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1520885401735786-212/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1520885401735786-212/raft_consensus-itest-cluster/master-0/wal uuid: "edc74cc9e16f4934b5ef99cafc75d0b9" format_stamp: "Formatted at 2018-03-12 20:12:00 on dist-test-slave-dist-test-slave-f27p" I0312 20:12:08.550879 6208 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1520885401735786-212/raft_consensus-itest-cluster/master-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0312 20:12:08.740464 6208 env_posix.cc:1620] Not raising this process' running threads per effective uid limit of 60284; it is already as high as it can go I0312 20:12:08.741102 6208 master_main.cc:76] Starting Master server... I0312 20:12:08.742312 6225 process_memory.cc:182] Process hard memory limit is 11.777631 GB I0312 20:12:08.742460 6225 process_memory.cc:184] Process soft memory limit is 9.422105 GB I0312 20:12:08.742558 6225 process_memory.cc:187] Process memory pressure threshold is 7.066578 GB I0312 20:12:08.781527 6208 rpc_server.cc:201] RPC server started. Bound to: 127.0.53.62:53264 I0312 20:12:08.807320 6208 webserver.cc:173] Starting webserver on 127.0.53.62:36897 I0312 20:12:08.807435 6208 webserver.cc:184] Document root disabled I0312 20:12:08.808670 6208 webserver.cc:311] Webserver started. Bound to: http://127.0.53.62:36897/ I0312 20:12:08.809502 6208 server_base.cc:607] Dumped server information to /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1520885401735786-212/raft_consensus-itest-cluster/master-0/data/info.pb I0312 20:12:08.813835 212 external_mini_cluster.cc:867] Started /tmp/dist-test-task3bwPeE/build/tsan/bin/kudu-master as pid 6208 I0312 20:12:08.815239 212 external_mini_cluster.cc:805] Running /tmp/dist-test-task3bwPeE/build/tsan/bin/kudu-tserver /tmp/dist-test-task3bwPeE/build/tsan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1520885401735786-212/raft_consensus-itest-cluster/ts-1/wal --fs_data_dirs=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1520885401735786-212/raft_consensus-itest-cluster/ts-1/data --block_manager=log --rpc_bind_addresses=127.0.53.2:47310 --local_ip_for_outbound_sockets=127.0.53.2 --webserver_port=50074 --webserver_interface=127.0.53.2 --tserver_master_addrs=127.0.53.62:53264 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1520885401735786-212/raft_consensus-itest-cluster/ts-1/logs --server_dump_info_path=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1520885401735786-212/raft_consensus-itest-cluster/ts-1/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=false --log_compression_codec=none --log_cache_size_limit_mb=1 --log_segment_size_mb=1 --log_async_preallocate_segments=false --log_min_segments_to_retain=1 --log_max_segments_to_retain=3 --maintenance_manager_polling_interval_ms=100 --log_target_replay_size_mb=1 --max_cell_size_bytes=1000000 with env {} I0312 20:12:08.820595 6269 tablet_bootstrap.cc:436] T 00000000000000000000000000000000 P edc74cc9e16f4934b5ef99cafc75d0b9: Bootstrap starting. I0312 20:12:08.846017 6269 log.cc:520] T 00000000000000000000000000000000 P edc74cc9e16f4934b5ef99cafc75d0b9: Log is configured to *not* fsync() on all Append() calls I0312 20:12:08.860512 6269 tablet_bootstrap.cc:436] T 00000000000000000000000000000000 P edc74cc9e16f4934b5ef99cafc75d0b9: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=1} inserts{seen=4 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates I0312 20:12:08.861111 6269 tablet_bootstrap.cc:436] T 00000000000000000000000000000000 P edc74cc9e16f4934b5ef99cafc75d0b9: Bootstrap complete. I0312 20:12:08.879082 6269 raft_consensus.cc:304] T 00000000000000000000000000000000 P edc74cc9e16f4934b5ef99cafc75d0b9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edc74cc9e16f4934b5ef99cafc75d0b9" member_type: VOTER } I0312 20:12:08.880441 6269 raft_consensus.cc:605] T 00000000000000000000000000000000 P edc74cc9e16f4934b5ef99cafc75d0b9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: edc74cc9e16f4934b5ef99cafc75d0b9, State: Initialized, Role: FOLLOWER I0312 20:12:08.880924 6269 consensus_queue.cc:219] T 00000000000000000000000000000000 P edc74cc9e16f4934b5ef99cafc75d0b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edc74cc9e16f4934b5ef99cafc75d0b9" member_type: VOTER } I0312 20:12:08.881302 6269 raft_consensus.cc:344] T 00000000000000000000000000000000 P edc74cc9e16f4934b5ef99cafc75d0b9 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0312 20:12:08.881446 6269 raft_consensus.cc:436] T 00000000000000000000000000000000 P edc74cc9e16f4934b5ef99cafc75d0b9 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0312 20:12:08.881587 6269 raft_consensus.cc:2700] T 00000000000000000000000000000000 P edc74cc9e16f4934b5ef99cafc75d0b9: Snoozing failure detection for 2.203s (starting election) I0312 20:12:08.881731 6269 raft_consensus.cc:2752] T 00000000000000000000000000000000 P edc74cc9e16f4934b5ef99cafc75d0b9 [term 1 FOLLOWER]: Advancing to term 2 I0312 20:12:08.886059 6269 raft_consensus.cc:458] T 00000000000000000000000000000000 P edc74cc9e16f4934b5ef99cafc75d0b9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edc74cc9e16f4934b5ef99cafc75d0b9" member_type: VOTER } I0312 20:12:08.886462 6269 leader_election.cc:258] T 00000000000000000000000000000000 P edc74cc9e16f4934b5ef99cafc75d0b9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. I0312 20:12:08.888038 6273 raft_consensus.cc:2700] T 00000000000000000000000000000000 P edc74cc9e16f4934b5ef99cafc75d0b9: Snoozing failure detection for 2.090s (election complete) I0312 20:12:08.888294 6273 raft_consensus.cc:2507] T 00000000000000000000000000000000 P edc74cc9e16f4934b5ef99cafc75d0b9 [term 2 FOLLOWER]: Leader election won for term 2 I0312 20:12:08.890350 6273 raft_consensus.cc:572] T 00000000000000000000000000000000 P edc74cc9e16f4934b5ef99cafc75d0b9 [term 2 LEADER]: Becoming Leader. State: Replica: edc74cc9e16f4934b5ef99cafc75d0b9, State: Running, Role: LEADER I0312 20:12:08.890799 6273 consensus_queue.cc:196] T 00000000000000000000000000000000 P edc74cc9e16f4934b5ef99cafc75d0b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edc74cc9e16f4934b5ef99cafc75d0b9" member_type: VOTER } I0312 20:12:08.894145 6274 sys_catalog.cc:335] T 00000000000000000000000000000000 P edc74cc9e16f4934b5ef99cafc75d0b9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader edc74cc9e16f4934b5ef99cafc75d0b9. Latest consensus state: current_term: 2 leader_uuid: "edc74cc9e16f4934b5ef99cafc75d0b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edc74cc9e16f4934b5ef99cafc75d0b9" member_type: VOTER } } I0312 20:12:08.894194 6273 sys_catalog.cc:335] T 00000000000000000000000000000000 P edc74cc9e16f4934b5ef99cafc75d0b9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "edc74cc9e16f4934b5ef99cafc75d0b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edc74cc9e16f4934b5ef99cafc75d0b9" member_type: VOTER } } I0312 20:12:08.894556 6274 sys_catalog.cc:338] T 00000000000000000000000000000000 P edc74cc9e16f4934b5ef99cafc75d0b9 [sys.catalog]: This master's current role is: LEADER I0312 20:12:08.894599 6273 sys_catalog.cc:338] T 00000000000000000000000000000000 P edc74cc9e16f4934b5ef99cafc75d0b9 [sys.catalog]: This master's current role is: LEADER I0312 20:12:08.897948 6269 sys_catalog.cc:335] T 00000000000000000000000000000000 P edc74cc9e16f4934b5ef99cafc75d0b9 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 2 leader_uuid: "edc74cc9e16f4934b5ef99cafc75d0b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edc74cc9e16f4934b5ef99cafc75d0b9" member_type: VOTER } } I0312 20:12:08.898250 6269 sys_catalog.cc:338] T 00000000000000000000000000000000 P edc74cc9e16f4934b5ef99cafc75d0b9 [sys.catalog]: This master's current role is: LEADER I0312 20:12:08.899158 6269 sys_catalog.cc:422] T 00000000000000000000000000000000 P edc74cc9e16f4934b5ef99cafc75d0b9 [sys.catalog]: configured and running, proceeding with master startup. I0312 20:12:08.899448 6277 catalog_manager.cc:972] Loading table and tablet metadata into memory... I0312 20:12:08.901222 6208 master_main.cc:79] Master server successfully started. I0312 20:12:08.904289 6277 catalog_manager.cc:319] Loaded metadata for table TestTable [id=ce3d9a5ec62f446d8f8a4df6c8c754d8] I0312 20:12:08.906325 6277 catalog_manager.cc:373] Loaded metadata for tablet 9a02c73fbfaa40d5bf986d34f5b785f9 (table TestTable [id=ce3d9a5ec62f446d8f8a4df6c8c754d8]) I0312 20:12:08.906723 6277 catalog_manager.cc:983] Initializing Kudu internal certificate authority... I0312 20:12:08.910190 6277 catalog_manager.cc:992] Loading token signing keys... I0312 20:12:08.911644 6277 catalog_manager.cc:3879] T 00000000000000000000000000000000 P edc74cc9e16f4934b5ef99cafc75d0b9: Loaded TSK: 0 WARNING: Logging before InitGoogleLogging() is written to STDERR W0312 20:12:08.962687 6270 flags.cc:412] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0312 20:12:08.963161 6270 flags.cc:412] Enabled unsafe flag: --max_cell_size_bytes=1000000 W0312 20:12:08.963351 6270 flags.cc:412] Enabled unsafe flag: --never_fsync=true W0312 20:12:08.987231 6270 flags.cc:412] Enabled experimental flag: --log_compression_codec=none W0312 20:12:08.987429 6270 flags.cc:412] Enabled experimental flag: --log_max_segments_to_retain=3 W0312 20:12:08.987674 6270 flags.cc:412] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false W0312 20:12:08.987888 6270 flags.cc:412] Enabled experimental flag: --ipki_server_key_size=1024 W0312 20:12:08.988322 6270 flags.cc:412] Enabled experimental flag: --log_target_replay_size_mb=1 W0312 20:12:08.988538 6270 flags.cc:412] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.53.2 I0312 20:12:09.020278 6270 tablet_server_main.cc:72] Tablet server non-default flags: --log_compression_codec=none --log_max_segments_to_retain=3 --log_cache_size_limit_mb=1 --log_async_preallocate_segments=false --log_segment_size_mb=1 --raft_prepare_replacement_before_eviction=false --fs_data_dirs=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1520885401735786-212/raft_consensus-itest-cluster/ts-1/data --fs_wal_dir=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1520885401735786-212/raft_consensus-itest-cluster/ts-1/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.0.53.2:47310 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1520885401735786-212/raft_consensus-itest-cluster/ts-1/data/info.pb --webserver_interface=127.0.53.2 --webserver_port=50074 --max_cell_size_bytes=1000000 --tserver_master_addrs=127.0.53.62:53264 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --log_target_replay_size_mb=1 --maintenance_manager_polling_interval_ms=100 --enable_minidumps=false --local_ip_for_outbound_sockets=127.0.53.2 --log_dir=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1520885401735786-212/raft_consensus-itest-cluster/ts-1/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.7.0-SNAPSHOT revision 13e10c63c46de30ac85f34b543cdd6fb44881f61 build type FASTDEBUG built by None at 12 Mar 2018 20:06:40 UTC on jenkins-slave-vn0g build id 12388 TSAN enabled I0312 20:12:09.035383 6270 tablet_server_main.cc:79] Initializing tablet server... I0312 20:12:09.037798 6270 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 1131776us I0312 20:12:09.038449 6270 fs_manager.cc:260] Metadata directory not provided I0312 20:12:09.038635 6270 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0312 20:12:09.045428 6270 fs_manager.cc:406] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s I0312 20:12:09.045760 6270 env_posix.cc:1620] Not raising this process' open files per process limit of 65536; it is already as high as it can go I0312 20:12:09.046128 6270 file_cache.cc:470] Constructed file cache lbm with capacity 26214 I0312 20:12:09.064118 6270 fs_manager.cc:418] Time spent opening block manager: real 0.009s user 0.000s sys 0.000s I0312 20:12:09.064285 6270 fs_manager.cc:429] Opened local filesystem: /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1520885401735786-212/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1520885401735786-212/raft_consensus-itest-cluster/ts-1/wal uuid: "1183c4bf4e494b75a57827848998cacc" format_stamp: "Formatted at 2018-03-12 20:12:01 on dist-test-slave-dist-test-slave-f27p" I0312 20:12:09.065996 6270 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1520885401735786-212/raft_consensus-itest-cluster/ts-1/data/data Total live blocks: 5 Total live bytes: 136507 Total live bytes (after alignment): 155648 Total number of LBM containers: 6 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0312 20:12:09.278129 6270 env_posix.cc:1620] Not raising this process' running threads per effective uid limit of 60284; it is already as high as it can go I0312 20:12:09.280056 6270 ts_tablet_manager.cc:337] Loading tablet metadata (0/1 complete) I0312 20:12:09.280241 6270 ts_tablet_manager.cc:995] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 1183c4bf4e494b75a57827848998cacc: Loading tablet metadata I0312 20:12:09.287485 6270 ts_tablet_manager.cc:349] Loaded tablet metadata (1 live tablets) I0312 20:12:09.292281 6270 ts_tablet_manager.cc:1163] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 1183c4bf4e494b75a57827848998cacc: Registered tablet (data state: TABLET_DATA_READY) I0312 20:12:09.296025 6298 ts_tablet_manager.cc:1018] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 1183c4bf4e494b75a57827848998cacc: Bootstrapping tablet I0312 20:12:09.296481 6298 tablet_bootstrap.cc:436] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 1183c4bf4e494b75a57827848998cacc: Bootstrap starting. I0312 20:12:09.316570 6298 log.cc:520] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 1183c4bf4e494b75a57827848998cacc: Log is configured to *not* fsync() on all Append() calls I0312 20:12:09.305706 6270 tablet_server_main.cc:84] Starting tablet server... I0312 20:12:09.424468 6298 tablet_bootstrap.cc:436] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 1183c4bf4e494b75a57827848998cacc: Bootstrap replayed 1/3 log segments. Stats: ops{read=7 overwritten=0 applied=5 ignored=5} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=3. Pending: 2 replicates I0312 20:12:09.430352 6298 log.cc:644] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 1183c4bf4e494b75a57827848998cacc: Max segment size reached. Starting new segment allocation I0312 20:12:09.431974 6298 log.cc:571] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 1183c4bf4e494b75a57827848998cacc: Rolled over to a new log segment at /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1520885401735786-212/raft_consensus-itest-cluster/ts-1/wal/wals/9a02c73fbfaa40d5bf986d34f5b785f9/wal-000000002 I0312 20:12:09.469027 6298 tablet_bootstrap.cc:436] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 1183c4bf4e494b75a57827848998cacc: Bootstrap replayed 2/3 log segments. Stats: ops{read=14 overwritten=0 applied=11 ignored=11} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=3. Pending: 3 replicates I0312 20:12:09.475750 6298 log.cc:644] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 1183c4bf4e494b75a57827848998cacc: Max segment size reached. Starting new segment allocation I0312 20:12:09.477021 6298 log.cc:571] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 1183c4bf4e494b75a57827848998cacc: Rolled over to a new log segment at /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1520885401735786-212/raft_consensus-itest-cluster/ts-1/wal/wals/9a02c73fbfaa40d5bf986d34f5b785f9/wal-000000003 I0312 20:12:09.498564 6298 tablet_bootstrap.cc:436] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 1183c4bf4e494b75a57827848998cacc: Bootstrap replayed 3/3 log segments. Stats: ops{read=17 overwritten=0 applied=17 ignored=12} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=3. Pending: 0 replicates I0312 20:12:09.499696 6298 tablet_bootstrap.cc:436] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 1183c4bf4e494b75a57827848998cacc: Bootstrap complete. I0312 20:12:09.500803 6298 ts_tablet_manager.cc:1035] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 1183c4bf4e494b75a57827848998cacc: Time spent bootstrapping tablet: real 0.205s user 0.060s sys 0.068s I0312 20:12:09.523478 6298 raft_consensus.cc:304] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 1183c4bf4e494b75a57827848998cacc [term 1 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: 39 OBSOLETE_local: false peers { permanent_uuid: "ac1592be0a6240378933768820f57ed3" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 59968 } } peers { permanent_uuid: "1183c4bf4e494b75a57827848998cacc" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 47310 } } peers { permanent_uuid: "0c2c01e0fc8840bdb67647734ba4c3f0" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 60846 } attrs { } } I0312 20:12:09.526010 6298 raft_consensus.cc:605] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 1183c4bf4e494b75a57827848998cacc [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1183c4bf4e494b75a57827848998cacc, State: Initialized, Role: FOLLOWER I0312 20:12:09.526827 6298 consensus_queue.cc:219] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 1183c4bf4e494b75a57827848998cacc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 39, Last appended: 1.39, Last appended by leader: 39, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 39 OBSOLETE_local: false peers { permanent_uuid: "ac1592be0a6240378933768820f57ed3" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 59968 } } peers { permanent_uuid: "1183c4bf4e494b75a57827848998cacc" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 47310 } } peers { permanent_uuid: "0c2c01e0fc8840bdb67647734ba4c3f0" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 60846 } attrs { } } I0312 20:12:09.529839 6298 ts_tablet_manager.cc:1063] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 1183c4bf4e494b75a57827848998cacc: Time spent starting tablet: real 0.029s user 0.016s sys 0.012s I0312 20:12:09.575094 6270 rpc_server.cc:201] RPC server started. Bound to: 127.0.53.2:47310 I0312 20:12:09.575994 6270 webserver.cc:173] Starting webserver on 127.0.53.2:50074 I0312 20:12:09.576099 6270 webserver.cc:184] Document root disabled I0312 20:12:09.577507 6270 webserver.cc:311] Webserver started. Bound to: http://127.0.53.2:50074/ I0312 20:12:09.578169 6270 server_base.cc:607] Dumped server information to /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1520885401735786-212/raft_consensus-itest-cluster/ts-1/data/info.pb I0312 20:12:09.586043 6270 tablet_server_main.cc:87] Tablet server successfully started. I0312 20:12:09.586812 212 external_mini_cluster.cc:867] Started /tmp/dist-test-task3bwPeE/build/tsan/bin/kudu-tserver as pid 6270 I0312 20:12:09.587028 6406 process_memory.cc:182] Process hard memory limit is 11.777631 GB I0312 20:12:09.587196 6406 process_memory.cc:184] Process soft memory limit is 9.422105 GB I0312 20:12:09.587301 6406 process_memory.cc:187] Process memory pressure threshold is 7.066578 GB I0312 20:12:09.587462 6406 maintenance_manager.cc:302] P 1183c4bf4e494b75a57827848998cacc: Scheduling UndoDeltaBlockGCOp(9a02c73fbfaa40d5bf986d34f5b785f9): 506 bytes on disk I0312 20:12:09.588083 212 external_mini_cluster.cc:805] Running /tmp/dist-test-task3bwPeE/build/tsan/bin/kudu-tserver /tmp/dist-test-task3bwPeE/build/tsan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1520885401735786-212/raft_consensus-itest-cluster/ts-2/wal --fs_data_dirs=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1520885401735786-212/raft_consensus-itest-cluster/ts-2/data --block_manager=log --rpc_bind_addresses=127.0.53.3:60846 --local_ip_for_outbound_sockets=127.0.53.3 --webserver_port=44251 --webserver_interface=127.0.53.3 --tserver_master_addrs=127.0.53.62:53264 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1520885401735786-212/raft_consensus-itest-cluster/ts-2/logs --server_dump_info_path=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1520885401735786-212/raft_consensus-itest-cluster/ts-2/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=false --log_compression_codec=none --log_cache_size_limit_mb=1 --log_segment_size_mb=1 --log_async_preallocate_segments=false --log_min_segments_to_retain=1 --log_max_segments_to_retain=3 --maintenance_manager_polling_interval_ms=100 --log_target_replay_size_mb=1 --max_cell_size_bytes=1000000 with env {} I0312 20:12:09.589037 6287 maintenance_manager.cc:492] P 1183c4bf4e494b75a57827848998cacc: UndoDeltaBlockGCOp(9a02c73fbfaa40d5bf986d34f5b785f9) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":221,"lbm_reads_lt_1ms":4} I0312 20:12:09.604513 6405 heartbeater.cc:346] Connected to a master server at 127.0.53.62:53264 I0312 20:12:09.604784 6405 heartbeater.cc:426] Registering TS with master... I0312 20:12:09.605283 6405 heartbeater.cc:475] Master 127.0.53.62:53264 requested a full tablet report, sending... I0312 20:12:09.607316 6236 ts_manager.cc:81] Registered new tserver with Master: 1183c4bf4e494b75a57827848998cacc (127.0.53.2:47310) I0312 20:12:09.609637 6236 master_service.cc:243] Signed X509 certificate for tserver {username='slave'} at 127.0.53.2:48632 WARNING: Logging before InitGoogleLogging() is written to STDERR W0312 20:12:09.718824 6409 flags.cc:412] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0312 20:12:09.719159 6409 flags.cc:412] Enabled unsafe flag: --max_cell_size_bytes=1000000 W0312 20:12:09.719290 6409 flags.cc:412] Enabled unsafe flag: --never_fsync=true W0312 20:12:09.733695 6409 flags.cc:412] Enabled experimental flag: --log_compression_codec=none W0312 20:12:09.733836 6409 flags.cc:412] Enabled experimental flag: --log_max_segments_to_retain=3 W0312 20:12:09.734002 6409 flags.cc:412] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false W0312 20:12:09.734150 6409 flags.cc:412] Enabled experimental flag: --ipki_server_key_size=1024 W0312 20:12:09.734432 6409 flags.cc:412] Enabled experimental flag: --log_target_replay_size_mb=1 W0312 20:12:09.734565 6409 flags.cc:412] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.53.3 I0312 20:12:09.763296 6409 tablet_server_main.cc:72] Tablet server non-default flags: --log_compression_codec=none --log_max_segments_to_retain=3 --log_cache_size_limit_mb=1 --log_async_preallocate_segments=false --log_segment_size_mb=1 --raft_prepare_replacement_before_eviction=false --fs_data_dirs=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1520885401735786-212/raft_consensus-itest-cluster/ts-2/data --fs_wal_dir=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1520885401735786-212/raft_consensus-itest-cluster/ts-2/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.0.53.3:60846 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1520885401735786-212/raft_consensus-itest-cluster/ts-2/data/info.pb --webserver_interface=127.0.53.3 --webserver_port=44251 --max_cell_size_bytes=1000000 --tserver_master_addrs=127.0.53.62:53264 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --log_target_replay_size_mb=1 --maintenance_manager_polling_interval_ms=100 --enable_minidumps=false --local_ip_for_outbound_sockets=127.0.53.3 --log_dir=/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1520885401735786-212/raft_consensus-itest-cluster/ts-2/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.7.0-SNAPSHOT revision 13e10c63c46de30ac85f34b543cdd6fb44881f61 build type FASTDEBUG built by None at 12 Mar 2018 20:06:40 UTC on jenkins-slave-vn0g build id 12388 TSAN enabled I0312 20:12:09.778492 6409 tablet_server_main.cc:79] Initializing tablet server... I0312 20:12:09.780894 6409 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 1131776us I0312 20:12:09.781591 6409 fs_manager.cc:260] Metadata directory not provided I0312 20:12:09.781790 6409 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0312 20:12:09.786725 6409 fs_manager.cc:406] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s I0312 20:12:09.787008 6409 env_posix.cc:1620] Not raising this process' open files per process limit of 65536; it is already as high as it can go I0312 20:12:09.787263 6409 file_cache.cc:470] Constructed file cache lbm with capacity 26214 I0312 20:12:09.793975 6409 fs_manager.cc:418] Time spent opening block manager: real 0.002s user 0.000s sys 0.004s I0312 20:12:09.794095 6409 fs_manager.cc:429] Opened local filesystem: /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1520885401735786-212/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1520885401735786-212/raft_consensus-itest-cluster/ts-2/wal uuid: "0c2c01e0fc8840bdb67647734ba4c3f0" format_stamp: "Formatted at 2018-03-12 20:12:02 on dist-test-slave-dist-test-slave-f27p" I0312 20:12:09.795603 6409 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1520885401735786-212/raft_consensus-itest-cluster/ts-2/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0312 20:12:10.015209 6409 env_posix.cc:1620] Not raising this process' running threads per effective uid limit of 60284; it is already as high as it can go I0312 20:12:10.016546 6409 ts_tablet_manager.cc:337] Loading tablet metadata (0/1 complete) I0312 20:12:10.016685 6409 ts_tablet_manager.cc:995] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 0c2c01e0fc8840bdb67647734ba4c3f0: Loading tablet metadata W0312 20:12:10.020817 6409 ts_tablet_manager.cc:1307] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 0c2c01e0fc8840bdb67647734ba4c3f0: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_COPYING I0312 20:12:10.024816 6409 ts_tablet_manager.cc:1360] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 0c2c01e0fc8840bdb67647734ba4c3f0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED I0312 20:12:10.035058 6409 ts_tablet_manager.cc:1373] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 0c2c01e0fc8840bdb67647734ba4c3f0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2 I0312 20:12:10.035625 6409 ts_tablet_manager.cc:1163] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 0c2c01e0fc8840bdb67647734ba4c3f0: Registered tablet (data state: TABLET_DATA_TOMBSTONED) I0312 20:12:10.035794 6409 ts_tablet_manager.cc:349] Loaded tablet metadata (0 live tablets) I0312 20:12:10.036916 6409 tablet_server_main.cc:84] Starting tablet server... I0312 20:12:10.142508 6409 rpc_server.cc:201] RPC server started. Bound to: 127.0.53.3:60846 I0312 20:12:10.143411 6409 webserver.cc:173] Starting webserver on 127.0.53.3:44251 I0312 20:12:10.143528 6409 webserver.cc:184] Document root disabled I0312 20:12:10.144923 6409 webserver.cc:311] Webserver started. Bound to: http://127.0.53.3:44251/ I0312 20:12:10.145637 6409 server_base.cc:607] Dumped server information to /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1520885401735786-212/raft_consensus-itest-cluster/ts-2/data/info.pb I0312 20:12:10.148680 6531 process_memory.cc:182] Process hard memory limit is 11.777631 GB I0312 20:12:10.151522 6531 process_memory.cc:184] Process soft memory limit is 9.422105 GB I0312 20:12:10.151660 6531 process_memory.cc:187] Process memory pressure threshold is 7.066578 GB I0312 20:12:10.152752 212 external_mini_cluster.cc:867] Started /tmp/dist-test-task3bwPeE/build/tsan/bin/kudu-tserver as pid 6409 I0312 20:12:10.153244 212 test_util.cc:192] Using random seed: -1044066044 I0312 20:12:10.169513 6530 heartbeater.cc:346] Connected to a master server at 127.0.53.62:53264 I0312 20:12:10.169782 6530 heartbeater.cc:426] Registering TS with master... I0312 20:12:10.170327 6530 heartbeater.cc:475] Master 127.0.53.62:53264 requested a full tablet report, sending... I0312 20:12:10.172626 6236 ts_manager.cc:81] Registered new tserver with Master: 0c2c01e0fc8840bdb67647734ba4c3f0 (127.0.53.3:60846) I0312 20:12:10.174994 6236 master_service.cc:243] Signed X509 certificate for tserver {username='slave'} at 127.0.53.3:50578 I0312 20:12:10.152050 6409 tablet_server_main.cc:87] Tablet server successfully started. I0312 20:12:10.199447 212 test_workload.cc:306] TestWorkload: Skipping table creation because table TestTable already exists W0312 20:12:10.305086 6537 meta_cache.cc:218] Tablet 9a02c73fbfaa40d5bf986d34f5b785f9: Replica ac1592be0a6240378933768820f57ed3 (127.0.53.1:59968) has failed: Network error: Client connection negotiation failed: client connection to 127.0.53.1:59968: connect: Connection refused (error 111) I0312 20:12:10.612644 6405 heartbeater.cc:467] Master 127.0.53.62:53264 was elected leader, sending a full tablet report... I0312 20:12:10.745353 6547 raft_consensus.cc:436] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 1183c4bf4e494b75a57827848998cacc [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0312 20:12:10.745766 6547 raft_consensus.cc:2700] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 1183c4bf4e494b75a57827848998cacc: Snoozing failure detection for 2.230s (starting election) I0312 20:12:10.746112 6547 raft_consensus.cc:458] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 1183c4bf4e494b75a57827848998cacc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 39 OBSOLETE_local: false peers { permanent_uuid: "ac1592be0a6240378933768820f57ed3" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 59968 } } peers { permanent_uuid: "1183c4bf4e494b75a57827848998cacc" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 47310 } } peers { permanent_uuid: "0c2c01e0fc8840bdb67647734ba4c3f0" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 60846 } attrs { } } I0312 20:12:10.747488 6547 leader_election.cc:231] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 1183c4bf4e494b75a57827848998cacc [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer ac1592be0a6240378933768820f57ed3 I0312 20:12:10.747987 6547 leader_election.cc:231] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 1183c4bf4e494b75a57827848998cacc [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 0c2c01e0fc8840bdb67647734ba4c3f0 W0312 20:12:10.751297 6294 leader_election.cc:287] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 1183c4bf4e494b75a57827848998cacc [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ac1592be0a6240378933768820f57ed3: Network error: Client connection negotiation failed: client connection to 127.0.53.1:59968: connect: Connection refused (error 111) I0312 20:12:10.759882 6487 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "9a02c73fbfaa40d5bf986d34f5b785f9" candidate_uuid: "1183c4bf4e494b75a57827848998cacc" candidate_term: 2 candidate_status { last_received { term: 1 index: 39 } } ignore_live_leader: false dest_uuid: "0c2c01e0fc8840bdb67647734ba4c3f0" is_pre_election: true I0312 20:12:10.766031 6487 raft_consensus.cc:1562] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 0c2c01e0fc8840bdb67647734ba4c3f0 [term 1 FOLLOWER]: voting while tombstoned based on last-logged opid 1.2 I0312 20:12:10.767457 6487 raft_consensus.cc:2215] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 0c2c01e0fc8840bdb67647734ba4c3f0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1183c4bf4e494b75a57827848998cacc in term 1. I0312 20:12:10.768183 6294 leader_election.cc:363] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 1183c4bf4e494b75a57827848998cacc [CANDIDATE]: Term 2 pre-election: Vote granted by peer 0c2c01e0fc8840bdb67647734ba4c3f0 I0312 20:12:10.768335 6294 leader_election.cc:258] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 1183c4bf4e494b75a57827848998cacc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. I0312 20:12:10.768707 6547 raft_consensus.cc:2700] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 1183c4bf4e494b75a57827848998cacc: Snoozing failure detection for 1.904s (election complete) I0312 20:12:10.768882 6547 raft_consensus.cc:2507] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 1183c4bf4e494b75a57827848998cacc [term 1 FOLLOWER]: Leader pre-election won for term 2 I0312 20:12:10.769026 6547 raft_consensus.cc:436] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 1183c4bf4e494b75a57827848998cacc [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0312 20:12:10.769170 6547 raft_consensus.cc:2700] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 1183c4bf4e494b75a57827848998cacc: Snoozing failure detection for 2.115s (starting election) I0312 20:12:10.769307 6547 raft_consensus.cc:2752] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 1183c4bf4e494b75a57827848998cacc [term 1 FOLLOWER]: Advancing to term 2 I0312 20:12:10.772610 6547 raft_consensus.cc:458] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 1183c4bf4e494b75a57827848998cacc [term 2 FOLLOWER]: Starting leader election with config: opid_index: 39 OBSOLETE_local: false peers { permanent_uuid: "ac1592be0a6240378933768820f57ed3" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 59968 } } peers { permanent_uuid: "1183c4bf4e494b75a57827848998cacc" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 47310 } } peers { permanent_uuid: "0c2c01e0fc8840bdb67647734ba4c3f0" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 60846 } attrs { } } I0312 20:12:10.773350 6547 leader_election.cc:231] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 1183c4bf4e494b75a57827848998cacc [CANDIDATE]: Term 2 election: Requesting vote from peer ac1592be0a6240378933768820f57ed3 I0312 20:12:10.773627 6547 leader_election.cc:231] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 1183c4bf4e494b75a57827848998cacc [CANDIDATE]: Term 2 election: Requesting vote from peer 0c2c01e0fc8840bdb67647734ba4c3f0 W0312 20:12:10.774287 6294 leader_election.cc:287] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 1183c4bf4e494b75a57827848998cacc [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer ac1592be0a6240378933768820f57ed3: Network error: Client connection negotiation failed: client connection to 127.0.53.1:59968: connect: Connection refused (error 111) I0312 20:12:10.775137 6487 tablet_service.cc:993] Received RequestConsensusVote() RPC: tablet_id: "9a02c73fbfaa40d5bf986d34f5b785f9" candidate_uuid: "1183c4bf4e494b75a57827848998cacc" candidate_term: 2 candidate_status { last_received { term: 1 index: 39 } } ignore_live_leader: false dest_uuid: "0c2c01e0fc8840bdb67647734ba4c3f0" I0312 20:12:10.775482 6487 raft_consensus.cc:1562] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 0c2c01e0fc8840bdb67647734ba4c3f0 [term 1 FOLLOWER]: voting while tombstoned based on last-logged opid 1.2 I0312 20:12:10.775676 6487 raft_consensus.cc:2752] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 0c2c01e0fc8840bdb67647734ba4c3f0 [term 1 FOLLOWER]: Advancing to term 2 I0312 20:12:10.778852 6487 raft_consensus.cc:2215] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 0c2c01e0fc8840bdb67647734ba4c3f0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1183c4bf4e494b75a57827848998cacc in term 2. I0312 20:12:10.779544 6294 leader_election.cc:363] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 1183c4bf4e494b75a57827848998cacc [CANDIDATE]: Term 2 election: Vote granted by peer 0c2c01e0fc8840bdb67647734ba4c3f0 I0312 20:12:10.779690 6294 leader_election.cc:258] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 1183c4bf4e494b75a57827848998cacc [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. I0312 20:12:10.780150 6547 raft_consensus.cc:2700] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 1183c4bf4e494b75a57827848998cacc: Snoozing failure detection for 2.150s (election complete) I0312 20:12:10.780375 6547 raft_consensus.cc:2507] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 1183c4bf4e494b75a57827848998cacc [term 2 FOLLOWER]: Leader election won for term 2 I0312 20:12:10.781513 6547 raft_consensus.cc:572] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 1183c4bf4e494b75a57827848998cacc [term 2 LEADER]: Becoming Leader. State: Replica: 1183c4bf4e494b75a57827848998cacc, State: Running, Role: LEADER I0312 20:12:10.781862 6547 consensus_queue.cc:196] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 1183c4bf4e494b75a57827848998cacc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 39, Committed index: 39, Last appended: 1.39, Last appended by leader: 39, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 39 OBSOLETE_local: false peers { permanent_uuid: "ac1592be0a6240378933768820f57ed3" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 59968 } } peers { permanent_uuid: "1183c4bf4e494b75a57827848998cacc" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 47310 } } peers { permanent_uuid: "0c2c01e0fc8840bdb67647734ba4c3f0" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 60846 } attrs { } } I0312 20:12:10.785930 6236 catalog_manager.cc:3592] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 1183c4bf4e494b75a57827848998cacc reported cstate change: term changed from 1 to 2. New cstate: current_term: 2 leader_uuid: "1183c4bf4e494b75a57827848998cacc" committed_config { opid_index: 39 OBSOLETE_local: false peers { permanent_uuid: "ac1592be0a6240378933768820f57ed3" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 59968 } } peers { permanent_uuid: "1183c4bf4e494b75a57827848998cacc" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 47310 } } peers { permanent_uuid: "0c2c01e0fc8840bdb67647734ba4c3f0" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 60846 } attrs { } } } W0312 20:12:10.811326 6294 consensus_peers.cc:436] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 1183c4bf4e494b75a57827848998cacc -> Peer ac1592be0a6240378933768820f57ed3 (127.0.53.1:59968): Couldn't send request to peer ac1592be0a6240378933768820f57ed3 for tablet 9a02c73fbfaa40d5bf986d34f5b785f9. Status: Network error: Client connection negotiation failed: client connection to 127.0.53.1:59968: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 1 times. W0312 20:12:10.812300 6294 consensus_peers.cc:436] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 1183c4bf4e494b75a57827848998cacc -> Peer 0c2c01e0fc8840bdb67647734ba4c3f0 (127.0.53.3:60846): Couldn't send request to peer 0c2c01e0fc8840bdb67647734ba4c3f0 for tablet 9a02c73fbfaa40d5bf986d34f5b785f9. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: INITIALIZED. Retrying in the next heartbeat period. Already tried 1 times. I0312 20:12:11.178117 6530 heartbeater.cc:467] Master 127.0.53.62:53264 was elected leader, sending a full tablet report... W0312 20:12:11.213409 6294 consensus_peers.cc:436] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 1183c4bf4e494b75a57827848998cacc -> Peer ac1592be0a6240378933768820f57ed3 (127.0.53.1:59968): Couldn't send request to peer ac1592be0a6240378933768820f57ed3 for tablet 9a02c73fbfaa40d5bf986d34f5b785f9. Status: Network error: Client connection negotiation failed: client connection to 127.0.53.1:59968: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 2 times. I0312 20:12:11.429903 6557 tablet_replica.cc:262] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 0c2c01e0fc8840bdb67647734ba4c3f0: stopping tablet replica I0312 20:12:11.430109 6557 raft_consensus.cc:2012] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 0c2c01e0fc8840bdb67647734ba4c3f0 [term 2 FOLLOWER]: Raft consensus shutting down. I0312 20:12:11.430271 6557 raft_consensus.cc:2039] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 0c2c01e0fc8840bdb67647734ba4c3f0 [term 2 FOLLOWER]: Raft consensus is shut down! I0312 20:12:11.430506 6557 ts_tablet_manager.cc:662] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 0c2c01e0fc8840bdb67647734ba4c3f0: Initiating tablet copy from peer 1183c4bf4e494b75a57827848998cacc (127.0.53.2:47310) I0312 20:12:11.430761 6557 tablet_copy_client.cc:237] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 0c2c01e0fc8840bdb67647734ba4c3f0: tablet copy: Beginning tablet copy session from remote peer at address 127.0.53.2:47310 I0312 20:12:11.439975 6382 tablet_copy_service.cc:133] P 1183c4bf4e494b75a57827848998cacc: Received BeginTabletCopySession request for tablet 9a02c73fbfaa40d5bf986d34f5b785f9 from peer 0c2c01e0fc8840bdb67647734ba4c3f0 ({username='slave'} at 127.0.53.3:42871) I0312 20:12:11.440243 6382 tablet_copy_service.cc:154] P 1183c4bf4e494b75a57827848998cacc: Beginning new tablet copy session on tablet 9a02c73fbfaa40d5bf986d34f5b785f9 from peer 0c2c01e0fc8840bdb67647734ba4c3f0 at {username='slave'} at 127.0.53.3:42871: session id = 0c2c01e0fc8840bdb67647734ba4c3f0-9a02c73fbfaa40d5bf986d34f5b785f9 I0312 20:12:11.443764 6382 tablet_copy_source_session.cc:202] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 1183c4bf4e494b75a57827848998cacc: Tablet Copy: opened 5 blocks and 3 log segments I0312 20:12:11.446063 6557 ts_tablet_manager.cc:1360] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 0c2c01e0fc8840bdb67647734ba4c3f0: Deleting tablet data with delete state TABLET_DATA_COPYING I0312 20:12:11.451742 6557 ts_tablet_manager.cc:1373] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 0c2c01e0fc8840bdb67647734ba4c3f0: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.2 I0312 20:12:11.452005 6557 data_dirs.cc:931] Could only allocate 1 dirs of requested 3 for tablet 9a02c73fbfaa40d5bf986d34f5b785f9. 1 dirs total, 0 dirs full, 0 dirs failed I0312 20:12:11.454638 6557 ts_tablet_manager.cc:1163] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 0c2c01e0fc8840bdb67647734ba4c3f0: Registered tablet (data state: TABLET_DATA_COPYING) I0312 20:12:11.454958 6557 tablet_copy_client.cc:529] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 0c2c01e0fc8840bdb67647734ba4c3f0: tablet copy: Starting download of 5 data blocks... I0312 20:12:11.469159 6557 tablet_copy_client.cc:492] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 0c2c01e0fc8840bdb67647734ba4c3f0: tablet copy: Starting download of 3 WAL segments... I0312 20:12:11.534859 6557 tablet_copy_client.cc:397] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 0c2c01e0fc8840bdb67647734ba4c3f0: tablet copy: Tablet Copy complete. Replacing tablet superblock. I0312 20:12:11.538219 6557 ts_tablet_manager.cc:1018] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 0c2c01e0fc8840bdb67647734ba4c3f0: Bootstrapping tablet I0312 20:12:11.538542 6557 tablet_bootstrap.cc:436] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 0c2c01e0fc8840bdb67647734ba4c3f0: Bootstrap starting. I0312 20:12:11.548714 6557 log.cc:520] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 0c2c01e0fc8840bdb67647734ba4c3f0: Log is configured to *not* fsync() on all Append() calls I0312 20:12:11.575645 6557 tablet_bootstrap.cc:436] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 0c2c01e0fc8840bdb67647734ba4c3f0: Bootstrap replayed 1/3 log segments. Stats: ops{read=7 overwritten=0 applied=5 ignored=5} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 2 replicates I0312 20:12:11.578219 6557 log.cc:644] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 0c2c01e0fc8840bdb67647734ba4c3f0: Max segment size reached. Starting new segment allocation I0312 20:12:11.579154 6557 log.cc:571] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 0c2c01e0fc8840bdb67647734ba4c3f0: Rolled over to a new log segment at /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1520885401735786-212/raft_consensus-itest-cluster/ts-2/wal/wals/9a02c73fbfaa40d5bf986d34f5b785f9/wal-000000002 I0312 20:12:11.598379 6557 tablet_bootstrap.cc:436] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 0c2c01e0fc8840bdb67647734ba4c3f0: Bootstrap replayed 2/3 log segments. Stats: ops{read=14 overwritten=0 applied=11 ignored=11} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 3 replicates I0312 20:12:11.600945 6557 log.cc:644] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 0c2c01e0fc8840bdb67647734ba4c3f0: Max segment size reached. Starting new segment allocation I0312 20:12:11.601794 6557 log.cc:571] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 0c2c01e0fc8840bdb67647734ba4c3f0: Rolled over to a new log segment at /tmp/dist-test-task3bwPeE/test-tmp/raft_consensus_election-itest.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1520885401735786-212/raft_consensus-itest-cluster/ts-2/wal/wals/9a02c73fbfaa40d5bf986d34f5b785f9/wal-000000003 I0312 20:12:11.613919 6557 tablet_bootstrap.cc:436] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 0c2c01e0fc8840bdb67647734ba4c3f0: Bootstrap replayed 3/3 log segments. Stats: ops{read=22 overwritten=0 applied=17 ignored=12} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 5 replicates I0312 20:12:11.614549 6557 tablet_bootstrap.cc:436] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 0c2c01e0fc8840bdb67647734ba4c3f0: Bootstrap complete. I0312 20:12:11.615245 6557 ts_tablet_manager.cc:1035] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 0c2c01e0fc8840bdb67647734ba4c3f0: Time spent bootstrapping tablet: real 0.077s user 0.052s sys 0.024s I0312 20:12:11.617749 6557 raft_consensus.cc:304] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 0c2c01e0fc8840bdb67647734ba4c3f0 [term 2 FOLLOWER]: Replica starting. Triggering 5 pending transactions. Active config: opid_index: 39 OBSOLETE_local: false peers { permanent_uuid: "ac1592be0a6240378933768820f57ed3" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 59968 } } peers { permanent_uuid: "1183c4bf4e494b75a57827848998cacc" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 47310 } } peers { permanent_uuid: "0c2c01e0fc8840bdb67647734ba4c3f0" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 60846 } attrs { } } I0312 20:12:11.622337 6557 raft_consensus.cc:605] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 0c2c01e0fc8840bdb67647734ba4c3f0 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c2c01e0fc8840bdb67647734ba4c3f0, State: Initialized, Role: FOLLOWER I0312 20:12:11.622774 6557 consensus_queue.cc:219] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 0c2c01e0fc8840bdb67647734ba4c3f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 39, Last appended: 2.44, Last appended by leader: 44, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 39 OBSOLETE_local: false peers { permanent_uuid: "ac1592be0a6240378933768820f57ed3" member_type: VOTER last_known_addr { host: "127.0.53.1" port: 59968 } } peers { permanent_uuid: "1183c4bf4e494b75a57827848998cacc" member_type: VOTER last_known_addr { host: "127.0.53.2" port: 47310 } } peers { permanent_uuid: "0c2c01e0fc8840bdb67647734ba4c3f0" member_type: VOTER last_known_addr { host: "127.0.53.3" port: 60846 } attrs { } } I0312 20:12:11.625175 6557 ts_tablet_manager.cc:1063] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 0c2c01e0fc8840bdb67647734ba4c3f0: Time spent starting tablet: real 0.010s user 0.012s sys 0.000s I0312 20:12:11.626366 6382 tablet_copy_service.cc:323] P 1183c4bf4e494b75a57827848998cacc: Request end of tablet copy session 0c2c01e0fc8840bdb67647734ba4c3f0-9a02c73fbfaa40d5bf986d34f5b785f9 received from {username='slave'} at 127.0.53.3:42871 I0312 20:12:11.626619 6382 tablet_copy_service.cc:411] P 1183c4bf4e494b75a57827848998cacc: Ending tablet copy session 0c2c01e0fc8840bdb67647734ba4c3f0-9a02c73fbfaa40d5bf986d34f5b785f9 on tablet 9a02c73fbfaa40d5bf986d34f5b785f9 with peer 0c2c01e0fc8840bdb67647734ba4c3f0 I0312 20:12:11.654436 6531 maintenance_manager.cc:302] P 0c2c01e0fc8840bdb67647734ba4c3f0: Scheduling UndoDeltaBlockGCOp(9a02c73fbfaa40d5bf986d34f5b785f9): 506 bytes on disk I0312 20:12:11.655712 6415 maintenance_manager.cc:492] P 0c2c01e0fc8840bdb67647734ba4c3f0: UndoDeltaBlockGCOp(9a02c73fbfaa40d5bf986d34f5b785f9) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":159,"lbm_reads_lt_1ms":4} W0312 20:12:11.726056 6294 consensus_peers.cc:436] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 1183c4bf4e494b75a57827848998cacc -> Peer ac1592be0a6240378933768820f57ed3 (127.0.53.1:59968): Couldn't send request to peer ac1592be0a6240378933768820f57ed3 for tablet 9a02c73fbfaa40d5bf986d34f5b785f9. Status: Network error: Client connection negotiation failed: client connection to 127.0.53.1:59968: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 3 times. I0312 20:12:11.870477 6487 raft_consensus.cc:1003] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 0c2c01e0fc8840bdb67647734ba4c3f0 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 1.39->[2.40-2.44] Dedup: 2.44->[] I0312 20:12:11.883576 6565 rpcz_store.cc:265] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:57441 (ReqId={client: f43607971b344bf7a3cc849720a25c7a, seq_no=3, attempt_no=26}) took 1078ms. Request Metrics: {"child_traces":[["txn",{"apply.queue_time_us":1062,"cfile_cache_miss":2,"cfile_cache_miss_bytes":4127,"cfile_init":1,"lbm_read_time_us":347,"lbm_reads_lt_1ms":6,"mutex_wait_us":27,"num_ops":50,"prepare.queue_time_us":806,"prepare.run_cpu_time_us":3140,"prepare.run_wall_time_us":3730,"raft.queue_time_us":138,"raft.run_cpu_time_us":923,"raft.run_wall_time_us":1006,"replication_time_us":1064399,"spinlock_wait_cycles":456576,"thread_start_us":1474,"threads_started":2}]]} I0312 20:12:11.907939 6567 rpcz_store.cc:265] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:57441 (ReqId={client: f43607971b344bf7a3cc849720a25c7a, seq_no=1, attempt_no=27}) took 1079ms. Request Metrics: {"child_traces":[["txn",{"apply.queue_time_us":7647,"cfile_cache_hit":2,"cfile_cache_hit_bytes":4127,"mutex_wait_us":37,"num_ops":50,"prepare.queue_time_us":59,"prepare.run_cpu_time_us":2673,"prepare.run_wall_time_us":2779,"raft.queue_time_us":102,"raft.run_cpu_time_us":116,"raft.run_wall_time_us":125,"replication_time_us":1043191,"spinlock_wait_cycles":240512,"thread_start_us":7436,"threads_started":1}]]} I0312 20:12:11.916473 6565 rpcz_store.cc:265] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:57441 (ReqId={client: f43607971b344bf7a3cc849720a25c7a, seq_no=2, attempt_no=28}) took 1038ms. Request Metrics: {"child_traces":[["txn",{"apply.queue_time_us":11011,"mutex_wait_us":14,"num_ops":50,"prepare.queue_time_us":62,"prepare.run_cpu_time_us":2721,"prepare.run_wall_time_us":2880,"raft.queue_time_us":136,"raft.run_cpu_time_us":123,"raft.run_wall_time_us":132,"replication_time_us":1001072,"spinlock_wait_cycles":332544,"thread_start_us":22059,"threads_started":1}]]} W0312 20:12:12.162968 6294 consensus_peers.cc:436] T 9a02c73fbfaa40d5bf986d34f5b785f9 P 1183c4bf4e494b75a57827848998cacc -> Peer ac1592be0a6240378933768820f57ed3 (127.0.53.1:59968): Couldn't send request to peer ac1592be0a6240378933768820f57ed3 for tablet 9a02c73fbfaa40d5bf986d34f5b785f9. Status: Network error: Client connection negotiation failed: client connection to 127.0.53.1:59968: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 4 times. I0312 20:12:12.282200 212 external_mini_cluster.cc:997] Killing /tmp/dist-test-task3bwPeE/build/tsan/bin/kudu-tserver with pid 6270 W0312 20:12:12.310430 6420 connection.cc:513] client connection to 127.0.53.2:47310 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0312 20:12:12.310452 6217 connection.cc:513] server connection from 127.0.53.2:48632 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) I0312 20:12:12.310693 212 external_mini_cluster.cc:997] Killing /tmp/dist-test-task3bwPeE/build/tsan/bin/kudu-tserver with pid 6409 W0312 20:12:12.340710 6219 connection.cc:513] server connection from 127.0.53.3:50578 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) I0312 20:12:12.342291 212 external_mini_cluster.cc:997] Killing /tmp/dist-test-task3bwPeE/build/tsan/bin/kudu-master with pid 6208 [ OK ] RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy (11928 ms) [----------] 10 tests from RaftConsensusElectionITest (130570 ms total) [----------] Global test environment tear-down [==========] 10 tests from 1 test case ran. (130570 ms total) [ PASSED ] 10 tests. ThreadSanitizer: reported 2 warnings