[ RUN ] RaftConsensusITest.TestCatchupAfterOpsEvicted I0221 01:26:15.985647 8186 test_util.cc:173] Using random seed: -121166356 I0221 01:26:15.986207 8186 ts_itest-base.h:91] Starting cluster with: I0221 01:26:15.986215 8186 ts_itest-base.h:92] -------------- I0221 01:26:15.986218 8186 ts_itest-base.h:93] 3 tablet servers I0221 01:26:15.986220 8186 ts_itest-base.h:94] 3 replicas per TS I0221 01:26:15.986222 8186 ts_itest-base.h:95] -------------- I0221 01:26:15.986559 8186 external_mini_cluster.cc:681] Running /home/mpercy/src/kudu/build/dynclang/bin/kudu-master kudu-master --ipki_ca_key_size=1024 --fs_wal_dir=/tmp/kudutest-1000/raft_consensus-itest.RaftConsensusITest.TestCatchupAfterOpsEvicted.1487669166730301-8186/raft_consensus-itest-cluster/master-0/data --fs_data_dirs=/tmp/kudutest-1000/raft_consensus-itest.RaftConsensusITest.TestCatchupAfterOpsEvicted.1487669166730301-8186/raft_consensus-itest-cluster/master-0/data --rpc_bind_addresses=127.0.0.1:0 --webserver_interface=localhost --webserver_port=0 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --log_redact_user_data=false --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/kudutest-1000/raft_consensus-itest.RaftConsensusITest.TestCatchupAfterOpsEvicted.1487669166730301-8186/raft_consensus-itest-cluster/master-0/logs --server_dump_info_path=/tmp/kudutest-1000/raft_consensus-itest.RaftConsensusITest.TestCatchupAfterOpsEvicted.1487669166730301-8186/raft_consensus-itest-cluster/master-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W0221 01:26:16.011499 11677 flags.cc:335] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0221 01:26:16.011602 11677 flags.cc:335] Enabled unsafe flag: --never_fsync=true W0221 01:26:16.012388 11677 flags.cc:335] Enabled experimental flag: --ipki_ca_key_size=1024 W0221 01:26:16.012398 11677 flags.cc:335] Enabled experimental flag: --ipki_server_key_size=1024 W0221 01:26:16.012425 11677 flags.cc:335] Enabled experimental flag: --log_redact_user_data=false I0221 01:26:16.013604 11677 master_main.cc:60] Master server non-default flags: --fs_data_dirs=/tmp/kudutest-1000/raft_consensus-itest.RaftConsensusITest.TestCatchupAfterOpsEvicted.1487669166730301-8186/raft_consensus-itest-cluster/master-0/data --fs_wal_dir=/tmp/kudutest-1000/raft_consensus-itest.RaftConsensusITest.TestCatchupAfterOpsEvicted.1487669166730301-8186/raft_consensus-itest-cluster/master-0/data --ipki_ca_key_size=1024 --ipki_server_key_size=1024 --rpc_bind_addresses=127.0.0.1:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/kudutest-1000/raft_consensus-itest.RaftConsensusITest.TestCatchupAfterOpsEvicted.1487669166730301-8186/raft_consensus-itest-cluster/master-0/data/info.pb --webserver_interface=localhost --webserver_port=0 --never_fsync=true --heap_profile_path=/tmp/kudu-master.11677 --unlock_experimental_flags=true --unlock_unsafe_flags=true --log_redact_user_data=false --enable_minidumps=false --log_dir=/tmp/kudutest-1000/raft_consensus-itest.RaftConsensusITest.TestCatchupAfterOpsEvicted.1487669166730301-8186/raft_consensus-itest-cluster/master-0/logs --logbuflevel=-1 --logtostderr=true Master server version: kudu 1.3.0-SNAPSHOT revision 0767cd78007dc4073f32511a6a194f925d52f614 build type DEBUG built by mpercy at 21 Feb 2017 01:21:47 PST on mpercy-T460p I0221 01:26:16.014125 11677 mem_tracker.cc:140] MemTracker: hard memory limit is 25.031622 GB I0221 01:26:16.014147 11677 mem_tracker.cc:142] MemTracker: soft memory limit is 15.018973 GB I0221 01:26:16.017243 11677 master_main.cc:67] Initializing master server... I0221 01:26:16.017479 11677 hybrid_clock.cc:177] HybridClock initialized. Resolution in nanos?: 1 Wait times tolerance adjustment: 1.0005 Current error: 564594 I0221 01:26:16.019258 11677 env_posix.cc:1316] Not raising process file limit of 65536; it is already as high as it can go I0221 01:26:16.019366 11677 file_cache.cc:401] Constructed file cache lbm with capacity 32768 W0221 01:26:16.019500 11677 fs_manager.cc:516] Unable to read directory: Not found: /tmp/kudutest-1000/raft_consensus-itest.RaftConsensusITest.TestCatchupAfterOpsEvicted.1487669166730301-8186/raft_consensus-itest-cluster/master-0/data: No such file or directory (error 2) W0221 01:26:16.019564 11677 fs_manager.cc:525] could not check and fix permissions for path: /tmp/kudutest-1000/raft_consensus-itest.RaftConsensusITest.TestCatchupAfterOpsEvicted.1487669166730301-8186/raft_consensus-itest-cluster/master-0/data: Not found: stat: No such file or directory (error 2) I0221 01:26:16.019618 11677 server_base.cc:181] Could not load existing FS layout: Not found: /tmp/kudutest-1000/raft_consensus-itest.RaftConsensusITest.TestCatchupAfterOpsEvicted.1487669166730301-8186/raft_consensus-itest-cluster/master-0/data/instance: No such file or directory (error 2) I0221 01:26:16.019632 11677 server_base.cc:182] Creating new FS layout I0221 01:26:16.021889 11677 fs_manager.cc:370] Generated new instance metadata in path /tmp/kudutest-1000/raft_consensus-itest.RaftConsensusITest.TestCatchupAfterOpsEvicted.1487669166730301-8186/raft_consensus-itest-cluster/master-0/data/instance: uuid: "e112fba79af54e179eff2a3e09683ff6" format_stamp: "Formatted at 2017-02-21 09:26:16 on mpercy-T460p" I0221 01:26:16.024890 11677 fs_manager.cc:256] Time spent opening block manager: real 0.001s user 0.004s sys 0.000s I0221 01:26:16.024917 11677 fs_manager.cc:259] Opened local filesystem: /tmp/kudutest-1000/raft_consensus-itest.RaftConsensusITest.TestCatchupAfterOpsEvicted.1487669166730301-8186/raft_consensus-itest-cluster/master-0/data uuid: "e112fba79af54e179eff2a3e09683ff6" format_stamp: "Formatted at 2017-02-21 09:26:16 on mpercy-T460p" I0221 01:26:16.227527 11677 master_main.cc:70] Starting Master server... I0221 01:26:16.232620 11677 rpc_server.cc:164] RPC server started. Bound to: 127.0.0.1:44820 I0221 01:26:16.232818 11677 webserver.cc:129] Starting webserver on localhost:0 I0221 01:26:16.232826 11677 webserver.cc:134] Document root: /home/mpercy/src/kudu/www I0221 01:26:16.233149 11677 webserver.cc:256] Webserver started. Bound to: http://127.0.0.1:35407/ I0221 01:26:16.233439 11677 server_base.cc:260] Dumped server information to /tmp/kudutest-1000/raft_consensus-itest.RaftConsensusITest.TestCatchupAfterOpsEvicted.1487669166730301-8186/raft_consensus-itest-cluster/master-0/data/info.pb I0221 01:26:16.236779 11736 tablet_bootstrap.cc:382] T 00000000000000000000000000000000 P e112fba79af54e179eff2a3e09683ff6: Bootstrap starting. I0221 01:26:16.237543 11736 tablet_bootstrap.cc:541] T 00000000000000000000000000000000 P e112fba79af54e179eff2a3e09683ff6: Time spent opening tablet: real 0.000s user 0.000s sys 0.000s I0221 01:26:16.237633 11736 tablet_bootstrap.cc:484] T 00000000000000000000000000000000 P e112fba79af54e179eff2a3e09683ff6: No blocks or log segments found. Creating new log. I0221 01:26:16.237923 11736 log.cc:375] T 00000000000000000000000000000000 P e112fba79af54e179eff2a3e09683ff6: Log is configured to *not* fsync() on all Append() calls I0221 01:26:16.238607 11736 tablet_bootstrap.cc:382] T 00000000000000000000000000000000 P e112fba79af54e179eff2a3e09683ff6: No bootstrap required, opened a new log I0221 01:26:16.239574 11736 raft_consensus.cc:288] T 00000000000000000000000000000000 P e112fba79af54e179eff2a3e09683ff6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e112fba79af54e179eff2a3e09683ff6" member_type: VOTER } I0221 01:26:16.239835 11736 raft_consensus.cc:319] T 00000000000000000000000000000000 P e112fba79af54e179eff2a3e09683ff6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0221 01:26:16.239859 11736 raft_consensus.cc:548] T 00000000000000000000000000000000 P e112fba79af54e179eff2a3e09683ff6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e112fba79af54e179eff2a3e09683ff6, State: 1, Role: FOLLOWER I0221 01:26:16.239886 11736 consensus_queue.cc:176] T 00000000000000000000000000000000 P e112fba79af54e179eff2a3e09683ff6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I0221 01:26:16.239917 11736 raft_consensus.cc:332] T 00000000000000000000000000000000 P e112fba79af54e179eff2a3e09683ff6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0221 01:26:16.239926 11736 raft_consensus.cc:411] T 00000000000000000000000000000000 P e112fba79af54e179eff2a3e09683ff6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0221 01:26:16.239944 11736 raft_consensus.cc:2162] T 00000000000000000000000000000000 P e112fba79af54e179eff2a3e09683ff6 [term 0 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.6 s I0221 01:26:16.239962 11736 raft_consensus.cc:2206] T 00000000000000000000000000000000 P e112fba79af54e179eff2a3e09683ff6 [term 0 FOLLOWER]: Advancing to term 1 I0221 01:26:16.240149 8186 external_mini_cluster.cc:717] Started /home/mpercy/src/kudu/build/dynclang/bin/kudu-master as pid 11677 I0221 01:26:16.240293 11736 raft_consensus.cc:435] T 00000000000000000000000000000000 P e112fba79af54e179eff2a3e09683ff6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e112fba79af54e179eff2a3e09683ff6" member_type: VOTER } I0221 01:26:16.240305 8186 external_mini_cluster.cc:681] Running /home/mpercy/src/kudu/build/dynclang/bin/kudu-tserver kudu-tserver --fs_wal_dir=/tmp/kudutest-1000/raft_consensus-itest.RaftConsensusITest.TestCatchupAfterOpsEvicted.1487669166730301-8186/raft_consensus-itest-cluster/ts-0/data --fs_data_dirs=/tmp/kudutest-1000/raft_consensus-itest.RaftConsensusITest.TestCatchupAfterOpsEvicted.1487669166730301-8186/raft_consensus-itest-cluster/ts-0/data --rpc_bind_addresses=127.31.250.0:0 --local_ip_for_outbound_sockets=127.31.250.0 --webserver_interface=127.31.250.0 --webserver_port=0 --tserver_master_addrs=127.0.0.1:44820 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --log_redact_user_data=false --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/kudutest-1000/raft_consensus-itest.RaftConsensusITest.TestCatchupAfterOpsEvicted.1487669166730301-8186/raft_consensus-itest-cluster/ts-0/logs --server_dump_info_path=/tmp/kudutest-1000/raft_consensus-itest.RaftConsensusITest.TestCatchupAfterOpsEvicted.1487669166730301-8186/raft_consensus-itest-cluster/ts-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --log_cache_size_limit_mb=1 --consensus_max_batch_size_bytes=500000 --log_segment_size_mb=1 --log_async_preallocate_segments=false --maintenance_manager_polling_interval_ms=100 --log_target_replay_size_mb=1 --max_cell_size_bytes=1000000 --log_compression_codec=none with env {} I0221 01:26:16.240432 11736 leader_election.cc:243] T 00000000000000000000000000000000 P e112fba79af54e179eff2a3e09683ff6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0221 01:26:16.240689 11740 raft_consensus.cc:2162] T 00000000000000000000000000000000 P e112fba79af54e179eff2a3e09683ff6 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.85 s I0221 01:26:16.240722 11740 raft_consensus.cc:1971] T 00000000000000000000000000000000 P e112fba79af54e179eff2a3e09683ff6 [term 1 FOLLOWER]: Leader election won for term 1 I0221 01:26:16.241065 11740 raft_consensus.cc:518] T 00000000000000000000000000000000 P e112fba79af54e179eff2a3e09683ff6 [term 1 LEADER]: Becoming Leader. State: Replica: e112fba79af54e179eff2a3e09683ff6, State: 1, Role: LEADER I0221 01:26:16.241183 11740 consensus_queue.cc:158] T 00000000000000000000000000000000 P e112fba79af54e179eff2a3e09683ff6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Current term: 1, Majority size: 1, State: 1, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e112fba79af54e179eff2a3e09683ff6" member_type: VOTER } I0221 01:26:16.242609 11736 sys_catalog.cc:264] T 00000000000000000000000000000000 P e112fba79af54e179eff2a3e09683ff6 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletPeer. Latest consensus state: current_term: 1 leader_uuid: "e112fba79af54e179eff2a3e09683ff6" config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e112fba79af54e179eff2a3e09683ff6" member_type: VOTER } } I0221 01:26:16.242650 11736 sys_catalog.cc:267] T 00000000000000000000000000000000 P e112fba79af54e179eff2a3e09683ff6 [sys.catalog]: This master's current role is: LEADER I0221 01:26:16.242643 11743 sys_catalog.cc:264] T 00000000000000000000000000000000 P e112fba79af54e179eff2a3e09683ff6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e112fba79af54e179eff2a3e09683ff6. Latest consensus state: current_term: 1 leader_uuid: "e112fba79af54e179eff2a3e09683ff6" config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e112fba79af54e179eff2a3e09683ff6" member_type: VOTER } } I0221 01:26:16.242666 11743 sys_catalog.cc:267] T 00000000000000000000000000000000 P e112fba79af54e179eff2a3e09683ff6 [sys.catalog]: This master's current role is: LEADER I0221 01:26:16.242682 11742 sys_catalog.cc:264] T 00000000000000000000000000000000 P e112fba79af54e179eff2a3e09683ff6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e112fba79af54e179eff2a3e09683ff6" config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e112fba79af54e179eff2a3e09683ff6" member_type: VOTER } } I0221 01:26:16.242710 11742 sys_catalog.cc:267] T 00000000000000000000000000000000 P e112fba79af54e179eff2a3e09683ff6 [sys.catalog]: This master's current role is: LEADER I0221 01:26:16.243245 11736 sys_catalog.cc:342] T 00000000000000000000000000000000 P e112fba79af54e179eff2a3e09683ff6 [sys.catalog]: configured and running, proceeding with master startup. I0221 01:26:16.243301 11744 catalog_manager.cc:772] Loading table and tablet metadata into memory... I0221 01:26:16.243351 11746 catalog_manager.cc:398] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: 1 I0221 01:26:16.243576 11677 master_main.cc:73] Master server successfully started. I0221 01:26:16.244738 11744 catalog_manager.cc:780] Loading CA info into memory... WARNING: Logging before InitGoogleLogging() is written to STDERR W0221 01:26:16.263063 11741 flags.cc:335] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0221 01:26:16.263171 11741 flags.cc:335] Enabled unsafe flag: --max_cell_size_bytes=1000000 W0221 01:26:16.263190 11741 flags.cc:335] Enabled unsafe flag: --never_fsync=true W0221 01:26:16.263833 11741 flags.cc:335] Enabled experimental flag: --log_compression_codec=none W0221 01:26:16.263852 11741 flags.cc:335] Enabled experimental flag: --ipki_server_key_size=1024 W0221 01:26:16.263891 11741 flags.cc:335] Enabled experimental flag: --log_redact_user_data=false W0221 01:26:16.263897 11741 flags.cc:335] Enabled experimental flag: --log_target_replay_size_mb=1 W0221 01:26:16.263908 11741 flags.cc:335] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.250.0 I0221 01:26:16.264966 11741 tablet_server_main.cc:56] Tablet server non-default flags: --consensus_max_batch_size_bytes=500000 --log_compression_codec=none --log_cache_size_limit_mb=1 --log_async_preallocate_segments=false --log_segment_size_mb=1 --fs_data_dirs=/tmp/kudutest-1000/raft_consensus-itest.RaftConsensusITest.TestCatchupAfterOpsEvicted.1487669166730301-8186/raft_consensus-itest-cluster/ts-0/data --fs_wal_dir=/tmp/kudutest-1000/raft_consensus-itest.RaftConsensusITest.TestCatchupAfterOpsEvicted.1487669166730301-8186/raft_consensus-itest-cluster/ts-0/data --ipki_server_key_size=1024 --rpc_bind_addresses=127.31.250.0:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/kudutest-1000/raft_consensus-itest.RaftConsensusITest.TestCatchupAfterOpsEvicted.1487669166730301-8186/raft_consensus-itest-cluster/ts-0/data/info.pb --webserver_interface=127.31.250.0 --webserver_port=0 --max_cell_size_bytes=1000000 --tserver_master_addrs=127.0.0.1:44820 --never_fsync=true --heap_profile_path=/tmp/kudu-tserver.11741 --unlock_experimental_flags=true --unlock_unsafe_flags=true --log_redact_user_data=false --log_target_replay_size_mb=1 --maintenance_manager_polling_interval_ms=100 --enable_minidumps=false --local_ip_for_outbound_sockets=127.31.250.0 --log_dir=/tmp/kudutest-1000/raft_consensus-itest.RaftConsensusITest.TestCatchupAfterOpsEvicted.1487669166730301-8186/raft_consensus-itest-cluster/ts-0/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.3.0-SNAPSHOT revision 0767cd78007dc4073f32511a6a194f925d52f614 build type DEBUG built by mpercy at 21 Feb 2017 01:21:47 PST on mpercy-T460p I0221 01:26:16.265409 11741 mem_tracker.cc:140] MemTracker: hard memory limit is 25.031622 GB I0221 01:26:16.265426 11741 mem_tracker.cc:142] MemTracker: soft memory limit is 15.018973 GB I0221 01:26:16.268321 11741 tablet_server_main.cc:63] Initializing tablet server... I0221 01:26:16.268692 11741 hybrid_clock.cc:177] HybridClock initialized. Resolution in nanos?: 1 Wait times tolerance adjustment: 1.0005 Current error: 564594 I0221 01:26:16.270185 11741 env_posix.cc:1316] Not raising process file limit of 65536; it is already as high as it can go I0221 01:26:16.270267 11741 file_cache.cc:401] Constructed file cache lbm with capacity 32768 W0221 01:26:16.270426 11741 fs_manager.cc:516] Unable to read directory: Not found: /tmp/kudutest-1000/raft_consensus-itest.RaftConsensusITest.TestCatchupAfterOpsEvicted.1487669166730301-8186/raft_consensus-itest-cluster/ts-0/data: No such file or directory (error 2) W0221 01:26:16.270473 11741 fs_manager.cc:525] could not check and fix permissions for path: /tmp/kudutest-1000/raft_consensus-itest.RaftConsensusITest.TestCatchupAfterOpsEvicted.1487669166730301-8186/raft_consensus-itest-cluster/ts-0/data: Not found: stat: No such file or directory (error 2) I0221 01:26:16.270524 11741 server_base.cc:181] Could not load existing FS layout: Not found: /tmp/kudutest-1000/raft_consensus-itest.RaftConsensusITest.TestCatchupAfterOpsEvicted.1487669166730301-8186/raft_consensus-itest-cluster/ts-0/data/instance: No such file or directory (error 2) I0221 01:26:16.270537 11741 server_base.cc:182] Creating new FS layout I0221 01:26:16.272631 11741 fs_manager.cc:370] Generated new instance metadata in path /tmp/kudutest-1000/raft_consensus-itest.RaftConsensusITest.TestCatchupAfterOpsEvicted.1487669166730301-8186/raft_consensus-itest-cluster/ts-0/data/instance: uuid: "1eb837892f874e99a16858e7958a1356" format_stamp: "Formatted at 2017-02-21 09:26:16 on mpercy-T460p" I0221 01:26:16.275574 11741 fs_manager.cc:256] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s I0221 01:26:16.275601 11741 fs_manager.cc:259] Opened local filesystem: /tmp/kudutest-1000/raft_consensus-itest.RaftConsensusITest.TestCatchupAfterOpsEvicted.1487669166730301-8186/raft_consensus-itest-cluster/ts-0/data uuid: "1eb837892f874e99a16858e7958a1356" format_stamp: "Formatted at 2017-02-21 09:26:16 on mpercy-T460p" I0221 01:26:16.276828 11744 catalog_manager.cc:718] Wrote cert authority information into the system table. I0221 01:26:16.311023 11741 tablet_server_main.cc:66] Starting tablet server... I0221 01:26:16.317842 11741 rpc_server.cc:164] RPC server started. Bound to: 127.31.250.0:35707 I0221 01:26:16.318064 11741 webserver.cc:129] Starting webserver on 127.31.250.0:0 I0221 01:26:16.318074 11741 webserver.cc:134] Document root: /home/mpercy/src/kudu/www I0221 01:26:16.318357 11741 webserver.cc:256] Webserver started. Bound to: http://127.31.250.0:33005/ I0221 01:26:16.318653 11741 server_base.cc:260] Dumped server information to /tmp/kudutest-1000/raft_consensus-itest.RaftConsensusITest.TestCatchupAfterOpsEvicted.1487669166730301-8186/raft_consensus-itest-cluster/ts-0/data/info.pb I0221 01:26:16.319185 11741 tablet_server_main.cc:69] Tablet server successfully started. I0221 01:26:16.322695 8186 external_mini_cluster.cc:717] Started /home/mpercy/src/kudu/build/dynclang/bin/kudu-tserver as pid 11741 I0221 01:26:16.322886 8186 external_mini_cluster.cc:681] Running /home/mpercy/src/kudu/build/dynclang/bin/kudu-tserver kudu-tserver --fs_wal_dir=/tmp/kudutest-1000/raft_consensus-itest.RaftConsensusITest.TestCatchupAfterOpsEvicted.1487669166730301-8186/raft_consensus-itest-cluster/ts-1/data --fs_data_dirs=/tmp/kudutest-1000/raft_consensus-itest.RaftConsensusITest.TestCatchupAfterOpsEvicted.1487669166730301-8186/raft_consensus-itest-cluster/ts-1/data --rpc_bind_addresses=127.31.250.1:0 --local_ip_for_outbound_sockets=127.31.250.1 --webserver_interface=127.31.250.1 --webserver_port=0 --tserver_master_addrs=127.0.0.1:44820 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --log_redact_user_data=false --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/kudutest-1000/raft_consensus-itest.RaftConsensusITest.TestCatchupAfterOpsEvicted.1487669166730301-8186/raft_consensus-itest-cluster/ts-1/logs --server_dump_info_path=/tmp/kudutest-1000/raft_consensus-itest.RaftConsensusITest.TestCatchupAfterOpsEvicted.1487669166730301-8186/raft_consensus-itest-cluster/ts-1/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --log_cache_size_limit_mb=1 --consensus_max_batch_size_bytes=500000 --log_segment_size_mb=1 --log_async_preallocate_segments=false --maintenance_manager_polling_interval_ms=100 --log_target_replay_size_mb=1 --max_cell_size_bytes=1000000 --log_compression_codec=none with env {} I0221 01:26:16.324326 11869 heartbeater.cc:291] Connected to a master server at 127.0.0.1:44820 I0221 01:26:16.324411 11869 heartbeater.cc:359] Registering TS with master... I0221 01:26:16.324559 11869 heartbeater.cc:389] Master 127.0.0.1:44820 requested a full tablet report, sending... I0221 01:26:16.325186 11703 ts_manager.cc:78] Registered new tserver with Master: 1eb837892f874e99a16858e7958a1356 (127.31.250.0:35707) I0221 01:26:16.325769 11703 master_service.cc:162] Signed X509 certificate for tserver {real_user=mpercy} at 127.31.250.0:35398 WARNING: Logging before InitGoogleLogging() is written to STDERR W0221 01:26:16.346796 11873 flags.cc:335] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0221 01:26:16.346922 11873 flags.cc:335] Enabled unsafe flag: --max_cell_size_bytes=1000000 W0221 01:26:16.346936 11873 flags.cc:335] Enabled unsafe flag: --never_fsync=true W0221 01:26:16.347646 11873 flags.cc:335] Enabled experimental flag: --log_compression_codec=none W0221 01:26:16.347666 11873 flags.cc:335] Enabled experimental flag: --ipki_server_key_size=1024 W0221 01:26:16.347700 11873 flags.cc:335] Enabled experimental flag: --log_redact_user_data=false W0221 01:26:16.347705 11873 flags.cc:335] Enabled experimental flag: --log_target_replay_size_mb=1 W0221 01:26:16.347714 11873 flags.cc:335] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.250.1 I0221 01:26:16.348847 11873 tablet_server_main.cc:56] Tablet server non-default flags: --consensus_max_batch_size_bytes=500000 --log_compression_codec=none --log_cache_size_limit_mb=1 --log_async_preallocate_segments=false --log_segment_size_mb=1 --fs_data_dirs=/tmp/kudutest-1000/raft_consensus-itest.RaftConsensusITest.TestCatchupAfterOpsEvicted.1487669166730301-8186/raft_consensus-itest-cluster/ts-1/data --fs_wal_dir=/tmp/kudutest-1000/raft_consensus-itest.RaftConsensusITest.TestCatchupAfterOpsEvicted.1487669166730301-8186/raft_consensus-itest-cluster/ts-1/data --ipki_server_key_size=1024 --rpc_bind_addresses=127.31.250.1:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/kudutest-1000/raft_consensus-itest.RaftConsensusITest.TestCatchupAfterOpsEvicted.1487669166730301-8186/raft_consensus-itest-cluster/ts-1/data/info.pb --webserver_interface=127.31.250.1 --webserver_port=0 --max_cell_size_bytes=1000000 --tserver_master_addrs=127.0.0.1:44820 --never_fsync=true --heap_profile_path=/tmp/kudu-tserver.11873 --unlock_experimental_flags=true --unlock_unsafe_flags=true --log_redact_user_data=false --log_target_replay_size_mb=1 --maintenance_manager_polling_interval_ms=100 --enable_minidumps=false --local_ip_for_outbound_sockets=127.31.250.1 --log_dir=/tmp/kudutest-1000/raft_consensus-itest.RaftConsensusITest.TestCatchupAfterOpsEvicted.1487669166730301-8186/raft_consensus-itest-cluster/ts-1/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.3.0-SNAPSHOT revision 0767cd78007dc4073f32511a6a194f925d52f614 build type DEBUG built by mpercy at 21 Feb 2017 01:21:47 PST on mpercy-T460p I0221 01:26:16.349334 11873 mem_tracker.cc:140] MemTracker: hard memory limit is 25.031622 GB I0221 01:26:16.349354 11873 mem_tracker.cc:142] MemTracker: soft memory limit is 15.018973 GB I0221 01:26:16.352748 11873 tablet_server_main.cc:63] Initializing tablet server... I0221 01:26:16.353185 11873 hybrid_clock.cc:177] HybridClock initialized. Resolution in nanos?: 1 Wait times tolerance adjustment: 1.0005 Current error: 564594 I0221 01:26:16.354861 11873 env_posix.cc:1316] Not raising process file limit of 65536; it is already as high as it can go I0221 01:26:16.354953 11873 file_cache.cc:401] Constructed file cache lbm with capacity 32768 W0221 01:26:16.355124 11873 fs_manager.cc:516] Unable to read directory: Not found: /tmp/kudutest-1000/raft_consensus-itest.RaftConsensusITest.TestCatchupAfterOpsEvicted.1487669166730301-8186/raft_consensus-itest-cluster/ts-1/data: No such file or directory (error 2) W0221 01:26:16.355175 11873 fs_manager.cc:525] could not check and fix permissions for path: /tmp/kudutest-1000/raft_consensus-itest.RaftConsensusITest.TestCatchupAfterOpsEvicted.1487669166730301-8186/raft_consensus-itest-cluster/ts-1/data: Not found: stat: No such file or directory (error 2) I0221 01:26:16.355248 11873 server_base.cc:181] Could not load existing FS layout: Not found: /tmp/kudutest-1000/raft_consensus-itest.RaftConsensusITest.TestCatchupAfterOpsEvicted.1487669166730301-8186/raft_consensus-itest-cluster/ts-1/data/instance: No such file or directory (error 2) I0221 01:26:16.355262 11873 server_base.cc:182] Creating new FS layout I0221 01:26:16.357735 11873 fs_manager.cc:370] Generated new instance metadata in path /tmp/kudutest-1000/raft_consensus-itest.RaftConsensusITest.TestCatchupAfterOpsEvicted.1487669166730301-8186/raft_consensus-itest-cluster/ts-1/data/instance: uuid: "c8ec78660e8642b4b82113d62a3042cb" format_stamp: "Formatted at 2017-02-21 09:26:16 on mpercy-T460p" I0221 01:26:16.361299 11873 fs_manager.cc:256] Time spent opening block manager: real 0.001s user 0.004s sys 0.000s I0221 01:26:16.361341 11873 fs_manager.cc:259] Opened local filesystem: /tmp/kudutest-1000/raft_consensus-itest.RaftConsensusITest.TestCatchupAfterOpsEvicted.1487669166730301-8186/raft_consensus-itest-cluster/ts-1/data uuid: "c8ec78660e8642b4b82113d62a3042cb" format_stamp: "Formatted at 2017-02-21 09:26:16 on mpercy-T460p" I0221 01:26:16.393599 11873 tablet_server_main.cc:66] Starting tablet server... I0221 01:26:16.400194 11873 rpc_server.cc:164] RPC server started. Bound to: 127.31.250.1:40370 I0221 01:26:16.400400 11873 webserver.cc:129] Starting webserver on 127.31.250.1:0 I0221 01:26:16.400409 11873 webserver.cc:134] Document root: /home/mpercy/src/kudu/www I0221 01:26:16.400684 11873 webserver.cc:256] Webserver started. Bound to: http://127.31.250.1:38895/ I0221 01:26:16.400985 11873 server_base.cc:260] Dumped server information to /tmp/kudutest-1000/raft_consensus-itest.RaftConsensusITest.TestCatchupAfterOpsEvicted.1487669166730301-8186/raft_consensus-itest-cluster/ts-1/data/info.pb I0221 01:26:16.401394 11873 tablet_server_main.cc:69] Tablet server successfully started. I0221 01:26:16.405268 12006 heartbeater.cc:291] Connected to a master server at 127.0.0.1:44820 I0221 01:26:16.405345 12006 heartbeater.cc:359] Registering TS with master... I0221 01:26:16.405488 12006 heartbeater.cc:389] Master 127.0.0.1:44820 requested a full tablet report, sending... I0221 01:26:16.405515 8186 external_mini_cluster.cc:717] Started /home/mpercy/src/kudu/build/dynclang/bin/kudu-tserver as pid 11873 I0221 01:26:16.405747 8186 external_mini_cluster.cc:681] Running /home/mpercy/src/kudu/build/dynclang/bin/kudu-tserver kudu-tserver --fs_wal_dir=/tmp/kudutest-1000/raft_consensus-itest.RaftConsensusITest.TestCatchupAfterOpsEvicted.1487669166730301-8186/raft_consensus-itest-cluster/ts-2/data --fs_data_dirs=/tmp/kudutest-1000/raft_consensus-itest.RaftConsensusITest.TestCatchupAfterOpsEvicted.1487669166730301-8186/raft_consensus-itest-cluster/ts-2/data --rpc_bind_addresses=127.31.250.2:0 --local_ip_for_outbound_sockets=127.31.250.2 --webserver_interface=127.31.250.2 --webserver_port=0 --tserver_master_addrs=127.0.0.1:44820 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --log_redact_user_data=false --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/kudutest-1000/raft_consensus-itest.RaftConsensusITest.TestCatchupAfterOpsEvicted.1487669166730301-8186/raft_consensus-itest-cluster/ts-2/logs --server_dump_info_path=/tmp/kudutest-1000/raft_consensus-itest.RaftConsensusITest.TestCatchupAfterOpsEvicted.1487669166730301-8186/raft_consensus-itest-cluster/ts-2/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --log_cache_size_limit_mb=1 --consensus_max_batch_size_bytes=500000 --log_segment_size_mb=1 --log_async_preallocate_segments=false --maintenance_manager_polling_interval_ms=100 --log_target_replay_size_mb=1 --max_cell_size_bytes=1000000 --log_compression_codec=none with env {} I0221 01:26:16.405849 11703 ts_manager.cc:78] Registered new tserver with Master: c8ec78660e8642b4b82113d62a3042cb (127.31.250.1:40370) I0221 01:26:16.406159 11703 master_service.cc:162] Signed X509 certificate for tserver {real_user=mpercy} at 127.31.250.1:36151 WARNING: Logging before InitGoogleLogging() is written to STDERR W0221 01:26:16.430145 12009 flags.cc:335] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0221 01:26:16.430248 12009 flags.cc:335] Enabled unsafe flag: --max_cell_size_bytes=1000000 W0221 01:26:16.430259 12009 flags.cc:335] Enabled unsafe flag: --never_fsync=true W0221 01:26:16.430922 12009 flags.cc:335] Enabled experimental flag: --log_compression_codec=none W0221 01:26:16.430941 12009 flags.cc:335] Enabled experimental flag: --ipki_server_key_size=1024 W0221 01:26:16.430974 12009 flags.cc:335] Enabled experimental flag: --log_redact_user_data=false W0221 01:26:16.430980 12009 flags.cc:335] Enabled experimental flag: --log_target_replay_size_mb=1 W0221 01:26:16.430990 12009 flags.cc:335] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.250.2 I0221 01:26:16.432124 12009 tablet_server_main.cc:56] Tablet server non-default flags: --consensus_max_batch_size_bytes=500000 --log_compression_codec=none --log_cache_size_limit_mb=1 --log_async_preallocate_segments=false --log_segment_size_mb=1 --fs_data_dirs=/tmp/kudutest-1000/raft_consensus-itest.RaftConsensusITest.TestCatchupAfterOpsEvicted.1487669166730301-8186/raft_consensus-itest-cluster/ts-2/data --fs_wal_dir=/tmp/kudutest-1000/raft_consensus-itest.RaftConsensusITest.TestCatchupAfterOpsEvicted.1487669166730301-8186/raft_consensus-itest-cluster/ts-2/data --ipki_server_key_size=1024 --rpc_bind_addresses=127.31.250.2:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/kudutest-1000/raft_consensus-itest.RaftConsensusITest.TestCatchupAfterOpsEvicted.1487669166730301-8186/raft_consensus-itest-cluster/ts-2/data/info.pb --webserver_interface=127.31.250.2 --webserver_port=0 --max_cell_size_bytes=1000000 --tserver_master_addrs=127.0.0.1:44820 --never_fsync=true --heap_profile_path=/tmp/kudu-tserver.12009 --unlock_experimental_flags=true --unlock_unsafe_flags=true --log_redact_user_data=false --log_target_replay_size_mb=1 --maintenance_manager_polling_interval_ms=100 --enable_minidumps=false --local_ip_for_outbound_sockets=127.31.250.2 --log_dir=/tmp/kudutest-1000/raft_consensus-itest.RaftConsensusITest.TestCatchupAfterOpsEvicted.1487669166730301-8186/raft_consensus-itest-cluster/ts-2/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.3.0-SNAPSHOT revision 0767cd78007dc4073f32511a6a194f925d52f614 build type DEBUG built by mpercy at 21 Feb 2017 01:21:47 PST on mpercy-T460p I0221 01:26:16.432600 12009 mem_tracker.cc:140] MemTracker: hard memory limit is 25.031622 GB I0221 01:26:16.432617 12009 mem_tracker.cc:142] MemTracker: soft memory limit is 15.018973 GB I0221 01:26:16.435963 12009 tablet_server_main.cc:63] Initializing tablet server... I0221 01:26:16.436429 12009 hybrid_clock.cc:177] HybridClock initialized. Resolution in nanos?: 1 Wait times tolerance adjustment: 1.0005 Current error: 564594 I0221 01:26:16.438094 12009 env_posix.cc:1316] Not raising process file limit of 65536; it is already as high as it can go I0221 01:26:16.438201 12009 file_cache.cc:401] Constructed file cache lbm with capacity 32768 W0221 01:26:16.438364 12009 fs_manager.cc:516] Unable to read directory: Not found: /tmp/kudutest-1000/raft_consensus-itest.RaftConsensusITest.TestCatchupAfterOpsEvicted.1487669166730301-8186/raft_consensus-itest-cluster/ts-2/data: No such file or directory (error 2) W0221 01:26:16.438413 12009 fs_manager.cc:525] could not check and fix permissions for path: /tmp/kudutest-1000/raft_consensus-itest.RaftConsensusITest.TestCatchupAfterOpsEvicted.1487669166730301-8186/raft_consensus-itest-cluster/ts-2/data: Not found: stat: No such file or directory (error 2) I0221 01:26:16.438464 12009 server_base.cc:181] Could not load existing FS layout: Not found: /tmp/kudutest-1000/raft_consensus-itest.RaftConsensusITest.TestCatchupAfterOpsEvicted.1487669166730301-8186/raft_consensus-itest-cluster/ts-2/data/instance: No such file or directory (error 2) I0221 01:26:16.438478 12009 server_base.cc:182] Creating new FS layout I0221 01:26:16.440775 12009 fs_manager.cc:370] Generated new instance metadata in path /tmp/kudutest-1000/raft_consensus-itest.RaftConsensusITest.TestCatchupAfterOpsEvicted.1487669166730301-8186/raft_consensus-itest-cluster/ts-2/data/instance: uuid: "33e7b3c7fdd64255b209cc8123db9646" format_stamp: "Formatted at 2017-02-21 09:26:16 on mpercy-T460p" I0221 01:26:16.444104 12009 fs_manager.cc:256] Time spent opening block manager: real 0.001s user 0.004s sys 0.000s I0221 01:26:16.444134 12009 fs_manager.cc:259] Opened local filesystem: /tmp/kudutest-1000/raft_consensus-itest.RaftConsensusITest.TestCatchupAfterOpsEvicted.1487669166730301-8186/raft_consensus-itest-cluster/ts-2/data uuid: "33e7b3c7fdd64255b209cc8123db9646" format_stamp: "Formatted at 2017-02-21 09:26:16 on mpercy-T460p" I0221 01:26:16.463539 12009 tablet_server_main.cc:66] Starting tablet server... I0221 01:26:16.470569 12009 rpc_server.cc:164] RPC server started. Bound to: 127.31.250.2:46079 I0221 01:26:16.470778 12009 webserver.cc:129] Starting webserver on 127.31.250.2:0 I0221 01:26:16.470787 12009 webserver.cc:134] Document root: /home/mpercy/src/kudu/www I0221 01:26:16.471034 12009 webserver.cc:256] Webserver started. Bound to: http://127.31.250.2:44840/ I0221 01:26:16.471328 12009 server_base.cc:260] Dumped server information to /tmp/kudutest-1000/raft_consensus-itest.RaftConsensusITest.TestCatchupAfterOpsEvicted.1487669166730301-8186/raft_consensus-itest-cluster/ts-2/data/info.pb I0221 01:26:16.471768 12009 tablet_server_main.cc:69] Tablet server successfully started. I0221 01:26:16.475692 12129 heartbeater.cc:291] Connected to a master server at 127.0.0.1:44820 I0221 01:26:16.475771 12129 heartbeater.cc:359] Registering TS with master... I0221 01:26:16.475930 12129 heartbeater.cc:389] Master 127.0.0.1:44820 requested a full tablet report, sending... I0221 01:26:16.476339 11703 ts_manager.cc:78] Registered new tserver with Master: 33e7b3c7fdd64255b209cc8123db9646 (127.31.250.2:46079) I0221 01:26:16.476675 11703 master_service.cc:162] Signed X509 certificate for tserver {real_user=mpercy} at 127.31.250.2:50990 I0221 01:26:16.478482 8186 external_mini_cluster.cc:717] Started /home/mpercy/src/kudu/build/dynclang/bin/kudu-tserver as pid 12009 I0221 01:26:16.481156 8186 external_mini_cluster.cc:408] 3 TS(s) registered with all masters I0221 01:26:16.487388 11703 catalog_manager.cc:1000] CreateTable from {real_user=mpercy} at 127.0.0.1:48988: name: "TestTable" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 3 split_rows_range_bounds { } partition_schema { range_schema { columns { name: "key" } } } I0221 01:26:16.494776 11943 tablet_service.cc:631] Processing CreateTablet for tablet 2c6176fb99ed4fb4a027584aeeb0a708 (table=TestTable [id=9eaab9113ca144d08f6c5035b204b27f]), partition=RANGE (key) PARTITION UNBOUNDED I0221 01:26:16.494778 12065 tablet_service.cc:631] Processing CreateTablet for tablet 2c6176fb99ed4fb4a027584aeeb0a708 (table=TestTable [id=9eaab9113ca144d08f6c5035b204b27f]), partition=RANGE (key) PARTITION UNBOUNDED I0221 01:26:16.495031 11806 tablet_service.cc:631] Processing CreateTablet for tablet 2c6176fb99ed4fb4a027584aeeb0a708 (table=TestTable [id=9eaab9113ca144d08f6c5035b204b27f]), partition=RANGE (key) PARTITION UNBOUNDED I0221 01:26:16.496824 11943 ts_tablet_manager.cc:876] T 2c6176fb99ed4fb4a027584aeeb0a708 P c8ec78660e8642b4b82113d62a3042cb: Registered tablet (data state: TABLET_DATA_READY) I0221 01:26:16.496824 12065 ts_tablet_manager.cc:876] T 2c6176fb99ed4fb4a027584aeeb0a708 P 33e7b3c7fdd64255b209cc8123db9646: Registered tablet (data state: TABLET_DATA_READY) I0221 01:26:16.496909 11806 ts_tablet_manager.cc:876] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356: Registered tablet (data state: TABLET_DATA_READY) I0221 01:26:16.497033 12140 ts_tablet_manager.cc:735] T 2c6176fb99ed4fb4a027584aeeb0a708 P 33e7b3c7fdd64255b209cc8123db9646: Bootstrapping tablet I0221 01:26:16.497069 12141 ts_tablet_manager.cc:735] T 2c6176fb99ed4fb4a027584aeeb0a708 P c8ec78660e8642b4b82113d62a3042cb: Bootstrapping tablet I0221 01:26:16.497125 12142 ts_tablet_manager.cc:735] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356: Bootstrapping tablet I0221 01:26:16.497409 12140 tablet_bootstrap.cc:382] T 2c6176fb99ed4fb4a027584aeeb0a708 P 33e7b3c7fdd64255b209cc8123db9646: Bootstrap starting. I0221 01:26:16.497426 12141 tablet_bootstrap.cc:382] T 2c6176fb99ed4fb4a027584aeeb0a708 P c8ec78660e8642b4b82113d62a3042cb: Bootstrap starting. I0221 01:26:16.497509 12142 tablet_bootstrap.cc:382] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356: Bootstrap starting. I0221 01:26:16.497992 12142 tablet_bootstrap.cc:541] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356: Time spent opening tablet: real 0.000s user 0.000s sys 0.000s I0221 01:26:16.497992 12141 tablet_bootstrap.cc:541] T 2c6176fb99ed4fb4a027584aeeb0a708 P c8ec78660e8642b4b82113d62a3042cb: Time spent opening tablet: real 0.000s user 0.000s sys 0.000s I0221 01:26:16.497990 12140 tablet_bootstrap.cc:541] T 2c6176fb99ed4fb4a027584aeeb0a708 P 33e7b3c7fdd64255b209cc8123db9646: Time spent opening tablet: real 0.000s user 0.000s sys 0.000s I0221 01:26:16.498086 12142 tablet_bootstrap.cc:484] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356: No blocks or log segments found. Creating new log. I0221 01:26:16.498086 12140 tablet_bootstrap.cc:484] T 2c6176fb99ed4fb4a027584aeeb0a708 P 33e7b3c7fdd64255b209cc8123db9646: No blocks or log segments found. Creating new log. I0221 01:26:16.498088 12141 tablet_bootstrap.cc:484] T 2c6176fb99ed4fb4a027584aeeb0a708 P c8ec78660e8642b4b82113d62a3042cb: No blocks or log segments found. Creating new log. I0221 01:26:16.498379 12140 log.cc:375] T 2c6176fb99ed4fb4a027584aeeb0a708 P 33e7b3c7fdd64255b209cc8123db9646: Log is configured to *not* fsync() on all Append() calls I0221 01:26:16.498379 12141 log.cc:375] T 2c6176fb99ed4fb4a027584aeeb0a708 P c8ec78660e8642b4b82113d62a3042cb: Log is configured to *not* fsync() on all Append() calls I0221 01:26:16.498383 12142 log.cc:375] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356: Log is configured to *not* fsync() on all Append() calls I0221 01:26:16.499028 12142 tablet_bootstrap.cc:382] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356: No bootstrap required, opened a new log I0221 01:26:16.499028 12140 tablet_bootstrap.cc:382] T 2c6176fb99ed4fb4a027584aeeb0a708 P 33e7b3c7fdd64255b209cc8123db9646: No bootstrap required, opened a new log I0221 01:26:16.499066 12141 tablet_bootstrap.cc:382] T 2c6176fb99ed4fb4a027584aeeb0a708 P c8ec78660e8642b4b82113d62a3042cb: No bootstrap required, opened a new log I0221 01:26:16.499099 12140 ts_tablet_manager.cc:740] T 2c6176fb99ed4fb4a027584aeeb0a708 P 33e7b3c7fdd64255b209cc8123db9646: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.000s I0221 01:26:16.499099 12142 ts_tablet_manager.cc:740] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.000s I0221 01:26:16.499122 12141 ts_tablet_manager.cc:740] T 2c6176fb99ed4fb4a027584aeeb0a708 P c8ec78660e8642b4b82113d62a3042cb: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.000s I0221 01:26:16.500075 12141 raft_consensus.cc:288] T 2c6176fb99ed4fb4a027584aeeb0a708 P c8ec78660e8642b4b82113d62a3042cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8ec78660e8642b4b82113d62a3042cb" member_type: VOTER last_known_addr { host: "127.31.250.1" port: 40370 } } peers { permanent_uuid: "33e7b3c7fdd64255b209cc8123db9646" member_type: VOTER last_known_addr { host: "127.31.250.2" port: 46079 } } peers { permanent_uuid: "1eb837892f874e99a16858e7958a1356" member_type: VOTER last_known_addr { host: "127.31.250.0" port: 35707 } } I0221 01:26:16.500196 12142 raft_consensus.cc:288] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8ec78660e8642b4b82113d62a3042cb" member_type: VOTER last_known_addr { host: "127.31.250.1" port: 40370 } } peers { permanent_uuid: "33e7b3c7fdd64255b209cc8123db9646" member_type: VOTER last_known_addr { host: "127.31.250.2" port: 46079 } } peers { permanent_uuid: "1eb837892f874e99a16858e7958a1356" member_type: VOTER last_known_addr { host: "127.31.250.0" port: 35707 } } I0221 01:26:16.500272 12140 raft_consensus.cc:288] T 2c6176fb99ed4fb4a027584aeeb0a708 P 33e7b3c7fdd64255b209cc8123db9646 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8ec78660e8642b4b82113d62a3042cb" member_type: VOTER last_known_addr { host: "127.31.250.1" port: 40370 } } peers { permanent_uuid: "33e7b3c7fdd64255b209cc8123db9646" member_type: VOTER last_known_addr { host: "127.31.250.2" port: 46079 } } peers { permanent_uuid: "1eb837892f874e99a16858e7958a1356" member_type: VOTER last_known_addr { host: "127.31.250.0" port: 35707 } } I0221 01:26:16.500397 12141 raft_consensus.cc:319] T 2c6176fb99ed4fb4a027584aeeb0a708 P c8ec78660e8642b4b82113d62a3042cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0221 01:26:16.500425 12141 raft_consensus.cc:548] T 2c6176fb99ed4fb4a027584aeeb0a708 P c8ec78660e8642b4b82113d62a3042cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8ec78660e8642b4b82113d62a3042cb, State: 1, Role: FOLLOWER I0221 01:26:16.500459 12141 consensus_queue.cc:176] T 2c6176fb99ed4fb4a027584aeeb0a708 P c8ec78660e8642b4b82113d62a3042cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I0221 01:26:16.500568 12142 raft_consensus.cc:319] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0221 01:26:16.500604 12142 raft_consensus.cc:548] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1eb837892f874e99a16858e7958a1356, State: 1, Role: FOLLOWER I0221 01:26:16.500619 12140 raft_consensus.cc:319] T 2c6176fb99ed4fb4a027584aeeb0a708 P 33e7b3c7fdd64255b209cc8123db9646 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0221 01:26:16.500653 12140 raft_consensus.cc:548] T 2c6176fb99ed4fb4a027584aeeb0a708 P 33e7b3c7fdd64255b209cc8123db9646 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33e7b3c7fdd64255b209cc8123db9646, State: 1, Role: FOLLOWER I0221 01:26:16.500646 12142 consensus_queue.cc:176] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I0221 01:26:16.500694 12140 consensus_queue.cc:176] T 2c6176fb99ed4fb4a027584aeeb0a708 P 33e7b3c7fdd64255b209cc8123db9646 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Current term: 0, Majority size: -1, State: 1, Mode: NON_LEADER I0221 01:26:16.500751 12006 heartbeater.cc:381] Master 127.0.0.1:44820 was elected leader, sending a full tablet report... I0221 01:26:16.500810 12141 ts_tablet_manager.cc:767] T 2c6176fb99ed4fb4a027584aeeb0a708 P c8ec78660e8642b4b82113d62a3042cb: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s I0221 01:26:16.500900 11869 heartbeater.cc:381] Master 127.0.0.1:44820 was elected leader, sending a full tablet report... I0221 01:26:16.500951 12129 heartbeater.cc:381] Master 127.0.0.1:44820 was elected leader, sending a full tablet report... I0221 01:26:16.501013 12142 ts_tablet_manager.cc:767] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s I0221 01:26:16.501062 12140 ts_tablet_manager.cc:767] T 2c6176fb99ed4fb4a027584aeeb0a708 P 33e7b3c7fdd64255b209cc8123db9646: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s I0221 01:26:17.199460 12149 raft_consensus.cc:411] T 2c6176fb99ed4fb4a027584aeeb0a708 P c8ec78660e8642b4b82113d62a3042cb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0221 01:26:17.199548 12149 raft_consensus.cc:2162] T 2c6176fb99ed4fb4a027584aeeb0a708 P c8ec78660e8642b4b82113d62a3042cb [term 0 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.61 s I0221 01:26:17.199592 12149 raft_consensus.cc:435] T 2c6176fb99ed4fb4a027584aeeb0a708 P c8ec78660e8642b4b82113d62a3042cb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8ec78660e8642b4b82113d62a3042cb" member_type: VOTER last_known_addr { host: "127.31.250.1" port: 40370 } } peers { permanent_uuid: "33e7b3c7fdd64255b209cc8123db9646" member_type: VOTER last_known_addr { host: "127.31.250.2" port: 46079 } } peers { permanent_uuid: "1eb837892f874e99a16858e7958a1356" member_type: VOTER last_known_addr { host: "127.31.250.0" port: 35707 } } I0221 01:26:17.200212 12149 leader_election.cc:216] T 2c6176fb99ed4fb4a027584aeeb0a708 P c8ec78660e8642b4b82113d62a3042cb [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 33e7b3c7fdd64255b209cc8123db9646 I0221 01:26:17.200386 12149 leader_election.cc:216] T 2c6176fb99ed4fb4a027584aeeb0a708 P c8ec78660e8642b4b82113d62a3042cb [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 1eb837892f874e99a16858e7958a1356 I0221 01:26:17.204154 12085 raft_consensus.cc:2162] T 2c6176fb99ed4fb4a027584aeeb0a708 P 33e7b3c7fdd64255b209cc8123db9646 [term 0 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.89 s I0221 01:26:17.204216 12085 raft_consensus.cc:1747] T 2c6176fb99ed4fb4a027584aeeb0a708 P 33e7b3c7fdd64255b209cc8123db9646 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8ec78660e8642b4b82113d62a3042cb in term 0. I0221 01:26:17.204545 11895 leader_election.cc:348] T 2c6176fb99ed4fb4a027584aeeb0a708 P c8ec78660e8642b4b82113d62a3042cb [CANDIDATE]: Term 1 pre-election: Vote granted by peer 33e7b3c7fdd64255b209cc8123db9646 I0221 01:26:17.204593 11895 leader_election.cc:243] T 2c6176fb99ed4fb4a027584aeeb0a708 P c8ec78660e8642b4b82113d62a3042cb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. I0221 01:26:17.204640 11826 raft_consensus.cc:2162] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 0 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.63 s I0221 01:26:17.204679 12152 raft_consensus.cc:2162] T 2c6176fb99ed4fb4a027584aeeb0a708 P c8ec78660e8642b4b82113d62a3042cb [term 0 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 2.23 s I0221 01:26:17.204691 11826 raft_consensus.cc:1747] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8ec78660e8642b4b82113d62a3042cb in term 0. I0221 01:26:17.204715 12152 raft_consensus.cc:1971] T 2c6176fb99ed4fb4a027584aeeb0a708 P c8ec78660e8642b4b82113d62a3042cb [term 0 FOLLOWER]: Leader pre-election won for term 1 I0221 01:26:17.204738 12152 raft_consensus.cc:411] T 2c6176fb99ed4fb4a027584aeeb0a708 P c8ec78660e8642b4b82113d62a3042cb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0221 01:26:17.204751 12152 raft_consensus.cc:2162] T 2c6176fb99ed4fb4a027584aeeb0a708 P c8ec78660e8642b4b82113d62a3042cb [term 0 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.68 s I0221 01:26:17.204761 12152 raft_consensus.cc:2206] T 2c6176fb99ed4fb4a027584aeeb0a708 P c8ec78660e8642b4b82113d62a3042cb [term 0 FOLLOWER]: Advancing to term 1 I0221 01:26:17.204942 11895 leader_election.cc:348] T 2c6176fb99ed4fb4a027584aeeb0a708 P c8ec78660e8642b4b82113d62a3042cb [CANDIDATE]: Term 1 pre-election: Vote granted by peer 1eb837892f874e99a16858e7958a1356 I0221 01:26:17.205332 12152 raft_consensus.cc:435] T 2c6176fb99ed4fb4a027584aeeb0a708 P c8ec78660e8642b4b82113d62a3042cb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8ec78660e8642b4b82113d62a3042cb" member_type: VOTER last_known_addr { host: "127.31.250.1" port: 40370 } } peers { permanent_uuid: "33e7b3c7fdd64255b209cc8123db9646" member_type: VOTER last_known_addr { host: "127.31.250.2" port: 46079 } } peers { permanent_uuid: "1eb837892f874e99a16858e7958a1356" member_type: VOTER last_known_addr { host: "127.31.250.0" port: 35707 } } I0221 01:26:17.205541 12152 leader_election.cc:216] T 2c6176fb99ed4fb4a027584aeeb0a708 P c8ec78660e8642b4b82113d62a3042cb [CANDIDATE]: Term 1 election: Requesting vote from peer 33e7b3c7fdd64255b209cc8123db9646 I0221 01:26:17.205595 12152 leader_election.cc:216] T 2c6176fb99ed4fb4a027584aeeb0a708 P c8ec78660e8642b4b82113d62a3042cb [CANDIDATE]: Term 1 election: Requesting vote from peer 1eb837892f874e99a16858e7958a1356 I0221 01:26:17.205893 12085 raft_consensus.cc:2206] T 2c6176fb99ed4fb4a027584aeeb0a708 P 33e7b3c7fdd64255b209cc8123db9646 [term 0 FOLLOWER]: Advancing to term 1 I0221 01:26:17.205938 11826 raft_consensus.cc:2206] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 0 FOLLOWER]: Advancing to term 1 I0221 01:26:17.205955 12085 raft_consensus.cc:2162] T 2c6176fb99ed4fb4a027584aeeb0a708 P 33e7b3c7fdd64255b209cc8123db9646 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.91 s I0221 01:26:17.205991 11826 raft_consensus.cc:2162] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.8 s I0221 01:26:17.206410 12085 raft_consensus.cc:1747] T 2c6176fb99ed4fb4a027584aeeb0a708 P 33e7b3c7fdd64255b209cc8123db9646 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8ec78660e8642b4b82113d62a3042cb in term 1. I0221 01:26:17.206465 11826 raft_consensus.cc:1747] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8ec78660e8642b4b82113d62a3042cb in term 1. I0221 01:26:17.206655 11895 leader_election.cc:348] T 2c6176fb99ed4fb4a027584aeeb0a708 P c8ec78660e8642b4b82113d62a3042cb [CANDIDATE]: Term 1 election: Vote granted by peer 33e7b3c7fdd64255b209cc8123db9646 I0221 01:26:17.206681 11895 leader_election.cc:243] T 2c6176fb99ed4fb4a027584aeeb0a708 P c8ec78660e8642b4b82113d62a3042cb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0221 01:26:17.206748 11895 leader_election.cc:348] T 2c6176fb99ed4fb4a027584aeeb0a708 P c8ec78660e8642b4b82113d62a3042cb [CANDIDATE]: Term 1 election: Vote granted by peer 1eb837892f874e99a16858e7958a1356 I0221 01:26:17.206776 12152 raft_consensus.cc:2162] T 2c6176fb99ed4fb4a027584aeeb0a708 P c8ec78660e8642b4b82113d62a3042cb [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.67 s I0221 01:26:17.206810 12152 raft_consensus.cc:1971] T 2c6176fb99ed4fb4a027584aeeb0a708 P c8ec78660e8642b4b82113d62a3042cb [term 1 FOLLOWER]: Leader election won for term 1 I0221 01:26:17.206970 12152 raft_consensus.cc:518] T 2c6176fb99ed4fb4a027584aeeb0a708 P c8ec78660e8642b4b82113d62a3042cb [term 1 LEADER]: Becoming Leader. State: Replica: c8ec78660e8642b4b82113d62a3042cb, State: 1, Role: LEADER I0221 01:26:17.207088 12152 consensus_queue.cc:158] T 2c6176fb99ed4fb4a027584aeeb0a708 P c8ec78660e8642b4b82113d62a3042cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Current term: 1, Majority size: 2, State: 1, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8ec78660e8642b4b82113d62a3042cb" member_type: VOTER last_known_addr { host: "127.31.250.1" port: 40370 } } peers { permanent_uuid: "33e7b3c7fdd64255b209cc8123db9646" member_type: VOTER last_known_addr { host: "127.31.250.2" port: 46079 } } peers { permanent_uuid: "1eb837892f874e99a16858e7958a1356" member_type: VOTER last_known_addr { host: "127.31.250.0" port: 35707 } } I0221 01:26:17.209662 11701 catalog_manager.cc:2265] T 2c6176fb99ed4fb4a027584aeeb0a708 reported consensus state change: term changed from 0 to 1, leader changed from to c8ec78660e8642b4b82113d62a3042cb (127.31.250.1). New consensus state: current_term: 1 leader_uuid: "c8ec78660e8642b4b82113d62a3042cb" config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8ec78660e8642b4b82113d62a3042cb" member_type: VOTER last_known_addr { host: "127.31.250.1" port: 40370 } } peers { permanent_uuid: "33e7b3c7fdd64255b209cc8123db9646" member_type: VOTER last_known_addr { host: "127.31.250.2" port: 46079 } } peers { permanent_uuid: "1eb837892f874e99a16858e7958a1356" member_type: VOTER last_known_addr { host: "127.31.250.0" port: 35707 } } } I0221 01:26:17.358778 8186 external_mini_cluster.cc:408] 3 TS(s) registered with all masters I0221 01:26:17.360693 8186 ts_itest-base.h:194] Waiting for 1 tablets on tserver 1eb837892f874e99a16858e7958a1356 to finish bootstrapping I0221 01:26:17.367717 8186 ts_itest-base.h:194] Waiting for 1 tablets on tserver c8ec78660e8642b4b82113d62a3042cb to finish bootstrapping I0221 01:26:17.374301 8186 ts_itest-base.h:194] Waiting for 1 tablets on tserver 33e7b3c7fdd64255b209cc8123db9646 to finish bootstrapping I0221 01:26:17.378798 8186 raft_consensus-itest.cc:669] Paused one of the replicas, starting to write. I0221 01:26:17.387207 12085 raft_consensus.cc:918] T 2c6176fb99ed4fb4a027584aeeb0a708 P 33e7b3c7fdd64255b209cc8123db9646 [term 1 FOLLOWER]: Refusing update from remote peer c8ec78660e8642b4b82113d62a3042cb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0221 01:26:17.387660 12161 consensus_queue.cc:695] T 2c6176fb99ed4fb4a027584aeeb0a708 P c8ec78660e8642b4b82113d62a3042cb [LEADER]: Connected to new peer: Peer: 33e7b3c7fdd64255b209cc8123db9646, Is new: false, Last received: 0.0, Next index: 1, Last known committed idx: 0, Last exchange result: ERROR, Needs tablet copy: false I0221 01:26:17.408660 12148 log.cc:541] T 2c6176fb99ed4fb4a027584aeeb0a708 P c8ec78660e8642b4b82113d62a3042cb: Max segment size reached. Starting new segment allocation I0221 01:26:17.409210 12148 log.cc:426] T 2c6176fb99ed4fb4a027584aeeb0a708 P c8ec78660e8642b4b82113d62a3042cb: Rolled over to a new log segment at /tmp/kudutest-1000/raft_consensus-itest.RaftConsensusITest.TestCatchupAfterOpsEvicted.1487669166730301-8186/raft_consensus-itest-cluster/ts-1/data/wals/2c6176fb99ed4fb4a027584aeeb0a708/wal-000000002 I0221 01:26:17.409561 12146 log.cc:541] T 2c6176fb99ed4fb4a027584aeeb0a708 P 33e7b3c7fdd64255b209cc8123db9646: Max segment size reached. Starting new segment allocation I0221 01:26:17.410074 12146 log.cc:426] T 2c6176fb99ed4fb4a027584aeeb0a708 P 33e7b3c7fdd64255b209cc8123db9646: Rolled over to a new log segment at /tmp/kudutest-1000/raft_consensus-itest.RaftConsensusITest.TestCatchupAfterOpsEvicted.1487669166730301-8186/raft_consensus-itest-cluster/ts-2/data/wals/2c6176fb99ed4fb4a027584aeeb0a708/wal-000000002 I0221 01:26:17.426504 12148 log.cc:541] T 2c6176fb99ed4fb4a027584aeeb0a708 P c8ec78660e8642b4b82113d62a3042cb: Max segment size reached. Starting new segment allocation I0221 01:26:17.426775 12148 log.cc:426] T 2c6176fb99ed4fb4a027584aeeb0a708 P c8ec78660e8642b4b82113d62a3042cb: Rolled over to a new log segment at /tmp/kudutest-1000/raft_consensus-itest.RaftConsensusITest.TestCatchupAfterOpsEvicted.1487669166730301-8186/raft_consensus-itest-cluster/ts-1/data/wals/2c6176fb99ed4fb4a027584aeeb0a708/wal-000000003 I0221 01:26:17.427160 12146 log.cc:541] T 2c6176fb99ed4fb4a027584aeeb0a708 P 33e7b3c7fdd64255b209cc8123db9646: Max segment size reached. Starting new segment allocation I0221 01:26:17.427434 12146 log.cc:426] T 2c6176fb99ed4fb4a027584aeeb0a708 P 33e7b3c7fdd64255b209cc8123db9646: Rolled over to a new log segment at /tmp/kudutest-1000/raft_consensus-itest.RaftConsensusITest.TestCatchupAfterOpsEvicted.1487669166730301-8186/raft_consensus-itest-cluster/ts-2/data/wals/2c6176fb99ed4fb4a027584aeeb0a708/wal-000000003 I0221 01:26:17.443451 12148 log.cc:541] T 2c6176fb99ed4fb4a027584aeeb0a708 P c8ec78660e8642b4b82113d62a3042cb: Max segment size reached. Starting new segment allocation I0221 01:26:17.443795 12148 log.cc:426] T 2c6176fb99ed4fb4a027584aeeb0a708 P c8ec78660e8642b4b82113d62a3042cb: Rolled over to a new log segment at /tmp/kudutest-1000/raft_consensus-itest.RaftConsensusITest.TestCatchupAfterOpsEvicted.1487669166730301-8186/raft_consensus-itest-cluster/ts-1/data/wals/2c6176fb99ed4fb4a027584aeeb0a708/wal-000000004 I0221 01:26:17.444093 12146 log.cc:541] T 2c6176fb99ed4fb4a027584aeeb0a708 P 33e7b3c7fdd64255b209cc8123db9646: Max segment size reached. Starting new segment allocation I0221 01:26:17.444396 12146 log.cc:426] T 2c6176fb99ed4fb4a027584aeeb0a708 P 33e7b3c7fdd64255b209cc8123db9646: Rolled over to a new log segment at /tmp/kudutest-1000/raft_consensus-itest.RaftConsensusITest.TestCatchupAfterOpsEvicted.1487669166730301-8186/raft_consensus-itest-cluster/ts-2/data/wals/2c6176fb99ed4fb4a027584aeeb0a708/wal-000000004 I0221 01:26:17.462019 12148 log.cc:541] T 2c6176fb99ed4fb4a027584aeeb0a708 P c8ec78660e8642b4b82113d62a3042cb: Max segment size reached. Starting new segment allocation I0221 01:26:17.462395 12148 log.cc:426] T 2c6176fb99ed4fb4a027584aeeb0a708 P c8ec78660e8642b4b82113d62a3042cb: Rolled over to a new log segment at /tmp/kudutest-1000/raft_consensus-itest.RaftConsensusITest.TestCatchupAfterOpsEvicted.1487669166730301-8186/raft_consensus-itest-cluster/ts-1/data/wals/2c6176fb99ed4fb4a027584aeeb0a708/wal-000000005 I0221 01:26:17.462716 12146 log.cc:541] T 2c6176fb99ed4fb4a027584aeeb0a708 P 33e7b3c7fdd64255b209cc8123db9646: Max segment size reached. Starting new segment allocation I0221 01:26:17.462975 12146 log.cc:426] T 2c6176fb99ed4fb4a027584aeeb0a708 P 33e7b3c7fdd64255b209cc8123db9646: Rolled over to a new log segment at /tmp/kudutest-1000/raft_consensus-itest.RaftConsensusITest.TestCatchupAfterOpsEvicted.1487669166730301-8186/raft_consensus-itest-cluster/ts-2/data/wals/2c6176fb99ed4fb4a027584aeeb0a708/wal-000000005 I0221 01:26:17.476363 12014 tablet.cc:933] T 2c6176fb99ed4fb4a027584aeeb0a708 P 33e7b3c7fdd64255b209cc8123db9646: Flush: entering stage 1 (old memrowset already frozen for inserts) I0221 01:26:17.476393 12014 compaction.cc:877] Selected 1 rowsets to compact: I0221 01:26:17.476404 12014 compaction.cc:880] memrowset(current size on disk: ~0 bytes) I0221 01:26:17.476408 12014 tablet.cc:935] T 2c6176fb99ed4fb4a027584aeeb0a708 P 33e7b3c7fdd64255b209cc8123db9646: Memstore in-memory size: 8257801 bytes I0221 01:26:17.476425 12014 tablet.cc:1190] T 2c6176fb99ed4fb4a027584aeeb0a708 P 33e7b3c7fdd64255b209cc8123db9646: Flush: entering phase 1 (flushing snapshot). Phase 1 snapshot: MvccSnapshot[committed={T|T < 6093492950936788992}] I0221 01:26:17.477262 12148 log.cc:541] T 2c6176fb99ed4fb4a027584aeeb0a708 P c8ec78660e8642b4b82113d62a3042cb: Max segment size reached. Starting new segment allocation *** Aborted at 1487669177 (unix time) try "date -d @1487669177" if you are using GNU date *** I0221 01:26:17.477612 12148 log.cc:426] T 2c6176fb99ed4fb4a027584aeeb0a708 P c8ec78660e8642b4b82113d62a3042cb: Rolled over to a new log segment at /tmp/kudutest-1000/raft_consensus-itest.RaftConsensusITest.TestCatchupAfterOpsEvicted.1487669166730301-8186/raft_consensus-itest-cluster/ts-1/data/wals/2c6176fb99ed4fb4a027584aeeb0a708/wal-000000006 I0221 01:26:17.478018 12146 log.cc:541] T 2c6176fb99ed4fb4a027584aeeb0a708 P 33e7b3c7fdd64255b209cc8123db9646: Max segment size reached. Starting new segment allocation PC: @ 0x0 (unknown) I0221 01:26:17.478278 12146 log.cc:426] T 2c6176fb99ed4fb4a027584aeeb0a708 P 33e7b3c7fdd64255b209cc8123db9646: Rolled over to a new log segment at /tmp/kudutest-1000/raft_consensus-itest.RaftConsensusITest.TestCatchupAfterOpsEvicted.1487669166730301-8186/raft_consensus-itest-cluster/ts-2/data/wals/2c6176fb99ed4fb4a027584aeeb0a708/wal-000000006 *** SIGSEGV (@0x0) received by PID 12009 (TID 0x7fe150a20700) from PID 0; stack trace: *** @ 0x7fe1551d7390 (unknown) at ??:0 @ 0x0 (unknown) @ 0x7fe154711f05 kudu::(anonymous namespace)::ShardedLRUCache::Insert() at ??:0 @ 0x7fe154772b5b kudu::internal::BaseDescriptor<>::InsertIntoCache() at ??:0 @ 0x7fe1547728d7 kudu::internal::Descriptor<>::ReopenFileIfNecessary() at ??:0 @ 0x7fe154772540 kudu::internal::Descriptor<>::InitOnce() at ??:0 @ 0x7fe154772620 kudu::internal::InitCb<>() at ??:0 @ 0x7fe154294b0b GoogleOnceInternalInit() at ??:0 @ 0x7fe157f37069 GoogleOnceDynamic::Init<>() at ??:0 @ 0x7fe1547724f3 kudu::KuduOnceDynamic::Init<>() at ??:0 @ 0x7fe15476ec55 kudu::internal::Descriptor<>::Init() at ??:0 @ 0x7fe15476e4d9 kudu::FileCache<>::OpenExistingFile() at ??:0 @ 0x7fe154a3b26b kudu::fs::internal::LogBlockContainer::Create() at ??:0 @ 0x7fe154a4753b kudu::fs::LogBlockManager::GetOrCreateContainer() at ??:0 @ 0x7fe154a46ead kudu::fs::LogBlockManager::CreateBlock() at ??:0 @ 0x7fe154a478ce kudu::fs::LogBlockManager::CreateBlock() at ??:0 @ 0x7fe154a325ea kudu::FsManager::CreateNewBlock() at ??:0 @ 0x7fe157ee3090 kudu::tablet::MultiColumnWriter::Open() at ??:0 @ 0x7fe157eb861c kudu::tablet::DiskRowSetWriter::Open() at ??:0 @ 0x7fe157ebb936 kudu::tablet::RollingDiskRowSetWriter::RollWriter() at ??:0 @ 0x7fe157ebb4cf kudu::tablet::RollingDiskRowSetWriter::Open() at ??:0 @ 0x7fe157e024a2 kudu::tablet::Tablet::DoMergeCompactionOrFlush() at ??:0 @ 0x7fe157e019aa kudu::tablet::Tablet::FlushInternal() at ??:0 @ 0x7fe157e00bc6 kudu::tablet::Tablet::FlushUnlocked() at ??:0 @ 0x7fe157e5a0a1 kudu::tablet::FlushMRSOp::Perform() at ??:0 @ 0x7fe1547a541a kudu::MaintenanceManager::LaunchOp() at ??:0 @ 0x7fe1547abee1 boost::_mfi::mf1<>::operator()() at ??:0 @ 0x7fe1547abe3f boost::_bi::list2<>::operator()<>() at ??:0 @ 0x7fe1547abdca boost::_bi::bind_t<>::operator()() at ??:0 @ 0x7fe1547abb50 boost::detail::function::void_function_obj_invoker0<>::invoke() at ??:0 @ 0x7fe1577d0888 boost::function0<>::operator()() at ??:0 @ 0x7fe1548250f9 kudu::FunctionRunnable::Run() at ??:0 I0221 01:26:17.504663 11878 tablet.cc:933] T 2c6176fb99ed4fb4a027584aeeb0a708 P c8ec78660e8642b4b82113d62a3042cb: Flush: entering stage 1 (old memrowset already frozen for inserts) I0221 01:26:17.504683 11878 compaction.cc:877] Selected 1 rowsets to compact: I0221 01:26:17.504693 11878 compaction.cc:880] memrowset(current size on disk: ~0 bytes) I0221 01:26:17.504696 11878 tablet.cc:935] T 2c6176fb99ed4fb4a027584aeeb0a708 P c8ec78660e8642b4b82113d62a3042cb: Memstore in-memory size: 8257801 bytes I0221 01:26:17.504708 11878 tablet.cc:1190] T 2c6176fb99ed4fb4a027584aeeb0a708 P c8ec78660e8642b4b82113d62a3042cb: Flush: entering phase 1 (flushing snapshot). Phase 1 snapshot: MvccSnapshot[committed={T|T < 6093492950989471744 or (T in {6093492950989471744})}] *** Aborted at 1487669177 (unix time) try "date -d @1487669177" if you are using GNU date *** PC: @ 0x0 (unknown) *** SIGSEGV (@0x0) received by PID 11873 (TID 0x7f789ffff700) from PID 0; stack trace: *** @ 0x7f78b0f6f390 (unknown) at ??:0 @ 0x0 (unknown) @ 0x7f78b04a9f05 kudu::(anonymous namespace)::ShardedLRUCache::Insert() at ??:0 @ 0x7f78b050ab5b kudu::internal::BaseDescriptor<>::InsertIntoCache() at ??:0 @ 0x7f78b050a8d7 kudu::internal::Descriptor<>::ReopenFileIfNecessary() at ??:0 @ 0x7f78b050a540 kudu::internal::Descriptor<>::InitOnce() at ??:0 @ 0x7f78b050a620 kudu::internal::InitCb<>() at ??:0 @ 0x7f78b002cb0b GoogleOnceInternalInit() at ??:0 @ 0x7f78b3ccf069 GoogleOnceDynamic::Init<>() at ??:0 @ 0x7f78b050a4f3 kudu::KuduOnceDynamic::Init<>() at ??:0 @ 0x7f78b0506c55 kudu::internal::Descriptor<>::Init() at ??:0 @ 0x7f78b05064d9 kudu::FileCache<>::OpenExistingFile() at ??:0 @ 0x7f78b07d326b kudu::fs::internal::LogBlockContainer::Create() at ??:0 @ 0x7f78b07df53b kudu::fs::LogBlockManager::GetOrCreateContainer() at ??:0 @ 0x7f78b07deead kudu::fs::LogBlockManager::CreateBlock() at ??:0 @ 0x7f78b07df8ce kudu::fs::LogBlockManager::CreateBlock() at ??:0 @ 0x7f78b07ca5ea kudu::FsManager::CreateNewBlock() at ??:0 @ 0x7f78b3c7b090 kudu::tablet::MultiColumnWriter::Open() at ??:0 @ 0x7f78b3c5061c kudu::tablet::DiskRowSetWriter::Open() at ??:0 @ 0x7f78b3c53936 kudu::tablet::RollingDiskRowSetWriter::RollWriter() at ??:0 @ 0x7f78b3c534cf kudu::tablet::RollingDiskRowSetWriter::Open() at ??:0 @ 0x7f78b3b9a4a2 kudu::tablet::Tablet::DoMergeCompactionOrFlush() at ??:0 @ 0x7f78b3b999aa kudu::tablet::Tablet::FlushInternal() at ??:0 @ 0x7f78b3b98bc6 kudu::tablet::Tablet::FlushUnlocked() at ??:0 @ 0x7f78b3bf20a1 kudu::tablet::FlushMRSOp::Perform() at ??:0 @ 0x7f78b053d41a kudu::MaintenanceManager::LaunchOp() at ??:0 @ 0x7f78b0543ee1 boost::_mfi::mf1<>::operator()() at ??:0 @ 0x7f78b0543e3f boost::_bi::list2<>::operator()<>() at ??:0 @ 0x7f78b0543dca boost::_bi::bind_t<>::operator()() at ??:0 @ 0x7f78b0543b50 boost::detail::function::void_function_obj_invoker0<>::invoke() at ??:0 @ 0x7f78b3568888 boost::function0<>::operator()() at ??:0 @ 0x7f78b05bd0f9 kudu::FunctionRunnable::Run() at ??:0 W0221 01:26:17.658171 11676 connection.cc:466] client connection to 127.31.250.1:40370 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0221 01:26:17.658171 11674 connection.cc:466] client connection to 127.31.250.1:40370 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0221 01:26:17.658215 11686 connection.cc:466] server connection from 127.31.250.1:36151 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0221 01:26:17.658515 11687 connection.cc:466] client connection to 127.31.250.1:40370 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0221 01:26:18.490001 11759 connection.cc:466] server connection from 127.31.250.1:39428 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) I0221 01:26:18.490190 11826 raft_consensus.cc:918] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Refusing update from remote peer c8ec78660e8642b4b82113d62a3042cb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) W0221 01:26:18.490386 11759 connection.cc:380] Connection torn down before Call kudu.consensus.ConsensusService.UpdateConsensus from 127.31.250.1:39428 (request call id 3) could send its response Connected to the Master WARNING: Unable to connect to Tablet Server c8ec78660e8642b4b82113d62a3042cb (127.31.250.1:40370): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.1:40370: connect: Connection refused (error 111) W0221 01:26:18.580319 11685 connection.cc:466] server connection from 127.31.250.2:50990 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0221 01:26:18.580319 11676 connection.cc:466] client connection to 127.31.250.2:46079 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0221 01:26:18.580319 11686 connection.cc:466] client connection to 127.31.250.2:46079 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0221 01:26:18.581043 12276 negotiation.cc:250] Failed RPC negotiation. Trace: 0221 01:26:18.500986 (+ 0us) reactor.cc:373] Submitting negotiation task for client connection to 127.31.250.2:46079 0221 01:26:18.501143 (+ 157us) negotiation.cc:82] Waiting for socket to connect 0221 01:26:18.501147 (+ 4us) client_negotiation.cc:152] Beginning negotiation 0221 01:26:18.501169 (+ 22us) client_negotiation.cc:227] Sending NEGOTIATE NegotiatePB request 0221 01:26:18.580357 (+ 79188us) negotiation.cc:241] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.250.2:46079: BlockingRecv error: recv error: Connection reset by peer (error 104) Metrics: {"negotiator.queue_time_us":117,"thread_start_us":91,"threads_started":1} WARNING: Unable to connect to Tablet Server 33e7b3c7fdd64255b209cc8123db9646 (127.31.250.2:46079): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.2:46079: BlockingRecv error: recv error: Connection reset by peer (error 104) WARNING: Fetched info from 1 Tablet Servers, 2 weren't reachable I0221 01:26:18.581506 8186 cluster_verifier.cc:72] Check not successful yet, sleeping and retrying: Network error: Not all Tablet Servers are reachable Connected to the Master WARNING: Unable to connect to Tablet Server c8ec78660e8642b4b82113d62a3042cb (127.31.250.1:40370): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.1:40370: connect: Connection refused (error 111) WARNING: Unable to connect to Tablet Server 33e7b3c7fdd64255b209cc8123db9646 (127.31.250.2:46079): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.2:46079: connect: Connection refused (error 111) WARNING: Fetched info from 1 Tablet Servers, 2 weren't reachable I0221 01:26:18.757513 8186 cluster_verifier.cc:72] Check not successful yet, sleeping and retrying: Network error: Not all Tablet Servers are reachable Connected to the Master WARNING: Unable to connect to Tablet Server 33e7b3c7fdd64255b209cc8123db9646 (127.31.250.2:46079): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.2:46079: connect: Connection refused (error 111) WARNING: Unable to connect to Tablet Server c8ec78660e8642b4b82113d62a3042cb (127.31.250.1:40370): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.1:40370: connect: Connection refused (error 111) WARNING: Fetched info from 1 Tablet Servers, 2 weren't reachable I0221 01:26:19.021263 8186 cluster_verifier.cc:72] Check not successful yet, sleeping and retrying: Network error: Not all Tablet Servers are reachable Connected to the Master WARNING: Unable to connect to Tablet Server c8ec78660e8642b4b82113d62a3042cb (127.31.250.1:40370): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.1:40370: connect: Connection refused (error 111) WARNING: Unable to connect to Tablet Server 33e7b3c7fdd64255b209cc8123db9646 (127.31.250.2:46079): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.2:46079: connect: Connection refused (error 111) WARNING: Fetched info from 1 Tablet Servers, 2 weren't reachable I0221 01:26:19.393502 8186 cluster_verifier.cc:72] Check not successful yet, sleeping and retrying: Network error: Not all Tablet Servers are reachable Connected to the Master WARNING: Unable to connect to Tablet Server c8ec78660e8642b4b82113d62a3042cb (127.31.250.1:40370): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.1:40370: connect: Connection refused (error 111) WARNING: Unable to connect to Tablet Server 33e7b3c7fdd64255b209cc8123db9646 (127.31.250.2:46079): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.2:46079: connect: Connection refused (error 111) WARNING: Fetched info from 1 Tablet Servers, 2 weren't reachable I0221 01:26:19.935421 8186 cluster_verifier.cc:72] Check not successful yet, sleeping and retrying: Network error: Not all Tablet Servers are reachable Connected to the Master WARNING: Unable to connect to Tablet Server c8ec78660e8642b4b82113d62a3042cb (127.31.250.1:40370): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.1:40370: connect: Connection refused (error 111) WARNING: Unable to connect to Tablet Server 33e7b3c7fdd64255b209cc8123db9646 (127.31.250.2:46079): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.2:46079: connect: Connection refused (error 111) WARNING: Fetched info from 1 Tablet Servers, 2 weren't reachable I0221 01:26:20.733366 8186 cluster_verifier.cc:72] Check not successful yet, sleeping and retrying: Network error: Not all Tablet Servers are reachable I0221 01:26:20.954998 12150 raft_consensus.cc:411] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0221 01:26:20.955090 12150 raft_consensus.cc:2162] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 2.08 s I0221 01:26:20.955130 12150 raft_consensus.cc:435] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8ec78660e8642b4b82113d62a3042cb" member_type: VOTER last_known_addr { host: "127.31.250.1" port: 40370 } } peers { permanent_uuid: "33e7b3c7fdd64255b209cc8123db9646" member_type: VOTER last_known_addr { host: "127.31.250.2" port: 46079 } } peers { permanent_uuid: "1eb837892f874e99a16858e7958a1356" member_type: VOTER last_known_addr { host: "127.31.250.0" port: 35707 } } I0221 01:26:20.956099 12150 leader_election.cc:216] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer c8ec78660e8642b4b82113d62a3042cb I0221 01:26:20.956292 12150 leader_election.cc:216] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 33e7b3c7fdd64255b209cc8123db9646 W0221 01:26:20.957094 11759 leader_election.cc:272] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c8ec78660e8642b4b82113d62a3042cb: Network error: Client connection negotiation failed: client connection to 127.31.250.1:40370: connect: Connection refused (error 111) W0221 01:26:20.957350 11758 leader_election.cc:272] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 33e7b3c7fdd64255b209cc8123db9646: Network error: Client connection negotiation failed: client connection to 127.31.250.2:46079: connect: Connection refused (error 111) I0221 01:26:20.957474 11758 leader_election.cc:243] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. I0221 01:26:20.957839 12397 raft_consensus.cc:2162] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.7 s I0221 01:26:20.957931 12397 raft_consensus.cc:1908] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority Connected to the Master WARNING: Unable to connect to Tablet Server c8ec78660e8642b4b82113d62a3042cb (127.31.250.1:40370): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.1:40370: connect: Connection refused (error 111) WARNING: Unable to connect to Tablet Server 33e7b3c7fdd64255b209cc8123db9646 (127.31.250.2:46079): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.2:46079: connect: Connection refused (error 111) WARNING: Fetched info from 1 Tablet Servers, 2 weren't reachable I0221 01:26:21.750715 8186 cluster_verifier.cc:72] Check not successful yet, sleeping and retrying: Network error: Not all Tablet Servers are reachable Connected to the Master WARNING: Unable to connect to Tablet Server c8ec78660e8642b4b82113d62a3042cb (127.31.250.1:40370): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.1:40370: connect: Connection refused (error 111) WARNING: Unable to connect to Tablet Server 33e7b3c7fdd64255b209cc8123db9646 (127.31.250.2:46079): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.2:46079: connect: Connection refused (error 111) WARNING: Fetched info from 1 Tablet Servers, 2 weren't reachable I0221 01:26:22.767665 8186 cluster_verifier.cc:72] Check not successful yet, sleeping and retrying: Network error: Not all Tablet Servers are reachable Connected to the Master WARNING: Unable to connect to Tablet Server c8ec78660e8642b4b82113d62a3042cb (127.31.250.1:40370): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.1:40370: connect: Connection refused (error 111) WARNING: Unable to connect to Tablet Server 33e7b3c7fdd64255b209cc8123db9646 (127.31.250.2:46079): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.2:46079: connect: Connection refused (error 111) WARNING: Fetched info from 1 Tablet Servers, 2 weren't reachable I0221 01:26:23.809671 8186 cluster_verifier.cc:72] Check not successful yet, sleeping and retrying: Network error: Not all Tablet Servers are reachable I0221 01:26:24.310782 12150 raft_consensus.cc:411] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0221 01:26:24.310871 12150 raft_consensus.cc:2162] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 2.5 s I0221 01:26:24.310914 12150 raft_consensus.cc:435] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8ec78660e8642b4b82113d62a3042cb" member_type: VOTER last_known_addr { host: "127.31.250.1" port: 40370 } } peers { permanent_uuid: "33e7b3c7fdd64255b209cc8123db9646" member_type: VOTER last_known_addr { host: "127.31.250.2" port: 46079 } } peers { permanent_uuid: "1eb837892f874e99a16858e7958a1356" member_type: VOTER last_known_addr { host: "127.31.250.0" port: 35707 } } I0221 01:26:24.311477 12150 leader_election.cc:216] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer c8ec78660e8642b4b82113d62a3042cb I0221 01:26:24.311640 12150 leader_election.cc:216] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 33e7b3c7fdd64255b209cc8123db9646 W0221 01:26:24.312578 11759 leader_election.cc:272] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c8ec78660e8642b4b82113d62a3042cb: Network error: Client connection negotiation failed: client connection to 127.31.250.1:40370: connect: Connection refused (error 111) W0221 01:26:24.312799 11758 leader_election.cc:272] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 33e7b3c7fdd64255b209cc8123db9646: Network error: Client connection negotiation failed: client connection to 127.31.250.2:46079: connect: Connection refused (error 111) I0221 01:26:24.312875 11758 leader_election.cc:243] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. I0221 01:26:24.313177 12481 raft_consensus.cc:2162] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 2.17 s I0221 01:26:24.313261 12481 raft_consensus.cc:1908] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority Connected to the Master WARNING: Unable to connect to Tablet Server c8ec78660e8642b4b82113d62a3042cb (127.31.250.1:40370): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.1:40370: connect: Connection refused (error 111) WARNING: Unable to connect to Tablet Server 33e7b3c7fdd64255b209cc8123db9646 (127.31.250.2:46079): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.2:46079: connect: Connection refused (error 111) WARNING: Fetched info from 1 Tablet Servers, 2 weren't reachable I0221 01:26:24.852035 8186 cluster_verifier.cc:72] Check not successful yet, sleeping and retrying: Network error: Not all Tablet Servers are reachable Connected to the Master WARNING: Unable to connect to Tablet Server c8ec78660e8642b4b82113d62a3042cb (127.31.250.1:40370): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.1:40370: connect: Connection refused (error 111) WARNING: Unable to connect to Tablet Server 33e7b3c7fdd64255b209cc8123db9646 (127.31.250.2:46079): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.2:46079: connect: Connection refused (error 111) WARNING: Fetched info from 1 Tablet Servers, 2 weren't reachable I0221 01:26:25.873232 8186 cluster_verifier.cc:72] Check not successful yet, sleeping and retrying: Network error: Not all Tablet Servers are reachable Connected to the Master WARNING: Unable to connect to Tablet Server c8ec78660e8642b4b82113d62a3042cb (127.31.250.1:40370): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.1:40370: connect: Connection refused (error 111) WARNING: Unable to connect to Tablet Server 33e7b3c7fdd64255b209cc8123db9646 (127.31.250.2:46079): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.2:46079: connect: Connection refused (error 111) WARNING: Fetched info from 1 Tablet Servers, 2 weren't reachable I0221 01:26:26.889463 8186 cluster_verifier.cc:72] Check not successful yet, sleeping and retrying: Network error: Not all Tablet Servers are reachable Connected to the Master WARNING: Unable to connect to Tablet Server c8ec78660e8642b4b82113d62a3042cb (127.31.250.1:40370): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.1:40370: connect: Connection refused (error 111) WARNING: Unable to connect to Tablet Server 33e7b3c7fdd64255b209cc8123db9646 (127.31.250.2:46079): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.2:46079: connect: Connection refused (error 111) WARNING: Fetched info from 1 Tablet Servers, 2 weren't reachable I0221 01:26:27.931341 8186 cluster_verifier.cc:72] Check not successful yet, sleeping and retrying: Network error: Not all Tablet Servers are reachable I0221 01:26:28.559437 12150 raft_consensus.cc:411] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0221 01:26:28.559530 12150 raft_consensus.cc:2162] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.96 s I0221 01:26:28.559569 12150 raft_consensus.cc:435] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8ec78660e8642b4b82113d62a3042cb" member_type: VOTER last_known_addr { host: "127.31.250.1" port: 40370 } } peers { permanent_uuid: "33e7b3c7fdd64255b209cc8123db9646" member_type: VOTER last_known_addr { host: "127.31.250.2" port: 46079 } } peers { permanent_uuid: "1eb837892f874e99a16858e7958a1356" member_type: VOTER last_known_addr { host: "127.31.250.0" port: 35707 } } I0221 01:26:28.560070 12150 leader_election.cc:216] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer c8ec78660e8642b4b82113d62a3042cb I0221 01:26:28.560241 12150 leader_election.cc:216] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 33e7b3c7fdd64255b209cc8123db9646 W0221 01:26:28.561157 11759 leader_election.cc:272] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c8ec78660e8642b4b82113d62a3042cb: Network error: Client connection negotiation failed: client connection to 127.31.250.1:40370: connect: Connection refused (error 111) W0221 01:26:28.561358 11758 leader_election.cc:272] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 33e7b3c7fdd64255b209cc8123db9646: Network error: Client connection negotiation failed: client connection to 127.31.250.2:46079: connect: Connection refused (error 111) I0221 01:26:28.561434 11758 leader_election.cc:243] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. I0221 01:26:28.561733 12665 raft_consensus.cc:2162] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.57 s I0221 01:26:28.561831 12665 raft_consensus.cc:1908] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority Connected to the Master WARNING: Unable to connect to Tablet Server 33e7b3c7fdd64255b209cc8123db9646 (127.31.250.2:46079): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.2:46079: connect: Connection refused (error 111) WARNING: Unable to connect to Tablet Server c8ec78660e8642b4b82113d62a3042cb (127.31.250.1:40370): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.1:40370: connect: Connection refused (error 111) WARNING: Fetched info from 1 Tablet Servers, 2 weren't reachable I0221 01:26:28.973217 8186 cluster_verifier.cc:72] Check not successful yet, sleeping and retrying: Network error: Not all Tablet Servers are reachable Connected to the Master WARNING: Unable to connect to Tablet Server c8ec78660e8642b4b82113d62a3042cb (127.31.250.1:40370): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.1:40370: connect: Connection refused (error 111) WARNING: Unable to connect to Tablet Server 33e7b3c7fdd64255b209cc8123db9646 (127.31.250.2:46079): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.2:46079: connect: Connection refused (error 111) WARNING: Fetched info from 1 Tablet Servers, 2 weren't reachable I0221 01:26:30.013222 8186 cluster_verifier.cc:72] Check not successful yet, sleeping and retrying: Network error: Not all Tablet Servers are reachable Connected to the Master WARNING: Unable to connect to Tablet Server c8ec78660e8642b4b82113d62a3042cb (127.31.250.1:40370): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.1:40370: connect: Connection refused (error 111) WARNING: Unable to connect to Tablet Server 33e7b3c7fdd64255b209cc8123db9646 (127.31.250.2:46079): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.2:46079: connect: Connection refused (error 111) WARNING: Fetched info from 1 Tablet Servers, 2 weren't reachable I0221 01:26:31.055794 8186 cluster_verifier.cc:72] Check not successful yet, sleeping and retrying: Network error: Not all Tablet Servers are reachable I0221 01:26:31.712813 12150 raft_consensus.cc:411] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0221 01:26:31.712913 12150 raft_consensus.cc:2162] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.72 s I0221 01:26:31.712954 12150 raft_consensus.cc:435] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8ec78660e8642b4b82113d62a3042cb" member_type: VOTER last_known_addr { host: "127.31.250.1" port: 40370 } } peers { permanent_uuid: "33e7b3c7fdd64255b209cc8123db9646" member_type: VOTER last_known_addr { host: "127.31.250.2" port: 46079 } } peers { permanent_uuid: "1eb837892f874e99a16858e7958a1356" member_type: VOTER last_known_addr { host: "127.31.250.0" port: 35707 } } I0221 01:26:31.713436 12150 leader_election.cc:216] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer c8ec78660e8642b4b82113d62a3042cb I0221 01:26:31.713585 12150 leader_election.cc:216] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 33e7b3c7fdd64255b209cc8123db9646 W0221 01:26:31.714831 11759 leader_election.cc:272] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c8ec78660e8642b4b82113d62a3042cb: Network error: Client connection negotiation failed: client connection to 127.31.250.1:40370: connect: Connection refused (error 111) W0221 01:26:31.714967 11758 leader_election.cc:272] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 33e7b3c7fdd64255b209cc8123db9646: Network error: Client connection negotiation failed: client connection to 127.31.250.2:46079: connect: Connection refused (error 111) I0221 01:26:31.715049 11758 leader_election.cc:243] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. I0221 01:26:31.715462 12748 raft_consensus.cc:2162] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 5.15 s I0221 01:26:31.715569 12748 raft_consensus.cc:1908] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority Connected to the Master WARNING: Unable to connect to Tablet Server c8ec78660e8642b4b82113d62a3042cb (127.31.250.1:40370): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.1:40370: connect: Connection refused (error 111) WARNING: Unable to connect to Tablet Server 33e7b3c7fdd64255b209cc8123db9646 (127.31.250.2:46079): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.2:46079: connect: Connection refused (error 111) WARNING: Fetched info from 1 Tablet Servers, 2 weren't reachable I0221 01:26:32.098668 8186 cluster_verifier.cc:72] Check not successful yet, sleeping and retrying: Network error: Not all Tablet Servers are reachable Connected to the Master WARNING: WARNING: Unable to connect to Tablet Server c8ec78660e8642b4b82113d62a3042cb (127.31.250.1:40370): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.1:40370: connect: Connection refused (error 111) Unable to connect to Tablet Server 33e7b3c7fdd64255b209cc8123db9646 (127.31.250.2:46079): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.2:46079: connect: Connection refused (error 111) WARNING: Fetched info from 1 Tablet Servers, 2 weren't reachable I0221 01:26:33.138648 8186 cluster_verifier.cc:72] Check not successful yet, sleeping and retrying: Network error: Not all Tablet Servers are reachable Connected to the Master WARNING: WARNING: Unable to connect to Tablet Server c8ec78660e8642b4b82113d62a3042cb (127.31.250.1:40370): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.1:40370: connect: Connection refused (error 111) Unable to connect to Tablet Server 33e7b3c7fdd64255b209cc8123db9646 (127.31.250.2:46079): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.2:46079: connect: Connection refused (error 111) WARNING: Fetched info from 1 Tablet Servers, 2 weren't reachable I0221 01:26:34.179771 8186 cluster_verifier.cc:72] Check not successful yet, sleeping and retrying: Network error: Not all Tablet Servers are reachable Connected to the Master WARNING: WARNING: Unable to connect to Tablet Server c8ec78660e8642b4b82113d62a3042cb (127.31.250.1:40370): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.1:40370: connect: Connection refused (error 111)Unable to connect to Tablet Server 33e7b3c7fdd64255b209cc8123db9646 (127.31.250.2:46079): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.2:46079: connect: Connection refused (error 111) WARNING: Fetched info from 1 Tablet Servers, 2 weren't reachable I0221 01:26:35.221237 8186 cluster_verifier.cc:72] Check not successful yet, sleeping and retrying: Network error: Not all Tablet Servers are reachable Connected to the Master WARNING: Unable to connect to Tablet Server c8ec78660e8642b4b82113d62a3042cb (127.31.250.1:40370): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.1:40370: connect: Connection refused (error 111) WARNING: Unable to connect to Tablet Server 33e7b3c7fdd64255b209cc8123db9646 (127.31.250.2:46079): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.2:46079: connect: Connection refused (error 111) WARNING: Fetched info from 1 Tablet Servers, 2 weren't reachable I0221 01:26:36.262089 8186 cluster_verifier.cc:72] Check not successful yet, sleeping and retrying: Network error: Not all Tablet Servers are reachable Connected to the Master WARNING: Unable to connect to Tablet Server c8ec78660e8642b4b82113d62a3042cb (127.31.250.1:40370): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.1:40370: connect: Connection refused (error 111) WARNING: Unable to connect to Tablet Server 33e7b3c7fdd64255b209cc8123db9646 (127.31.250.2:46079): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.2:46079: connect: Connection refused (error 111) WARNING: Fetched info from 1 Tablet Servers, 2 weren't reachable I0221 01:26:37.303316 8186 cluster_verifier.cc:72] Check not successful yet, sleeping and retrying: Network error: Not all Tablet Servers are reachable Connected to the Master WARNING: Unable to connect to Tablet Server c8ec78660e8642b4b82113d62a3042cb (127.31.250.1:40370): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.1:40370: connect: Connection refused (error 111) WARNING: Unable to connect to Tablet Server 33e7b3c7fdd64255b209cc8123db9646 (127.31.250.2:46079): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.2:46079: connect: Connection refused (error 111) WARNING: Fetched info from 1 Tablet Servers, 2 weren't reachable I0221 01:26:38.345381 8186 cluster_verifier.cc:72] Check not successful yet, sleeping and retrying: Network error: Not all Tablet Servers are reachable I0221 01:26:39.064232 12150 raft_consensus.cc:411] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0221 01:26:39.064318 12150 raft_consensus.cc:2162] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 6.8 s I0221 01:26:39.064359 12150 raft_consensus.cc:435] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8ec78660e8642b4b82113d62a3042cb" member_type: VOTER last_known_addr { host: "127.31.250.1" port: 40370 } } peers { permanent_uuid: "33e7b3c7fdd64255b209cc8123db9646" member_type: VOTER last_known_addr { host: "127.31.250.2" port: 46079 } } peers { permanent_uuid: "1eb837892f874e99a16858e7958a1356" member_type: VOTER last_known_addr { host: "127.31.250.0" port: 35707 } } I0221 01:26:39.064868 12150 leader_election.cc:216] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer c8ec78660e8642b4b82113d62a3042cb I0221 01:26:39.065024 12150 leader_election.cc:216] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 33e7b3c7fdd64255b209cc8123db9646 W0221 01:26:39.065933 11759 leader_election.cc:272] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c8ec78660e8642b4b82113d62a3042cb: Network error: Client connection negotiation failed: client connection to 127.31.250.1:40370: connect: Connection refused (error 111) W0221 01:26:39.066243 11758 leader_election.cc:272] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 33e7b3c7fdd64255b209cc8123db9646: Network error: Client connection negotiation failed: client connection to 127.31.250.2:46079: connect: Connection refused (error 111) I0221 01:26:39.066326 11758 leader_election.cc:243] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. I0221 01:26:39.066642 12962 raft_consensus.cc:2162] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 9.69 s I0221 01:26:39.066728 12962 raft_consensus.cc:1908] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority Connected to the Master WARNING: Unable to connect to Tablet Server c8ec78660e8642b4b82113d62a3042cb (127.31.250.1:40370): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.1:40370: connect: Connection refused (error 111) WARNING: Unable to connect to Tablet Server 33e7b3c7fdd64255b209cc8123db9646 (127.31.250.2:46079): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.2:46079: connect: Connection refused (error 111) WARNING: Fetched info from 1 Tablet Servers, 2 weren't reachable I0221 01:26:39.386509 8186 cluster_verifier.cc:72] Check not successful yet, sleeping and retrying: Network error: Not all Tablet Servers are reachable Connected to the Master WARNING: Unable to connect to Tablet Server c8ec78660e8642b4b82113d62a3042cb (127.31.250.1:40370): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.1:40370: connect: Connection refused (error 111) WARNING: Unable to connect to Tablet Server 33e7b3c7fdd64255b209cc8123db9646 (127.31.250.2:46079): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.2:46079: connect: Connection refused (error 111) WARNING: Fetched info from 1 Tablet Servers, 2 weren't reachable I0221 01:26:40.404422 8186 cluster_verifier.cc:72] Check not successful yet, sleeping and retrying: Network error: Not all Tablet Servers are reachable Connected to the Master WARNING: Unable to connect to Tablet Server c8ec78660e8642b4b82113d62a3042cb (127.31.250.1:40370): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.1:40370: connect: Connection refused (error 111) WARNING: Unable to connect to Tablet Server 33e7b3c7fdd64255b209cc8123db9646 (127.31.250.2:46079): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.2:46079: connect: Connection refused (error 111) WARNING: Fetched info from 1 Tablet Servers, 2 weren't reachable I0221 01:26:41.445209 8186 cluster_verifier.cc:72] Check not successful yet, sleeping and retrying: Network error: Not all Tablet Servers are reachable Connected to the Master WARNING: Unable to connect to Tablet Server c8ec78660e8642b4b82113d62a3042cb (127.31.250.1:40370): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.1:40370: connect: Connection refused (error 111) WARNING: Unable to connect to Tablet Server 33e7b3c7fdd64255b209cc8123db9646 (127.31.250.2:46079): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.2:46079: connect: Connection refused (error 111) WARNING: Fetched info from 1 Tablet Servers, 2 weren't reachable I0221 01:26:42.461343 8186 cluster_verifier.cc:72] Check not successful yet, sleeping and retrying: Network error: Not all Tablet Servers are reachable Connected to the Master WARNING: Unable to connect to Tablet Server c8ec78660e8642b4b82113d62a3042cb (127.31.250.1:40370): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.1:40370: connect: Connection refused (error 111) WARNING: Unable to connect to Tablet Server 33e7b3c7fdd64255b209cc8123db9646 (127.31.250.2:46079): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.2:46079: connect: Connection refused (error 111) WARNING: Fetched info from 1 Tablet Servers, 2 weren't reachable I0221 01:26:43.501631 8186 cluster_verifier.cc:72] Check not successful yet, sleeping and retrying: Network error: Not all Tablet Servers are reachable Connected to the Master WARNING: Unable to connect to Tablet Server c8ec78660e8642b4b82113d62a3042cb (127.31.250.1:40370): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.1:40370: connect: Connection refused (error 111) WARNING: Unable to connect to Tablet Server 33e7b3c7fdd64255b209cc8123db9646 (127.31.250.2:46079): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.2:46079: connect: Connection refused (error 111) WARNING: Fetched info from 1 Tablet Servers, 2 weren't reachable I0221 01:26:44.517161 8186 cluster_verifier.cc:72] Check not successful yet, sleeping and retrying: Network error: Not all Tablet Servers are reachable Connected to the Master WARNING: Unable to connect to Tablet Server 33e7b3c7fdd64255b209cc8123db9646 (127.31.250.2:46079): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.2:46079: connect: Connection refused (error 111)WARNING: Unable to connect to Tablet Server c8ec78660e8642b4b82113d62a3042cb (127.31.250.1:40370): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.1:40370: connect: Connection refused (error 111) WARNING: Fetched info from 1 Tablet Servers, 2 weren't reachable I0221 01:26:45.558616 8186 cluster_verifier.cc:72] Check not successful yet, sleeping and retrying: Network error: Not all Tablet Servers are reachable Connected to the Master WARNING: Unable to connect to Tablet Server c8ec78660e8642b4b82113d62a3042cb (127.31.250.1:40370): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.1:40370: connect: Connection refused (error 111) WARNING: Unable to connect to Tablet Server 33e7b3c7fdd64255b209cc8123db9646 (127.31.250.2:46079): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.2:46079: connect: Connection refused (error 111) WARNING: Fetched info from 1 Tablet Servers, 2 weren't reachable I0221 01:26:46.602468 8186 cluster_verifier.cc:72] Check not successful yet, sleeping and retrying: Network error: Not all Tablet Servers are reachable Connected to the Master WARNING: WARNING: Unable to connect to Tablet Server 33e7b3c7fdd64255b209cc8123db9646 (127.31.250.2:46079): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.2:46079: connect: Connection refused (error 111) Unable to connect to Tablet Server c8ec78660e8642b4b82113d62a3042cb (127.31.250.1:40370): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.1:40370: connect: Connection refused (error 111) WARNING: Fetched info from 1 Tablet Servers, 2 weren't reachable I0221 01:26:47.643805 8186 cluster_verifier.cc:72] Check not successful yet, sleeping and retrying: Network error: Not all Tablet Servers are reachable Connected to the Master WARNING: Unable to connect to Tablet Server c8ec78660e8642b4b82113d62a3042cb (127.31.250.1:40370): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.1:40370: connect: Connection refused (error 111) WARNING: Unable to connect to Tablet Server 33e7b3c7fdd64255b209cc8123db9646 (127.31.250.2:46079): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.2:46079: connect: Connection refused (error 111) WARNING: Fetched info from 1 Tablet Servers, 2 weren't reachable I0221 01:26:48.686806 8186 cluster_verifier.cc:72] Check not successful yet, sleeping and retrying: Network error: Not all Tablet Servers are reachable Connected to the Master WARNING: Unable to connect to Tablet Server c8ec78660e8642b4b82113d62a3042cb (127.31.250.1:40370): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.1:40370: connect: Connection refused (error 111) WARNING: Unable to connect to Tablet Server 33e7b3c7fdd64255b209cc8123db9646 (127.31.250.2:46079): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.2:46079: connect: Connection refused (error 111) WARNING: Fetched info from 1 Tablet Servers, 2 weren't reachable I0221 01:26:49.727731 8186 cluster_verifier.cc:72] Check not successful yet, sleeping and retrying: Network error: Not all Tablet Servers are reachable Connected to the Master WARNING: Unable to connect to Tablet Server c8ec78660e8642b4b82113d62a3042cb (127.31.250.1:40370): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.1:40370: connect: Connection refused (error 111) WARNING: Unable to connect to Tablet Server 33e7b3c7fdd64255b209cc8123db9646 (127.31.250.2:46079): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.2:46079: connect: Connection refused (error 111) WARNING: Fetched info from 1 Tablet Servers, 2 weren't reachable I0221 01:26:50.753259 8186 cluster_verifier.cc:72] Check not successful yet, sleeping and retrying: Network error: Not all Tablet Servers are reachable I0221 01:26:50.898278 12150 raft_consensus.cc:411] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0221 01:26:50.898391 12150 raft_consensus.cc:2162] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 7.71 s I0221 01:26:50.898454 12150 raft_consensus.cc:435] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8ec78660e8642b4b82113d62a3042cb" member_type: VOTER last_known_addr { host: "127.31.250.1" port: 40370 } } peers { permanent_uuid: "33e7b3c7fdd64255b209cc8123db9646" member_type: VOTER last_known_addr { host: "127.31.250.2" port: 46079 } } peers { permanent_uuid: "1eb837892f874e99a16858e7958a1356" member_type: VOTER last_known_addr { host: "127.31.250.0" port: 35707 } } I0221 01:26:50.898947 12150 leader_election.cc:216] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer c8ec78660e8642b4b82113d62a3042cb I0221 01:26:50.899114 12150 leader_election.cc:216] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 33e7b3c7fdd64255b209cc8123db9646 W0221 01:26:50.900163 11759 leader_election.cc:272] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c8ec78660e8642b4b82113d62a3042cb: Network error: Client connection negotiation failed: client connection to 127.31.250.1:40370: connect: Connection refused (error 111) W0221 01:26:50.900266 11758 leader_election.cc:272] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 33e7b3c7fdd64255b209cc8123db9646: Network error: Client connection negotiation failed: client connection to 127.31.250.2:46079: connect: Connection refused (error 111) I0221 01:26:50.900352 11758 leader_election.cc:243] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. I0221 01:26:50.900660 13316 raft_consensus.cc:2162] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 6.22 s I0221 01:26:50.900750 13316 raft_consensus.cc:1908] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority Connected to the Master WARNING: Unable to connect to Tablet Server c8ec78660e8642b4b82113d62a3042cb (127.31.250.1:40370): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.1:40370: connect: Connection refused (error 111) WARNING: Unable to connect to Tablet Server 33e7b3c7fdd64255b209cc8123db9646 (127.31.250.2:46079): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.2:46079: connect: Connection refused (error 111) WARNING: Fetched info from 1 Tablet Servers, 2 weren't reachable I0221 01:26:51.797313 8186 cluster_verifier.cc:72] Check not successful yet, sleeping and retrying: Network error: Not all Tablet Servers are reachable Connected to the Master WARNING: WARNING: Unable to connect to Tablet Server c8ec78660e8642b4b82113d62a3042cb (127.31.250.1:40370): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.1:40370: connect: Connection refused (error 111) Unable to connect to Tablet Server 33e7b3c7fdd64255b209cc8123db9646 (127.31.250.2:46079): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.2:46079: connect: Connection refused (error 111) WARNING: Fetched info from 1 Tablet Servers, 2 weren't reachable I0221 01:26:52.839295 8186 cluster_verifier.cc:72] Check not successful yet, sleeping and retrying: Network error: Not all Tablet Servers are reachable Connected to the Master WARNING: Unable to connect to Tablet Server c8ec78660e8642b4b82113d62a3042cb (127.31.250.1:40370): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.1:40370: connect: Connection refused (error 111) WARNING: Unable to connect to Tablet Server 33e7b3c7fdd64255b209cc8123db9646 (127.31.250.2:46079): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.2:46079: connect: Connection refused (error 111) WARNING: Fetched info from 1 Tablet Servers, 2 weren't reachable I0221 01:26:53.859352 8186 cluster_verifier.cc:72] Check not successful yet, sleeping and retrying: Network error: Not all Tablet Servers are reachable Connected to the Master WARNING: Unable to connect to Tablet Server c8ec78660e8642b4b82113d62a3042cb (127.31.250.1:40370): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.1:40370: connect: Connection refused (error 111) WARNING: Unable to connect to Tablet Server 33e7b3c7fdd64255b209cc8123db9646 (127.31.250.2:46079): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.2:46079: connect: Connection refused (error 111) WARNING: Fetched info from 1 Tablet Servers, 2 weren't reachable I0221 01:26:54.901257 8186 cluster_verifier.cc:72] Check not successful yet, sleeping and retrying: Network error: Not all Tablet Servers are reachable Connected to the Master WARNING: WARNING: Unable to connect to Tablet Server c8ec78660e8642b4b82113d62a3042cb (127.31.250.1:40370): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.1:40370: connect: Connection refused (error 111)Unable to connect to Tablet Server 33e7b3c7fdd64255b209cc8123db9646 (127.31.250.2:46079): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.2:46079: connect: Connection refused (error 111) WARNING: Fetched info from 1 Tablet Servers, 2 weren't reachable I0221 01:26:55.942996 8186 cluster_verifier.cc:72] Check not successful yet, sleeping and retrying: Network error: Not all Tablet Servers are reachable Connected to the Master WARNING: Unable to connect to Tablet Server c8ec78660e8642b4b82113d62a3042cb (127.31.250.1:40370): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.1:40370: connect: Connection refused (error 111) WARNING: Unable to connect to Tablet Server 33e7b3c7fdd64255b209cc8123db9646 (127.31.250.2:46079): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.2:46079: connect: Connection refused (error 111) WARNING: Fetched info from 1 Tablet Servers, 2 weren't reachable I0221 01:26:56.958652 8186 cluster_verifier.cc:72] Check not successful yet, sleeping and retrying: Network error: Not all Tablet Servers are reachable Connected to the Master WARNING: Unable to connect to Tablet Server 33e7b3c7fdd64255b209cc8123db9646 (127.31.250.2:46079): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.2:46079: connect: Connection refused (error 111) WARNING: Unable to connect to Tablet Server c8ec78660e8642b4b82113d62a3042cb (127.31.250.1:40370): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.1:40370: connect: Connection refused (error 111) WARNING: Fetched info from 1 Tablet Servers, 2 weren't reachable I0221 01:26:58.003196 8186 cluster_verifier.cc:72] Check not successful yet, sleeping and retrying: Network error: Not all Tablet Servers are reachable I0221 01:26:58.890451 12150 raft_consensus.cc:411] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0221 01:26:58.890537 12150 raft_consensus.cc:2162] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 9.46 s I0221 01:26:58.890576 12150 raft_consensus.cc:435] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8ec78660e8642b4b82113d62a3042cb" member_type: VOTER last_known_addr { host: "127.31.250.1" port: 40370 } } peers { permanent_uuid: "33e7b3c7fdd64255b209cc8123db9646" member_type: VOTER last_known_addr { host: "127.31.250.2" port: 46079 } } peers { permanent_uuid: "1eb837892f874e99a16858e7958a1356" member_type: VOTER last_known_addr { host: "127.31.250.0" port: 35707 } } I0221 01:26:58.891065 12150 leader_election.cc:216] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer c8ec78660e8642b4b82113d62a3042cb I0221 01:26:58.891214 12150 leader_election.cc:216] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 33e7b3c7fdd64255b209cc8123db9646 W0221 01:26:58.892316 11758 leader_election.cc:272] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 33e7b3c7fdd64255b209cc8123db9646: Network error: Client connection negotiation failed: client connection to 127.31.250.2:46079: connect: Connection refused (error 111) W0221 01:26:58.892419 11759 leader_election.cc:272] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c8ec78660e8642b4b82113d62a3042cb: Network error: Client connection negotiation failed: client connection to 127.31.250.1:40370: connect: Connection refused (error 111) I0221 01:26:58.892498 11759 leader_election.cc:243] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. I0221 01:26:58.892802 13562 raft_consensus.cc:2162] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 3.92 s I0221 01:26:58.892884 13562 raft_consensus.cc:1908] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority Connected to the Master WARNING: WARNING: Unable to connect to Tablet Server 33e7b3c7fdd64255b209cc8123db9646 (127.31.250.2:46079): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.2:46079: connect: Connection refused (error 111) Unable to connect to Tablet Server c8ec78660e8642b4b82113d62a3042cb (127.31.250.1:40370): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.1:40370: connect: Connection refused (error 111) WARNING: Fetched info from 1 Tablet Servers, 2 weren't reachable I0221 01:26:59.045128 8186 cluster_verifier.cc:72] Check not successful yet, sleeping and retrying: Network error: Not all Tablet Servers are reachable Connected to the Master WARNING: Unable to connect to Tablet Server 33e7b3c7fdd64255b209cc8123db9646 (127.31.250.2:46079): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.2:46079: connect: Connection refused (error 111) WARNING: Unable to connect to Tablet Server c8ec78660e8642b4b82113d62a3042cb (127.31.250.1:40370): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.1:40370: connect: Connection refused (error 111) WARNING: Fetched info from 1 Tablet Servers, 2 weren't reachable I0221 01:27:00.087200 8186 cluster_verifier.cc:72] Check not successful yet, sleeping and retrying: Network error: Not all Tablet Servers are reachable Connected to the Master WARNING: Unable to connect to Tablet Server c8ec78660e8642b4b82113d62a3042cb (127.31.250.1:40370): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.1:40370: connect: Connection refused (error 111) WARNING: Unable to connect to Tablet Server 33e7b3c7fdd64255b209cc8123db9646 (127.31.250.2:46079): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.2:46079: connect: Connection refused (error 111) WARNING: Fetched info from 1 Tablet Servers, 2 weren't reachable I0221 01:27:01.104801 8186 cluster_verifier.cc:72] Check not successful yet, sleeping and retrying: Network error: Not all Tablet Servers are reachable Connected to the Master WARNING: Unable to connect to Tablet Server c8ec78660e8642b4b82113d62a3042cb (127.31.250.1:40370): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.1:40370: connect: Connection refused (error 111) WARNING: Unable to connect to Tablet Server 33e7b3c7fdd64255b209cc8123db9646 (127.31.250.2:46079): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.2:46079: connect: Connection refused (error 111) WARNING: Fetched info from 1 Tablet Servers, 2 weren't reachable I0221 01:27:02.127480 8186 cluster_verifier.cc:72] Check not successful yet, sleeping and retrying: Network error: Not all Tablet Servers are reachable Connected to the Master WARNING: Unable to connect to Tablet Server c8ec78660e8642b4b82113d62a3042cb (127.31.250.1:40370): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.1:40370: connect: Connection refused (error 111) WARNING: Unable to connect to Tablet Server 33e7b3c7fdd64255b209cc8123db9646 (127.31.250.2:46079): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.2:46079: connect: Connection refused (error 111) WARNING: Fetched info from 1 Tablet Servers, 2 weren't reachable I0221 01:27:03.169867 8186 cluster_verifier.cc:72] Check not successful yet, sleeping and retrying: Network error: Not all Tablet Servers are reachable Connected to the Master WARNING: WARNING: Unable to connect to Tablet Server c8ec78660e8642b4b82113d62a3042cb (127.31.250.1:40370): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.1:40370: connect: Connection refused (error 111)Unable to connect to Tablet Server 33e7b3c7fdd64255b209cc8123db9646 (127.31.250.2:46079): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.2:46079: connect: Connection refused (error 111) WARNING: Fetched info from 1 Tablet Servers, 2 weren't reachable I0221 01:27:04.212004 8186 cluster_verifier.cc:72] Check not successful yet, sleeping and retrying: Network error: Not all Tablet Servers are reachable I0221 01:27:04.336443 12150 raft_consensus.cc:411] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0221 01:27:04.336527 12150 raft_consensus.cc:2162] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 12.2 s I0221 01:27:04.336570 12150 raft_consensus.cc:435] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8ec78660e8642b4b82113d62a3042cb" member_type: VOTER last_known_addr { host: "127.31.250.1" port: 40370 } } peers { permanent_uuid: "33e7b3c7fdd64255b209cc8123db9646" member_type: VOTER last_known_addr { host: "127.31.250.2" port: 46079 } } peers { permanent_uuid: "1eb837892f874e99a16858e7958a1356" member_type: VOTER last_known_addr { host: "127.31.250.0" port: 35707 } } I0221 01:27:04.337057 12150 leader_election.cc:216] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer c8ec78660e8642b4b82113d62a3042cb I0221 01:27:04.337206 12150 leader_election.cc:216] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 33e7b3c7fdd64255b209cc8123db9646 W0221 01:27:04.337898 11759 leader_election.cc:272] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c8ec78660e8642b4b82113d62a3042cb: Network error: Client connection negotiation failed: client connection to 127.31.250.1:40370: connect: Connection refused (error 111) W0221 01:27:04.338088 11758 leader_election.cc:272] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 33e7b3c7fdd64255b209cc8123db9646: Network error: Client connection negotiation failed: client connection to 127.31.250.2:46079: connect: Connection refused (error 111) I0221 01:27:04.338166 11758 leader_election.cc:243] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. I0221 01:27:04.338457 13730 raft_consensus.cc:2162] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 15.7 s I0221 01:27:04.338539 13730 raft_consensus.cc:1908] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority Connected to the Master WARNING: Unable to connect to Tablet Server c8ec78660e8642b4b82113d62a3042cb (127.31.250.1:40370): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.1:40370: connect: Connection refused (error 111) WARNING: Unable to connect to Tablet Server 33e7b3c7fdd64255b209cc8123db9646 (127.31.250.2:46079): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.2:46079: connect: Connection refused (error 111) WARNING: Fetched info from 1 Tablet Servers, 2 weren't reachable I0221 01:27:05.255689 8186 cluster_verifier.cc:72] Check not successful yet, sleeping and retrying: Network error: Not all Tablet Servers are reachable Connected to the Master WARNING: Unable to connect to Tablet Server c8ec78660e8642b4b82113d62a3042cb (127.31.250.1:40370): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.1:40370: connect: Connection refused (error 111) WARNING: Unable to connect to Tablet Server 33e7b3c7fdd64255b209cc8123db9646 (127.31.250.2:46079): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.2:46079: connect: Connection refused (error 111) WARNING: Fetched info from 1 Tablet Servers, 2 weren't reachable I0221 01:27:06.298413 8186 cluster_verifier.cc:72] Check not successful yet, sleeping and retrying: Network error: Not all Tablet Servers are reachable Connected to the Master WARNING: Unable to connect to Tablet Server c8ec78660e8642b4b82113d62a3042cb (127.31.250.1:40370): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.1:40370: connect: Connection refused (error 111) WARNING: Unable to connect to Tablet Server 33e7b3c7fdd64255b209cc8123db9646 (127.31.250.2:46079): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.2:46079: connect: Connection refused (error 111) WARNING: Fetched info from 1 Tablet Servers, 2 weren't reachable I0221 01:27:07.339364 8186 cluster_verifier.cc:72] Check not successful yet, sleeping and retrying: Network error: Not all Tablet Servers are reachable Connected to the Master WARNING: WARNING: Unable to connect to Tablet Server c8ec78660e8642b4b82113d62a3042cb (127.31.250.1:40370): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.1:40370: connect: Connection refused (error 111) Unable to connect to Tablet Server 33e7b3c7fdd64255b209cc8123db9646 (127.31.250.2:46079): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.2:46079: connect: Connection refused (error 111) WARNING: Fetched info from 1 Tablet Servers, 2 weren't reachable I0221 01:27:08.381494 8186 cluster_verifier.cc:72] Check not successful yet, sleeping and retrying: Network error: Not all Tablet Servers are reachable Connected to the Master WARNING: Unable to connect to Tablet Server c8ec78660e8642b4b82113d62a3042cb (127.31.250.1:40370): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.1:40370: connect: Connection refused (error 111) WARNING: Unable to connect to Tablet Server 33e7b3c7fdd64255b209cc8123db9646 (127.31.250.2:46079): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.2:46079: connect: Connection refused (error 111) WARNING: Fetched info from 1 Tablet Servers, 2 weren't reachable I0221 01:27:09.398035 8186 cluster_verifier.cc:72] Check not successful yet, sleeping and retrying: Network error: Not all Tablet Servers are reachable Connected to the Master WARNING: Unable to connect to Tablet Server 33e7b3c7fdd64255b209cc8123db9646 (127.31.250.2:46079): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.2:46079: connect: Connection refused (error 111) WARNING: Unable to connect to Tablet Server c8ec78660e8642b4b82113d62a3042cb (127.31.250.1:40370): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.1:40370: connect: Connection refused (error 111) WARNING: Fetched info from 1 Tablet Servers, 2 weren't reachable I0221 01:27:10.441366 8186 cluster_verifier.cc:72] Check not successful yet, sleeping and retrying: Network error: Not all Tablet Servers are reachable Connected to the Master WARNING: WARNING: Unable to connect to Tablet Server 33e7b3c7fdd64255b209cc8123db9646 (127.31.250.2:46079): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.2:46079: connect: Connection refused (error 111) Unable to connect to Tablet Server c8ec78660e8642b4b82113d62a3042cb (127.31.250.1:40370): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.1:40370: connect: Connection refused (error 111) WARNING: Fetched info from 1 Tablet Servers, 2 weren't reachable I0221 01:27:11.482172 8186 cluster_verifier.cc:72] Check not successful yet, sleeping and retrying: Network error: Not all Tablet Servers are reachable Connected to the Master WARNING: Unable to connect to Tablet Server c8ec78660e8642b4b82113d62a3042cb (127.31.250.1:40370): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.1:40370: connect: Connection refused (error 111) WARNING: Unable to connect to Tablet Server 33e7b3c7fdd64255b209cc8123db9646 (127.31.250.2:46079): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.2:46079: connect: Connection refused (error 111) WARNING: Fetched info from 1 Tablet Servers, 2 weren't reachable I0221 01:27:12.496207 8186 cluster_verifier.cc:72] Check not successful yet, sleeping and retrying: Network error: Not all Tablet Servers are reachable Connected to the Master WARNING: Unable to connect to Tablet Server c8ec78660e8642b4b82113d62a3042cb (127.31.250.1:40370): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.1:40370: connect: Connection refused (error 111) WARNING: Unable to connect to Tablet Server 33e7b3c7fdd64255b209cc8123db9646 (127.31.250.2:46079): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.2:46079: connect: Connection refused (error 111) WARNING: Fetched info from 1 Tablet Servers, 2 weren't reachable I0221 01:27:13.538341 8186 cluster_verifier.cc:72] Check not successful yet, sleeping and retrying: Network error: Not all Tablet Servers are reachable Connected to the Master WARNING: Unable to connect to Tablet Server c8ec78660e8642b4b82113d62a3042cb (127.31.250.1:40370): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.1:40370: connect: Connection refused (error 111) WARNING: Unable to connect to Tablet Server 33e7b3c7fdd64255b209cc8123db9646 (127.31.250.2:46079): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.2:46079: connect: Connection refused (error 111) WARNING: Fetched info from 1 Tablet Servers, 2 weren't reachable I0221 01:27:14.555047 8186 cluster_verifier.cc:72] Check not successful yet, sleeping and retrying: Network error: Not all Tablet Servers are reachable Connected to the Master WARNING: Unable to connect to Tablet Server 33e7b3c7fdd64255b209cc8123db9646 (127.31.250.2:46079): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.2:46079: connect: Connection refused (error 111) WARNING: Unable to connect to Tablet Server c8ec78660e8642b4b82113d62a3042cb (127.31.250.1:40370): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.1:40370: connect: Connection refused (error 111) WARNING: Fetched info from 1 Tablet Servers, 2 weren't reachable I0221 01:27:15.575523 8186 cluster_verifier.cc:72] Check not successful yet, sleeping and retrying: Network error: Not all Tablet Servers are reachable Connected to the Master WARNING: Unable to connect to Tablet Server c8ec78660e8642b4b82113d62a3042cb (127.31.250.1:40370): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.1:40370: connect: Connection refused (error 111) WARNING: Unable to connect to Tablet Server 33e7b3c7fdd64255b209cc8123db9646 (127.31.250.2:46079): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.2:46079: connect: Connection refused (error 111) WARNING: Fetched info from 1 Tablet Servers, 2 weren't reachable I0221 01:27:16.617671 8186 cluster_verifier.cc:72] Check not successful yet, sleeping and retrying: Network error: Not all Tablet Servers are reachable Connected to the Master WARNING: Unable to connect to Tablet Server c8ec78660e8642b4b82113d62a3042cb (127.31.250.1:40370): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.1:40370: connect: Connection refused (error 111) WARNING: Unable to connect to Tablet Server 33e7b3c7fdd64255b209cc8123db9646 (127.31.250.2:46079): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.2:46079: connect: Connection refused (error 111) WARNING: Fetched info from 1 Tablet Servers, 2 weren't reachable I0221 01:27:17.658808 8186 cluster_verifier.cc:72] Check not successful yet, sleeping and retrying: Network error: Not all Tablet Servers are reachable Connected to the Master WARNING: Unable to connect to Tablet Server c8ec78660e8642b4b82113d62a3042cb (127.31.250.1:40370): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.1:40370: connect: Connection refused (error 111) WARNING: Unable to connect to Tablet Server 33e7b3c7fdd64255b209cc8123db9646 (127.31.250.2:46079): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.2:46079: connect: Connection refused (error 111) WARNING: Fetched info from 1 Tablet Servers, 2 weren't reachable I0221 01:27:18.701608 8186 cluster_verifier.cc:72] Check not successful yet, sleeping and retrying: Network error: Not all Tablet Servers are reachable Connected to the Master WARNING: Unable to connect to Tablet Server c8ec78660e8642b4b82113d62a3042cb (127.31.250.1:40370): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.1:40370: connect: Connection refused (error 111) WARNING: Unable to connect to Tablet Server 33e7b3c7fdd64255b209cc8123db9646 (127.31.250.2:46079): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.2:46079: connect: Connection refused (error 111) WARNING: Fetched info from 1 Tablet Servers, 2 weren't reachable I0221 01:27:19.742651 8186 cluster_verifier.cc:72] Check not successful yet, sleeping and retrying: Network error: Not all Tablet Servers are reachable Connected to the Master WARNING: Unable to connect to Tablet Server c8ec78660e8642b4b82113d62a3042cb (127.31.250.1:40370): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.1:40370: connect: Connection refused (error 111) WARNING: Unable to connect to Tablet Server 33e7b3c7fdd64255b209cc8123db9646 (127.31.250.2:46079): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.2:46079: connect: Connection refused (error 111) WARNING: Fetched info from 1 Tablet Servers, 2 weren't reachable I0221 01:27:20.784378 8186 cluster_verifier.cc:72] Check not successful yet, sleeping and retrying: Network error: Not all Tablet Servers are reachable I0221 01:27:21.720962 12150 raft_consensus.cc:411] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0221 01:27:21.721043 12150 raft_consensus.cc:2162] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 13.7 s I0221 01:27:21.721086 12150 raft_consensus.cc:435] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8ec78660e8642b4b82113d62a3042cb" member_type: VOTER last_known_addr { host: "127.31.250.1" port: 40370 } } peers { permanent_uuid: "33e7b3c7fdd64255b209cc8123db9646" member_type: VOTER last_known_addr { host: "127.31.250.2" port: 46079 } } peers { permanent_uuid: "1eb837892f874e99a16858e7958a1356" member_type: VOTER last_known_addr { host: "127.31.250.0" port: 35707 } } I0221 01:27:21.721570 12150 leader_election.cc:216] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer c8ec78660e8642b4b82113d62a3042cb I0221 01:27:21.721719 12150 leader_election.cc:216] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 33e7b3c7fdd64255b209cc8123db9646 W0221 01:27:21.722975 11759 leader_election.cc:272] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c8ec78660e8642b4b82113d62a3042cb: Network error: Client connection negotiation failed: client connection to 127.31.250.1:40370: connect: Connection refused (error 111) W0221 01:27:21.723095 11758 leader_election.cc:272] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 33e7b3c7fdd64255b209cc8123db9646: Network error: Client connection negotiation failed: client connection to 127.31.250.2:46079: connect: Connection refused (error 111) I0221 01:27:21.723177 11758 leader_election.cc:243] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. I0221 01:27:21.723510 14234 raft_consensus.cc:2162] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 14.5 s I0221 01:27:21.723592 14234 raft_consensus.cc:1908] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority Connected to the Master WARNING: Unable to connect to Tablet Server 33e7b3c7fdd64255b209cc8123db9646 (127.31.250.2:46079): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.2:46079: connect: Connection refused (error 111) WARNING: Unable to connect to Tablet Server c8ec78660e8642b4b82113d62a3042cb (127.31.250.1:40370): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.1:40370: connect: Connection refused (error 111) WARNING: Fetched info from 1 Tablet Servers, 2 weren't reachable I0221 01:27:21.830335 8186 cluster_verifier.cc:72] Check not successful yet, sleeping and retrying: Network error: Not all Tablet Servers are reachable Connected to the Master WARNING: Unable to connect to Tablet Server 33e7b3c7fdd64255b209cc8123db9646 (127.31.250.2:46079): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.2:46079: connect: Connection refused (error 111) WARNING: Unable to connect to Tablet Server c8ec78660e8642b4b82113d62a3042cb (127.31.250.1:40370): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.1:40370: connect: Connection refused (error 111) WARNING: Fetched info from 1 Tablet Servers, 2 weren't reachable I0221 01:27:22.872757 8186 cluster_verifier.cc:72] Check not successful yet, sleeping and retrying: Network error: Not all Tablet Servers are reachable Connected to the Master WARNING: Unable to connect to Tablet Server c8ec78660e8642b4b82113d62a3042cb (127.31.250.1:40370): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.1:40370: connect: Connection refused (error 111) WARNING: Unable to connect to Tablet Server 33e7b3c7fdd64255b209cc8123db9646 (127.31.250.2:46079): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.2:46079: connect: Connection refused (error 111) WARNING: Fetched info from 1 Tablet Servers, 2 weren't reachable I0221 01:27:23.914394 8186 cluster_verifier.cc:72] Check not successful yet, sleeping and retrying: Network error: Not all Tablet Servers are reachable Connected to the Master WARNING: Unable to connect to Tablet Server c8ec78660e8642b4b82113d62a3042cb (127.31.250.1:40370): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.1:40370: connect: Connection refused (error 111) WARNING: Unable to connect to Tablet Server 33e7b3c7fdd64255b209cc8123db9646 (127.31.250.2:46079): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.2:46079: connect: Connection refused (error 111) WARNING: Fetched info from 1 Tablet Servers, 2 weren't reachable I0221 01:27:24.956240 8186 cluster_verifier.cc:72] Check not successful yet, sleeping and retrying: Network error: Not all Tablet Servers are reachable Connected to the Master WARNING: Unable to connect to Tablet Server c8ec78660e8642b4b82113d62a3042cb (127.31.250.1:40370): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.1:40370: connect: Connection refused (error 111) WARNING: Unable to connect to Tablet Server 33e7b3c7fdd64255b209cc8123db9646 (127.31.250.2:46079): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.2:46079: connect: Connection refused (error 111) WARNING: Fetched info from 1 Tablet Servers, 2 weren't reachable I0221 01:27:25.998512 8186 cluster_verifier.cc:72] Check not successful yet, sleeping and retrying: Network error: Not all Tablet Servers are reachable Connected to the Master WARNING: Unable to connect to Tablet Server c8ec78660e8642b4b82113d62a3042cb (127.31.250.1:40370): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.1:40370: connect: Connection refused (error 111) WARNING: Unable to connect to Tablet Server 33e7b3c7fdd64255b209cc8123db9646 (127.31.250.2:46079): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.2:46079: connect: Connection refused (error 111) WARNING: Fetched info from 1 Tablet Servers, 2 weren't reachable I0221 01:27:27.041041 8186 cluster_verifier.cc:72] Check not successful yet, sleeping and retrying: Network error: Not all Tablet Servers are reachable Connected to the Master WARNING: Unable to connect to Tablet Server 33e7b3c7fdd64255b209cc8123db9646 (127.31.250.2:46079): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.2:46079: connect: Connection refused (error 111) WARNING: Unable to connect to Tablet Server c8ec78660e8642b4b82113d62a3042cb (127.31.250.1:40370): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.1:40370: connect: Connection refused (error 111) WARNING: Fetched info from 1 Tablet Servers, 2 weren't reachable I0221 01:27:28.082511 8186 cluster_verifier.cc:72] Check not successful yet, sleeping and retrying: Network error: Not all Tablet Servers are reachable Connected to the Master WARNING: WARNING: Unable to connect to Tablet Server 33e7b3c7fdd64255b209cc8123db9646 (127.31.250.2:46079): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.2:46079: connect: Connection refused (error 111) Unable to connect to Tablet Server c8ec78660e8642b4b82113d62a3042cb (127.31.250.1:40370): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.1:40370: connect: Connection refused (error 111) WARNING: Fetched info from 1 Tablet Servers, 2 weren't reachable I0221 01:27:29.126729 8186 cluster_verifier.cc:72] Check not successful yet, sleeping and retrying: Network error: Not all Tablet Servers are reachable Connected to the Master WARNING: Unable to connect to Tablet Server c8ec78660e8642b4b82113d62a3042cb (127.31.250.1:40370): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.1:40370: connect: Connection refused (error 111)WARNING: Unable to connect to Tablet Server 33e7b3c7fdd64255b209cc8123db9646 (127.31.250.2:46079): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.2:46079: connect: Connection refused (error 111) WARNING: Fetched info from 1 Tablet Servers, 2 weren't reachable I0221 01:27:30.167979 8186 cluster_verifier.cc:72] Check not successful yet, sleeping and retrying: Network error: Not all Tablet Servers are reachable Connected to the Master WARNING: Unable to connect to Tablet Server c8ec78660e8642b4b82113d62a3042cb (127.31.250.1:40370): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.1:40370: connect: Connection refused (error 111) WARNING: Unable to connect to Tablet Server 33e7b3c7fdd64255b209cc8123db9646 (127.31.250.2:46079): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.2:46079: connect: Connection refused (error 111) WARNING: Fetched info from 1 Tablet Servers, 2 weren't reachable I0221 01:27:31.209640 8186 cluster_verifier.cc:72] Check not successful yet, sleeping and retrying: Network error: Not all Tablet Servers are reachable Connected to the Master WARNING: Unable to connect to Tablet Server c8ec78660e8642b4b82113d62a3042cb (127.31.250.1:40370): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.1:40370: connect: Connection refused (error 111) WARNING: Unable to connect to Tablet Server 33e7b3c7fdd64255b209cc8123db9646 (127.31.250.2:46079): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.2:46079: connect: Connection refused (error 111) WARNING: Fetched info from 1 Tablet Servers, 2 weren't reachable I0221 01:27:32.251487 8186 cluster_verifier.cc:72] Check not successful yet, sleeping and retrying: Network error: Not all Tablet Servers are reachable Connected to the Master WARNING: Unable to connect to Tablet Server c8ec78660e8642b4b82113d62a3042cb (127.31.250.1:40370): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.1:40370: connect: Connection refused (error 111) WARNING: Unable to connect to Tablet Server 33e7b3c7fdd64255b209cc8123db9646 (127.31.250.2:46079): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.2:46079: connect: Connection refused (error 111) WARNING: Fetched info from 1 Tablet Servers, 2 weren't reachable I0221 01:27:33.292690 8186 cluster_verifier.cc:72] Check not successful yet, sleeping and retrying: Network error: Not all Tablet Servers are reachable Connected to the Master WARNING: Unable to connect to Tablet Server 33e7b3c7fdd64255b209cc8123db9646 (127.31.250.2:46079): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.2:46079: connect: Connection refused (error 111) WARNING: Unable to connect to Tablet Server c8ec78660e8642b4b82113d62a3042cb (127.31.250.1:40370): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.1:40370: connect: Connection refused (error 111) WARNING: Fetched info from 1 Tablet Servers, 2 weren't reachable I0221 01:27:34.333627 8186 cluster_verifier.cc:72] Check not successful yet, sleeping and retrying: Network error: Not all Tablet Servers are reachable Connected to the Master WARNING: Unable to connect to Tablet Server 33e7b3c7fdd64255b209cc8123db9646 (127.31.250.2:46079): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.2:46079: connect: Connection refused (error 111) WARNING: Unable to connect to Tablet Server c8ec78660e8642b4b82113d62a3042cb (127.31.250.1:40370): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.1:40370: connect: Connection refused (error 111) WARNING: Fetched info from 1 Tablet Servers, 2 weren't reachable I0221 01:27:35.363414 8186 cluster_verifier.cc:72] Check not successful yet, sleeping and retrying: Network error: Not all Tablet Servers are reachable Connected to the Master WARNING: WARNING: Unable to connect to Tablet Server c8ec78660e8642b4b82113d62a3042cb (127.31.250.1:40370): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.1:40370: connect: Connection refused (error 111) Unable to connect to Tablet Server 33e7b3c7fdd64255b209cc8123db9646 (127.31.250.2:46079): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.2:46079: connect: Connection refused (error 111) WARNING: Fetched info from 1 Tablet Servers, 2 weren't reachable I0221 01:27:36.407562 8186 cluster_verifier.cc:72] Check not successful yet, sleeping and retrying: Network error: Not all Tablet Servers are reachable Connected to the Master WARNING: Unable to connect to Tablet Server c8ec78660e8642b4b82113d62a3042cb (127.31.250.1:40370): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.1:40370: connect: Connection refused (error 111) WARNING: Unable to connect to Tablet Server 33e7b3c7fdd64255b209cc8123db9646 (127.31.250.2:46079): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.2:46079: connect: Connection refused (error 111) WARNING: Fetched info from 1 Tablet Servers, 2 weren't reachable I0221 01:27:37.449707 8186 cluster_verifier.cc:72] Check not successful yet, sleeping and retrying: Network error: Not all Tablet Servers are reachable I0221 01:27:38.279029 12150 raft_consensus.cc:411] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0221 01:27:38.279120 12150 raft_consensus.cc:2162] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 2.67 s I0221 01:27:38.279160 12150 raft_consensus.cc:435] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8ec78660e8642b4b82113d62a3042cb" member_type: VOTER last_known_addr { host: "127.31.250.1" port: 40370 } } peers { permanent_uuid: "33e7b3c7fdd64255b209cc8123db9646" member_type: VOTER last_known_addr { host: "127.31.250.2" port: 46079 } } peers { permanent_uuid: "1eb837892f874e99a16858e7958a1356" member_type: VOTER last_known_addr { host: "127.31.250.0" port: 35707 } } I0221 01:27:38.279747 12150 leader_election.cc:216] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer c8ec78660e8642b4b82113d62a3042cb I0221 01:27:38.279919 12150 leader_election.cc:216] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 33e7b3c7fdd64255b209cc8123db9646 W0221 01:27:38.281165 11759 leader_election.cc:272] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c8ec78660e8642b4b82113d62a3042cb: Network error: Client connection negotiation failed: client connection to 127.31.250.1:40370: connect: Connection refused (error 111) W0221 01:27:38.281275 11758 leader_election.cc:272] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 33e7b3c7fdd64255b209cc8123db9646: Network error: Client connection negotiation failed: client connection to 127.31.250.2:46079: connect: Connection refused (error 111) I0221 01:27:38.281365 11758 leader_election.cc:243] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. I0221 01:27:38.281672 14710 raft_consensus.cc:2162] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 14.9 s I0221 01:27:38.281756 14710 raft_consensus.cc:1908] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority Connected to the Master WARNING: Unable to connect to Tablet Server c8ec78660e8642b4b82113d62a3042cb (127.31.250.1:40370): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.1:40370: connect: Connection refused (error 111) WARNING: Unable to connect to Tablet Server 33e7b3c7fdd64255b209cc8123db9646 (127.31.250.2:46079): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.2:46079: connect: Connection refused (error 111) WARNING: Fetched info from 1 Tablet Servers, 2 weren't reachable I0221 01:27:38.469285 8186 cluster_verifier.cc:72] Check not successful yet, sleeping and retrying: Network error: Not all Tablet Servers are reachable Connected to the Master WARNING: Unable to connect to Tablet Server c8ec78660e8642b4b82113d62a3042cb (127.31.250.1:40370): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.1:40370: connect: Connection refused (error 111) WARNING: Unable to connect to Tablet Server 33e7b3c7fdd64255b209cc8123db9646 (127.31.250.2:46079): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.2:46079: connect: Connection refused (error 111) WARNING: Fetched info from 1 Tablet Servers, 2 weren't reachable I0221 01:27:39.512446 8186 cluster_verifier.cc:72] Check not successful yet, sleeping and retrying: Network error: Not all Tablet Servers are reachable Connected to the Master WARNING: Unable to connect to Tablet Server c8ec78660e8642b4b82113d62a3042cb (127.31.250.1:40370): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.1:40370: connect: Connection refused (error 111) WARNING: Unable to connect to Tablet Server 33e7b3c7fdd64255b209cc8123db9646 (127.31.250.2:46079): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.2:46079: connect: Connection refused (error 111) WARNING: Fetched info from 1 Tablet Servers, 2 weren't reachable I0221 01:27:40.526623 8186 cluster_verifier.cc:72] Check not successful yet, sleeping and retrying: Network error: Not all Tablet Servers are reachable Connected to the Master WARNING: Unable to connect to Tablet Server c8ec78660e8642b4b82113d62a3042cb (127.31.250.1:40370): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.1:40370: connect: Connection refused (error 111) WARNING: Unable to connect to Tablet Server 33e7b3c7fdd64255b209cc8123db9646 (127.31.250.2:46079): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.2:46079: connect: Connection refused (error 111) WARNING: Fetched info from 1 Tablet Servers, 2 weren't reachable I0221 01:27:41.541978 8186 cluster_verifier.cc:72] Check not successful yet, sleeping and retrying: Network error: Not all Tablet Servers are reachable Connected to the Master WARNING: Unable to connect to Tablet Server c8ec78660e8642b4b82113d62a3042cb (127.31.250.1:40370): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.1:40370: connect: Connection refused (error 111) WARNING: Unable to connect to Tablet Server 33e7b3c7fdd64255b209cc8123db9646 (127.31.250.2:46079): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.2:46079: connect: Connection refused (error 111) WARNING: Fetched info from 1 Tablet Servers, 2 weren't reachable I0221 01:27:42.556211 8186 cluster_verifier.cc:72] Check not successful yet, sleeping and retrying: Network error: Not all Tablet Servers are reachable Connected to the Master WARNING: Unable to connect to Tablet Server c8ec78660e8642b4b82113d62a3042cb (127.31.250.1:40370): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.1:40370: connect: Connection refused (error 111) WARNING: Unable to connect to Tablet Server 33e7b3c7fdd64255b209cc8123db9646 (127.31.250.2:46079): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.2:46079: connect: Connection refused (error 111) WARNING: Fetched info from 1 Tablet Servers, 2 weren't reachable I0221 01:27:43.599958 8186 cluster_verifier.cc:72] Check not successful yet, sleeping and retrying: Network error: Not all Tablet Servers are reachable Connected to the Master WARNING: Unable to connect to Tablet Server c8ec78660e8642b4b82113d62a3042cb (127.31.250.1:40370): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.1:40370: connect: Connection refused (error 111) WARNING: Unable to connect to Tablet Server 33e7b3c7fdd64255b209cc8123db9646 (127.31.250.2:46079): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.2:46079: connect: Connection refused (error 111) WARNING: Fetched info from 1 Tablet Servers, 2 weren't reachable I0221 01:27:44.615469 8186 cluster_verifier.cc:72] Check not successful yet, sleeping and retrying: Network error: Not all Tablet Servers are reachable Connected to the Master WARNING: Unable to connect to Tablet Server c8ec78660e8642b4b82113d62a3042cb (127.31.250.1:40370): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.1:40370: connect: Connection refused (error 111) WARNING: Unable to connect to Tablet Server 33e7b3c7fdd64255b209cc8123db9646 (127.31.250.2:46079): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.2:46079: connect: Connection refused (error 111) WARNING: Fetched info from 1 Tablet Servers, 2 weren't reachable I0221 01:27:45.631301 8186 cluster_verifier.cc:72] Check not successful yet, sleeping and retrying: Network error: Not all Tablet Servers are reachable Connected to the Master WARNING: Unable to connect to Tablet Server c8ec78660e8642b4b82113d62a3042cb (127.31.250.1:40370): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.1:40370: connect: Connection refused (error 111) WARNING: Unable to connect to Tablet Server 33e7b3c7fdd64255b209cc8123db9646 (127.31.250.2:46079): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.2:46079: connect: Connection refused (error 111) WARNING: Fetched info from 1 Tablet Servers, 2 weren't reachable I0221 01:27:46.671514 8186 cluster_verifier.cc:72] Check not successful yet, sleeping and retrying: Network error: Not all Tablet Servers are reachable Connected to the Master WARNING: Unable to connect to Tablet Server 33e7b3c7fdd64255b209cc8123db9646 (127.31.250.2:46079): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.2:46079: connect: Connection refused (error 111) WARNING: Unable to connect to Tablet Server c8ec78660e8642b4b82113d62a3042cb (127.31.250.1:40370): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.1:40370: connect: Connection refused (error 111) WARNING: Fetched info from 1 Tablet Servers, 2 weren't reachable I0221 01:27:47.714020 8186 cluster_verifier.cc:72] Check not successful yet, sleeping and retrying: Network error: Not all Tablet Servers are reachable Connected to the Master WARNING: Unable to connect to Tablet Server 33e7b3c7fdd64255b209cc8123db9646 (127.31.250.2:46079): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.2:46079: connect: Connection refused (error 111) WARNING: Unable to connect to Tablet Server c8ec78660e8642b4b82113d62a3042cb (127.31.250.1:40370): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.1:40370: connect: Connection refused (error 111) WARNING: Fetched info from 1 Tablet Servers, 2 weren't reachable I0221 01:27:48.755862 8186 cluster_verifier.cc:72] Check not successful yet, sleeping and retrying: Network error: Not all Tablet Servers are reachable Connected to the Master WARNING: Unable to connect to Tablet Server c8ec78660e8642b4b82113d62a3042cb (127.31.250.1:40370): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.1:40370: connect: Connection refused (error 111) WARNING: Unable to connect to Tablet Server 33e7b3c7fdd64255b209cc8123db9646 (127.31.250.2:46079): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.2:46079: connect: Connection refused (error 111) WARNING: Fetched info from 1 Tablet Servers, 2 weren't reachable I0221 01:27:49.770169 8186 cluster_verifier.cc:72] Check not successful yet, sleeping and retrying: Network error: Not all Tablet Servers are reachable Connected to the Master WARNING: Unable to connect to Tablet Server c8ec78660e8642b4b82113d62a3042cb (127.31.250.1:40370): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.1:40370: connect: Connection refused (error 111) WARNING: Unable to connect to Tablet Server 33e7b3c7fdd64255b209cc8123db9646 (127.31.250.2:46079): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.2:46079: connect: Connection refused (error 111) WARNING: Fetched info from 1 Tablet Servers, 2 weren't reachable I0221 01:27:50.812942 8186 cluster_verifier.cc:72] Check not successful yet, sleeping and retrying: Network error: Not all Tablet Servers are reachable Connected to the Master WARNING: Unable to connect to Tablet Server c8ec78660e8642b4b82113d62a3042cb (127.31.250.1:40370): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.1:40370: connect: Connection refused (error 111) WARNING: Unable to connect to Tablet Server 33e7b3c7fdd64255b209cc8123db9646 (127.31.250.2:46079): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.2:46079: connect: Connection refused (error 111) WARNING: Fetched info from 1 Tablet Servers, 2 weren't reachable I0221 01:27:51.856027 8186 cluster_verifier.cc:72] Check not successful yet, sleeping and retrying: Network error: Not all Tablet Servers are reachable Connected to the Master WARNING: Unable to connect to Tablet Server c8ec78660e8642b4b82113d62a3042cb (127.31.250.1:40370): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.1:40370: connect: Connection refused (error 111) WARNING: Unable to connect to Tablet Server 33e7b3c7fdd64255b209cc8123db9646 (127.31.250.2:46079): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.2:46079: connect: Connection refused (error 111) WARNING: Fetched info from 1 Tablet Servers, 2 weren't reachable I0221 01:27:52.895733 8186 cluster_verifier.cc:72] Check not successful yet, sleeping and retrying: Network error: Not all Tablet Servers are reachable Connected to the Master WARNING: Unable to connect to Tablet Server c8ec78660e8642b4b82113d62a3042cb (127.31.250.1:40370): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.1:40370: connect: Connection refused (error 111) WARNING: Unable to connect to Tablet Server 33e7b3c7fdd64255b209cc8123db9646 (127.31.250.2:46079): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.2:46079: connect: Connection refused (error 111) WARNING: Fetched info from 1 Tablet Servers, 2 weren't reachable I0221 01:27:53.942589 8186 cluster_verifier.cc:72] Check not successful yet, sleeping and retrying: Network error: Not all Tablet Servers are reachable Connected to the Master WARNING: Unable to connect to Tablet Server c8ec78660e8642b4b82113d62a3042cb (127.31.250.1:40370): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.1:40370: connect: Connection refused (error 111) WARNING: Unable to connect to Tablet Server 33e7b3c7fdd64255b209cc8123db9646 (127.31.250.2:46079): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.2:46079: connect: Connection refused (error 111) WARNING: Fetched info from 1 Tablet Servers, 2 weren't reachable I0221 01:27:54.987404 8186 cluster_verifier.cc:72] Check not successful yet, sleeping and retrying: Network error: Not all Tablet Servers are reachable I0221 01:27:55.451495 12150 raft_consensus.cc:411] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0221 01:27:55.451587 12150 raft_consensus.cc:2162] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 8.95 s I0221 01:27:55.451627 12150 raft_consensus.cc:435] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8ec78660e8642b4b82113d62a3042cb" member_type: VOTER last_known_addr { host: "127.31.250.1" port: 40370 } } peers { permanent_uuid: "33e7b3c7fdd64255b209cc8123db9646" member_type: VOTER last_known_addr { host: "127.31.250.2" port: 46079 } } peers { permanent_uuid: "1eb837892f874e99a16858e7958a1356" member_type: VOTER last_known_addr { host: "127.31.250.0" port: 35707 } } I0221 01:27:55.452127 12150 leader_election.cc:216] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer c8ec78660e8642b4b82113d62a3042cb I0221 01:27:55.452283 12150 leader_election.cc:216] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 33e7b3c7fdd64255b209cc8123db9646 W0221 01:27:55.453060 11759 leader_election.cc:272] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c8ec78660e8642b4b82113d62a3042cb: Network error: Client connection negotiation failed: client connection to 127.31.250.1:40370: connect: Connection refused (error 111) W0221 01:27:55.453357 11758 leader_election.cc:272] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 33e7b3c7fdd64255b209cc8123db9646: Network error: Client connection negotiation failed: client connection to 127.31.250.2:46079: connect: Connection refused (error 111) I0221 01:27:55.453433 11758 leader_election.cc:243] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. I0221 01:27:55.453770 15248 raft_consensus.cc:2162] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 2.66 s I0221 01:27:55.453855 15248 raft_consensus.cc:1908] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority Connected to the Master WARNING: Unable to connect to Tablet Server c8ec78660e8642b4b82113d62a3042cb (127.31.250.1:40370): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.1:40370: connect: Connection refused (error 111) WARNING: Unable to connect to Tablet Server 33e7b3c7fdd64255b209cc8123db9646 (127.31.250.2:46079): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.2:46079: connect: Connection refused (error 111) WARNING: Fetched info from 1 Tablet Servers, 2 weren't reachable I0221 01:27:56.034523 8186 cluster_verifier.cc:72] Check not successful yet, sleeping and retrying: Network error: Not all Tablet Servers are reachable Connected to the Master WARNING: Unable to connect to Tablet Server c8ec78660e8642b4b82113d62a3042cb (127.31.250.1:40370): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.1:40370: connect: Connection refused (error 111) WARNING: Unable to connect to Tablet Server 33e7b3c7fdd64255b209cc8123db9646 (127.31.250.2:46079): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.2:46079: connect: Connection refused (error 111) WARNING: Fetched info from 1 Tablet Servers, 2 weren't reachable I0221 01:27:57.078821 8186 cluster_verifier.cc:72] Check not successful yet, sleeping and retrying: Network error: Not all Tablet Servers are reachable Connected to the Master WARNING: Unable to connect to Tablet Server c8ec78660e8642b4b82113d62a3042cb (127.31.250.1:40370): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.1:40370: connect: Connection refused (error 111) WARNING: Unable to connect to Tablet Server 33e7b3c7fdd64255b209cc8123db9646 (127.31.250.2:46079): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.2:46079: connect: Connection refused (error 111) WARNING: Fetched info from 1 Tablet Servers, 2 weren't reachable I0221 01:27:58.110152 8186 cluster_verifier.cc:72] Check not successful yet, sleeping and retrying: Network error: Not all Tablet Servers are reachable Connected to the Master WARNING: Unable to connect to Tablet Server c8ec78660e8642b4b82113d62a3042cb (127.31.250.1:40370): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.1:40370: connect: Connection refused (error 111) WARNING: Unable to connect to Tablet Server 33e7b3c7fdd64255b209cc8123db9646 (127.31.250.2:46079): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.2:46079: connect: Connection refused (error 111) WARNING: Fetched info from 1 Tablet Servers, 2 weren't reachable I0221 01:27:59.144754 8186 cluster_verifier.cc:72] Check not successful yet, sleeping and retrying: Network error: Not all Tablet Servers are reachable I0221 01:27:59.960110 12150 raft_consensus.cc:411] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0221 01:27:59.960201 12150 raft_consensus.cc:2162] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 12 s I0221 01:27:59.960242 12150 raft_consensus.cc:435] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8ec78660e8642b4b82113d62a3042cb" member_type: VOTER last_known_addr { host: "127.31.250.1" port: 40370 } } peers { permanent_uuid: "33e7b3c7fdd64255b209cc8123db9646" member_type: VOTER last_known_addr { host: "127.31.250.2" port: 46079 } } peers { permanent_uuid: "1eb837892f874e99a16858e7958a1356" member_type: VOTER last_known_addr { host: "127.31.250.0" port: 35707 } } I0221 01:27:59.960741 12150 leader_election.cc:216] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer c8ec78660e8642b4b82113d62a3042cb I0221 01:27:59.960902 12150 leader_election.cc:216] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 33e7b3c7fdd64255b209cc8123db9646 W0221 01:27:59.962218 11759 leader_election.cc:272] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c8ec78660e8642b4b82113d62a3042cb: Network error: Client connection negotiation failed: client connection to 127.31.250.1:40370: connect: Connection refused (error 111) W0221 01:27:59.962312 11758 leader_election.cc:272] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 33e7b3c7fdd64255b209cc8123db9646: Network error: Client connection negotiation failed: client connection to 127.31.250.2:46079: connect: Connection refused (error 111) I0221 01:27:59.962385 11758 leader_election.cc:243] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. I0221 01:27:59.962671 15369 raft_consensus.cc:2162] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 19.5 s I0221 01:27:59.962767 15369 raft_consensus.cc:1908] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority Connected to the Master WARNING: Unable to connect to Tablet Server c8ec78660e8642b4b82113d62a3042cb (127.31.250.1:40370): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.1:40370: connect: Connection refused (error 111)WARNING: Unable to connect to Tablet Server 33e7b3c7fdd64255b209cc8123db9646 (127.31.250.2:46079): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.2:46079: connect: Connection refused (error 111) WARNING: Fetched info from 1 Tablet Servers, 2 weren't reachable I0221 01:28:00.193814 8186 cluster_verifier.cc:72] Check not successful yet, sleeping and retrying: Network error: Not all Tablet Servers are reachable Connected to the Master WARNING: Unable to connect to Tablet Server 33e7b3c7fdd64255b209cc8123db9646 (127.31.250.2:46079): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.2:46079: connect: Connection refused (error 111) WARNING: Unable to connect to Tablet Server c8ec78660e8642b4b82113d62a3042cb (127.31.250.1:40370): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.1:40370: connect: Connection refused (error 111) WARNING: Fetched info from 1 Tablet Servers, 2 weren't reachable I0221 01:28:01.237432 8186 cluster_verifier.cc:72] Check not successful yet, sleeping and retrying: Network error: Not all Tablet Servers are reachable Connected to the Master WARNING: WARNING: Unable to connect to Tablet Server 33e7b3c7fdd64255b209cc8123db9646 (127.31.250.2:46079): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.2:46079: connect: Connection refused (error 111) Unable to connect to Tablet Server c8ec78660e8642b4b82113d62a3042cb (127.31.250.1:40370): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.1:40370: connect: Connection refused (error 111) WARNING: Fetched info from 1 Tablet Servers, 2 weren't reachable I0221 01:28:02.283043 8186 cluster_verifier.cc:72] Check not successful yet, sleeping and retrying: Network error: Not all Tablet Servers are reachable Connected to the Master WARNING: Unable to connect to Tablet Server c8ec78660e8642b4b82113d62a3042cb (127.31.250.1:40370): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.1:40370: connect: Connection refused (error 111) WARNING: Unable to connect to Tablet Server 33e7b3c7fdd64255b209cc8123db9646 (127.31.250.2:46079): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.2:46079: connect: Connection refused (error 111) WARNING: Fetched info from 1 Tablet Servers, 2 weren't reachable I0221 01:28:03.326091 8186 cluster_verifier.cc:72] Check not successful yet, sleeping and retrying: Network error: Not all Tablet Servers are reachable Connected to the Master WARNING: Unable to connect to Tablet Server c8ec78660e8642b4b82113d62a3042cb (127.31.250.1:40370): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.1:40370: connect: Connection refused (error 111) WARNING: Unable to connect to Tablet Server 33e7b3c7fdd64255b209cc8123db9646 (127.31.250.2:46079): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.2:46079: connect: Connection refused (error 111) WARNING: Fetched info from 1 Tablet Servers, 2 weren't reachable I0221 01:28:04.369840 8186 cluster_verifier.cc:72] Check not successful yet, sleeping and retrying: Network error: Not all Tablet Servers are reachable Connected to the Master WARNING: Unable to connect to Tablet Server c8ec78660e8642b4b82113d62a3042cb (127.31.250.1:40370): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.1:40370: connect: Connection refused (error 111) WARNING: Unable to connect to Tablet Server 33e7b3c7fdd64255b209cc8123db9646 (127.31.250.2:46079): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.2:46079: connect: Connection refused (error 111) WARNING: Fetched info from 1 Tablet Servers, 2 weren't reachable I0221 01:28:05.411103 8186 cluster_verifier.cc:72] Check not successful yet, sleeping and retrying: Network error: Not all Tablet Servers are reachable Connected to the Master WARNING: Unable to connect to Tablet Server c8ec78660e8642b4b82113d62a3042cb (127.31.250.1:40370): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.1:40370: connect: Connection refused (error 111) WARNING: Unable to connect to Tablet Server 33e7b3c7fdd64255b209cc8123db9646 (127.31.250.2:46079): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.2:46079: connect: Connection refused (error 111) WARNING: Fetched info from 1 Tablet Servers, 2 weren't reachable I0221 01:28:06.452466 8186 cluster_verifier.cc:72] Check not successful yet, sleeping and retrying: Network error: Not all Tablet Servers are reachable Connected to the Master WARNING: Unable to connect to Tablet Server c8ec78660e8642b4b82113d62a3042cb (127.31.250.1:40370): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.1:40370: connect: Connection refused (error 111) WARNING: Unable to connect to Tablet Server 33e7b3c7fdd64255b209cc8123db9646 (127.31.250.2:46079): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.2:46079: connect: Connection refused (error 111) WARNING: Fetched info from 1 Tablet Servers, 2 weren't reachable I0221 01:28:07.495183 8186 cluster_verifier.cc:72] Check not successful yet, sleeping and retrying: Network error: Not all Tablet Servers are reachable Connected to the Master WARNING: WARNING: Unable to connect to Tablet Server c8ec78660e8642b4b82113d62a3042cb (127.31.250.1:40370): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.1:40370: connect: Connection refused (error 111)Unable to connect to Tablet Server 33e7b3c7fdd64255b209cc8123db9646 (127.31.250.2:46079): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.2:46079: connect: Connection refused (error 111) WARNING: Fetched info from 1 Tablet Servers, 2 weren't reachable I0221 01:28:08.526393 8186 cluster_verifier.cc:72] Check not successful yet, sleeping and retrying: Network error: Not all Tablet Servers are reachable Connected to the Master WARNING: Unable to connect to Tablet Server c8ec78660e8642b4b82113d62a3042cb (127.31.250.1:40370): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.1:40370: connect: Connection refused (error 111) WARNING: Unable to connect to Tablet Server 33e7b3c7fdd64255b209cc8123db9646 (127.31.250.2:46079): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.2:46079: connect: Connection refused (error 111) WARNING: Fetched info from 1 Tablet Servers, 2 weren't reachable I0221 01:28:09.556123 8186 cluster_verifier.cc:72] Check not successful yet, sleeping and retrying: Network error: Not all Tablet Servers are reachable Connected to the Master WARNING: Unable to connect to Tablet Server c8ec78660e8642b4b82113d62a3042cb (127.31.250.1:40370): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.1:40370: connect: Connection refused (error 111) WARNING: Unable to connect to Tablet Server 33e7b3c7fdd64255b209cc8123db9646 (127.31.250.2:46079): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.2:46079: connect: Connection refused (error 111) WARNING: Fetched info from 1 Tablet Servers, 2 weren't reachable I0221 01:28:10.571241 8186 cluster_verifier.cc:72] Check not successful yet, sleeping and retrying: Network error: Not all Tablet Servers are reachable Connected to the Master WARNING: WARNING: Unable to connect to Tablet Server c8ec78660e8642b4b82113d62a3042cb (127.31.250.1:40370): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.1:40370: connect: Connection refused (error 111) Unable to connect to Tablet Server 33e7b3c7fdd64255b209cc8123db9646 (127.31.250.2:46079): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.2:46079: connect: Connection refused (error 111) WARNING: Fetched info from 1 Tablet Servers, 2 weren't reachable I0221 01:28:11.619349 8186 cluster_verifier.cc:72] Check not successful yet, sleeping and retrying: Network error: Not all Tablet Servers are reachable Connected to the Master WARNING: Unable to connect to Tablet Server c8ec78660e8642b4b82113d62a3042cb (127.31.250.1:40370): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.1:40370: connect: Connection refused (error 111) WARNING: Unable to connect to Tablet Server 33e7b3c7fdd64255b209cc8123db9646 (127.31.250.2:46079): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.2:46079: connect: Connection refused (error 111) WARNING: Fetched info from 1 Tablet Servers, 2 weren't reachable I0221 01:28:12.658176 8186 cluster_verifier.cc:72] Check not successful yet, sleeping and retrying: Network error: Not all Tablet Servers are reachable Connected to the Master WARNING: WARNING: Unable to connect to Tablet Server c8ec78660e8642b4b82113d62a3042cb (127.31.250.1:40370): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.1:40370: connect: Connection refused (error 111) Unable to connect to Tablet Server 33e7b3c7fdd64255b209cc8123db9646 (127.31.250.2:46079): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.2:46079: connect: Connection refused (error 111) WARNING: Fetched info from 1 Tablet Servers, 2 weren't reachable I0221 01:28:13.702400 8186 cluster_verifier.cc:72] Check not successful yet, sleeping and retrying: Network error: Not all Tablet Servers are reachable Connected to the Master WARNING: Unable to connect to Tablet Server c8ec78660e8642b4b82113d62a3042cb (127.31.250.1:40370): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.1:40370: connect: Connection refused (error 111) WARNING: Unable to connect to Tablet Server 33e7b3c7fdd64255b209cc8123db9646 (127.31.250.2:46079): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.2:46079: connect: Connection refused (error 111) WARNING: Fetched info from 1 Tablet Servers, 2 weren't reachable I0221 01:28:14.745596 8186 cluster_verifier.cc:72] Check not successful yet, sleeping and retrying: Network error: Not all Tablet Servers are reachable Connected to the Master WARNING: Unable to connect to Tablet Server c8ec78660e8642b4b82113d62a3042cb (127.31.250.1:40370): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.1:40370: connect: Connection refused (error 111) WARNING: Unable to connect to Tablet Server 33e7b3c7fdd64255b209cc8123db9646 (127.31.250.2:46079): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 127.31.250.2:46079: connect: Connection refused (error 111) WARNING: Fetched info from 1 Tablet Servers, 2 weren't reachable I0221 01:28:15.766387 8186 cluster_verifier.cc:72] Check not successful yet, sleeping and retrying: Network error: Not all Tablet Servers are reachable I0221 01:28:16.420493 11682 tablet.cc:933] T 00000000000000000000000000000000 P e112fba79af54e179eff2a3e09683ff6: Flush: entering stage 1 (old memrowset already frozen for inserts) I0221 01:28:16.420570 11682 compaction.cc:877] Selected 1 rowsets to compact: I0221 01:28:16.420635 11682 compaction.cc:880] memrowset(current size on disk: ~0 bytes) I0221 01:28:16.420657 11682 tablet.cc:935] T 00000000000000000000000000000000 P e112fba79af54e179eff2a3e09683ff6: Memstore in-memory size: 10311 bytes I0221 01:28:16.420706 11682 tablet.cc:1190] T 00000000000000000000000000000000 P e112fba79af54e179eff2a3e09683ff6: Flush: entering phase 1 (flushing snapshot). Phase 1 snapshot: MvccSnapshot[committed={T|T < 6093492950601797632 or (T in {6093492950601797632})}] *** Aborted at 1487669296 (unix time) try "date -d @1487669296" if you are using GNU date *** PC: @ 0x0 (unknown) *** SIGSEGV (@0x0) received by PID 11677 (TID 0x7fcf2b7fe700) from PID 0; stack trace: *** @ 0x7fcf350be390 (unknown) at ??:0 @ 0x0 (unknown) @ 0x7fcf3373ef05 kudu::(anonymous namespace)::ShardedLRUCache::Insert() at ??:0 @ 0x7fcf3379fb5b kudu::internal::BaseDescriptor<>::InsertIntoCache() at ??:0 @ 0x7fcf3379f8d7 kudu::internal::Descriptor<>::ReopenFileIfNecessary() at ??:0 @ 0x7fcf3379f540 kudu::internal::Descriptor<>::InitOnce() at ??:0 @ 0x7fcf3379f620 kudu::internal::InitCb<>() at ??:0 @ 0x7fcf332c1b0b GoogleOnceInternalInit() at ??:0 @ 0x7fcf36f01069 GoogleOnceDynamic::Init<>() at ??:0 @ 0x7fcf3379f4f3 kudu::KuduOnceDynamic::Init<>() at ??:0 @ 0x7fcf3379bc55 kudu::internal::Descriptor<>::Init() at ??:0 @ 0x7fcf3379b4d9 kudu::FileCache<>::OpenExistingFile() at ??:0 @ 0x7fcf34d2126b kudu::fs::internal::LogBlockContainer::Create() at ??:0 @ 0x7fcf34d2d53b kudu::fs::LogBlockManager::GetOrCreateContainer() at ??:0 @ 0x7fcf34d2cead kudu::fs::LogBlockManager::CreateBlock() at ??:0 @ 0x7fcf34d2d8ce kudu::fs::LogBlockManager::CreateBlock() at ??:0 @ 0x7fcf34d185ea kudu::FsManager::CreateNewBlock() at ??:0 @ 0x7fcf36ead090 kudu::tablet::MultiColumnWriter::Open() at ??:0 @ 0x7fcf36e8261c kudu::tablet::DiskRowSetWriter::Open() at ??:0 @ 0x7fcf36e85936 kudu::tablet::RollingDiskRowSetWriter::RollWriter() at ??:0 @ 0x7fcf36e854cf kudu::tablet::RollingDiskRowSetWriter::Open() at ??:0 @ 0x7fcf36dcc4a2 kudu::tablet::Tablet::DoMergeCompactionOrFlush() at ??:0 @ 0x7fcf36dcb9aa kudu::tablet::Tablet::FlushInternal() at ??:0 @ 0x7fcf36dcabc6 kudu::tablet::Tablet::FlushUnlocked() at ??:0 @ 0x7fcf36e240a1 kudu::tablet::FlushMRSOp::Perform() at ??:0 @ 0x7fcf337d241a kudu::MaintenanceManager::LaunchOp() at ??:0 @ 0x7fcf337d8ee1 boost::_mfi::mf1<>::operator()() at ??:0 @ 0x7fcf337d8e3f boost::_bi::list2<>::operator()<>() at ??:0 @ 0x7fcf337d8dca boost::_bi::bind_t<>::operator()() at ??:0 @ 0x7fcf337d8b50 boost::detail::function::void_function_obj_invoker0<>::invoke() at ??:0 @ 0x7fcf34956888 boost::function0<>::operator()() at ??:0 @ 0x7fcf338520f9 kudu::FunctionRunnable::Run() at ??:0 W0221 01:28:16.570904 11757 connection.cc:466] client connection to 127.0.0.1:44820 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0221 01:28:16.683327 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:28:17.684872 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:28:18.686977 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:28:19.688896 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:28:20.690470 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) I0221 01:28:21.511386 12150 raft_consensus.cc:411] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0221 01:28:21.511468 12150 raft_consensus.cc:2162] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 14.9 s I0221 01:28:21.511508 12150 raft_consensus.cc:435] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8ec78660e8642b4b82113d62a3042cb" member_type: VOTER last_known_addr { host: "127.31.250.1" port: 40370 } } peers { permanent_uuid: "33e7b3c7fdd64255b209cc8123db9646" member_type: VOTER last_known_addr { host: "127.31.250.2" port: 46079 } } peers { permanent_uuid: "1eb837892f874e99a16858e7958a1356" member_type: VOTER last_known_addr { host: "127.31.250.0" port: 35707 } } I0221 01:28:21.512034 12150 leader_election.cc:216] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer c8ec78660e8642b4b82113d62a3042cb I0221 01:28:21.512208 12150 leader_election.cc:216] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 33e7b3c7fdd64255b209cc8123db9646 W0221 01:28:21.513418 11758 leader_election.cc:272] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 33e7b3c7fdd64255b209cc8123db9646: Network error: Client connection negotiation failed: client connection to 127.31.250.2:46079: connect: Connection refused (error 111) W0221 01:28:21.513522 11759 leader_election.cc:272] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c8ec78660e8642b4b82113d62a3042cb: Network error: Client connection negotiation failed: client connection to 127.31.250.1:40370: connect: Connection refused (error 111) I0221 01:28:21.513604 11759 leader_election.cc:243] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. I0221 01:28:21.513972 15932 raft_consensus.cc:2162] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 10.7 s I0221 01:28:21.514057 15932 raft_consensus.cc:1908] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority W0221 01:28:21.691927 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:28:22.693496 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:28:23.695057 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:28:24.696957 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:28:25.698817 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:28:26.700433 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:28:27.702044 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:28:28.703960 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:28:29.705605 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:28:30.707650 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:28:31.709597 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:28:32.711185 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:28:33.711886 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) I0221 01:28:33.906000 12150 raft_consensus.cc:411] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0221 01:28:33.906074 12150 raft_consensus.cc:2162] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 7.85 s I0221 01:28:33.906105 12150 raft_consensus.cc:435] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8ec78660e8642b4b82113d62a3042cb" member_type: VOTER last_known_addr { host: "127.31.250.1" port: 40370 } } peers { permanent_uuid: "33e7b3c7fdd64255b209cc8123db9646" member_type: VOTER last_known_addr { host: "127.31.250.2" port: 46079 } } peers { permanent_uuid: "1eb837892f874e99a16858e7958a1356" member_type: VOTER last_known_addr { host: "127.31.250.0" port: 35707 } } I0221 01:28:33.906464 12150 leader_election.cc:216] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer c8ec78660e8642b4b82113d62a3042cb I0221 01:28:33.906584 12150 leader_election.cc:216] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 33e7b3c7fdd64255b209cc8123db9646 W0221 01:28:33.907119 11759 leader_election.cc:272] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c8ec78660e8642b4b82113d62a3042cb: Network error: Client connection negotiation failed: client connection to 127.31.250.1:40370: connect: Connection refused (error 111) W0221 01:28:33.907444 11758 leader_election.cc:272] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 33e7b3c7fdd64255b209cc8123db9646: Network error: Client connection negotiation failed: client connection to 127.31.250.2:46079: connect: Connection refused (error 111) I0221 01:28:33.907505 11758 leader_election.cc:243] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. I0221 01:28:33.907863 16121 raft_consensus.cc:2162] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 4.41 s I0221 01:28:33.907937 16121 raft_consensus.cc:1908] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority W0221 01:28:34.713307 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:28:35.714828 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:28:36.716385 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:28:37.717875 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:28:38.719977 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:28:39.721823 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) I0221 01:28:39.975035 12150 raft_consensus.cc:411] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0221 01:28:39.975126 12150 raft_consensus.cc:2162] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 3.58 s I0221 01:28:39.975164 12150 raft_consensus.cc:435] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8ec78660e8642b4b82113d62a3042cb" member_type: VOTER last_known_addr { host: "127.31.250.1" port: 40370 } } peers { permanent_uuid: "33e7b3c7fdd64255b209cc8123db9646" member_type: VOTER last_known_addr { host: "127.31.250.2" port: 46079 } } peers { permanent_uuid: "1eb837892f874e99a16858e7958a1356" member_type: VOTER last_known_addr { host: "127.31.250.0" port: 35707 } } I0221 01:28:39.975744 12150 leader_election.cc:216] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer c8ec78660e8642b4b82113d62a3042cb I0221 01:28:39.975921 12150 leader_election.cc:216] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 33e7b3c7fdd64255b209cc8123db9646 W0221 01:28:39.976994 11758 leader_election.cc:272] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 33e7b3c7fdd64255b209cc8123db9646: Network error: Client connection negotiation failed: client connection to 127.31.250.2:46079: connect: Connection refused (error 111) W0221 01:28:39.977097 11759 leader_election.cc:272] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c8ec78660e8642b4b82113d62a3042cb: Network error: Client connection negotiation failed: client connection to 127.31.250.1:40370: connect: Connection refused (error 111) I0221 01:28:39.977190 11759 leader_election.cc:243] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. I0221 01:28:39.977514 16200 raft_consensus.cc:2162] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 17.9 s I0221 01:28:39.977603 16200 raft_consensus.cc:1908] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority W0221 01:28:40.723242 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:28:41.724774 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:28:42.726356 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:28:43.727867 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:28:44.729382 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:28:45.730908 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:28:46.732875 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) I0221 01:28:46.826710 8186 cluster_verifier.cc:72] Check not successful yet, sleeping and retrying: Timed out: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.0.0.1:44820, num_attempts: 241) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:28:47.734763 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:28:48.736702 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:28:49.738502 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:28:50.740208 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:28:51.742209 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:28:52.743741 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:28:53.745368 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:28:54.746862 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:28:55.748612 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:28:56.750588 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:28:57.751329 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:28:58.753401 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:28:59.754901 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) I0221 01:28:59.807584 12150 raft_consensus.cc:411] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0221 01:28:59.807672 12150 raft_consensus.cc:2162] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.97 s I0221 01:28:59.807713 12150 raft_consensus.cc:435] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8ec78660e8642b4b82113d62a3042cb" member_type: VOTER last_known_addr { host: "127.31.250.1" port: 40370 } } peers { permanent_uuid: "33e7b3c7fdd64255b209cc8123db9646" member_type: VOTER last_known_addr { host: "127.31.250.2" port: 46079 } } peers { permanent_uuid: "1eb837892f874e99a16858e7958a1356" member_type: VOTER last_known_addr { host: "127.31.250.0" port: 35707 } } I0221 01:28:59.808214 12150 leader_election.cc:216] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer c8ec78660e8642b4b82113d62a3042cb I0221 01:28:59.808382 12150 leader_election.cc:216] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 33e7b3c7fdd64255b209cc8123db9646 W0221 01:28:59.809447 11758 leader_election.cc:272] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 33e7b3c7fdd64255b209cc8123db9646: Network error: Client connection negotiation failed: client connection to 127.31.250.2:46079: connect: Connection refused (error 111) W0221 01:28:59.809545 11759 leader_election.cc:272] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c8ec78660e8642b4b82113d62a3042cb: Network error: Client connection negotiation failed: client connection to 127.31.250.1:40370: connect: Connection refused (error 111) I0221 01:28:59.809631 11759 leader_election.cc:243] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. I0221 01:28:59.809999 16386 raft_consensus.cc:2162] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 10.1 s I0221 01:28:59.810097 16386 raft_consensus.cc:1908] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority W0221 01:29:00.756700 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:29:01.758589 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:29:02.759932 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:29:03.761665 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:29:04.763762 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:29:05.765314 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:29:06.767416 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:29:07.768635 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:29:08.770584 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:29:09.772572 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:29:10.774385 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) I0221 01:29:11.698462 12150 raft_consensus.cc:411] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0221 01:29:11.698551 12150 raft_consensus.cc:2162] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 12.5 s I0221 01:29:11.698593 12150 raft_consensus.cc:435] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8ec78660e8642b4b82113d62a3042cb" member_type: VOTER last_known_addr { host: "127.31.250.1" port: 40370 } } peers { permanent_uuid: "33e7b3c7fdd64255b209cc8123db9646" member_type: VOTER last_known_addr { host: "127.31.250.2" port: 46079 } } peers { permanent_uuid: "1eb837892f874e99a16858e7958a1356" member_type: VOTER last_known_addr { host: "127.31.250.0" port: 35707 } } I0221 01:29:11.699123 12150 leader_election.cc:216] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer c8ec78660e8642b4b82113d62a3042cb I0221 01:29:11.699339 12150 leader_election.cc:216] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 33e7b3c7fdd64255b209cc8123db9646 W0221 01:29:11.700433 11759 leader_election.cc:272] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c8ec78660e8642b4b82113d62a3042cb: Network error: Client connection negotiation failed: client connection to 127.31.250.1:40370: connect: Connection refused (error 111) W0221 01:29:11.700531 11758 leader_election.cc:272] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 33e7b3c7fdd64255b209cc8123db9646: Network error: Client connection negotiation failed: client connection to 127.31.250.2:46079: connect: Connection refused (error 111) I0221 01:29:11.700614 11758 leader_election.cc:243] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. I0221 01:29:11.700904 16502 raft_consensus.cc:2162] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 12.9 s I0221 01:29:11.700999 16502 raft_consensus.cc:1908] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority W0221 01:29:11.776232 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:29:12.777904 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:29:13.779753 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:29:14.781492 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:29:15.782974 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:29:16.784786 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:29:17.786792 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) I0221 01:29:17.887728 8186 cluster_verifier.cc:72] Check not successful yet, sleeping and retrying: Timed out: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.0.0.1:44820, num_attempts: 241) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:29:18.788373 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:29:19.789959 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:29:20.791400 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:29:21.793249 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:29:22.795210 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:29:23.796911 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:29:24.798426 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:29:25.800045 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) I0221 01:29:26.511499 12150 raft_consensus.cc:411] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0221 01:29:26.511586 12150 raft_consensus.cc:2162] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 12 s I0221 01:29:26.511629 12150 raft_consensus.cc:435] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8ec78660e8642b4b82113d62a3042cb" member_type: VOTER last_known_addr { host: "127.31.250.1" port: 40370 } } peers { permanent_uuid: "33e7b3c7fdd64255b209cc8123db9646" member_type: VOTER last_known_addr { host: "127.31.250.2" port: 46079 } } peers { permanent_uuid: "1eb837892f874e99a16858e7958a1356" member_type: VOTER last_known_addr { host: "127.31.250.0" port: 35707 } } I0221 01:29:26.512137 12150 leader_election.cc:216] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer c8ec78660e8642b4b82113d62a3042cb I0221 01:29:26.512297 12150 leader_election.cc:216] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 33e7b3c7fdd64255b209cc8123db9646 W0221 01:29:26.513208 11759 leader_election.cc:272] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c8ec78660e8642b4b82113d62a3042cb: Network error: Client connection negotiation failed: client connection to 127.31.250.1:40370: connect: Connection refused (error 111) W0221 01:29:26.513514 11758 leader_election.cc:272] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 33e7b3c7fdd64255b209cc8123db9646: Network error: Client connection negotiation failed: client connection to 127.31.250.2:46079: connect: Connection refused (error 111) I0221 01:29:26.513588 11758 leader_election.cc:243] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. I0221 01:29:26.513895 16638 raft_consensus.cc:2162] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 6.82 s I0221 01:29:26.513979 16638 raft_consensus.cc:1908] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority W0221 01:29:26.801479 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:29:27.803390 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:29:28.805001 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:29:29.806354 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:29:30.807554 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:29:31.809001 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:29:32.810617 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:29:33.812170 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:29:34.813597 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) I0221 01:29:34.882495 12150 raft_consensus.cc:411] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0221 01:29:34.882575 12150 raft_consensus.cc:2162] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 19.2 s I0221 01:29:34.882614 12150 raft_consensus.cc:435] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8ec78660e8642b4b82113d62a3042cb" member_type: VOTER last_known_addr { host: "127.31.250.1" port: 40370 } } peers { permanent_uuid: "33e7b3c7fdd64255b209cc8123db9646" member_type: VOTER last_known_addr { host: "127.31.250.2" port: 46079 } } peers { permanent_uuid: "1eb837892f874e99a16858e7958a1356" member_type: VOTER last_known_addr { host: "127.31.250.0" port: 35707 } } I0221 01:29:34.883081 12150 leader_election.cc:216] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer c8ec78660e8642b4b82113d62a3042cb I0221 01:29:34.883220 12150 leader_election.cc:216] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 33e7b3c7fdd64255b209cc8123db9646 W0221 01:29:34.883955 11759 leader_election.cc:272] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c8ec78660e8642b4b82113d62a3042cb: Network error: Client connection negotiation failed: client connection to 127.31.250.1:40370: connect: Connection refused (error 111) W0221 01:29:34.884145 11758 leader_election.cc:272] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 33e7b3c7fdd64255b209cc8123db9646: Network error: Client connection negotiation failed: client connection to 127.31.250.2:46079: connect: Connection refused (error 111) I0221 01:29:34.884222 11758 leader_election.cc:243] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. I0221 01:29:34.884537 16743 raft_consensus.cc:2162] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 6.31 s I0221 01:29:34.884615 16743 raft_consensus.cc:1908] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority W0221 01:29:35.815008 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:29:36.816493 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:29:37.817876 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:29:38.819350 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:29:39.824398 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:29:40.825889 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:29:41.827239 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:29:42.828586 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) I0221 01:29:43.249964 12150 raft_consensus.cc:411] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0221 01:29:43.250030 12150 raft_consensus.cc:2162] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 14.4 s I0221 01:29:43.250062 12150 raft_consensus.cc:435] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8ec78660e8642b4b82113d62a3042cb" member_type: VOTER last_known_addr { host: "127.31.250.1" port: 40370 } } peers { permanent_uuid: "33e7b3c7fdd64255b209cc8123db9646" member_type: VOTER last_known_addr { host: "127.31.250.2" port: 46079 } } peers { permanent_uuid: "1eb837892f874e99a16858e7958a1356" member_type: VOTER last_known_addr { host: "127.31.250.0" port: 35707 } } I0221 01:29:43.250458 12150 leader_election.cc:216] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer c8ec78660e8642b4b82113d62a3042cb I0221 01:29:43.250582 12150 leader_election.cc:216] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 33e7b3c7fdd64255b209cc8123db9646 W0221 01:29:43.251564 11758 leader_election.cc:272] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 33e7b3c7fdd64255b209cc8123db9646: Network error: Client connection negotiation failed: client connection to 127.31.250.2:46079: connect: Connection refused (error 111) W0221 01:29:43.251644 11759 leader_election.cc:272] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c8ec78660e8642b4b82113d62a3042cb: Network error: Client connection negotiation failed: client connection to 127.31.250.1:40370: connect: Connection refused (error 111) I0221 01:29:43.251703 11759 leader_election.cc:243] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. I0221 01:29:43.252014 16818 raft_consensus.cc:2162] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 12.3 s I0221 01:29:43.252084 16818 raft_consensus.cc:1908] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority W0221 01:29:43.830096 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:29:44.831439 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:29:45.832816 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:29:46.834130 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:29:47.835659 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:29:48.837153 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) I0221 01:29:48.946213 8186 cluster_verifier.cc:72] Check not successful yet, sleeping and retrying: Timed out: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.0.0.1:44820, num_attempts: 241) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:29:49.838536 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:29:50.840019 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:29:51.841490 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:29:52.842874 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:29:53.844386 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:29:54.845738 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:29:55.847111 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:29:56.848580 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) I0221 01:29:57.663439 12150 raft_consensus.cc:411] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0221 01:29:57.663530 12150 raft_consensus.cc:2162] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 19.5 s I0221 01:29:57.663575 12150 raft_consensus.cc:435] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8ec78660e8642b4b82113d62a3042cb" member_type: VOTER last_known_addr { host: "127.31.250.1" port: 40370 } } peers { permanent_uuid: "33e7b3c7fdd64255b209cc8123db9646" member_type: VOTER last_known_addr { host: "127.31.250.2" port: 46079 } } peers { permanent_uuid: "1eb837892f874e99a16858e7958a1356" member_type: VOTER last_known_addr { host: "127.31.250.0" port: 35707 } } I0221 01:29:57.664070 12150 leader_election.cc:216] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer c8ec78660e8642b4b82113d62a3042cb I0221 01:29:57.664224 12150 leader_election.cc:216] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 33e7b3c7fdd64255b209cc8123db9646 W0221 01:29:57.665464 11759 leader_election.cc:272] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c8ec78660e8642b4b82113d62a3042cb: Network error: Client connection negotiation failed: client connection to 127.31.250.1:40370: connect: Connection refused (error 111) W0221 01:29:57.665565 11758 leader_election.cc:272] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 33e7b3c7fdd64255b209cc8123db9646: Network error: Client connection negotiation failed: client connection to 127.31.250.2:46079: connect: Connection refused (error 111) I0221 01:29:57.665668 11758 leader_election.cc:243] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. I0221 01:29:57.666036 16944 raft_consensus.cc:2162] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 9.16 s I0221 01:29:57.666146 16944 raft_consensus.cc:1908] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority W0221 01:29:57.849952 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:29:58.851419 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:29:59.852834 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:30:00.854338 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:30:01.855731 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:30:02.857689 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:30:03.859666 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:30:04.861167 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:30:05.862666 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:30:06.864280 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:30:07.865847 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) I0221 01:30:08.673064 12150 raft_consensus.cc:411] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0221 01:30:08.673102 12150 raft_consensus.cc:2162] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 16.7 s I0221 01:30:08.673120 12150 raft_consensus.cc:435] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8ec78660e8642b4b82113d62a3042cb" member_type: VOTER last_known_addr { host: "127.31.250.1" port: 40370 } } peers { permanent_uuid: "33e7b3c7fdd64255b209cc8123db9646" member_type: VOTER last_known_addr { host: "127.31.250.2" port: 46079 } } peers { permanent_uuid: "1eb837892f874e99a16858e7958a1356" member_type: VOTER last_known_addr { host: "127.31.250.0" port: 35707 } } I0221 01:30:08.673332 12150 leader_election.cc:216] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer c8ec78660e8642b4b82113d62a3042cb I0221 01:30:08.673399 12150 leader_election.cc:216] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 33e7b3c7fdd64255b209cc8123db9646 W0221 01:30:08.673991 11759 leader_election.cc:272] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c8ec78660e8642b4b82113d62a3042cb: Network error: Client connection negotiation failed: client connection to 127.31.250.1:40370: connect: Connection refused (error 111) W0221 01:30:08.674049 11758 leader_election.cc:272] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 33e7b3c7fdd64255b209cc8123db9646: Network error: Client connection negotiation failed: client connection to 127.31.250.2:46079: connect: Connection refused (error 111) I0221 01:30:08.674073 11758 leader_election.cc:243] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. I0221 01:30:08.674247 17054 raft_consensus.cc:2162] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 2.52 s I0221 01:30:08.674276 17054 raft_consensus.cc:1908] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority W0221 01:30:08.867223 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:30:09.868710 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:30:10.870201 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:30:11.872217 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:30:12.873664 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) I0221 01:30:13.215066 12150 raft_consensus.cc:411] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0221 01:30:13.215157 12150 raft_consensus.cc:2162] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.86 s I0221 01:30:13.215200 12150 raft_consensus.cc:435] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8ec78660e8642b4b82113d62a3042cb" member_type: VOTER last_known_addr { host: "127.31.250.1" port: 40370 } } peers { permanent_uuid: "33e7b3c7fdd64255b209cc8123db9646" member_type: VOTER last_known_addr { host: "127.31.250.2" port: 46079 } } peers { permanent_uuid: "1eb837892f874e99a16858e7958a1356" member_type: VOTER last_known_addr { host: "127.31.250.0" port: 35707 } } I0221 01:30:13.215744 12150 leader_election.cc:216] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer c8ec78660e8642b4b82113d62a3042cb I0221 01:30:13.215906 12150 leader_election.cc:216] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 33e7b3c7fdd64255b209cc8123db9646 W0221 01:30:13.216933 11759 leader_election.cc:272] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c8ec78660e8642b4b82113d62a3042cb: Network error: Client connection negotiation failed: client connection to 127.31.250.1:40370: connect: Connection refused (error 111) W0221 01:30:13.217061 11758 leader_election.cc:272] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 33e7b3c7fdd64255b209cc8123db9646: Network error: Client connection negotiation failed: client connection to 127.31.250.2:46079: connect: Connection refused (error 111) I0221 01:30:13.217152 11758 leader_election.cc:243] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. I0221 01:30:13.217466 17088 raft_consensus.cc:2162] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 8.03 s I0221 01:30:13.217557 17088 raft_consensus.cc:1908] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority W0221 01:30:13.875387 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:30:14.876708 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:30:15.878106 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:30:16.879891 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:30:17.881386 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:30:18.882815 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:30:19.884130 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) I0221 01:30:20.008674 8186 cluster_verifier.cc:72] Check not successful yet, sleeping and retrying: Timed out: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.0.0.1:44820, num_attempts: 241) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:30:20.885576 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:30:21.887064 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:30:22.888685 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) I0221 01:30:23.298818 12150 raft_consensus.cc:411] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0221 01:30:23.298907 12150 raft_consensus.cc:2162] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 14.7 s I0221 01:30:23.298949 12150 raft_consensus.cc:435] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8ec78660e8642b4b82113d62a3042cb" member_type: VOTER last_known_addr { host: "127.31.250.1" port: 40370 } } peers { permanent_uuid: "33e7b3c7fdd64255b209cc8123db9646" member_type: VOTER last_known_addr { host: "127.31.250.2" port: 46079 } } peers { permanent_uuid: "1eb837892f874e99a16858e7958a1356" member_type: VOTER last_known_addr { host: "127.31.250.0" port: 35707 } } I0221 01:30:23.299458 12150 leader_election.cc:216] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer c8ec78660e8642b4b82113d62a3042cb I0221 01:30:23.299603 12150 leader_election.cc:216] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 33e7b3c7fdd64255b209cc8123db9646 W0221 01:30:23.300465 11759 leader_election.cc:272] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c8ec78660e8642b4b82113d62a3042cb: Network error: Client connection negotiation failed: client connection to 127.31.250.1:40370: connect: Connection refused (error 111) W0221 01:30:23.300566 11758 leader_election.cc:272] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 33e7b3c7fdd64255b209cc8123db9646: Network error: Client connection negotiation failed: client connection to 127.31.250.2:46079: connect: Connection refused (error 111) I0221 01:30:23.300647 11758 leader_election.cc:243] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. I0221 01:30:23.300945 17201 raft_consensus.cc:2162] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 9.32 s I0221 01:30:23.301023 17201 raft_consensus.cc:1908] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority W0221 01:30:23.890590 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:30:24.892036 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:30:25.893002 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:30:26.894651 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:30:27.896266 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:30:28.898536 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:30:29.899976 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:30:30.901602 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:30:31.903615 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:30:32.905143 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:30:33.906599 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) I0221 01:30:34.419737 12150 raft_consensus.cc:411] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0221 01:30:34.419818 12150 raft_consensus.cc:2162] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 4 s I0221 01:30:34.419857 12150 raft_consensus.cc:435] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8ec78660e8642b4b82113d62a3042cb" member_type: VOTER last_known_addr { host: "127.31.250.1" port: 40370 } } peers { permanent_uuid: "33e7b3c7fdd64255b209cc8123db9646" member_type: VOTER last_known_addr { host: "127.31.250.2" port: 46079 } } peers { permanent_uuid: "1eb837892f874e99a16858e7958a1356" member_type: VOTER last_known_addr { host: "127.31.250.0" port: 35707 } } I0221 01:30:34.420338 12150 leader_election.cc:216] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer c8ec78660e8642b4b82113d62a3042cb I0221 01:30:34.420486 12150 leader_election.cc:216] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 33e7b3c7fdd64255b209cc8123db9646 W0221 01:30:34.421685 11759 leader_election.cc:272] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c8ec78660e8642b4b82113d62a3042cb: Network error: Client connection negotiation failed: client connection to 127.31.250.1:40370: connect: Connection refused (error 111) W0221 01:30:34.421782 11758 leader_election.cc:272] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 33e7b3c7fdd64255b209cc8123db9646: Network error: Client connection negotiation failed: client connection to 127.31.250.2:46079: connect: Connection refused (error 111) I0221 01:30:34.421860 11758 leader_election.cc:243] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. I0221 01:30:34.422201 17333 raft_consensus.cc:2162] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.53 s I0221 01:30:34.422291 17333 raft_consensus.cc:1908] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority W0221 01:30:34.908465 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:30:35.910009 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:30:36.911640 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) I0221 01:30:37.475152 12150 raft_consensus.cc:411] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0221 01:30:37.475239 12150 raft_consensus.cc:2162] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 9.21 s I0221 01:30:37.475316 12150 raft_consensus.cc:435] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8ec78660e8642b4b82113d62a3042cb" member_type: VOTER last_known_addr { host: "127.31.250.1" port: 40370 } } peers { permanent_uuid: "33e7b3c7fdd64255b209cc8123db9646" member_type: VOTER last_known_addr { host: "127.31.250.2" port: 46079 } } peers { permanent_uuid: "1eb837892f874e99a16858e7958a1356" member_type: VOTER last_known_addr { host: "127.31.250.0" port: 35707 } } I0221 01:30:37.475823 12150 leader_election.cc:216] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer c8ec78660e8642b4b82113d62a3042cb I0221 01:30:37.475983 12150 leader_election.cc:216] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 33e7b3c7fdd64255b209cc8123db9646 W0221 01:30:37.476913 11758 leader_election.cc:272] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 33e7b3c7fdd64255b209cc8123db9646: Network error: Client connection negotiation failed: client connection to 127.31.250.2:46079: connect: Connection refused (error 111) W0221 01:30:37.477169 11759 leader_election.cc:272] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c8ec78660e8642b4b82113d62a3042cb: Network error: Client connection negotiation failed: client connection to 127.31.250.1:40370: connect: Connection refused (error 111) I0221 01:30:37.477243 11759 leader_election.cc:243] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. I0221 01:30:37.477527 17366 raft_consensus.cc:2162] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 9.04 s I0221 01:30:37.477628 17366 raft_consensus.cc:1908] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority W0221 01:30:37.913084 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:30:38.914538 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:30:39.916100 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:30:40.918040 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:30:41.919627 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:30:42.921140 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:30:43.922675 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:30:44.924345 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:30:45.925746 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:30:46.927762 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:30:47.929327 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) I0221 01:30:48.144031 12150 raft_consensus.cc:411] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0221 01:30:48.144119 12150 raft_consensus.cc:2162] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 14.1 s I0221 01:30:48.144161 12150 raft_consensus.cc:435] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8ec78660e8642b4b82113d62a3042cb" member_type: VOTER last_known_addr { host: "127.31.250.1" port: 40370 } } peers { permanent_uuid: "33e7b3c7fdd64255b209cc8123db9646" member_type: VOTER last_known_addr { host: "127.31.250.2" port: 46079 } } peers { permanent_uuid: "1eb837892f874e99a16858e7958a1356" member_type: VOTER last_known_addr { host: "127.31.250.0" port: 35707 } } I0221 01:30:48.144667 12150 leader_election.cc:216] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer c8ec78660e8642b4b82113d62a3042cb I0221 01:30:48.144824 12150 leader_election.cc:216] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 33e7b3c7fdd64255b209cc8123db9646 W0221 01:30:48.145565 11759 leader_election.cc:272] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c8ec78660e8642b4b82113d62a3042cb: Network error: Client connection negotiation failed: client connection to 127.31.250.1:40370: connect: Connection refused (error 111) W0221 01:30:48.145851 11758 leader_election.cc:272] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 33e7b3c7fdd64255b209cc8123db9646: Network error: Client connection negotiation failed: client connection to 127.31.250.2:46079: connect: Connection refused (error 111) I0221 01:30:48.145927 11758 leader_election.cc:243] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. I0221 01:30:48.146245 17460 raft_consensus.cc:2162] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 11.7 s I0221 01:30:48.146333 17460 raft_consensus.cc:1908] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority W0221 01:30:48.930879 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:30:49.932703 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:30:50.934418 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) I0221 01:30:51.068675 8186 cluster_verifier.cc:72] Check not successful yet, sleeping and retrying: Timed out: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.0.0.1:44820, num_attempts: 241) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:30:51.936097 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:30:52.937988 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:30:53.939609 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:30:54.941110 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:30:55.942827 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:30:56.944299 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:30:57.946274 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:30:58.947803 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:30:59.952060 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:31:00.953745 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) I0221 01:31:01.716446 12150 raft_consensus.cc:411] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0221 01:31:01.716531 12150 raft_consensus.cc:2162] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 9.5 s I0221 01:31:01.716575 12150 raft_consensus.cc:435] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8ec78660e8642b4b82113d62a3042cb" member_type: VOTER last_known_addr { host: "127.31.250.1" port: 40370 } } peers { permanent_uuid: "33e7b3c7fdd64255b209cc8123db9646" member_type: VOTER last_known_addr { host: "127.31.250.2" port: 46079 } } peers { permanent_uuid: "1eb837892f874e99a16858e7958a1356" member_type: VOTER last_known_addr { host: "127.31.250.0" port: 35707 } } I0221 01:31:01.717058 12150 leader_election.cc:216] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer c8ec78660e8642b4b82113d62a3042cb I0221 01:31:01.717208 12150 leader_election.cc:216] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 33e7b3c7fdd64255b209cc8123db9646 W0221 01:31:01.718053 11759 leader_election.cc:272] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c8ec78660e8642b4b82113d62a3042cb: Network error: Client connection negotiation failed: client connection to 127.31.250.1:40370: connect: Connection refused (error 111) W0221 01:31:01.718402 11758 leader_election.cc:272] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 33e7b3c7fdd64255b209cc8123db9646: Network error: Client connection negotiation failed: client connection to 127.31.250.2:46079: connect: Connection refused (error 111) I0221 01:31:01.718477 11758 leader_election.cc:243] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. I0221 01:31:01.718791 17590 raft_consensus.cc:2162] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 18.1 s I0221 01:31:01.718868 17590 raft_consensus.cc:1908] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority W0221 01:31:01.955629 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:31:02.957309 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:31:03.959328 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:31:04.960846 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:31:05.962328 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:31:06.963783 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:31:07.965196 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:31:08.966671 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:31:09.968375 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:31:10.969777 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:31:11.971457 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:31:12.972956 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:31:13.974511 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:31:14.976220 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:31:15.978224 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:31:16.979945 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:31:17.981459 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:31:18.982151 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:31:19.983667 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:31:20.985322 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) I0221 01:31:21.720424 12150 raft_consensus.cc:411] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0221 01:31:21.720520 12150 raft_consensus.cc:2162] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 8.35 s I0221 01:31:21.720582 12150 raft_consensus.cc:435] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8ec78660e8642b4b82113d62a3042cb" member_type: VOTER last_known_addr { host: "127.31.250.1" port: 40370 } } peers { permanent_uuid: "33e7b3c7fdd64255b209cc8123db9646" member_type: VOTER last_known_addr { host: "127.31.250.2" port: 46079 } } peers { permanent_uuid: "1eb837892f874e99a16858e7958a1356" member_type: VOTER last_known_addr { host: "127.31.250.0" port: 35707 } } I0221 01:31:21.721099 12150 leader_election.cc:216] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer c8ec78660e8642b4b82113d62a3042cb I0221 01:31:21.721281 12150 leader_election.cc:216] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 33e7b3c7fdd64255b209cc8123db9646 W0221 01:31:21.722523 11758 leader_election.cc:272] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 33e7b3c7fdd64255b209cc8123db9646: Network error: Client connection negotiation failed: client connection to 127.31.250.2:46079: connect: Connection refused (error 111) W0221 01:31:21.722618 11759 leader_election.cc:272] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c8ec78660e8642b4b82113d62a3042cb: Network error: Client connection negotiation failed: client connection to 127.31.250.1:40370: connect: Connection refused (error 111) I0221 01:31:21.722702 11759 leader_election.cc:243] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. I0221 01:31:21.723043 17972 raft_consensus.cc:2162] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 2.13 s I0221 01:31:21.723129 17972 raft_consensus.cc:1908] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority W0221 01:31:21.987193 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) I0221 01:31:22.138133 8186 cluster_verifier.cc:72] Check not successful yet, sleeping and retrying: Timed out: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.0.0.1:44820, num_attempts: 241) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:31:22.988803 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:31:23.990340 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:31:24.991937 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) I0221 01:31:25.872660 12150 raft_consensus.cc:411] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0221 01:31:25.872697 12150 raft_consensus.cc:2162] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 16.5 s I0221 01:31:25.872719 12150 raft_consensus.cc:435] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8ec78660e8642b4b82113d62a3042cb" member_type: VOTER last_known_addr { host: "127.31.250.1" port: 40370 } } peers { permanent_uuid: "33e7b3c7fdd64255b209cc8123db9646" member_type: VOTER last_known_addr { host: "127.31.250.2" port: 46079 } } peers { permanent_uuid: "1eb837892f874e99a16858e7958a1356" member_type: VOTER last_known_addr { host: "127.31.250.0" port: 35707 } } I0221 01:31:25.872889 12150 leader_election.cc:216] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer c8ec78660e8642b4b82113d62a3042cb I0221 01:31:25.872946 12150 leader_election.cc:216] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 33e7b3c7fdd64255b209cc8123db9646 W0221 01:31:25.873479 11759 leader_election.cc:272] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c8ec78660e8642b4b82113d62a3042cb: Network error: Client connection negotiation failed: client connection to 127.31.250.1:40370: connect: Connection refused (error 111) W0221 01:31:25.873522 11758 leader_election.cc:272] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 33e7b3c7fdd64255b209cc8123db9646: Network error: Client connection negotiation failed: client connection to 127.31.250.2:46079: connect: Connection refused (error 111) I0221 01:31:25.873543 11758 leader_election.cc:243] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. I0221 01:31:25.873718 18024 raft_consensus.cc:2162] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 10.5 s I0221 01:31:25.873746 18024 raft_consensus.cc:1908] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority W0221 01:31:25.993353 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:31:26.994659 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:31:27.996217 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:31:28.997933 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:31:29.998759 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:31:31.000633 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:31:32.001260 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:31:33.002756 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:31:34.004757 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:31:35.006582 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:31:36.008165 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:31:37.010033 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) I0221 01:31:37.936324 12150 raft_consensus.cc:411] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0221 01:31:37.936405 12150 raft_consensus.cc:2162] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 7.6 s I0221 01:31:37.936444 12150 raft_consensus.cc:435] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8ec78660e8642b4b82113d62a3042cb" member_type: VOTER last_known_addr { host: "127.31.250.1" port: 40370 } } peers { permanent_uuid: "33e7b3c7fdd64255b209cc8123db9646" member_type: VOTER last_known_addr { host: "127.31.250.2" port: 46079 } } peers { permanent_uuid: "1eb837892f874e99a16858e7958a1356" member_type: VOTER last_known_addr { host: "127.31.250.0" port: 35707 } } I0221 01:31:37.936898 12150 leader_election.cc:216] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer c8ec78660e8642b4b82113d62a3042cb I0221 01:31:37.937038 12150 leader_election.cc:216] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 33e7b3c7fdd64255b209cc8123db9646 W0221 01:31:37.938197 11759 leader_election.cc:272] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c8ec78660e8642b4b82113d62a3042cb: Network error: Client connection negotiation failed: client connection to 127.31.250.1:40370: connect: Connection refused (error 111) W0221 01:31:37.938302 11758 leader_election.cc:272] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 33e7b3c7fdd64255b209cc8123db9646: Network error: Client connection negotiation failed: client connection to 127.31.250.2:46079: connect: Connection refused (error 111) I0221 01:31:37.938403 11758 leader_election.cc:243] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. I0221 01:31:37.938719 18177 raft_consensus.cc:2162] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 11.2 s I0221 01:31:37.938802 18177 raft_consensus.cc:1908] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority W0221 01:31:38.011962 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:31:39.013818 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:31:40.015362 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:31:41.016841 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:31:42.018395 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:31:43.019245 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:31:44.020913 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:31:45.022424 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:31:46.023958 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:31:47.025645 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:31:48.027143 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:31:49.028617 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:31:50.030165 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) I0221 01:31:50.831104 12150 raft_consensus.cc:411] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0221 01:31:50.831209 12150 raft_consensus.cc:2162] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 5.35 s I0221 01:31:50.831253 12150 raft_consensus.cc:435] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8ec78660e8642b4b82113d62a3042cb" member_type: VOTER last_known_addr { host: "127.31.250.1" port: 40370 } } peers { permanent_uuid: "33e7b3c7fdd64255b209cc8123db9646" member_type: VOTER last_known_addr { host: "127.31.250.2" port: 46079 } } peers { permanent_uuid: "1eb837892f874e99a16858e7958a1356" member_type: VOTER last_known_addr { host: "127.31.250.0" port: 35707 } } I0221 01:31:50.831809 12150 leader_election.cc:216] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer c8ec78660e8642b4b82113d62a3042cb I0221 01:31:50.831964 12150 leader_election.cc:216] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 33e7b3c7fdd64255b209cc8123db9646 W0221 01:31:50.833233 11759 leader_election.cc:272] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c8ec78660e8642b4b82113d62a3042cb: Network error: Client connection negotiation failed: client connection to 127.31.250.1:40370: connect: Connection refused (error 111) W0221 01:31:50.833348 11758 leader_election.cc:272] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 33e7b3c7fdd64255b209cc8123db9646: Network error: Client connection negotiation failed: client connection to 127.31.250.2:46079: connect: Connection refused (error 111) I0221 01:31:50.833427 11758 leader_election.cc:243] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. I0221 01:31:50.833726 18330 raft_consensus.cc:2162] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 15.5 s I0221 01:31:50.833816 18330 raft_consensus.cc:1908] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority W0221 01:31:51.031613 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:31:52.033138 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:31:53.034574 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) I0221 01:31:53.195698 8186 cluster_verifier.cc:72] Check not successful yet, sleeping and retrying: Timed out: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.0.0.1:44820, num_attempts: 241) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:31:54.036049 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:31:55.037807 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:31:56.039435 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:31:57.040858 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:31:58.042563 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:31:59.044580 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:32:00.046039 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:32:01.047332 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:32:02.048840 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:32:03.050351 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:32:04.051508 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:32:05.053014 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:32:06.054605 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:32:07.056438 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:32:08.057345 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) I0221 01:32:08.087379 12150 raft_consensus.cc:411] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0221 01:32:08.087448 12150 raft_consensus.cc:2162] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 3.87 s I0221 01:32:08.087482 12150 raft_consensus.cc:435] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8ec78660e8642b4b82113d62a3042cb" member_type: VOTER last_known_addr { host: "127.31.250.1" port: 40370 } } peers { permanent_uuid: "33e7b3c7fdd64255b209cc8123db9646" member_type: VOTER last_known_addr { host: "127.31.250.2" port: 46079 } } peers { permanent_uuid: "1eb837892f874e99a16858e7958a1356" member_type: VOTER last_known_addr { host: "127.31.250.0" port: 35707 } } I0221 01:32:08.087785 12150 leader_election.cc:216] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer c8ec78660e8642b4b82113d62a3042cb I0221 01:32:08.087879 12150 leader_election.cc:216] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 33e7b3c7fdd64255b209cc8123db9646 W0221 01:32:08.088536 11758 leader_election.cc:272] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 33e7b3c7fdd64255b209cc8123db9646: Network error: Client connection negotiation failed: client connection to 127.31.250.2:46079: connect: Connection refused (error 111) W0221 01:32:08.088591 11759 leader_election.cc:272] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c8ec78660e8642b4b82113d62a3042cb: Network error: Client connection negotiation failed: client connection to 127.31.250.1:40370: connect: Connection refused (error 111) I0221 01:32:08.088634 11759 leader_election.cc:243] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. I0221 01:32:08.088778 18522 raft_consensus.cc:2162] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 18.1 s I0221 01:32:08.088830 18522 raft_consensus.cc:1908] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority W0221 01:32:09.058801 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:32:10.060784 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:32:11.062361 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:32:12.063864 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:32:13.065083 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:32:14.066021 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:32:15.066783 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:32:16.068581 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:32:17.069382 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:32:18.070212 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:32:19.071787 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:32:20.072536 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:32:21.073993 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:32:22.074981 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:32:23.076557 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:32:24.078058 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) I0221 01:32:24.223572 8186 cluster_verifier.cc:72] Check not successful yet, sleeping and retrying: Timed out: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.0.0.1:44820, num_attempts: 241) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:32:25.080168 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:32:26.081208 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:32:27.082697 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) I0221 01:32:27.856139 12150 raft_consensus.cc:411] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0221 01:32:27.856256 12150 raft_consensus.cc:2162] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 16.4 s I0221 01:32:27.856325 12150 raft_consensus.cc:435] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8ec78660e8642b4b82113d62a3042cb" member_type: VOTER last_known_addr { host: "127.31.250.1" port: 40370 } } peers { permanent_uuid: "33e7b3c7fdd64255b209cc8123db9646" member_type: VOTER last_known_addr { host: "127.31.250.2" port: 46079 } } peers { permanent_uuid: "1eb837892f874e99a16858e7958a1356" member_type: VOTER last_known_addr { host: "127.31.250.0" port: 35707 } } I0221 01:32:27.856850 12150 leader_election.cc:216] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer c8ec78660e8642b4b82113d62a3042cb I0221 01:32:27.857026 12150 leader_election.cc:216] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 33e7b3c7fdd64255b209cc8123db9646 W0221 01:32:27.858337 11759 leader_election.cc:272] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c8ec78660e8642b4b82113d62a3042cb: Network error: Client connection negotiation failed: client connection to 127.31.250.1:40370: connect: Connection refused (error 111) W0221 01:32:27.858438 11758 leader_election.cc:272] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 33e7b3c7fdd64255b209cc8123db9646: Network error: Client connection negotiation failed: client connection to 127.31.250.2:46079: connect: Connection refused (error 111) I0221 01:32:27.858525 11758 leader_election.cc:243] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. I0221 01:32:27.858831 18767 raft_consensus.cc:2162] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 15 s I0221 01:32:27.858918 18767 raft_consensus.cc:1908] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority W0221 01:32:28.084126 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:32:29.085672 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:32:30.087028 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:32:31.089120 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:32:32.090589 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:32:33.092103 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:32:34.093575 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:32:35.095412 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:32:36.096937 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:32:37.098814 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:32:38.100419 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:32:39.101917 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:32:40.103440 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:32:41.104949 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:32:42.106358 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:32:43.108120 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:32:44.109771 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) I0221 01:32:44.435020 12150 raft_consensus.cc:411] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0221 01:32:44.435108 12150 raft_consensus.cc:2162] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 14.4 s I0221 01:32:44.435154 12150 raft_consensus.cc:435] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8ec78660e8642b4b82113d62a3042cb" member_type: VOTER last_known_addr { host: "127.31.250.1" port: 40370 } } peers { permanent_uuid: "33e7b3c7fdd64255b209cc8123db9646" member_type: VOTER last_known_addr { host: "127.31.250.2" port: 46079 } } peers { permanent_uuid: "1eb837892f874e99a16858e7958a1356" member_type: VOTER last_known_addr { host: "127.31.250.0" port: 35707 } } I0221 01:32:44.435696 12150 leader_election.cc:216] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer c8ec78660e8642b4b82113d62a3042cb I0221 01:32:44.435858 12150 leader_election.cc:216] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 33e7b3c7fdd64255b209cc8123db9646 W0221 01:32:44.436830 11759 leader_election.cc:272] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c8ec78660e8642b4b82113d62a3042cb: Network error: Client connection negotiation failed: client connection to 127.31.250.1:40370: connect: Connection refused (error 111) W0221 01:32:44.436931 11758 leader_election.cc:272] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 33e7b3c7fdd64255b209cc8123db9646: Network error: Client connection negotiation failed: client connection to 127.31.250.2:46079: connect: Connection refused (error 111) I0221 01:32:44.437011 11758 leader_election.cc:243] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. I0221 01:32:44.437412 18964 raft_consensus.cc:2162] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 16.5 s I0221 01:32:44.437513 18964 raft_consensus.cc:1908] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority W0221 01:32:45.110913 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:32:46.112818 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:32:47.114732 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:32:48.116276 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:32:49.118407 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:32:50.120291 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:32:51.121770 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:32:52.123265 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:32:53.124692 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:32:54.126260 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:32:55.128049 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) I0221 01:32:55.266943 8186 cluster_verifier.cc:72] Check not successful yet, sleeping and retrying: Timed out: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.0.0.1:44820, num_attempts: 241) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:32:56.130002 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:32:57.131790 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:32:58.133280 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:32:59.134706 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:33:00.136600 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:33:01.138373 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:33:02.140180 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) I0221 01:33:02.454345 12150 raft_consensus.cc:411] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0221 01:33:02.454450 12150 raft_consensus.cc:2162] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 19.7 s I0221 01:33:02.454497 12150 raft_consensus.cc:435] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8ec78660e8642b4b82113d62a3042cb" member_type: VOTER last_known_addr { host: "127.31.250.1" port: 40370 } } peers { permanent_uuid: "33e7b3c7fdd64255b209cc8123db9646" member_type: VOTER last_known_addr { host: "127.31.250.2" port: 46079 } } peers { permanent_uuid: "1eb837892f874e99a16858e7958a1356" member_type: VOTER last_known_addr { host: "127.31.250.0" port: 35707 } } I0221 01:33:02.455057 12150 leader_election.cc:216] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer c8ec78660e8642b4b82113d62a3042cb I0221 01:33:02.455217 12150 leader_election.cc:216] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 33e7b3c7fdd64255b209cc8123db9646 W0221 01:33:02.456327 11759 leader_election.cc:272] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c8ec78660e8642b4b82113d62a3042cb: Network error: Client connection negotiation failed: client connection to 127.31.250.1:40370: connect: Connection refused (error 111) W0221 01:33:02.456428 11758 leader_election.cc:272] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 33e7b3c7fdd64255b209cc8123db9646: Network error: Client connection negotiation failed: client connection to 127.31.250.2:46079: connect: Connection refused (error 111) I0221 01:33:02.456512 11758 leader_election.cc:243] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. I0221 01:33:02.456820 19171 raft_consensus.cc:2162] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 7.55 s I0221 01:33:02.456915 19171 raft_consensus.cc:1908] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority W0221 01:33:03.141705 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:33:04.143342 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:33:05.144973 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:33:06.146930 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:33:07.148624 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:33:08.149755 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:33:09.151751 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:33:10.153262 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:33:11.155220 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) I0221 01:33:11.562943 12150 raft_consensus.cc:411] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0221 01:33:11.563030 12150 raft_consensus.cc:2162] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 2.05 s I0221 01:33:11.563066 12150 raft_consensus.cc:435] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8ec78660e8642b4b82113d62a3042cb" member_type: VOTER last_known_addr { host: "127.31.250.1" port: 40370 } } peers { permanent_uuid: "33e7b3c7fdd64255b209cc8123db9646" member_type: VOTER last_known_addr { host: "127.31.250.2" port: 46079 } } peers { permanent_uuid: "1eb837892f874e99a16858e7958a1356" member_type: VOTER last_known_addr { host: "127.31.250.0" port: 35707 } } I0221 01:33:11.563588 12150 leader_election.cc:216] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer c8ec78660e8642b4b82113d62a3042cb I0221 01:33:11.563740 12150 leader_election.cc:216] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 33e7b3c7fdd64255b209cc8123db9646 W0221 01:33:11.564749 11758 leader_election.cc:272] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 33e7b3c7fdd64255b209cc8123db9646: Network error: Client connection negotiation failed: client connection to 127.31.250.2:46079: connect: Connection refused (error 111) W0221 01:33:11.564843 11759 leader_election.cc:272] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c8ec78660e8642b4b82113d62a3042cb: Network error: Client connection negotiation failed: client connection to 127.31.250.1:40370: connect: Connection refused (error 111) I0221 01:33:11.564937 11759 leader_election.cc:243] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. I0221 01:33:11.565251 19286 raft_consensus.cc:2162] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 8.94 s I0221 01:33:11.565330 19286 raft_consensus.cc:1908] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority W0221 01:33:12.157073 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:33:13.158610 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:33:14.160480 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:33:15.161422 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:33:16.162518 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:33:17.164419 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:33:18.165673 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:33:19.167138 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:33:20.168172 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:33:21.168880 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:33:22.169807 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) I0221 01:33:22.180850 12150 raft_consensus.cc:411] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0221 01:33:22.180896 12150 raft_consensus.cc:2162] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 17.2 s I0221 01:33:22.180915 12150 raft_consensus.cc:435] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8ec78660e8642b4b82113d62a3042cb" member_type: VOTER last_known_addr { host: "127.31.250.1" port: 40370 } } peers { permanent_uuid: "33e7b3c7fdd64255b209cc8123db9646" member_type: VOTER last_known_addr { host: "127.31.250.2" port: 46079 } } peers { permanent_uuid: "1eb837892f874e99a16858e7958a1356" member_type: VOTER last_known_addr { host: "127.31.250.0" port: 35707 } } I0221 01:33:22.181133 12150 leader_election.cc:216] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer c8ec78660e8642b4b82113d62a3042cb I0221 01:33:22.181202 12150 leader_election.cc:216] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 33e7b3c7fdd64255b209cc8123db9646 W0221 01:33:22.181879 11758 leader_election.cc:272] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 33e7b3c7fdd64255b209cc8123db9646: Network error: Client connection negotiation failed: client connection to 127.31.250.2:46079: connect: Connection refused (error 111) W0221 01:33:22.181939 11759 leader_election.cc:272] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c8ec78660e8642b4b82113d62a3042cb: Network error: Client connection negotiation failed: client connection to 127.31.250.1:40370: connect: Connection refused (error 111) I0221 01:33:22.181972 11759 leader_election.cc:243] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. I0221 01:33:22.182152 19415 raft_consensus.cc:2162] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 13.7 s I0221 01:33:22.182184 19415 raft_consensus.cc:1908] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority W0221 01:33:23.170994 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:33:24.172272 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:33:25.173771 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:33:26.175045 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) I0221 01:33:26.302189 8186 cluster_verifier.cc:72] Check not successful yet, sleeping and retrying: Timed out: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.0.0.1:44820, num_attempts: 241) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:33:27.177117 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:33:28.177876 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:33:29.179069 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:33:30.180574 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:33:31.182098 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:33:32.183706 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:33:33.185153 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:33:34.186185 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:33:35.187208 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:33:36.187969 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:33:37.188807 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) I0221 01:33:37.360448 12150 raft_consensus.cc:411] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0221 01:33:37.360481 12150 raft_consensus.cc:2162] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 5.91 s I0221 01:33:37.360498 12150 raft_consensus.cc:435] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8ec78660e8642b4b82113d62a3042cb" member_type: VOTER last_known_addr { host: "127.31.250.1" port: 40370 } } peers { permanent_uuid: "33e7b3c7fdd64255b209cc8123db9646" member_type: VOTER last_known_addr { host: "127.31.250.2" port: 46079 } } peers { permanent_uuid: "1eb837892f874e99a16858e7958a1356" member_type: VOTER last_known_addr { host: "127.31.250.0" port: 35707 } } I0221 01:33:37.360671 12150 leader_election.cc:216] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer c8ec78660e8642b4b82113d62a3042cb I0221 01:33:37.360724 12150 leader_election.cc:216] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 33e7b3c7fdd64255b209cc8123db9646 W0221 01:33:37.361114 11759 leader_election.cc:272] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c8ec78660e8642b4b82113d62a3042cb: Network error: Client connection negotiation failed: client connection to 127.31.250.1:40370: connect: Connection refused (error 111) W0221 01:33:37.361174 11758 leader_election.cc:272] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 33e7b3c7fdd64255b209cc8123db9646: Network error: Client connection negotiation failed: client connection to 127.31.250.2:46079: connect: Connection refused (error 111) I0221 01:33:37.361196 11758 leader_election.cc:243] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. I0221 01:33:37.361356 19622 raft_consensus.cc:2162] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 10.6 s I0221 01:33:37.361382 19622 raft_consensus.cc:1908] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority W0221 01:33:38.190843 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:33:39.192374 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:33:40.194329 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:33:41.196169 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:33:42.198161 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:33:43.200199 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:33:44.201851 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:33:45.203768 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:33:46.205355 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:33:47.207003 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:33:48.208564 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:33:49.209302 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) I0221 01:33:49.508527 12150 raft_consensus.cc:411] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0221 01:33:49.508586 12150 raft_consensus.cc:2162] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 2.42 s I0221 01:33:49.508611 12150 raft_consensus.cc:435] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8ec78660e8642b4b82113d62a3042cb" member_type: VOTER last_known_addr { host: "127.31.250.1" port: 40370 } } peers { permanent_uuid: "33e7b3c7fdd64255b209cc8123db9646" member_type: VOTER last_known_addr { host: "127.31.250.2" port: 46079 } } peers { permanent_uuid: "1eb837892f874e99a16858e7958a1356" member_type: VOTER last_known_addr { host: "127.31.250.0" port: 35707 } } I0221 01:33:49.508932 12150 leader_election.cc:216] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer c8ec78660e8642b4b82113d62a3042cb I0221 01:33:49.509033 12150 leader_election.cc:216] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 33e7b3c7fdd64255b209cc8123db9646 W0221 01:33:49.509694 11759 leader_election.cc:272] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c8ec78660e8642b4b82113d62a3042cb: Network error: Client connection negotiation failed: client connection to 127.31.250.1:40370: connect: Connection refused (error 111) W0221 01:33:49.509760 11758 leader_election.cc:272] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 33e7b3c7fdd64255b209cc8123db9646: Network error: Client connection negotiation failed: client connection to 127.31.250.2:46079: connect: Connection refused (error 111) I0221 01:33:49.509810 11758 leader_election.cc:243] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. I0221 01:33:49.510025 19749 raft_consensus.cc:2162] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 3.46 s I0221 01:33:49.510082 19749 raft_consensus.cc:1908] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority W0221 01:33:50.210448 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:33:51.211730 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:33:52.213239 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:33:53.215246 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:33:54.216992 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) I0221 01:33:54.995009 12150 raft_consensus.cc:411] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0221 01:33:54.995100 12150 raft_consensus.cc:2162] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 15.1 s I0221 01:33:54.995142 12150 raft_consensus.cc:435] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8ec78660e8642b4b82113d62a3042cb" member_type: VOTER last_known_addr { host: "127.31.250.1" port: 40370 } } peers { permanent_uuid: "33e7b3c7fdd64255b209cc8123db9646" member_type: VOTER last_known_addr { host: "127.31.250.2" port: 46079 } } peers { permanent_uuid: "1eb837892f874e99a16858e7958a1356" member_type: VOTER last_known_addr { host: "127.31.250.0" port: 35707 } } I0221 01:33:54.995676 12150 leader_election.cc:216] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer c8ec78660e8642b4b82113d62a3042cb I0221 01:33:54.995849 12150 leader_election.cc:216] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 33e7b3c7fdd64255b209cc8123db9646 W0221 01:33:54.996992 11758 leader_election.cc:272] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 33e7b3c7fdd64255b209cc8123db9646: Network error: Client connection negotiation failed: client connection to 127.31.250.2:46079: connect: Connection refused (error 111) W0221 01:33:54.997095 11759 leader_election.cc:272] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c8ec78660e8642b4b82113d62a3042cb: Network error: Client connection negotiation failed: client connection to 127.31.250.1:40370: connect: Connection refused (error 111) I0221 01:33:54.997179 11759 leader_election.cc:243] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. I0221 01:33:54.997485 19815 raft_consensus.cc:2162] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 12.3 s I0221 01:33:54.997570 19815 raft_consensus.cc:1908] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority W0221 01:33:55.218441 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:33:56.219173 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:33:57.220708 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) I0221 01:33:57.324842 8186 cluster_verifier.cc:72] Check not successful yet, sleeping and retrying: Timed out: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.0.0.1:44820, num_attempts: 241) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:33:58.222508 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:33:59.224123 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:34:00.226114 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:34:01.227756 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:34:02.229377 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:34:03.231359 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:34:04.232863 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:34:05.234426 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:34:06.235888 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:34:07.237850 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:34:08.239415 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) I0221 01:34:09.018299 12150 raft_consensus.cc:411] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0221 01:34:09.018345 12150 raft_consensus.cc:2162] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 10.5 s I0221 01:34:09.018365 12150 raft_consensus.cc:435] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8ec78660e8642b4b82113d62a3042cb" member_type: VOTER last_known_addr { host: "127.31.250.1" port: 40370 } } peers { permanent_uuid: "33e7b3c7fdd64255b209cc8123db9646" member_type: VOTER last_known_addr { host: "127.31.250.2" port: 46079 } } peers { permanent_uuid: "1eb837892f874e99a16858e7958a1356" member_type: VOTER last_known_addr { host: "127.31.250.0" port: 35707 } } I0221 01:34:09.018630 12150 leader_election.cc:216] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer c8ec78660e8642b4b82113d62a3042cb I0221 01:34:09.018712 12150 leader_election.cc:216] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 33e7b3c7fdd64255b209cc8123db9646 W0221 01:34:09.019332 11758 leader_election.cc:272] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 33e7b3c7fdd64255b209cc8123db9646: Network error: Client connection negotiation failed: client connection to 127.31.250.2:46079: connect: Connection refused (error 111) W0221 01:34:09.019378 11759 leader_election.cc:272] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c8ec78660e8642b4b82113d62a3042cb: Network error: Client connection negotiation failed: client connection to 127.31.250.1:40370: connect: Connection refused (error 111) I0221 01:34:09.019412 11759 leader_election.cc:243] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. I0221 01:34:09.019580 19987 raft_consensus.cc:2162] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 8.21 s I0221 01:34:09.019618 19987 raft_consensus.cc:1908] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority W0221 01:34:09.240214 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:34:10.241710 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:34:11.243738 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:34:12.245314 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:34:13.246904 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:34:14.248884 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:34:15.250746 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:34:16.251538 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:34:17.252924 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:34:18.254580 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) I0221 01:34:18.926409 12150 raft_consensus.cc:411] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0221 01:34:18.926491 12150 raft_consensus.cc:2162] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 13.3 s I0221 01:34:18.926532 12150 raft_consensus.cc:435] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8ec78660e8642b4b82113d62a3042cb" member_type: VOTER last_known_addr { host: "127.31.250.1" port: 40370 } } peers { permanent_uuid: "33e7b3c7fdd64255b209cc8123db9646" member_type: VOTER last_known_addr { host: "127.31.250.2" port: 46079 } } peers { permanent_uuid: "1eb837892f874e99a16858e7958a1356" member_type: VOTER last_known_addr { host: "127.31.250.0" port: 35707 } } I0221 01:34:18.927008 12150 leader_election.cc:216] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer c8ec78660e8642b4b82113d62a3042cb I0221 01:34:18.927155 12150 leader_election.cc:216] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 33e7b3c7fdd64255b209cc8123db9646 W0221 01:34:18.928163 11758 leader_election.cc:272] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 33e7b3c7fdd64255b209cc8123db9646: Network error: Client connection negotiation failed: client connection to 127.31.250.2:46079: connect: Connection refused (error 111) W0221 01:34:18.928356 11759 leader_election.cc:272] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c8ec78660e8642b4b82113d62a3042cb: Network error: Client connection negotiation failed: client connection to 127.31.250.1:40370: connect: Connection refused (error 111) I0221 01:34:18.928431 11759 leader_election.cc:243] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. I0221 01:34:18.928831 20145 raft_consensus.cc:2162] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 4.91 s I0221 01:34:18.928926 20145 raft_consensus.cc:1908] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority W0221 01:34:19.256053 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:34:20.257550 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:34:21.258460 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:34:22.259140 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:34:23.260007 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:34:24.261591 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:34:25.263111 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:34:26.263850 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) I0221 01:34:26.673557 12150 raft_consensus.cc:411] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0221 01:34:26.673593 12150 raft_consensus.cc:2162] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 3.54 s I0221 01:34:26.673609 12150 raft_consensus.cc:435] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8ec78660e8642b4b82113d62a3042cb" member_type: VOTER last_known_addr { host: "127.31.250.1" port: 40370 } } peers { permanent_uuid: "33e7b3c7fdd64255b209cc8123db9646" member_type: VOTER last_known_addr { host: "127.31.250.2" port: 46079 } } peers { permanent_uuid: "1eb837892f874e99a16858e7958a1356" member_type: VOTER last_known_addr { host: "127.31.250.0" port: 35707 } } I0221 01:34:26.673789 12150 leader_election.cc:216] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer c8ec78660e8642b4b82113d62a3042cb I0221 01:34:26.673846 12150 leader_election.cc:216] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 33e7b3c7fdd64255b209cc8123db9646 W0221 01:34:26.674345 11759 leader_election.cc:272] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c8ec78660e8642b4b82113d62a3042cb: Network error: Client connection negotiation failed: client connection to 127.31.250.1:40370: connect: Connection refused (error 111) W0221 01:34:26.674376 11758 leader_election.cc:272] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 33e7b3c7fdd64255b209cc8123db9646: Network error: Client connection negotiation failed: client connection to 127.31.250.2:46079: connect: Connection refused (error 111) I0221 01:34:26.674401 11758 leader_election.cc:243] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. I0221 01:34:26.674556 20223 raft_consensus.cc:2162] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 13.1 s I0221 01:34:26.674582 20223 raft_consensus.cc:1908] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority W0221 01:34:27.264508 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:34:28.265161 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) I0221 01:34:28.381700 8186 cluster_verifier.cc:72] Check not successful yet, sleeping and retrying: Timed out: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.0.0.1:44820, num_attempts: 241) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:34:29.266999 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:34:30.268692 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:34:31.270208 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:34:32.271896 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:34:33.273542 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:34:34.274335 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:34:35.275049 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:34:36.276136 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:34:37.277643 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:34:38.279136 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:34:39.280649 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:34:40.282060 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:34:41.283620 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) I0221 01:34:41.548578 12150 raft_consensus.cc:411] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0221 01:34:41.548661 12150 raft_consensus.cc:2162] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 11 s I0221 01:34:41.548702 12150 raft_consensus.cc:435] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8ec78660e8642b4b82113d62a3042cb" member_type: VOTER last_known_addr { host: "127.31.250.1" port: 40370 } } peers { permanent_uuid: "33e7b3c7fdd64255b209cc8123db9646" member_type: VOTER last_known_addr { host: "127.31.250.2" port: 46079 } } peers { permanent_uuid: "1eb837892f874e99a16858e7958a1356" member_type: VOTER last_known_addr { host: "127.31.250.0" port: 35707 } } I0221 01:34:41.549181 12150 leader_election.cc:216] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer c8ec78660e8642b4b82113d62a3042cb I0221 01:34:41.549330 12150 leader_election.cc:216] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 33e7b3c7fdd64255b209cc8123db9646 W0221 01:34:41.550330 11759 leader_election.cc:272] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c8ec78660e8642b4b82113d62a3042cb: Network error: Client connection negotiation failed: client connection to 127.31.250.1:40370: connect: Connection refused (error 111) W0221 01:34:41.550436 11758 leader_election.cc:272] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 33e7b3c7fdd64255b209cc8123db9646: Network error: Client connection negotiation failed: client connection to 127.31.250.2:46079: connect: Connection refused (error 111) I0221 01:34:41.550529 11758 leader_election.cc:243] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. I0221 01:34:41.550823 20437 raft_consensus.cc:2162] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 6.99 s I0221 01:34:41.550909 20437 raft_consensus.cc:1908] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority W0221 01:34:42.285260 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:34:43.286792 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:34:44.288525 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:34:45.290216 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:34:46.291893 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:34:47.292662 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:34:48.294301 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:34:49.296164 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) I0221 01:34:50.263455 12150 raft_consensus.cc:411] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0221 01:34:50.263545 12150 raft_consensus.cc:2162] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 14.3 s I0221 01:34:50.263586 12150 raft_consensus.cc:435] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8ec78660e8642b4b82113d62a3042cb" member_type: VOTER last_known_addr { host: "127.31.250.1" port: 40370 } } peers { permanent_uuid: "33e7b3c7fdd64255b209cc8123db9646" member_type: VOTER last_known_addr { host: "127.31.250.2" port: 46079 } } peers { permanent_uuid: "1eb837892f874e99a16858e7958a1356" member_type: VOTER last_known_addr { host: "127.31.250.0" port: 35707 } } I0221 01:34:50.264117 12150 leader_election.cc:216] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer c8ec78660e8642b4b82113d62a3042cb I0221 01:34:50.264283 12150 leader_election.cc:216] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 33e7b3c7fdd64255b209cc8123db9646 W0221 01:34:50.265420 11759 leader_election.cc:272] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c8ec78660e8642b4b82113d62a3042cb: Network error: Client connection negotiation failed: client connection to 127.31.250.1:40370: connect: Connection refused (error 111) W0221 01:34:50.265519 11758 leader_election.cc:272] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 33e7b3c7fdd64255b209cc8123db9646: Network error: Client connection negotiation failed: client connection to 127.31.250.2:46079: connect: Connection refused (error 111) I0221 01:34:50.265604 11758 leader_election.cc:243] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. I0221 01:34:50.265897 20540 raft_consensus.cc:2162] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 11.5 s I0221 01:34:50.265987 20540 raft_consensus.cc:1908] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority W0221 01:34:50.297786 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:34:51.299578 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:34:52.301057 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:34:53.303092 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:34:54.304656 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:34:55.306219 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:34:56.307970 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:34:57.309201 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:34:58.310747 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:34:59.312232 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) I0221 01:34:59.622208 8186 cluster_verifier.cc:72] Check not successful yet, sleeping and retrying: Timed out: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.0.0.1:44820, num_attempts: 242) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:35:00.313027 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:35:01.314510 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:35:02.316337 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:35:03.317903 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) I0221 01:35:03.344485 12150 raft_consensus.cc:411] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0221 01:35:03.344591 12150 raft_consensus.cc:2162] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 9.17 s I0221 01:35:03.344655 12150 raft_consensus.cc:435] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8ec78660e8642b4b82113d62a3042cb" member_type: VOTER last_known_addr { host: "127.31.250.1" port: 40370 } } peers { permanent_uuid: "33e7b3c7fdd64255b209cc8123db9646" member_type: VOTER last_known_addr { host: "127.31.250.2" port: 46079 } } peers { permanent_uuid: "1eb837892f874e99a16858e7958a1356" member_type: VOTER last_known_addr { host: "127.31.250.0" port: 35707 } } I0221 01:35:03.345206 12150 leader_election.cc:216] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer c8ec78660e8642b4b82113d62a3042cb I0221 01:35:03.345369 12150 leader_election.cc:216] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 33e7b3c7fdd64255b209cc8123db9646 W0221 01:35:03.346139 11759 leader_election.cc:272] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c8ec78660e8642b4b82113d62a3042cb: Network error: Client connection negotiation failed: client connection to 127.31.250.1:40370: connect: Connection refused (error 111) W0221 01:35:03.346599 11758 leader_election.cc:272] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 33e7b3c7fdd64255b209cc8123db9646: Network error: Client connection negotiation failed: client connection to 127.31.250.2:46079: connect: Connection refused (error 111) I0221 01:35:03.346673 11758 leader_election.cc:243] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. I0221 01:35:03.346953 20691 raft_consensus.cc:2162] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 15.4 s I0221 01:35:03.347038 20691 raft_consensus.cc:1908] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority W0221 01:35:04.319929 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:35:05.321487 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:35:06.323321 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:35:07.324757 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:35:08.326303 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:35:09.328311 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:35:10.329381 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:35:11.330476 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:35:12.332041 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:35:13.332821 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:35:14.334139 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:35:15.335674 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:35:16.337640 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:35:17.338991 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:35:18.340530 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:35:19.342512 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) I0221 01:35:20.338348 12150 raft_consensus.cc:411] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0221 01:35:20.338440 12150 raft_consensus.cc:2162] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 16 s I0221 01:35:20.338485 12150 raft_consensus.cc:435] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8ec78660e8642b4b82113d62a3042cb" member_type: VOTER last_known_addr { host: "127.31.250.1" port: 40370 } } peers { permanent_uuid: "33e7b3c7fdd64255b209cc8123db9646" member_type: VOTER last_known_addr { host: "127.31.250.2" port: 46079 } } peers { permanent_uuid: "1eb837892f874e99a16858e7958a1356" member_type: VOTER last_known_addr { host: "127.31.250.0" port: 35707 } } I0221 01:35:20.338979 12150 leader_election.cc:216] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer c8ec78660e8642b4b82113d62a3042cb I0221 01:35:20.339155 12150 leader_election.cc:216] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 33e7b3c7fdd64255b209cc8123db9646 W0221 01:35:20.340406 11758 leader_election.cc:272] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 33e7b3c7fdd64255b209cc8123db9646: Network error: Client connection negotiation failed: client connection to 127.31.250.2:46079: connect: Connection refused (error 111) W0221 01:35:20.340512 11759 leader_election.cc:272] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c8ec78660e8642b4b82113d62a3042cb: Network error: Client connection negotiation failed: client connection to 127.31.250.1:40370: connect: Connection refused (error 111) I0221 01:35:20.340593 11759 leader_election.cc:243] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. I0221 01:35:20.340888 20863 raft_consensus.cc:2162] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 15.6 s I0221 01:35:20.340977 20863 raft_consensus.cc:1908] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority W0221 01:35:20.343775 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:35:21.345404 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:35:22.347383 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:35:23.348801 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:35:24.349663 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:35:25.350432 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:35:26.351943 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:35:27.353477 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:35:28.354996 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:35:29.356972 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:35:30.358909 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) I0221 01:35:30.667999 8186 cluster_verifier.cc:72] Check not successful yet, sleeping and retrying: Timed out: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.0.0.1:44820, num_attempts: 241) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:35:31.360771 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:35:32.362681 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:35:33.363665 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:35:34.365176 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:35:35.367043 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:35:36.368500 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:35:37.369128 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) I0221 01:35:37.669991 12150 raft_consensus.cc:411] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0221 01:35:37.670092 12150 raft_consensus.cc:2162] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 7.21 s I0221 01:35:37.670130 12150 raft_consensus.cc:435] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8ec78660e8642b4b82113d62a3042cb" member_type: VOTER last_known_addr { host: "127.31.250.1" port: 40370 } } peers { permanent_uuid: "33e7b3c7fdd64255b209cc8123db9646" member_type: VOTER last_known_addr { host: "127.31.250.2" port: 46079 } } peers { permanent_uuid: "1eb837892f874e99a16858e7958a1356" member_type: VOTER last_known_addr { host: "127.31.250.0" port: 35707 } } I0221 01:35:37.670629 12150 leader_election.cc:216] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer c8ec78660e8642b4b82113d62a3042cb I0221 01:35:37.670795 12150 leader_election.cc:216] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 33e7b3c7fdd64255b209cc8123db9646 W0221 01:35:37.671638 11759 leader_election.cc:272] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c8ec78660e8642b4b82113d62a3042cb: Network error: Client connection negotiation failed: client connection to 127.31.250.1:40370: connect: Connection refused (error 111) W0221 01:35:37.671788 11758 leader_election.cc:272] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 33e7b3c7fdd64255b209cc8123db9646: Network error: Client connection negotiation failed: client connection to 127.31.250.2:46079: connect: Connection refused (error 111) I0221 01:35:37.671865 11758 leader_election.cc:243] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. I0221 01:35:37.672163 21051 raft_consensus.cc:2162] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 7.74 s I0221 01:35:37.672247 21051 raft_consensus.cc:1908] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority W0221 01:35:38.370690 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:35:39.372380 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:35:40.373955 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:35:41.375962 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:35:42.377498 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:35:43.378893 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:35:44.380398 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:35:45.381470 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:35:46.382287 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) I0221 01:35:46.989585 12150 raft_consensus.cc:411] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0221 01:35:46.989686 12150 raft_consensus.cc:2162] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 3.35 s I0221 01:35:46.989728 12150 raft_consensus.cc:435] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8ec78660e8642b4b82113d62a3042cb" member_type: VOTER last_known_addr { host: "127.31.250.1" port: 40370 } } peers { permanent_uuid: "33e7b3c7fdd64255b209cc8123db9646" member_type: VOTER last_known_addr { host: "127.31.250.2" port: 46079 } } peers { permanent_uuid: "1eb837892f874e99a16858e7958a1356" member_type: VOTER last_known_addr { host: "127.31.250.0" port: 35707 } } I0221 01:35:46.990236 12150 leader_election.cc:216] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer c8ec78660e8642b4b82113d62a3042cb I0221 01:35:46.990397 12150 leader_election.cc:216] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 33e7b3c7fdd64255b209cc8123db9646 W0221 01:35:46.991302 11759 leader_election.cc:272] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c8ec78660e8642b4b82113d62a3042cb: Network error: Client connection negotiation failed: client connection to 127.31.250.1:40370: connect: Connection refused (error 111) W0221 01:35:46.991611 11758 leader_election.cc:272] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 33e7b3c7fdd64255b209cc8123db9646: Network error: Client connection negotiation failed: client connection to 127.31.250.2:46079: connect: Connection refused (error 111) I0221 01:35:46.991688 11758 leader_election.cc:243] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. I0221 01:35:46.992069 21158 raft_consensus.cc:2162] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 8.45 s I0221 01:35:46.992158 21158 raft_consensus.cc:1908] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority W0221 01:35:47.383710 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:35:48.385637 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:35:49.387177 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:35:50.388738 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:35:51.390213 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:35:52.391808 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:35:53.393309 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:35:54.395035 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:35:55.396723 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:35:56.398190 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:35:57.398948 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) I0221 01:35:57.495434 12150 raft_consensus.cc:411] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0221 01:35:57.495538 12150 raft_consensus.cc:2162] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 10.9 s I0221 01:35:57.495581 12150 raft_consensus.cc:435] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8ec78660e8642b4b82113d62a3042cb" member_type: VOTER last_known_addr { host: "127.31.250.1" port: 40370 } } peers { permanent_uuid: "33e7b3c7fdd64255b209cc8123db9646" member_type: VOTER last_known_addr { host: "127.31.250.2" port: 46079 } } peers { permanent_uuid: "1eb837892f874e99a16858e7958a1356" member_type: VOTER last_known_addr { host: "127.31.250.0" port: 35707 } } I0221 01:35:57.496081 12150 leader_election.cc:216] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer c8ec78660e8642b4b82113d62a3042cb I0221 01:35:57.496255 12150 leader_election.cc:216] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 33e7b3c7fdd64255b209cc8123db9646 W0221 01:35:57.497056 11759 leader_election.cc:272] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c8ec78660e8642b4b82113d62a3042cb: Network error: Client connection negotiation failed: client connection to 127.31.250.1:40370: connect: Connection refused (error 111) W0221 01:35:57.497253 11758 leader_election.cc:272] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 33e7b3c7fdd64255b209cc8123db9646: Network error: Client connection negotiation failed: client connection to 127.31.250.2:46079: connect: Connection refused (error 111) I0221 01:35:57.497328 11758 leader_election.cc:243] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. I0221 01:35:57.497624 21262 raft_consensus.cc:2162] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 6.33 s I0221 01:35:57.497712 21262 raft_consensus.cc:1908] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority W0221 01:35:58.400503 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:35:59.401921 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:36:00.403389 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:36:01.404875 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) I0221 01:36:01.708024 8186 cluster_verifier.cc:72] Check not successful yet, sleeping and retrying: Timed out: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.0.0.1:44820, num_attempts: 241) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:36:02.405491 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:36:03.406103 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:36:04.407551 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:36:05.408939 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) I0221 01:36:05.717798 12150 raft_consensus.cc:411] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0221 01:36:05.717897 12150 raft_consensus.cc:2162] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 11.5 s I0221 01:36:05.717953 12150 raft_consensus.cc:435] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8ec78660e8642b4b82113d62a3042cb" member_type: VOTER last_known_addr { host: "127.31.250.1" port: 40370 } } peers { permanent_uuid: "33e7b3c7fdd64255b209cc8123db9646" member_type: VOTER last_known_addr { host: "127.31.250.2" port: 46079 } } peers { permanent_uuid: "1eb837892f874e99a16858e7958a1356" member_type: VOTER last_known_addr { host: "127.31.250.0" port: 35707 } } I0221 01:36:05.718508 12150 leader_election.cc:216] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer c8ec78660e8642b4b82113d62a3042cb I0221 01:36:05.718673 12150 leader_election.cc:216] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 33e7b3c7fdd64255b209cc8123db9646 W0221 01:36:05.719682 11758 leader_election.cc:272] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 33e7b3c7fdd64255b209cc8123db9646: Network error: Client connection negotiation failed: client connection to 127.31.250.2:46079: connect: Connection refused (error 111) W0221 01:36:05.719838 11759 leader_election.cc:272] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c8ec78660e8642b4b82113d62a3042cb: Network error: Client connection negotiation failed: client connection to 127.31.250.1:40370: connect: Connection refused (error 111) I0221 01:36:05.719913 11759 leader_election.cc:243] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. I0221 01:36:05.720254 21376 raft_consensus.cc:2162] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 17.9 s I0221 01:36:05.720345 21376 raft_consensus.cc:1908] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority W0221 01:36:06.410634 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:36:07.411970 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:36:08.412760 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:36:09.413511 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:36:10.415422 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:36:11.416954 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:36:12.418531 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:36:13.420104 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:36:14.422058 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:36:15.424047 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:36:16.425500 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:36:17.426457 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:36:18.428016 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:36:19.428943 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:36:20.430481 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:36:21.432003 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:36:22.433394 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:36:23.435186 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:36:24.437130 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) I0221 01:36:25.134147 12150 raft_consensus.cc:411] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0221 01:36:25.134239 12150 raft_consensus.cc:2162] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 13.5 s I0221 01:36:25.134282 12150 raft_consensus.cc:435] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8ec78660e8642b4b82113d62a3042cb" member_type: VOTER last_known_addr { host: "127.31.250.1" port: 40370 } } peers { permanent_uuid: "33e7b3c7fdd64255b209cc8123db9646" member_type: VOTER last_known_addr { host: "127.31.250.2" port: 46079 } } peers { permanent_uuid: "1eb837892f874e99a16858e7958a1356" member_type: VOTER last_known_addr { host: "127.31.250.0" port: 35707 } } I0221 01:36:25.134781 12150 leader_election.cc:216] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer c8ec78660e8642b4b82113d62a3042cb I0221 01:36:25.134958 12150 leader_election.cc:216] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 33e7b3c7fdd64255b209cc8123db9646 W0221 01:36:25.136047 11759 leader_election.cc:272] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c8ec78660e8642b4b82113d62a3042cb: Network error: Client connection negotiation failed: client connection to 127.31.250.1:40370: connect: Connection refused (error 111) W0221 01:36:25.136409 11758 leader_election.cc:272] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 33e7b3c7fdd64255b209cc8123db9646: Network error: Client connection negotiation failed: client connection to 127.31.250.2:46079: connect: Connection refused (error 111) I0221 01:36:25.136487 11758 leader_election.cc:243] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. I0221 01:36:25.136787 21582 raft_consensus.cc:2162] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 13.8 s I0221 01:36:25.136865 21582 raft_consensus.cc:1908] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority W0221 01:36:25.437989 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:36:26.438894 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:36:27.440691 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:36:28.441345 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:36:29.443181 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:36:30.445171 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:36:31.446717 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:36:32.448319 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) I0221 01:36:32.925112 8186 cluster_verifier.cc:72] Check not successful yet, sleeping and retrying: Timed out: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.0.0.1:44820, num_attempts: 242) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:36:33.449934 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:36:34.451793 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:36:35.453382 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:36:36.455021 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:36:37.456619 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:36:38.458050 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:36:39.459007 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:36:40.460968 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) I0221 01:36:40.802409 12150 raft_consensus.cc:411] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0221 01:36:40.802517 12150 raft_consensus.cc:2162] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 11.7 s I0221 01:36:40.802558 12150 raft_consensus.cc:435] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8ec78660e8642b4b82113d62a3042cb" member_type: VOTER last_known_addr { host: "127.31.250.1" port: 40370 } } peers { permanent_uuid: "33e7b3c7fdd64255b209cc8123db9646" member_type: VOTER last_known_addr { host: "127.31.250.2" port: 46079 } } peers { permanent_uuid: "1eb837892f874e99a16858e7958a1356" member_type: VOTER last_known_addr { host: "127.31.250.0" port: 35707 } } I0221 01:36:40.803061 12150 leader_election.cc:216] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer c8ec78660e8642b4b82113d62a3042cb I0221 01:36:40.803241 12150 leader_election.cc:216] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 33e7b3c7fdd64255b209cc8123db9646 W0221 01:36:40.804064 11759 leader_election.cc:272] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c8ec78660e8642b4b82113d62a3042cb: Network error: Client connection negotiation failed: client connection to 127.31.250.1:40370: connect: Connection refused (error 111) W0221 01:36:40.804285 11758 leader_election.cc:272] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 33e7b3c7fdd64255b209cc8123db9646: Network error: Client connection negotiation failed: client connection to 127.31.250.2:46079: connect: Connection refused (error 111) I0221 01:36:40.804359 11758 leader_election.cc:243] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. I0221 01:36:40.804641 21775 raft_consensus.cc:2162] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.57 s I0221 01:36:40.804735 21775 raft_consensus.cc:1908] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority W0221 01:36:41.462906 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:36:42.464349 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:36:43.466305 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) I0221 01:36:43.999260 12150 raft_consensus.cc:411] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0221 01:36:43.999408 12150 raft_consensus.cc:2162] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 6.17 s I0221 01:36:43.999477 12150 raft_consensus.cc:435] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8ec78660e8642b4b82113d62a3042cb" member_type: VOTER last_known_addr { host: "127.31.250.1" port: 40370 } } peers { permanent_uuid: "33e7b3c7fdd64255b209cc8123db9646" member_type: VOTER last_known_addr { host: "127.31.250.2" port: 46079 } } peers { permanent_uuid: "1eb837892f874e99a16858e7958a1356" member_type: VOTER last_known_addr { host: "127.31.250.0" port: 35707 } } I0221 01:36:44.000018 12150 leader_election.cc:216] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer c8ec78660e8642b4b82113d62a3042cb I0221 01:36:44.000210 12150 leader_election.cc:216] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 33e7b3c7fdd64255b209cc8123db9646 W0221 01:36:44.001121 11758 leader_election.cc:272] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 33e7b3c7fdd64255b209cc8123db9646: Network error: Client connection negotiation failed: client connection to 127.31.250.2:46079: connect: Connection refused (error 111) W0221 01:36:44.001346 11759 leader_election.cc:272] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c8ec78660e8642b4b82113d62a3042cb: Network error: Client connection negotiation failed: client connection to 127.31.250.1:40370: connect: Connection refused (error 111) I0221 01:36:44.001408 11759 leader_election.cc:243] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. I0221 01:36:44.001700 21818 raft_consensus.cc:2162] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 2.31 s I0221 01:36:44.001806 21818 raft_consensus.cc:1908] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority W0221 01:36:44.467726 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:36:45.469501 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:36:46.471431 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:36:47.472873 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) I0221 01:36:48.455127 12150 raft_consensus.cc:411] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0221 01:36:48.455217 12150 raft_consensus.cc:2162] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 12.1 s I0221 01:36:48.455312 12150 raft_consensus.cc:435] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8ec78660e8642b4b82113d62a3042cb" member_type: VOTER last_known_addr { host: "127.31.250.1" port: 40370 } } peers { permanent_uuid: "33e7b3c7fdd64255b209cc8123db9646" member_type: VOTER last_known_addr { host: "127.31.250.2" port: 46079 } } peers { permanent_uuid: "1eb837892f874e99a16858e7958a1356" member_type: VOTER last_known_addr { host: "127.31.250.0" port: 35707 } } I0221 01:36:48.455811 12150 leader_election.cc:216] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer c8ec78660e8642b4b82113d62a3042cb I0221 01:36:48.455968 12150 leader_election.cc:216] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 33e7b3c7fdd64255b209cc8123db9646 W0221 01:36:48.456907 11759 leader_election.cc:272] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c8ec78660e8642b4b82113d62a3042cb: Network error: Client connection negotiation failed: client connection to 127.31.250.1:40370: connect: Connection refused (error 111) W0221 01:36:48.457165 11758 leader_election.cc:272] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 33e7b3c7fdd64255b209cc8123db9646: Network error: Client connection negotiation failed: client connection to 127.31.250.2:46079: connect: Connection refused (error 111) I0221 01:36:48.457245 11758 leader_election.cc:243] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. I0221 01:36:48.457548 21910 raft_consensus.cc:2162] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 19.7 s I0221 01:36:48.457635 21910 raft_consensus.cc:1908] T 2c6176fb99ed4fb4a027584aeeb0a708 P 1eb837892f874e99a16858e7958a1356 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority W0221 01:36:48.474200 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111) W0221 01:36:49.475661 11869 heartbeater.cc:499] Failed to heartbeat to 127.0.0.1:44820: Network error: Failed to ping master at 127.0.0.1:44820: Client connection negotiation failed: client connection to 127.0.0.1:44820: connect: Connection refused (error 111)